feat(backup): show what a backup lacks in list, show, status and the --wait results - #40
Merged
Merged
Conversation
…--wait results
Arc 26.09.3 reports a backup's incompleteness on every backup endpoint:
the listing carries skipped_files, skipped_metadata_files and
unaddressable_files; the manifest names up to 32 skipped files in
skipped_sample and counts overlong keys in skipped_overlong_keys; the
status endpoint names a backup's skipped files as it already did for a
restore. arcli ignored all of it and its list help said the listing
could not tell whether a backup is incomplete.
The client types mirror the fields. "backup list" gains an INCOMPLETE
column: "-" when the server reports nothing missing, otherwise the
counts the server keeps apart ("3 skipped, 1 metadata, 2 unaddressable"),
never summed, because skipped data files are inside the file total,
metadata skips are not, and unaddressable files were never inventoried;
csv appends the three columns. "backup show" says the counts in words,
including how many skips were for a key too long to store and how many
outside-root Iceberg warehouse files were skipped, then names the files.
"backup status" and the final status of "--wait" list the skipped files
the server names, report files a backup could not list, and, for a
restore, what the restored backup already lacked and the warehouse files
left out. "create --wait" ends its success line with every gap, prints
the names, and points at "backup show". "restore" warns before restoring
a backup that lacks files for any reason the manifest records.
Against an Arc older than 26.09.3 everything reads as none reported.
…xists; restore warning in show's shape Review fix-up. A backup publishes its unaddressable count before copying but names the files only in its manifest, which exists once it has completed; while it runs, or after it failed, "backup status" now sends the operator to the server log instead of to a "backup show" that would answer not found. The restore pre-flight warning takes the same shape as "backup show": the overlong-key clause hangs on the joined skipped sentence, because the server counts overlong keys across data and metadata files alike, so a metadata-only overlong skip is no longer dropped or charged to the data files; plurals are right; and a backup whose only gap is outside-root Iceberg warehouse files warns too, as the release note already claimed. A completed "restore --wait" now says what the restored backup lacked and how many warehouse files were not restored, the one gap no earlier surface can show when the node has Iceberg off. The "backup show" hint after "create --wait" moves to stderr with backticks like the file's other hints; "N of M files" falls back to "N files" for a manifest from before the data/metadata split, where N can exceed M. Tests cover json passthrough, the metadata-only singular case, and the failed and running backup status hint.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to Basekick-Labs/arc#981 (arc#977), which makes a backup's incompleteness visible on every Arc backup endpoint. arcli's listing help said "the listing cannot tell whether a backup is incomplete"; now it can.
skipped_files,skipped_metadata_files,unaddressable_files; manifestskipped_metadata_files,skipped_sample,skipped_overlong_keys,unaddressable_files,unaddressable_sample,iceberg_warehouse.skipped_files; progressskipped_sample,unaddressable_files/unaddressable_sample,backup_skipped_files,backup_unaddressable_files,iceberg_warehouse_files_skipped. Absent keys (Arc < 26.09.3) read as zero.backup list: newINCOMPLETEcolumn,-when the server reports nothing missing, otherwise the counts the server keeps apart, e.g.3 skipped, 1 metadata, 2 unaddressable, never summed (skipped data files are insideFILES, metadata skips are not, unaddressable files were never inventoried).-o csvappendsskipped_files,skipped_metadata_files,unaddressable_filesaftertotal_size_bytes.-o jsonunchanged (raw passthrough).backup show: theINCOMPLETEline says the counts in words ("3 of 21 files and 1 metadata file were skipped while backing up (1 for a key too long to store); 2 files could not be listed (unaddressable); 1 Iceberg warehouse file was skipped"), followed by oneskipped:line per name in the manifest's sample and oneunaddressable:line per name in that sample.backup statusand the final status ofcreate --wait/restore --wait: list the skipped files the server names; report files a backup could not list (the manifest has their names); for a restore, say what the restored backup already lacked when it was taken and how many outside-root warehouse files were not restored. The server's error already carries a failed restore's counts, so only names and the backup's own gaps are added.create --waitends its success line with every gap ("(1 files skipped, 1 files could not be listed — the backup is incomplete)"), prints the names, and points atbackup showfor the full breakdown.restorewarns before restoring a backup that lacks files for any reason the manifest records (previously only skipped data files), and says how many of the skipped files had keys too long to store.docs/releases/v26.09.5.md. Docs: Basekick-Labs/docs.basekick.net (arcli backup page; also corrects the arc restore-error example to the server's current wording), docs(arcli): backup list INCOMPLETE column, show and status naming skipped files docs.basekick.net#95.Plan was reviewed adversarially before implementation; its findings (never sum the populations, the server's "unaddressable" vocabulary, unaddressable-only backups on status, restore pre-flight gate, print the sample on the success path) are in.
Test plan
TestBackup_IncompleteIsVisible(fixtures in the shapes Arc 26.09.3 produces; list table/csv/--no-header, show block and sample lines, backup status, restore pre-flight warning, failed-restore status) andTestBackup_CreateWaitReportsIncomplete(fake server toggle); both compile againstmainand fail there. ClientTestBackup_IncompleteShapesdecodes every field and reads zero where keys are absent.go test -race -count=1 ./...,gofmt -l .,go vet ./...clean.create --waitprints "completed: 23 files, 19.7 KiB (1 files skipped — the backup is incomplete)", theskipped:line and theshowhint;listshows1 skipped, csv…,1,0,0, json untouched;showprints "INCOMPLETE: 1 of 24 files was skipped while backing up" and the name;statusnames it; after a clean backup the new entry reads-.