feat: WAL time-travel snapshots - #31
Conversation
|
HyperFrames walkthrough of this PR. |
`materialize_at` — replay the newest checkpoint at or before a sequence plus the log entries through it, install the pack set that was live there — lived in `walgit-cli/src/wal_cmd.rs`, so nothing but the CLI could rewind. Move it to `walgit-wal::snapshot` (the crate both the CLI and the server already share) and give it the shape a caller other than a terminal needs: * it returns a `Snapshot` (at_seq, head_seq, from_seq, replayed entries, git dir, refs bounded at `REFS_MAX`, ref count, pack set with per-pack provenance) instead of printing; * narration goes to an `Arc<dyn Fn(String)>` sink, so the CLI prints and a server task turns the lines into notices; * errors are `WalError`, not `anyhow`, and no `unwrap` remains on the path; * the whole replay runs on the bulk runtime — downloads, 32 MiB copies and `git index-pack` waits must not sit on a caller's request workers (AGENTS §5, principle VI). `snapshot_at` is the second entry point over the same replay: it owns the `<cache.dir>/snapshots/<owner>/<name>/<seq>/` layout, rejects a zero or past-the-head sequence before touching the bucket, refuses to build a second pack set on an instance whose cache budget cannot hold the live one, and is idempotent — `snapshot.json` is written last, so a tree carrying it is complete and a tree without it is a partial run that gets rebuilt. The existing cold-rewind test moves with the code and grew the assertions the returned `Snapshot` makes possible (pack provenance, replay start, refs); a second test covers the `snapshot_at` layout, idempotency and seq validation. Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
The rewind that `walgit wal materialize --at-seq` does is now reachable over
HTTP on the instance that answers:
* `POST …/ops/snapshot?at_seq=<n>` is a non-mutating op (`mutating: false`) that
runs `walgit_wal::snapshot_at` as a task, narrated like every other op, and
returns `{built, snapshot}`. Missing, unparsable, zero and past-the-head
sequences are refused with the reason. Nothing is published and no live ref
moves: the answer carries `head_seq`, which is where the serving copy stays.
* `GET …/api/snapshot/{seq}` reads that rewind back with the overview's auth
(read). `no-store`, because a snapshot is a local cache like every other byte
on disk; `404` naming the op when this host has not built one.
`ops::run` now takes the narrator as an owned `Arc` and derives the borrowed
`Log` from it, because the WAL crate's replay outlives the call and runs on
another runtime.
New test `tests/snapshot.rs` (wired into `just test`): two pushes, rewind to
seq 1, the result's refs are the first commit, the rewound copy is a real
repository that does not contain the second commit, `info/refs` and the local
copy are still at seq 2, the GET round-trips, a second run rebuilds nothing,
and each bad `at_seq` is refused without leaving a tree behind.
Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
* WAL tab: the `snapshot` op's `at_seq` is a value, not a flag, so the ops table renders a number field (defaulting to the head) next to it plus a link to `…/api/snapshot/<n>`. `VALUE_PARAMS` replaces the hard-coded `strategy` check. Everything else — running, streaming, the recent-runs table — is the existing `runOp`/SSE path. * SDK: `repo.snapshot(seq)` and the `Snapshot`/`SnapshotRef`/`SnapshotPack` types (the UI stays an adapter over the SDK, D20). * `web/API.md`: the endpoint with its shape, the op that builds it, the 404 and the ref cap; `snapshot` added to the task kinds and the conformance list. * `docs/CONTRACT.md`: the `walgit_wal::snapshot` surface and the two routes; `WalError` gained the `Invalid`/`TooLarge` variants it already had in code. * `docs/ROUNDTRIPS.md`: a budget row for the rewind (operator-driven, off the critical path) — freshness GET, checkpoint + refs, the segments through the cut, and per missing pack a striped pack ∥ idx ∥ side-files; an already materialized sequence costs nothing past the freshness GET. * `AGENTS.md` §2.4: the provenance line now names the HTTP form of the rewind. Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
Two follow-ups on the lift: the local-pack copy builds `pack-<checksum>.<ext>` itself instead of going through `file_name()` (the name it wants is the one the lines below already build), and `walgit wal materialize` prints `ref_count`, not the length of the capped inline list. The WAL tab's at_seq field also stops nesting its explanation and the JSON link inside the input's own label. Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
The repo runs `cargo clippy --workspace --all-targets -- -D warnings`, and the new integration test tripped four of the panic-path restriction lints plus the duration-readability one. Propagate from `git_out` the way the sibling tests do, read the terminal packet's fields through `get` instead of indexing, and hoist the per-step ceiling into a named `STEP_TIMEOUT`.
fc29b98 to
8af44de
Compare
main removed bundles in the packfiles work (tobi#52-tobi#58), which is most of what conflicted here: - ui.rs: took main's file and re-applied only this branch's two additions (the snapshot route and the handler). The earlier bundle listing goes with the crate. - OverviewPage.tsx: dropped the bundle strategy picker and its state; VALUE_PARAMS is now just at_seq. The at_seq field and the snapshot JSON link are unchanged. - justfile / web/API.md / docs/CONTRACT.md: main's lists, minus the bundle entries, plus this branch's snapshot test, task kind and route. - walgit-wal/src/lib.rs: kept both modules. main's `snapshots` is manifest coverage (CoverageSnapshot, FetchView, validate_manifest); this branch's `snapshot` is the WAL rewind. Unrelated despite the names. just warnings, just clippy and just test are green, including the snapshot integration test.
|
Merged main. The packfiles work removed bundles, which is most of what conflicted.
One thing worth a look:
|

Walkthrough
HyperFrames walkthrough of this PR.
mp4
What this does
The WAL already keeps every state a repository has ever been in, and
walgit wal materialize --at-seqalready rebuilds one. That rewind was reachable only from a terminal with bucket credentials, because the code lived incrates/walgit-cli/src/wal_cmd.rs. This exposes the same replay over HTTP.The replay moved into
walgit-wal(crates/walgit-wal/src/snapshot.rs), the crate the CLI and the server already share — not copy-pasted intoops.rs. On the way it gained the shape a non-terminal caller needs: it returns aSnapshot(at_seq,head_seq,from_seq, replayedentries,git_dir,refs/ref_count,packswith per-packsource) instead of printing; narration goes to anArc<dyn Fn(String)>sink so the CLI prints and a server task emits notices; errors areWalErrorrather thananyhow; and nounwrapis left on the path. The whole replay runs on the bulk runtime (sync::on_bulk_runtime) so downloads, 32 MiB copies andgit index-packwaits never sit on a caller's request workers (principle VI).POST /{owner}/{repo}/api/ops/snapshot?at_seq=<n>— a new OPS entry,id = "snapshot",params = ["at_seq"],mutating: false. It runswalgit_wal::snapshot_atas an ordinary task (narrated, attachable, interrupted by the D31 drain like every other unit) and writes to<cache.dir>/snapshots/<owner>/<name>/<seq>/.snapshot.jsonis written last, so a tree carrying it is complete and one without it is a partial run that gets rebuilt: the op is idempotent and answers{built: bool, snapshot: {…}}.GET /{owner}/{repo}/api/snapshot/{seq}— reads that rewind back with the overview's auth (require_read),no-store,404naming the op when this host has not built one. An unparsable{seq}is a 400 from the route itself.WAL tab: the ops table renders
at_seqas a number field (defaulting to the head) plus a link to the JSON, because it is a value parameter and not a flag. Running, streaming and the recent-runs table are the existingrunOp/SSE path, unchanged.SDK + docs:
repo.snapshot(seq)and theSnapshottypes inweb/sdk/repos.ts; the endpoint, the op, the 404 and the ref cap inweb/API.md(plussnapshotin the task-kinds list and the conformance checklist); thewalgit_wal::snapshotsurface indocs/CONTRACT.md; a budget row indocs/ROUNDTRIPS.md; the provenance line inAGENTS.md§2.4.What it does not do
head_seqso a caller can see where the serving copy still is, and the test asserts it (refs endpoint, manifest head, applied seq and the local copy'srefs/heads/mainall stay at seq 2 after a rewind to seq 1).rematerializeis untouched. It has noat_seqand still means "catch this instance's copy up to the head".cache.dirthat an operator creates deliberately, andsnapshot_atrefuses to build one on an instance whose cache budget cannot hold even the live pack set (WalError::TooLarge→ 503 with the existing message). Bounding them across sequences would be a separate change.refsin the JSON is capped at 1000 entries withref_countalongside, so a 466 k-ref repository does not turn a request path into a 30 MB document. The rebuilt copy atgit_dirhas all of them.File pointers
crates/walgit-wal/src/snapshot.rsmaterialize_at), the cache-dir form (snapshot_at),snapshot_dir/read_snapshot,Snapshot/SnapshotRef/SnapshotPack, both unit testscrates/walgit-wal/src/lib.rspub mod snapshot+ re-exportscrates/walgit-cli/src/wal_cmd.rsWalAction::Materializenow calls the shared function and prints the summary; 348 lines of replay and its test left the CLIcrates/walgit-server/src/ops.rssnapshotOpSpec and arm;runtakes the narrator as an ownedArcand derives the borrowedLogfrom itcrates/walgit-server/src/web/ui.rssnapshot/{seq}route and handlercrates/walgit-server/src/web/v1.rs/api/v1discoverycrates/walgit-server/tests/snapshot.rsjust test)web/sdk/repos.ts,web/src/pages/OverviewPage.tsx,web/src/styles.cssat_seqfield, its input styleweb/API.md,web/sdk/README.md,docs/CONTRACT.md,docs/ROUNDTRIPS.md,AGENTS.mdRound trips
No hot path changes;
docs/ROUNDTRIPS.md§2 gained one row for the rewind, which is operator-driven: one conditional manifest GET (the refs sync that validatesat_seqagainst the head), then the checkpoint and its refs (2 GETs when one is at or before the cut), the log segments through the cut, and per pack not already local a striped pack ∥ idx ∥ advertised side-files. A sequence already materialized on the host costs nothing past the freshness GET. The sim suite's request budgets are unchanged and pass.Tests
crates/walgit-wal/src/snapshot.rs: the existing cold-rewind test moved with the code (not deleted) and grew the assertions the returnedSnapshotmakes possible — pack provenance (storevslocal), the replay start, the refs, and that the writer's copy kept its packs and its head. A second test covers thesnapshot_atlayout, idempotency (samebuilt_aton the second call), rebuild of a tree whose marker is gone, and rejection of seq 0 and past-the-head sequences without leaving a tree behind.crates/walgit-server/tests/snapshot.rs: two pushes (head seq 2) →POST ops/snapshot?at_seq=1→ the result's refs are the first commit,head_seqis 2, the rewound copy is a real repository at the first commit that does not contain the second,info/refs+ manifest + applied seq + the local copy are all still at seq 2,GET snapshot/1round-trips,GET snapshot/2is a 404 naming the op,GET snapshot/nopeis a 400, the second op run rebuilds nothing, and each badat_seq(missing, empty,abc,0,99) is refused with its reason.Run here on the pinned toolchain:
cargo fmt --checkclean;cargo test --workspace --lib --bins,cargo test -p walgit-store -p walgit-git -p walgit-wal -p walgit-bundle --tests, thewalgit-serverfast tier (now including--test snapshot), and--test simall green;cargo build --workspace --all-targetshas no rustc warnings;webpnpm run build(oxlint--deny-warnings+tsc --noEmit+ both vite builds) passes.Two pre-existing failures on this branch are also present on
mainat the same commit and are unrelated:e2e::lfs_roundtrip_when_available(verified failing on6d8fa54in a clean worktree here) ande2e::fetch_from_front_that_serves_the_base_remotely(the flake AGENTS.md documents; passes alone).cargo clippy -- -D warningsis also red onmainwith the pinned 1.97.1 toolchain — newer pedantic lints inwalgit-proto,walgit-configandwalgit-server/build.rs, ~1800 workspace-wide. The files this PR touches contribute none:crates/walgit-wal/src/snapshot.rsis clippy-clean, andcrates/walgit-server/tests/snapshot.rshas 6 findings of the classes every sibling test file already carries (indexing may panicetc.: 70 insim.rs, 61 inweb_api.rs). Fixing the pre-existing set would be a separate change.