Repository navigation
feat(snapshot): build shadowfork snapshots from a jochemnet release - #44
Conversation
A fresh mainnet snapshot advanced by one pre-run (EIP-8282 contracts on Osaka, Amsterdam at head + 3 s, 24 KiB + 64 KiB receivers at 100k per variant), its gas limit walked down to 200M, the fill flushed out of geth's journal and compacted, then the geth image and the replay bundle uploaded to ethpandaops-shadowfork-images. The other clients replay the bundle separately and latest is written once all five are up (shadowfork/README.md). Clean image: geth's path scheme journaled 1,354 blocks of the last fill on a clean stop, so the newest contracts were read from memory. The ramp-down runs with --cache.gc=0 (0-byte write buffer) and the step fails unless the journal holds only the last 128 diff layers; then geth db compact. Verified on a dev chain (layers 176 -> 128); rampdown.py built 256 Amsterdam blocks (V5, slot numbers, 11.5M -> 10M) whose bundle replayed on a second node to the same head. Runs on the synctest Disk4TB runners (the gasbenchmark hosts hold the jochemnet CI baselines); uses the repo's existing S3 credentials and probes them before downloading anything.
…#3304 qu0b/bench-images-v3 replaces the skylenet-based v2: upstream forks/amsterdam (2026-09-25) with the two open upstream PRs the fill needs, ethereum/execution-specs#3648 (spill payload lists to disk) and #3304 (the deployment script), plus the combined-sizes test and JUMPDEST at 24 KiB.
Replaces shadowfork-image-build and its scripts. geth replays the release's pre-run (base_bundle) and walks the gas limit down (fill: false), every other client replays release + ramp, each datadir is compacted (db compact) and exported (snapshot export) to <bucket>/jochemnet/<client>/<head>/, and latest is written once all are up. Runs on the jochemnet hosts and recovers their schelk scratch afterwards.
…head block snapshot export publishes the head block as-is, and its contract is the full-transaction eth_getBlockByNumber response.
…ots on it Found by the e2e prototype: geth on a flat instance dir opens the database but misses triedb/merkle.journal, then rewinds the head and truncates state history.
…he datadir Without --image the CLI pulled ethereum/client-go:stable (1.17.8 today) for a datadir written by v1.17.7 (found by the e2e prototype); a newer geth can migrate the database format on open.
…ing tail; compaction notes per client reth compacts (benchmarkoor #351: reth db copy --compact), ethrex does not: its TRANSACTION_LOCATIONS family has a merge operator ldb lacks, so the generic RocksDB compaction refuses the database.
…uts too From jochemnet's archives, extracted on the build droplets: reth v2 has static_files/ and rocksdb/ at the root, erigon's pruned archive is the datadir (chaindata/, snapshots/), ethrex's is the RocksDB directory that lives at <datadir>/chain-1.
There was a problem hiding this comment.
Summary
The PR replaces the old shadowfork-image-build workflow/scripts with benchmarkoor-snapshot.yaml, four helper scripts and a jochemnet snapshot builder config that replay a release, ramp the gas limit down, compact where a client command exists, and publish per-client snapshots. The latest commit correctly drops the external ldb compaction (nethermind/besu/reth/ethrex now skipped, geth/erigon use their pinned client image), and the earlier out-of-scope shadowfork-file and SNAPSHOT_PREFIX findings are gone. One blocker from the previous reviews remains: the benchmarkoor binary the workflow calls for compaction and export is deleted by the benchmarkoor action before those steps run.
Issues
- 🔴
.github/workflows/benchmarkoor-snapshot.yaml:203— benchmarkoor CLI is deleted by the action before db compact / snapshot export — see the thread on that line
Reviewed @ 28ac19d4
"First law of programming: whatever the bug, blame DNS."
| # --image: the release that wrote the datadir compacts it; the CLI default is the client's | ||
| # :stable, a newer geth that may migrate the database format on open. | ||
| - name: Compact | ||
| run: sudo -E "$RUNNER_TEMP/benchmarkoor" db compact --client "$CLIENT" --datadir "$DATADIR" --image "$SNAPSHOT_IMAGE" |
There was a problem hiding this comment.
🔴 benchmarkoor CLI is deleted by the action before db compact / snapshot export
The ethpandaops/benchmarkoor@master steps (lines 146 and 263) are a composite action whose final Cleanup step runs rm -f "${{ runner.temp }}/benchmarkoor" (benchmarkoor action.yaml:455, which is the same $RUNNER_TEMP/benchmarkoor the action's Extract benchmarkoor binary step wrote at line 135). By the time the Compact steps (line 206 and line 291) and snapshot-export.sh (line 22) run, that binary no longer exists, so every client fails with No such file or directory before any archive is published. Re-extract the binary (or run the CLI from the image) after the action step, before compacting/exporting.
…p-only pre-run (#349) A released benchmark snapshot can have a much higher block gas limit than the shadowfork needs. This lets a pre-run lower that limit—for example, from 1T to 200M—and record the new blocks for other clients to replay. - Uses Amsterdam's `targetGasLimit` to ramp down and hold the requested limit. A configured fork transition can complete before ramp-down starts; the block limit still applies. - Adds `fill: false` to skip EEST setup and `base_bundle` to continue from an existing recorded chain. The new bundle contains only the additional blocks, so consumers replay the base first. - Streams large replay bundles instead of loading them into memory, and continues the parent's Amsterdam slot number. - Gives every pre-run client time to flush its database on shutdown. A forced kill or failed shutdown fails the target. Validation: builder/config tests, fork-transition regressions, real Geth gas-limit checks, and replay of the 49.7 GB release bundle. CI passes. Used by [benchmarkoor-tests #44](ethpandaops/benchmarkoor-tests#44).
Adds `benchmarkoor snapshot export` to publish a stopped execution client's data directory as a snapshot that existing restore tools can consume. - Streams a compressed archive directly to S3-compatible storage, avoiding a second full copy on local disk. - Supports Geth, Nethermind, Besu, Reth, Erigon and ethrex, with their expected archive layouts and filenames. Excludes node identities, saved peers and temporary files. - Verifies the uploaded size and hashes of sampled ranges before publishing the head block and metadata. Updates `latest` only when explicitly requested. - Aborts failed or interrupted uploads. Handles symlinked data directories without producing an empty archive. Validation: packing and upload tests, MinIO integration tests, corruption/failure cases, and publication of all six mainnet-sized snapshots used for the msf-2 mock. CI passes. CLI options and layouts are documented in [docs/snapshot-export.md](https://github.com/ethpandaops/benchmarkoor/blob/qu0b/snapshot-export/docs/snapshot-export.md). Used by [benchmarkoor-tests #44](ethpandaops/benchmarkoor-tests#44).
Adds `benchmarkoor db compact` so snapshot builders can run the existing offline maintenance without starting a benchmark run. Only Geth and Erigon are supported. The command uses their own `db compact` commands in a client container. Use `--image` to select the version that wrote the datadir; the client must be stopped first. Other clients return an unsupported error. The CLI reuses the runner's inspection, optional preparation steps, timeout and error handling. It streams container output and does not write benchmark markers or reports. No new compaction implementation, database-file replacement, journal parser, helper image or image-publishing workflow is included. Snapshot workflow [#44](ethpandaops/benchmarkoor-tests#44) uses this command for Geth and Erigon and skips compaction for the other clients. Validation: client/config/runner/CLI package tests and golangci-lint pass. New tests cover pinned client images, native command execution, unsupported clients, and image-pull/compaction failures.
There was a problem hiding this comment.
The PR adds benchmarkoor-snapshot.yaml plus five helper scripts and a builder config that turn a jochemnet release into six shadowfork snapshots, and the follow-up commit under review restores the benchmarkoor CLI from the already pulled/built local image after the composite action deletes it. All previously reported findings are resolved: geth/erigon compaction now passes the pinned --image "$SNAPSHOT_IMAGE", the unused --compaction-image/SNAPSHOT_PREFIX paths are gone, the shadowfork/ files those earlier findings referenced were removed by this branch, and both jobs re-extract the binary before compact/export. I found no remaining real problem in the change.
Reviewed 8 changed file(s) @ d7da397d — no blocking issues found.
"First law of programming: whatever the bug, blame DNS."
Adds a workflow that turns a jochemnet benchmark release into six snapshots for a shadowfork. They include the release's deployed contracts and start at the same block, with a lower gas limit.
latestafter all six exports succeed. Recover each runner's scratch data afterward.Geth must reopen at the expected head. Nethermind uses snapshot-specific persistence settings and must pass an offline reopen check before export, including its head hash and state root.
After the benchmarkoor action cleans up, each job restores the CLI from the same cached image for compaction and export.
Dependencies (merged): benchmarkoor #349, #350 and #351.
Validation: workflow YAML, shell syntax and ShellCheck pass. CLI extraction was checked with a Docker stub for supplied/local images and cleanup on success/failure. Earlier end-to-end builds and the finalized mock used the previous compaction policy; the revised workflow has not had a full six-client run.