Skip to content

feat(snapshot): build shadowfork snapshots from a jochemnet release - #44

Merged
qu0b merged 14 commits into
masterfrom
qu0b/shadowfork-image-build
Oct 12, 2026
Merged

qu0b merged 14 commits into
masterfrom
qu0b/shadowfork-image-build

Conversation

@qu0b

@qu0b qu0b commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Adds a workflow that turns a jochemnet benchmark release into six snapshots for a shadowfork. They include the release's deployed contracts and start at the same block, with a lower gas limit.

  1. Verify the release bundle, replay it on Geth, and lower the gas limit to 200M by default.
  2. Replay the same blocks on the other five clients.
  3. Compact Geth and Erigon through their own commands. Explicitly skip compaction for Nethermind, Besu, Reth and Ethrex; no external database tools or manual file swaps.
  4. Export the datadirs and update latest after all six exports succeed. Recover each runner's scratch data afterward.

Geth must reopen at the expected head. Nethermind uses snapshot-specific persistence settings and must pass an offline reopen check before export, including its head hash and state root.

After the benchmarkoor action cleans up, each job restores the CLI from the same cached image for compaction and export.

Dependencies (merged): benchmarkoor #349, #350 and #351.

Validation: workflow YAML, shell syntax and ShellCheck pass. CLI extraction was checked with a Docker stub for supplied/local images and cleanup on success/failure. Earlier end-to-end builds and the finalized mock used the previous compaction policy; the revised workflow has not had a full six-client run.

A fresh mainnet snapshot advanced by one pre-run (EIP-8282 contracts on Osaka, Amsterdam at
head + 3 s, 24 KiB + 64 KiB receivers at 100k per variant), its gas limit walked down to 200M,
the fill flushed out of geth's journal and compacted, then the geth image and the replay bundle
uploaded to ethpandaops-shadowfork-images. The other clients replay the bundle separately and
latest is written once all five are up (shadowfork/README.md).

Clean image: geth's path scheme journaled 1,354 blocks of the last fill on a clean stop, so the
newest contracts were read from memory. The ramp-down runs with --cache.gc=0 (0-byte write
buffer) and the step fails unless the journal holds only the last 128 diff layers; then
geth db compact. Verified on a dev chain (layers 176 -> 128); rampdown.py built 256 Amsterdam
blocks (V5, slot numbers, 11.5M -> 10M) whose bundle replayed on a second node to the same head.

Runs on the synctest Disk4TB runners (the gasbenchmark hosts hold the jochemnet CI baselines);
uses the repo's existing S3 credentials and probes them before downloading anything.
@qu0b qu0b added the redpanda-review Ask redpandabot for a review of this PR label Oct 8, 2026
redpandabot[bot]

This comment was marked as outdated.

qu0b added 2 commits October 8, 2026 14:02
…#3304

qu0b/bench-images-v3 replaces the skylenet-based v2: upstream forks/amsterdam (2026-09-25) with
the two open upstream PRs the fill needs, ethereum/execution-specs#3648 (spill payload lists to
disk) and #3304 (the deployment script), plus the combined-sizes test and JUMPDEST at 24 KiB.
Replaces shadowfork-image-build and its scripts. geth replays the release's
pre-run (base_bundle) and walks the gas limit down (fill: false), every other
client replays release + ramp, each datadir is compacted (db compact) and
exported (snapshot export) to <bucket>/jochemnet/<client>/<head>/, and latest
is written once all are up. Runs on the jochemnet hosts and recovers their
schelk scratch afterwards.
@qu0b qu0b changed the title shadowfork-image-build: post-Gloas benchmark images (geth half), cleaned and gas-ramped benchmarkoor-snapshot: a jochemnet release as a compacted, gas-ramped shadowfork snapshot Oct 8, 2026
…head block

snapshot export publishes the head block as-is, and its contract is the
full-transaction eth_getBlockByNumber response.
redpandabot[bot]

This comment was marked as outdated.

…ots on it

Found by the e2e prototype: geth on a flat instance dir opens the database but
misses triedb/merkle.journal, then rewinds the head and truncates state history.
redpandabot[bot]

This comment was marked as outdated.

qu0b added 3 commits October 8, 2026 20:38
…he datadir

Without --image the CLI pulled ethereum/client-go:stable (1.17.8 today) for a
datadir written by v1.17.7 (found by the e2e prototype); a newer geth can
migrate the database format on open.
…ing tail; compaction notes per client

reth compacts (benchmarkoor #351: reth db copy --compact), ethrex does not: its
TRANSACTION_LOCATIONS family has a merge operator ldb lacks, so the generic
RocksDB compaction refuses the database.
…uts too

From jochemnet's archives, extracted on the build droplets: reth v2 has
static_files/ and rocksdb/ at the root, erigon's pruned archive is the datadir
(chaindata/, snapshots/), ethrex's is the RocksDB directory that lives at
<datadir>/chain-1.
redpandabot[bot]

This comment was marked as outdated.

@qu0b
qu0b marked this pull request as ready for review October 12, 2026 08:59

@redpandabot redpandabot Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

The PR replaces the old shadowfork-image-build workflow/scripts with benchmarkoor-snapshot.yaml, four helper scripts and a jochemnet snapshot builder config that replay a release, ramp the gas limit down, compact where a client command exists, and publish per-client snapshots. The latest commit correctly drops the external ldb compaction (nethermind/besu/reth/ethrex now skipped, geth/erigon use their pinned client image), and the earlier out-of-scope shadowfork-file and SNAPSHOT_PREFIX findings are gone. One blocker from the previous reviews remains: the benchmarkoor binary the workflow calls for compaction and export is deleted by the benchmarkoor action before those steps run.

Issues

  • 🔴 .github/workflows/benchmarkoor-snapshot.yaml:203 — benchmarkoor CLI is deleted by the action before db compact / snapshot export — see the thread on that line

Reviewed @ 28ac19d4
"First law of programming: whatever the bug, blame DNS."

# --image: the release that wrote the datadir compacts it; the CLI default is the client's
# :stable, a newer geth that may migrate the database format on open.
- name: Compact
run: sudo -E "$RUNNER_TEMP/benchmarkoor" db compact --client "$CLIENT" --datadir "$DATADIR" --image "$SNAPSHOT_IMAGE"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 benchmarkoor CLI is deleted by the action before db compact / snapshot export

The ethpandaops/benchmarkoor@master steps (lines 146 and 263) are a composite action whose final Cleanup step runs rm -f "${{ runner.temp }}/benchmarkoor" (benchmarkoor action.yaml:455, which is the same $RUNNER_TEMP/benchmarkoor the action's Extract benchmarkoor binary step wrote at line 135). By the time the Compact steps (line 206 and line 291) and snapshot-export.sh (line 22) run, that binary no longer exists, so every client fails with No such file or directory before any archive is published. Re-extract the binary (or run the CLI from the image) after the action step, before compacting/exporting.

qu0b added a commit to ethpandaops/benchmarkoor that referenced this pull request Oct 12, 2026
…p-only pre-run (#349)

A released benchmark snapshot can have a much higher block gas limit
than the shadowfork needs. This lets a pre-run lower that limit—for
example, from 1T to 200M—and record the new blocks for other clients to
replay.

- Uses Amsterdam's `targetGasLimit` to ramp down and hold the requested
limit. A configured fork transition can complete before ramp-down
starts; the block limit still applies.
- Adds `fill: false` to skip EEST setup and `base_bundle` to continue
from an existing recorded chain. The new bundle contains only the
additional blocks, so consumers replay the base first.
- Streams large replay bundles instead of loading them into memory, and
continues the parent's Amsterdam slot number.
- Gives every pre-run client time to flush its database on shutdown. A
forced kill or failed shutdown fails the target.

Validation: builder/config tests, fork-transition regressions, real Geth
gas-limit checks, and replay of the 49.7 GB release bundle. CI passes.

Used by [benchmarkoor-tests
#44](ethpandaops/benchmarkoor-tests#44).
qu0b added a commit to ethpandaops/benchmarkoor that referenced this pull request Oct 12, 2026
Adds `benchmarkoor snapshot export` to publish a stopped execution
client's data directory as a snapshot that existing restore tools can
consume.

- Streams a compressed archive directly to S3-compatible storage,
avoiding a second full copy on local disk.
- Supports Geth, Nethermind, Besu, Reth, Erigon and ethrex, with their
expected archive layouts and filenames. Excludes node identities, saved
peers and temporary files.
- Verifies the uploaded size and hashes of sampled ranges before
publishing the head block and metadata. Updates `latest` only when
explicitly requested.
- Aborts failed or interrupted uploads. Handles symlinked data
directories without producing an empty archive.

Validation: packing and upload tests, MinIO integration tests,
corruption/failure cases, and publication of all six mainnet-sized
snapshots used for the msf-2 mock. CI passes.

CLI options and layouts are documented in
[docs/snapshot-export.md](https://github.com/ethpandaops/benchmarkoor/blob/qu0b/snapshot-export/docs/snapshot-export.md).
Used by [benchmarkoor-tests
#44](ethpandaops/benchmarkoor-tests#44).
@qu0b qu0b changed the title benchmarkoor-snapshot: a jochemnet release as a compacted, gas-ramped shadowfork snapshot feat(snapshot): build shadowfork snapshots from a jochemnet release Oct 12, 2026
qu0b added a commit to ethpandaops/benchmarkoor that referenced this pull request Oct 12, 2026
Adds `benchmarkoor db compact` so snapshot builders can run the existing
offline maintenance without starting a benchmark run.

Only Geth and Erigon are supported. The command uses their own `db
compact` commands in a client container. Use `--image` to select the
version that wrote the datadir; the client must be stopped first. Other
clients return an unsupported error.

The CLI reuses the runner's inspection, optional preparation steps,
timeout and error handling. It streams container output and does not
write benchmark markers or reports.

No new compaction implementation, database-file replacement, journal
parser, helper image or image-publishing workflow is included. Snapshot
workflow
[#44](ethpandaops/benchmarkoor-tests#44) uses
this command for Geth and Erigon and skips compaction for the other
clients.

Validation: client/config/runner/CLI package tests and golangci-lint
pass. New tests cover pinned client images, native command execution,
unsupported clients, and image-pull/compaction failures.

@redpandabot redpandabot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR adds benchmarkoor-snapshot.yaml plus five helper scripts and a builder config that turn a jochemnet release into six shadowfork snapshots, and the follow-up commit under review restores the benchmarkoor CLI from the already pulled/built local image after the composite action deletes it. All previously reported findings are resolved: geth/erigon compaction now passes the pinned --image "$SNAPSHOT_IMAGE", the unused --compaction-image/SNAPSHOT_PREFIX paths are gone, the shadowfork/ files those earlier findings referenced were removed by this branch, and both jobs re-extract the binary before compact/export. I found no remaining real problem in the change.


Reviewed 8 changed file(s) @ d7da397d — no blocking issues found.
"First law of programming: whatever the bug, blame DNS."

@qu0b
qu0b merged commit 592e91b into master Oct 12, 2026
@qu0b
qu0b deleted the qu0b/shadowfork-image-build branch October 12, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

redpanda-review Ask redpandabot for a review of this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant