Skip to content

fix(fill): re-anchor start_block under --no-reset-between-tests - #6

Open
skylenet wants to merge 3 commits into
jochem-brouwer:feat-deploy-script-oom-fixfrom
skylenet:fix/no-reset-reanchor-start-block
Open

skylenet wants to merge 3 commits into
jochem-brouwer:feat-deploy-script-oom-fixfrom
skylenet:fix/no-reset-reanchor-start-block

Conversation

@skylenet

Copy link
Copy Markdown

Problem

--no-reset-between-tests keeps the chain moving between fills, but the
session anchor never moves with it.

_session_pre_run captures start_block once, from latest, after the
global setup:

@pytest.fixture(scope="session", autouse=True)
def _session_pre_run(...):
    ...
    # 4. Capture start block (head after global setup).
    start_block = eth_rpc.get_block_by_number("latest")
    client_backend.start_block = start_block

That is the only production write to client_backend.start_block. Every
test then chains its first block from it, in make_stateful_fixture.

Normally the invariant holds because _reset_chain_between_tests rewinds
the chain back to that block after each test. Under
--no-reset-between-tests the rewind is skipped and nothing takes its
place, so the anchor goes stale as soon as the first test builds a block.

The nonce side of the flag was already adapted — worker_key reads at
latest under the flag, with a comment explaining the inversion. The
anchor side was missed.

Symptoms

Two failures with the same root cause. Which one appears depends on where
the client's state-history window sits:

where it fails error
worker_key -> get_account historical state ... is not available
build_block (first block of a later test) parentHash is not current head

Seen while filling
tests/benchmark/stateful/bloatnet/test_setup_contracts.py on a
jochemnet snapshot. Both code_size parametrizations run in one session;
the first passes and advances the chain about 7,700 blocks, and the
second dies on the anchor it never updated.

Fix

Read the head before each test and re-anchor on it, so every test starts
from what the previous test actually left behind. That is the chain the
accumulating mode is built around.

Consumer impact

None. Consumers route pre-run setup by directory (pre_run/*.json,
applied once per session), not by matching a fixture's
start_block_hash. Each test's fixture now records the anchor it truly
chains from, which is more accurate than every test claiming the session
anchor while depending on its predecessors.

Testing

  • ruff check and ruff format --check pass in repo context.
  • pytest --collect-only over tests/benchmark/stateful/bloatnet
    collects 382 tests, unchanged from the base commit.
  • Behaviour verified against the failing fill described above.

The session captures start_block once, at the head that follows the
global setup. `_reset_chain_between_tests` rewinds to that block after
every test, which keeps the anchor correct for the next fill.

`--no-reset-between-tests` skips the rewind, but nothing advances the
anchor in its place. The first test moves the head, and every later test
still chains its first block from the stale anchor. The client then
rejects the parent ("parentHash is not current head"), or, on a long run
against a non-archive client, reports "historical state ... is not
available" once that block's state is pruned.

Read the head before each test and re-anchor on it. Consumers are not
affected: they route pre-run setup by directory (`pre_run/*.json`,
applied once per session), not by a fixture's `start_block_hash`.
skylenet added a commit to ethpandaops/benchmarkoor-tests that referenced this pull request Aug 27, 2026
Use skylenet/execution-specs fix/no-reset-reanchor-start-block. The
branch adds the start_block re-anchor for --no-reset-between-tests on top
of feat-deploy-script-oom-fix. See jochem-brouwer/execution-specs#6.
`hash` built the whole fixture as one JSON string and then again as UTF-8
bytes, on top of the `json_dict` copy it already holds. That is three
full-size copies alive at once, right after a fill has finished.

Feed the encoder's chunks into the digest instead. `iterencode` emits the
identical character stream, so the hash value does not change; 406
randomised documents (unicode, escapes, floats, deep nesting) hash the
same both ways.
A stateful benchmark fill produces one `FixtureEngineNewPayload` per
block. `make_stateful_fixture` held all of them, and `json_dict` then made
a full `model_dump` copy on top -- hex strings, so larger than the models.
A 41k-block fill reached 46 GiB and the kernel killed it partway through
the copy.

`PayloadBuffer` keeps the first 512 payloads in memory, which is every
ordinary fill, and behaves exactly as the list it replaces: the fixture
field is populated and nothing touches the disk. Past that it moves to a
temp file holding one payload per line of canonical JSON -- the exact form
`BaseFixture.hash` digests -- and the fixture field is left empty.

`hash` and the new `write_json` then stream that text back in at the right
place, one payload at a time. The hash is byte-identical either way, so a
fixture does not change identity because of how it was built.

Measured on a 1 GB fixture document: peak RSS 2957 MiB buffered vs 79 MiB
spilled, same hash.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant