test(spec): assert the rejection reason on expected-failure fixtures - #547
test(spec): assert the rejection reason on expected-failure fixtures#547MegaRedHand wants to merge 3 commits into
Conversation
Negative leanSpec fixtures name *why* their input must be rejected in a `rejectionReason` field, but the runners only asserted that some error came back. A fixture could therefore pass on a failure unrelated to the rule it exercises, which is what four state-transition fixtures were doing: a late state-root mismatch stood in for the attestation-data cap, the justification window and the block-slot check. Mirror leanSpec's `RejectionReason` vocabulary as a typed enum, classify client errors into it, and compare the two on every expected failure. An unclassified error and a reason string this build does not know both fail the fixture: accepting either would restore "any error will do". Unknown reasons stay deserializable as `Unknown(_)` so the Hive test driver still answers such a step over HTTP instead of rejecting the request; only the offline runners treat them as failures. `AggregateVerificationFailed` covered both an attestation aggregate and a block's merged proof, which the spec separates into `INVALID_SIGNATURE` and `INVALID_BLOCK_PROOF`, so block-proof verification now has its own variant.
🤖 Kimi Code ReviewThis PR significantly improves the spec-test infrastructure by introducing structured rejection reasons, ensuring tests fail for the correct reasons rather than any error. The implementation is sound and follows Rust best practices. Summary of Changes
Detailed Reviewcrates/common/test-fixtures/src/rejection.rs (New)Line 20-55: The Line 180-211: The Line 157-168: The crates/blockchain/src/spec_test_runner.rsLine 62-63: When classifying Line 67-95: The Line 98-102: Explicitly returning crates/blockchain/src/store.rsLine 1061-1062: Adding crates/blockchain/tests/forkchoice_spectests.rsLine 64-81: The anchor rejection handling correctly validates that the only expressible anchor failure is Line 173-196: The crates/net/rpc/src/test_driver.rsLine 234: Converting to Minor Suggestions
Consensus SafetyNo concerns. This PR only affects test infrastructure and error classification. No changes to:
The production code change (new Verdict: Approve. The PR is well-structured, exhaustively handles error variants, and prevents the test suite from accepting false positives. Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| } | ||
| (Err(_), None) => { | ||
| // Expected failure | ||
| // Expected failure. When the fixture names why, the transition must | ||
| // have failed for that reason: a state-root mismatch standing in for |
There was a problem hiding this comment.
When the existing negative fixtures exercise attestation limits, justification ranges, or skipped slot processing, this unconditional reason comparison receives a different transition error and fails four stf_spectests, leaving the required CI target red.
Knowledge Base Used: Blockchain core: fork choice, state transition, block building, sync
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/blockchain/state_transition/tests/stf_spectests.rs
Line: 82-84
Comment:
**STF fixtures now fail deterministically**
When the existing negative fixtures exercise attestation limits, justification ranges, or skipped slot processing, this unconditional reason comparison receives a different transition error and fails four `stf_spectests`, leaving the required CI target red.
**Knowledge Base Used:** [Blockchain core: fork choice, state transition, block building, sync](https://app.greptile.com/lambdaclass/-/custom-context/knowledge-base/lambdaclass/ethlambda/-/docs/blockchain-core.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| version.workspace = true | ||
|
|
||
| [dependencies] | ||
| ethlambda-state-transition.workspace = true |
There was a problem hiding this comment.
Adding ethlambda-state-transition as a regular dependency couples the common fixture model to a blockchain implementation crate, violating the repository's documented dependency direction and expanding the build surface for every fixture consumer.
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/common/test-fixtures/Cargo.toml
Line: 13
Comment:
**Common crate crosses dependency layers**
Adding `ethlambda-state-transition` as a regular dependency couples the common fixture model to a blockchain implementation crate, violating the repository's documented dependency direction and expanding the build surface for every fixture consumer.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Greptile SummaryThis PR adds typed rejection-reason assertions across the offline spec runners.
Confidence Score: 4/5This PR should not merge until the four deterministic STF spec-test failures are resolved or handled without weakening the new reason assertions. The new comparison correctly exposes rejection mismatches, but it is enabled unconditionally for fixtures the current STF replay path cannot satisfy, leaving a required test target red; the added common-to-blockchain dependency is also an actionable layering regression. Files Needing Attention: crates/blockchain/state_transition/tests/stf_spectests.rs and crates/common/test-fixtures/Cargo.toml
|
| Filename | Overview |
|---|---|
| crates/common/test-fixtures/src/rejection.rs | Adds the shared rejection-reason enum, comparison helper, and state-transition error classifier. |
| crates/blockchain/src/spec_test_runner.rs | Adds typed fork-choice step errors and classifies store failures for fixture assertions. |
| crates/blockchain/state_transition/tests/stf_spectests.rs | Enforces rejection-reason equality but deterministically leaves four existing fixtures failing. |
| crates/common/test-fixtures/Cargo.toml | Adds a higher-layer state-transition dependency to a common crate, contrary to the documented dependency direction. |
| crates/blockchain/src/store.rs | Splits block-proof verification failures into a dedicated StoreError variant. |
| crates/blockchain/tests/forkchoice_spectests.rs | Validates negative fork-choice steps against their expected typed rejection reasons. |
| crates/blockchain/tests/signature_spectests.rs | Validates signature-test failures against their expected rejection reasons. |
| crates/net/rpc/src/test_driver.rs | Converts typed StepError values to strings while preserving the existing HTTP response shape. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
Fixture["Negative fixture<br/>rejectionReason"] --> Runner["Offline spec runner"]
Runner --> Operation["Fork choice / STF / signature verification"]
Operation --> Error["Typed client error"]
Error --> Classifier["RejectionReason classifier"]
Fixture --> Compare["check_rejection_reason"]
Classifier --> Compare
Compare -->|match| Pass["Fixture passes"]
Compare -->|mismatch / unknown| Fail["Fixture fails"]
Prompt To Fix All With AI
### Issue 1
crates/blockchain/state_transition/tests/stf_spectests.rs:82-84
**STF fixtures now fail deterministically**
When the existing negative fixtures exercise attestation limits, justification ranges, or skipped slot processing, this unconditional reason comparison receives a different transition error and fails four `stf_spectests`, leaving the required CI target red.
### Issue 2
crates/common/test-fixtures/Cargo.toml:13
**Common crate crosses dependency layers**
Adding `ethlambda-state-transition` as a regular dependency couples the common fixture model to a blockchain implementation crate, violating the repository's documented dependency direction and expanding the build surface for every fixture consumer.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (1): Last reviewed commit: "test(spec): assert the rejection reason ..." | Re-trigger Greptile
🤖 Claude Code ReviewReview: PR #547 — assert rejection reason on expected-failure fixturesOverall: Solid, well-scoped test-infrastructure change. The core idea — mirroring leanSpec's
Minor observations (not blocking):
No security, memory-safety, or fork-choice/state-transition-logic issues found — this PR only changes how test failures are classified and reported, plus the one legitimate Automated review by Claude (Anthropic) · sonnet · custom prompt |
🤖 Codex Code ReviewFindings
Verification Static review only. I could not run Cargo checks in this environment because dependency resolution needs network access and a writable Cargo home. Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
Both are authored with leanSpec's `BlockSpec.skip_slot_processing`, which makes the filler call `process_block` alone, but the flag never reaches the emitted fixture: `StateTransitionFixture` carries only pre, blocks, post, postStateRoot and rejectionReason. Replaying `state_transition()` as the fixture format prescribes therefore runs `process_slots` first and dies on the placeholder zero state root instead of the rule under test, and nothing in the JSON marks the intended entry point. ream, zeam, gean, lantern and grandine all "pass" these two on that same state-root mismatch because they assert only that some error occurred; grandine's negative path even counts a state-root mismatch as the expected failure outright. Skipping states the limitation instead of pretending the assertion holds. The two remaining state-transition failures are real divergences, not replay artifacts, and stay red.
The macro saved one copy of the 36-entry table at the cost of making the enum invisible to readers, rustdoc and go-to-definition. Write the enum, `as_str` and `From<&str>` out directly. Drift between the two tables cannot pass silently: `as_str` is exhaustive, so a new variant fails to compile until it has a wire spelling, and a missing `From` arm lands the reason in `Unknown`, which every runner reports as a failure naming the string to add.
1 participant
Summary
Negative leanSpec fixtures name why their input must be rejected, in a
rejectionReasonfield. The spec-test runners parsed that field only sodeny_unknown_fieldswould accept it and asserted nothing more than "some errorcame back", so a fixture could pass on a failure unrelated to the rule it
exercises.
This wires the reason into the assertion for all three offline runners
(fork choice, state transition, verify signatures).
What changed
RejectionReason: typed mirror of leanSpec'sRejectionReasonStrEnum (36 reasons)crates/common/test-fixtures/src/rejection.rsStoreError→ reason classifier, exhaustive so new variants must be classifiedspec_test_runner::rejection_reasonstate_transition::Error→ reason, as a totalFromimpltest-fixtures/src/rejection.rsapply_fork_choice_stepnow returns a typedStepError(store rejection vs harness failure) instead ofStringspec_test_runnerrejection::check_rejection_reasonTwo failure modes are treated as test failures, not passes, since accepting
either restores "any error will do":
Unknown reasons still deserialize (as
Unknown(_)) so the Hive test driverkeeps answering a step over HTTP rather than 422-ing the request when leanSpec
adds a reason name; only the offline runners fail on them.
StoreError::AggregateVerificationFailedwas raised both for an attestationaggregate (
INVALID_SIGNATURE) and for a block's merged multi-message proof(
INVALID_BLOCK_PROOF). Since the spec separates those, block-proofverification now returns its own
BlockProofVerificationFailedvariant, whichalso makes production logs say which proof failed.
Results
exactly the reason their fixture names.
The 2 remaining failures are real divergences
They passed before for the wrong reason; the new assertion is what surfaces
them. Left failing rather than allowlisted, so the divergence stays visible:
test_block_exceeding_distinct_attestation_data_cap_rejects_blockTOO_MANY_ATTESTATION_DATASTATE_ROOT_MISMATCHMAX_ATTESTATIONS_DATAcap is enforced on block import (store.rs), not in the STF, so replaying the STF alone misses ittest_source_slot_beyond_tracked_range_rejects_blockJUSTIFIED_SLOT_OUT_OF_RANGESTATE_ROOT_MISMATCHis_valid_votetreats a source past the tracked justification window as "not justified" and skips the vote, where the spec rejects the blockThe first is a layering gap rather than live exposure: a node still rejects the
capped block at import. Both want follow-up PRs.
The 2 skips are unreplayable fixtures, not client bugs
test_block_with_wrong_slotandtest_block_at_parent_slot_rejected_when_slot_processing_skippedare authoredwith leanSpec's
BlockSpec.skip_slot_processing(block_spec.py:71), whichmakes the filler call
process_blockalone. That flag never reaches the emittedfixture —
StateTransitionFixturecarries onlypre,blocks,post,postStateRoot,rejectionReason— and the failing block is written with aplaceholder zero
stateRoot. Replayingstate_transition(), which is what thefixture format's own description prescribes, therefore advances the slot first
and then dies on the state root instead of reaching the rule under test.
Nothing in the JSON marks the entry point, so no client can reproduce these
two. Verified against fresh clones: ream, zeam, gean, lantern and grandine all
"pass" them on the same state-root mismatch because they assert only that some
error occurred (grandine's negative path counts a state-root mismatch as the
expected failure outright). qlean's 2-month-old vendored copy has the identical
shape under the older
expectException: AssertionErrorspelling, so this is nota regression: sharpening the fixture's claim to a specific reason is what made
it checkable, and therefore visibly unsatisfiable.
Upstream fix would be to emit the entry point per block (
skipSlotProcessing,or
entryPoint: state_transition | process_block), or to drop these two vectorssince they pin a Python-level API contract rather than cross-client behaviour.
Testing
Three unit tests cover reason parsing, the unknown-reason fallback and JSON
deserialization.