refactor(codegen): share generated tree walker engine - #341
Merged
Conversation
Copy/Paste DetectionFound 9 duplication(s) across 7 changed non-generated Rust file(s) (threshold: 100 tokens). Show duplicationsFound a 26 line (142 tokens) duplication in the following files:
atn.set_end_state(1, 4).expect("block end state");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 2 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 3 })
.expect("transition");
atn.add_transition(
2,
ParserTransitionSpec::Atom {
target: 4,
label: 1,
},
)
.expect("transition");
atn.add_transition(
3,
ParserTransitionSpec::Atom {
target: 4,
label: 2,
},
)
.expect("transition");
atn.add_transition(4, ParserTransitionSpec::Epsilon { target: 5 })
.expect("transition");
atn.add_decision_state(1).expect("decision state");
```rust
---
Found a 26 line (125 tokens) duplication in the following files:
* Starting at line 794 of crates/antlr-rust-runtime/src/generated.rs
* Starting at line 965 of crates/antlr-rust-runtime/src/generated.rs
```rust
$input: $crate::char_stream::CharStream,
$hooks: $crate::parser::SemanticHooks,
{
pub fn metadata() -> &'static $crate::generated::GrammarMetadata {
$metadata()
}
/// Adds a listener for lexer diagnostics.
pub fn add_error_listener<T>(&mut self, listener: T)
where
T: for<'a> $crate::errors::ErrorListener<dyn $crate::recognizer::Recognizer + 'a>
+ ::core::marker::Send
+ 'static,
{
$crate::recognizer::Recognizer::add_error_listener(&mut self.$base, listener);
}
/// Removes every lexer error listener, including the default console listener.
pub fn remove_error_listeners(&mut self) {
$crate::recognizer::Recognizer::remove_error_listeners(&mut self.$base);
}
/// Routes every token through ATN interpretation instead of the compiled
/// lexer DFA, so the learned-DFA trace (`lexer_dfa_string`) observes each
/// match.
pub fn set_force_interpreted(&mut self, force_interpreted: bool) {Found a 25 line (115 tokens) duplication in the following files:
atn.add_state(AtnStateKind::BlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
3
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))
.expect("state")
.index(),
4
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))
```rust
---
Found a 22 line (112 tokens) duplication in the following files:
* Starting at line 4286 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 4360 of crates/antlr-rust-codegen/src/generator/tests.rs
```rust
fn plus_loop_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::PlusBlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))Found a 27 line (110 tokens) duplication in the following files:
decision: 0,
alts: (1, 2),
track_alt_number: false,
allow_semantic_context: true,
force_context: false,
plus_loop: false,
fast_path: None,
body: &body,
},
0,
GeneratedStepRenderContext {
current_rule_index: 0,
embedded: None,
portable_locals: None,
decision_routing: DecisionRoutingRender::default(),
inline_action_statements: &BTreeMap::new(),
track_alt_numbers: false,
track_context_alt_numbers: false,
direct_generated_rule_calls: &[],
atn_preferred_rule_calls: &[],
adaptive_atn_preferred_rule_slots: &[],
adaptive_atn_probe_rule_slots: &[],
},
);
// The whole rendered star-loop captures the leading-predicate-to-exit-alt filtering.
insta::assert_snapshot!(
```rust
---
Found a 17 line (105 tokens) duplication in the following files:
* Starting at line 183 of crates/antlr-rust-runtime/src/generated.rs
* Starting at line 280 of crates/antlr-rust-runtime/src/generated.rs
```rust
fn __from_node_with_invocation_states(
node: $crate::RuleNodeView<'a>,
invocation_states: Option<Vec<isize>>,
) -> Self {
$(
let __default = <$attrs>::default();
let __attrs = node.generated_attrs::<$attrs>().unwrap_or(&__default);
)?
Self {
__node: __GeneratedRuleContext::Stored(node),
__invocation_states: invocation_states,
__state: std::marker::PhantomData,
$(
$($field: __attrs.$field.clone(),)+
)?
}
}Found a 25 line (104 tokens) duplication in the following files:
state: 1,
decision: 0,
track_alt_number: false,
allow_semantic_context: false,
force_context: false,
fast_path: None,
alts: &alts,
},
0,
GeneratedStepRenderContext {
current_rule_index: 0,
embedded: None,
portable_locals: None,
decision_routing: DecisionRoutingRender::default(),
inline_action_statements: &BTreeMap::new(),
track_alt_numbers: false,
track_context_alt_numbers: false,
direct_generated_rule_calls: &[],
atn_preferred_rule_calls: &[],
adaptive_atn_preferred_rule_slots: &[],
adaptive_atn_probe_rule_slots: &[],
},
);
assert!(rendered.contains("ll1_decision_prediction(atn(), 1)"));
```rust
---
Found a 28 line (102 tokens) duplication in the following files:
* Starting at line 3306 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 3466 of crates/antlr-rust-codegen/src/generator/tests.rs
```rust
state: 1,
decision: 0,
track_alt_number: false,
allow_semantic_context: true,
force_context: false,
fast_path: None,
alts: &alts,
},
0,
GeneratedStepRenderContext {
current_rule_index: 0,
embedded: None,
portable_locals: None,
decision_routing: DecisionRoutingRender::default(),
inline_action_statements: &BTreeMap::new(),
track_alt_numbers: false,
track_context_alt_numbers: false,
direct_generated_rule_calls: &[],
atn_preferred_rule_calls: &[],
adaptive_atn_preferred_rule_slots: &[],
adaptive_atn_probe_rule_slots: &[],
},
);
// One decision renders into a fresh String; snapshot the whole emitted control flow (the
// semantic-context gate, both predicate probes, the alt rewrite, the no-viable fallback)
// instead of six positive probes plus one negative guard.
insta::assert_snapshot!(Found a 16 line (101 tokens) duplication in the following files:
atn.set_loop_back_state(3, 4).expect("loop back state");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 2 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 3 })
.expect("transition");
atn.add_transition(
2,
ParserTransitionSpec::Atom {
target: 4,
label: 1,
},
)
.expect("transition");
atn.add_transition(4, ParserTransitionSpec::Epsilon { target: 1 })
```rust |
📝 WalkthroughWalkthroughThe runtime now provides ChangesGenerated tree walking consolidation
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant GeneratedTreeWalker
participant RuntimeWalkGenerated
participant ListenerCallbacks
GeneratedTreeWalker->>RuntimeWalkGenerated: pass parse tree and invocation states
RuntimeWalkGenerated->>ListenerCallbacks: emit enter, terminal, error, and exit callbacks
ListenerCallbacks-->>RuntimeWalkGenerated: return callback result
RuntimeWalkGenerated-->>GeneratedTreeWalker: return traversal result
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (3 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Code review — PR #341
|
Contributor
Merging this PR will not alter performance
Performance Changes
Comparing 
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@benches/benches/grammar_frontend.rs`:
- Around line 146-152: Update generated_listener_walk to wrap both parsed.tree()
and the result of TomlTreeWalker::walk in std::hint::black_box, preserving the
existing NoopTomlListener and successful-walk handling while preventing
traversal work from being optimized away.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: f62ef982-9d49-45de-9769-476bb14c8219
⛔ Files ignored due to path filters (14)
benches/Cargo.lockis excluded by!**/*.lockcrates/antlr-rust-codegen/src/generator/snapshots/antlr_rust_codegen__generator__tests__generated_module_file_header.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__cli__generated_codegen_api_checks.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__cli__generated_codegen_api_mismatch_diagnostic.snapis excluded by!**/*.snapcrates/antlr-rust-g4-parser/src/generated/antlr_v4_lexer.rsis excluded by!**/generated/**crates/antlr-rust-g4-parser/src/generated/antlr_v4_parser.rsis excluded by!**/generated/**crates/antlr-rust-rs-parser/src/generated/rust_lexer.rsis excluded by!**/generated/**crates/antlr-rust-rs-parser/src/generated/rust_parser.rsis excluded by!**/generated/**crates/antlr-rust-runtime/src/snapshots/antlr4_runtime__generated__tests__generated_walk_order_and_invocation_states.snapis excluded by!**/*.snapcrates/antlr-rust-runtime/src/snapshots/antlr4_runtime__generated__tests__generated_walk_short_circuit.snapis excluded by!**/*.snapcrates/antlr-rust-runtime/src/xpath/generated/x_path_lexer.rsis excluded by!**/generated/**crates/antlr-rust-toml-parser/src/generated/toml_lexer.rsis excluded by!**/generated/**crates/antlr-rust-toml-parser/src/generated/toml_parser.rsis excluded by!**/generated/**docs/migration.mdis excluded by!**/docs/**
📒 Files selected for processing (10)
README.mdbenches/Cargo.tomlbenches/benches/grammar_frontend.rscrates/antlr-rust-codegen/src/generator/tests.rscrates/antlr-rust-codegen/src/parser/surface/contexts.rscrates/antlr-rust-codegen/src/parser/surface/traversal.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/cli.rscrates/antlr-rust-runtime/src/generated.rscrates/antlr-rust-runtime/src/lib.rsthird_party/antlr-v4-grammar/self-hosted.sha256
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
tinovyatkin
force-pushed
the
issue-324-runtime-walk-engine
branch
from
August 12, 2026 13:52
e9a9175 to
61106c8
Compare
📊 Source Code Metrics (this PR vs
|
| File | Cyclomatic | Cognitive | Functions | LLOC | MI |
|---|---|---|---|---|---|
| crates/antlr-rust-codegen/src/generator/tests.rs | 292 ⚪ | 48 ⚪ | 207 ⚪ | 1336 (main: 1332) 🔴 | 0 ⚪ |
| crates/antlr-rust-runtime/src/generated.rs | 126 (main: 102) 🔴 | 33 (main: 14) 🔴 | 60 (main: 51) 🔴 | 151 (main: 101) 🔴 | 0 ⚪ |
| crates/antlr-rust-codegen/src/parser/surface/contexts.rs | 25 ⚪ | 22 ⚪ | 4 ⚪ | 69 ⚪ | 16.53 (main: 16.57) 🔴 |
| crates/antlr-rust-codegen/src/parser/surface/traversal.rs | 6 ⚪ | 2 ⚪ | 3 ⚪ | 36 ⚪ | 18.40 (main: 17.73) 🟢 |
| crates/antlr-rust-runtime/src/lib.rs | 4 ⚪ | 3 ⚪ | 1 ⚪ | 7 ⚪ | 29.44 (main: 29.52) 🔴 |
Generated by mehen v1.8.1 — the code quality watcher.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 61106c864e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Move iterative listener traversal into antlr-rust-runtime and keep only grammar-specific callback dispatch in generated parsers. Preserve public walker and listener APIs, visitation ordering, invocation-state threading, and callback error propagation. Use generic inlined callbacks after a dynamic-dispatch prototype regressed the hardened direct listener-walk benchmark by 16.7%. Add direct runtime coverage and a CodSpeed listener-walk benchmark that black-boxes its input and result. Bump the generated-code API to revision 13 while retaining revision-12 runtime compatibility, and regenerate the checked-in G4, Rust, TOML, and XPath recognizers together with compatibility documentation and snapshots.
tinovyatkin
force-pushed
the
issue-324-runtime-walk-engine
branch
from
August 12, 2026 13:59
61106c8 to
ecb4492
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
antlr-rust-runtimeNet effect
This is a modest consolidation, not a large feature or a large total-LOC
reduction.
That includes the new shared runtime implementation, compatibility support,
tests, benchmark, documentation, and generated-file churn.
and 31,617 bytes smaller.
antlr4-rust-genbinary linking all three parsers is 121,680bytes smaller, including 114,688 fewer
__TEXTbytes.The main long-term benefit is that traversal behavior now has one
grammar-agnostic runtime implementation instead of being maintained in every
generated plain and validated walker. Future generated parsers pay only for
their grammar-specific dispatch.
Measurements
antlr4-rust-genfile bytesantlr4-rust-gen__TEXTA trait-object callback prototype measured 3.208 us versus a 2.749 us
baseline (+16.7%), so the final runtime engine is generic over the callback
adapter and keeps the dispatch methods inlined. The benchmark black-boxes both
the tree input and walk result to prevent dead-code elimination.
Validation
cargo fmt -- --checkcargo test --locked --workspace --all-featurescargo clippy --locked --workspace --all-targets --all-features -- -D warningsListeners/*descriptorsCloses #324.