feat(codegen): centralize parse-driver and entry-point scaffolding in the runtime - #332
Merged
Conversation
… the runtime
Every generated parser repeated a 113-line parse-driver core and a
200-550-line entry-point block that were byte-identical across grammars
modulo the type-name prefix and exactly one divergent line per feature
axis. Generated modules now emit two thin macro invocations instead, so
the driver semantics have a single runtime-owned copy and the two
hand-maintained template variants disappear.
Runtime (generated-code API revision 10 -> 11):
- `__antlr4_rust_parser_driver!` expands the six driver methods
(`parse_rule`, `parse_rule_precedence`,
`parse_rule_precedence_from_generated`, `parse_rule_precedence_inner`,
`parse_interpreted_rule`, `parse_interpreted_rule_precedence`).
Generated code supplies only the interpreted-fallback binder block
that composes its grammar-specific `ParserRuntimeOptions`, plus an
`adaptive_direct` flag. Action dispatch is uniform: the driver always
routes deferred actions through the module's `run_action` (an empty
method for grammars without action states), replacing the divergent
`for action in actions { ... }` vs `let _ = actions;` template line.
- `__antlr4_rust_parser_entry_points!` expands the `parse` /
`parse_validated` / `parse_with_parser` / `parse_stream` /
`parse_stream_validated` / `parse_stream_with_parser` functions, the
`<Grammar>ParserParseOutput` alias of the new runtime
`GeneratedParseOutput<R, P>`, and the validation bridge behind the
doc-hidden `__GeneratedParserValidate` trait, keeping `validate()`
and the field/accessor surface source compatible.
- `GeneratedRuleError` moves to the runtime as a grammar-agnostic enum
whose `AdaptiveRetry` variant always exists; the adaptive-ATN retry
state bundle becomes `AdaptiveAtnRetryState<const RULES: usize>`
enabled by a const knob (`RULES = 0` when a grammar has no residual
adaptive routing; `retry_pending()` then constant-folds to `false`),
and rule bodies carry a uniform `retry [adaptive];` clause handled by
a new arm of `__antlr4_rust_generated_rule!`.
- Lexer `lex` / `lex_stream` become runtime generic functions that
generated lexer modules re-export.
The driver's entry-ordering invariants (fail-loud surfacing before Ok,
diagnostics before abort before semantic-miss draining, shared
`allow_generated_fallback` gate) were previously pinned per grammar by
generator rendered-text tests; they are now pinned once by a runtime
test over the macro source, and the generator tests assert the module
wiring instead. The accepted-revision arms retain 1-10 (older generated
source is self-contained and every surface it needs still exists); the
frozen revision-9 fixture continues to compile.
All checked-in recognizers are regenerated at revision 11. Generated
source shrinks by ~230 lines per parser and ~44 per lexer (toml parser
151,933 -> 141,165 bytes; rust parser 2,278,259 -> 2,268,215; g4 parser
451,066 -> 440,410), and grammars that do exercise the adaptive-retry
bundle now compile against the shared contract (covered by the
adaptive-routing CLI fixture end to end).
Closes #320
Copy/Paste DetectionFound 35 duplication(s) across 15 changed non-generated Rust file(s) (threshold: 100 tokens). Show duplicationsFound a 21 line (226 tokens) duplication in the following files:
(9, AtnStateKind::RuleStop),
] {
assert_eq!(atn.add_state(kind, Some(0)).expect("state").index(), state);
}
atn.set_left_recursive_rule(0)
.expect("left-recursive rule start");
atn.set_precedence_rule_decision(2)
.expect("precedence decision");
atn.set_loop_back_state(8, 7).expect("loop-back state");
atn.set_rule_to_start_state(vec![0])
.expect("rule start states");
atn.set_rule_to_stop_state(vec![9])
.expect("rule stop states");
for state in [1, 2, 3] {
atn.add_decision_state(state).expect("decision state");
}
for (source, target) in [(0, 1), (2, 3), (2, 8), (7, 2), (8, 9)] {
atn.add_transition(source, ParserTransitionSpec::Epsilon { target })
.expect("epsilon transition");
}
for (source, target, label) in [(1, 2, 1), (1, 2, 2), (4, 6, 4), (5, 6, 3), (6, 7, 1)] {
```rust
---
Found a 44 line (215 tokens) duplication in the following files:
* Starting at line 4285 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 17070 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn plus_loop_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::PlusBlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))
.expect("state")
.index(),
3
);
assert_eq!(
atn.add_state(AtnStateKind::PlusLoopBack, Some(0))
.expect("state")
.index(),
4
);
assert_eq!(
atn.add_state(AtnStateKind::LoopEnd, Some(0))
.expect("state")
.index(),
5
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))
.expect("state")
.index(),
6
);Found a 25 line (193 tokens) duplication in the following files:
let mut atn = ParserAtnBuilder::new(1);
for (state_number, kind) in [
(0, AtnStateKind::RuleStart),
(1, AtnStateKind::StarLoopEntry),
(2, AtnStateKind::Basic),
(3, AtnStateKind::Basic),
(4, AtnStateKind::StarLoopBack),
(5, AtnStateKind::LoopEnd),
(6, AtnStateKind::RuleStop),
] {
assert_eq!(
atn.add_state(kind, Some(0)).expect("state").index(),
state_number
);
}
atn.set_rule_to_start_state(vec![0])
.expect("rule start states");
atn.set_rule_to_stop_state(vec![6])
.expect("rule stop states");
atn.add_decision_state(1).expect("decision state");
atn.set_loop_back_state(5, 4).expect("loop back state");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("entry transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 2 })
.expect("loop body");
```rust
---
Found a 39 line (188 tokens) duplication in the following files:
* Starting at line 4149 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 16778 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn block_decision_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::BlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
3
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))
.expect("state")
.index(),
4
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))
.expect("state")
.index(),
5
);
atn.set_end_state(1, 4).expect("block end state");Found a 27 line (145 tokens) duplication in the following files:
fn generated_match_token_recovers_missing_token_from_context_follow() {
let atn = generated_match_recovery_atn();
let data = RecognizerData::new(
"Mini.g4",
Vocabulary::new(
[None, Some("'X'"), Some("'Y'")],
[None, Some("X"), Some("Y")],
[None::<&str>, None, None],
),
);
let mut parser = BaseParser::new(
CommonTokenStream::new(Source {
tokens: vec![TestToken::eof("parser-test", 3, 1, 3)],
index: 0,
}),
data,
);
parser.rule_context_stack = vec![
RuleContextFrame {
rule_index: 0,
invoking_state: 0,
},
RuleContextFrame {
rule_index: 1,
invoking_state: 1,
},
];
```rust
---
Found a 26 line (142 tokens) duplication in the following files:
* Starting at line 4187 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 4410 of crates/antlr-rust-codegen/src/generator/tests.rs
```rust
atn.set_end_state(1, 4).expect("block end state");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 2 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 3 })
.expect("transition");
atn.add_transition(
2,
ParserTransitionSpec::Atom {
target: 4,
label: 1,
},
)
.expect("transition");
atn.add_transition(
3,
ParserTransitionSpec::Atom {
target: 4,
label: 2,
},
)
.expect("transition");
atn.add_transition(4, ParserTransitionSpec::Epsilon { target: 5 })
.expect("transition");
atn.add_decision_state(1).expect("decision state");Found a 26 line (128 tokens) duplication in the following files:
fn linear_rule_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))
.expect("state")
.index(),
3
);
```rust
---
Found a 18 line (128 tokens) duplication in the following files:
* Starting at line 16324 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 16349 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn epsilon_cycle_atn() -> Atn {
let mut atn = ParserAtnBuilder::new(1);
for (state_number, kind) in [
(0, AtnStateKind::RuleStart),
(1, AtnStateKind::Basic),
(2, AtnStateKind::RuleStop),
] {
assert_eq!(
atn.add_state(kind, Some(0)).expect("state").index(),
state_number
);
}
atn.set_rule_to_start_state(vec![0])
.expect("rule start states");
atn.set_rule_to_stop_state(vec![2])
.expect("rule stop states");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("transition");Found a 27 line (127 tokens) duplication in the following files:
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::BlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
3
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))
```rust
---
Found a 27 line (127 tokens) duplication in the following files:
* Starting at line 16779 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 16851 of crates/antlr-rust-runtime/src/parser.rs
```rust
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::BlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
3
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))Found a 26 line (125 tokens) duplication in the following files:
$input: $crate::char_stream::CharStream,
$hooks: $crate::parser::SemanticHooks,
{
pub fn metadata() -> &'static $crate::generated::GrammarMetadata {
$metadata()
}
/// Adds a listener for lexer diagnostics.
pub fn add_error_listener<T>(&mut self, listener: T)
where
T: for<'a> $crate::errors::ErrorListener<dyn $crate::recognizer::Recognizer + 'a>
+ ::core::marker::Send
+ 'static,
{
$crate::recognizer::Recognizer::add_error_listener(&mut self.$base, listener);
}
/// Removes every lexer error listener, including the default console listener.
pub fn remove_error_listeners(&mut self) {
$crate::recognizer::Recognizer::remove_error_listeners(&mut self.$base);
}
/// Routes every token through ATN interpretation instead of the compiled
/// lexer DFA, so the learned-DFA trace (`lexer_dfa_string`) observes each
/// match.
pub fn set_force_interpreted(&mut self, force_interpreted: bool) {
```rust
---
Found a 22 line (125 tokens) duplication in the following files:
* Starting at line 16805 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 16877 of crates/antlr-rust-runtime/src/parser.rs
```rust
atn.add_state(AtnStateKind::BlockEnd, Some(0))
.expect("state")
.index(),
4
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))
.expect("state")
.index(),
5
);
atn.set_rule_to_start_state(vec![0])
.expect("rule start states");
atn.set_rule_to_stop_state(vec![5])
.expect("rule stop states");
atn.add_decision_state(1).expect("decision state");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("transition");
atn.add_transition(
1,
ParserTransitionSpec::Atom {
target: 2,Found a 34 line (119 tokens) duplication in the following files:
outcomes.extend(
self.recognize_state(
atn,
RecognizeRequest {
state_number: *target,
stop_state,
index,
rule_start_index,
decision_start_index: next_decision_start_index,
init_action_rules,
predicates,
semantics,
rule_args,
member_actions,
return_actions,
local_int_arg,
member_values: member_values.clone(),
return_values: return_values.clone(),
rule_alt_number: next_alt_number,
track_alt_numbers,
consumed_eof,
committed_decision: transition_committed,
precedence,
depth: depth + 1,
recovery_symbols: epsilon_recovery_symbols.clone(),
recovery_state: epsilon_recovery_state,
},
visiting,
memo,
expected,
)
.into_iter()
.map(|mut outcome| {
prepend_decision(&mut outcome, decision);
```rust
---
Found a 13 line (117 tokens) duplication in the following files:
* Starting at line 162 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 189 of crates/antlr-rust-runtime/src/parser.rs
```rust
ordinary $parser:ident, $state:expr, $rule:expr, $allow_fallback:expr,
$atn:expr, $fatal:path;
retry [$($retry:tt)*];
bind ($ctx:ident, $rule_start:ident, $consumed_eof:ident, $sync_error:ident);
setup { $($setup:tt)* }
body { $($body:tt)* }
success { $($success:tt)* }
recovery { $($recovery:tt)* }
) => {
$crate::__antlr4_rust_generated_rule! {
@body
parser $parser;
enter $parser.base.enter_rule($state, $rule);Found a 25 line (115 tokens) duplication in the following files:
atn.add_state(AtnStateKind::BlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
3
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))
.expect("state")
.index(),
4
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))
```rust
---
Found a 15 line (113 tokens) duplication in the following files:
* Starting at line 18205 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 18483 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn generated_match_token_counts_single_token_deletion_recovery() {
let atn = generated_match_recovery_atn();
let data = RecognizerData::new(
"Mini.g4",
Vocabulary::new(
[None, Some("'X'"), Some("'Y'"), Some("'Z'")],
[None, Some("X"), Some("Y"), Some("Z")],
[None::<&str>, None, None, None],
),
);
let mut parser = BaseParser::new(
CommonTokenStream::new(Source {
tokens: vec![
TestToken::new(3).with_text("z"),
TestToken::new(2).with_text("y"),Found a 22 line (112 tokens) duplication in the following files:
fn plus_loop_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::PlusBlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))
```rust
---
Found a 18 line (112 tokens) duplication in the following files:
* Starting at line 15134 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 17551 of crates/antlr-rust-runtime/src/parser.rs
```rust
(4, AtnStateKind::Basic, 0),
(5, AtnStateKind::RuleStop, 0),
(6, AtnStateKind::RuleStart, 1),
(7, AtnStateKind::Basic, 1),
(8, AtnStateKind::RuleStop, 1),
] {
assert_eq!(
atn.add_state(kind, Some(rule_index))
.expect("state")
.index(),
state_number
);
}
atn.set_rule_to_start_state(vec![0, 6])
.expect("rule start states");
atn.set_rule_to_stop_state(vec![5, 8])
.expect("rule stop states");
atn.add_decision_state(2).expect("decision state");Found a 12 line (112 tokens) duplication in the following files:
let mut atn = ParserAtnBuilder::new(1);
for (state, kind, rule) in [
(0, AtnStateKind::RuleStart, 0),
(1, AtnStateKind::StarLoopEntry, 0),
(2, AtnStateKind::Basic, 0), // ops hub
(3, AtnStateKind::Basic, 0), // shift prec
(4, AtnStateKind::Basic, 0), // shift first >
(5, AtnStateKind::Basic, 0), // shift second >
(6, AtnStateKind::Basic, 0), // rel prec
(7, AtnStateKind::Basic, 0), // rel >
(8, AtnStateKind::LoopEnd, 0),
(9, AtnStateKind::RuleStop, 0),
```rust
---
Found a 22 line (112 tokens) duplication in the following files:
* Starting at line 17325 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 17742 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn predicate_after_token_atn() -> Atn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))Found a 22 line (111 tokens) duplication in the following files:
fn left_recursive_loop_with_caller_follow_atn(caller_symbol: i32) -> Atn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(1))
```rust
---
Found a 27 line (110 tokens) duplication in the following files:
* Starting at line 3512 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 3624 of crates/antlr-rust-codegen/src/generator/tests.rs
```rust
decision: 0,
alts: (1, 2),
track_alt_number: false,
allow_semantic_context: true,
force_context: false,
plus_loop: false,
fast_path: None,
body: &body,
},
0,
GeneratedStepRenderContext {
current_rule_index: 0,
embedded: None,
portable_locals: None,
decision_routing: DecisionRoutingRender::default(),
inline_action_statements: &BTreeMap::new(),
track_alt_numbers: false,
track_context_alt_numbers: false,
direct_generated_rule_calls: &[],
atn_preferred_rule_calls: &[],
adaptive_atn_preferred_rule_slots: &[],
adaptive_atn_probe_rule_slots: &[],
},
);
// The whole rendered star-loop captures the leading-predicate-to-exit-alt filtering.
insta::assert_snapshot!(Found a 14 line (110 tokens) duplication in the following files:
fn parser_matches_token_and_reports_mismatch() {
let source = Source {
tokens: vec![
TestToken::new(1).with_text("x"),
TestToken::eof("parser-test", 1, 1, 1),
],
index: 0,
};
let data = RecognizerData::new(
"Mini.g4",
Vocabulary::new([None, Some("'x'")], [None, Some("X")], [None::<&str>, None]),
);
let mut parser = BaseParser::new(CommonTokenStream::new(source), data);
let matched = parser.match_token(1).expect("token 1 should match");
```rust
---
Found a 13 line (109 tokens) duplication in the following files:
* Starting at line 17875 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 21845 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn parser_matches_token_and_reports_mismatch() {
let source = Source {
tokens: vec![
TestToken::new(1).with_text("x"),
TestToken::eof("parser-test", 1, 1, 1),
],
index: 0,
};
let data = RecognizerData::new(
"Mini.g4",
Vocabulary::new([None, Some("'x'")], [None, Some("X")], [None::<&str>, None]),
);
let mut parser = BaseParser::new(CommonTokenStream::new(source), data);Found a 22 line (108 tokens) duplication in the following files:
fn linear_rule_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))
```rust
---
Found a 22 line (108 tokens) duplication in the following files:
* Starting at line 4359 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 17070 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn plus_block_decision_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::PlusBlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))Found a 22 line (108 tokens) duplication in the following files:
) -> Option<RecognizeOutcome> {
let (error_index, message) = self.expected_error_message(rule_index, start_index, expected);
let diagnostic = diagnostic_for_token(self.token_at(error_index), message);
let mut next_index = error_index;
loop {
let symbol = self.token_type_at(next_index);
if sync_symbols.contains(&symbol) {
if next_index == error_index {
return None;
}
break;
}
if symbol == TOKEN_EOF {
break;
}
let after = self.consume_index(next_index, symbol);
if after == next_index {
break;
}
next_index = after;
}
let mut nodes = NodeSeqId::EMPTY;
```rust
---
Found a 15 line (108 tokens) duplication in the following files:
* Starting at line 22116 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 22140 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn outcome_ties_keep_later_non_recursive_alternative() {
let arena = RecognitionArena::default();
let first = RecognizeOutcome {
index: 1,
consumed_eof: false,
alt_number: 0,
member_values: MemberEnv::new(),
return_values: BTreeMap::new(),
diagnostics: DiagnosticSeqId::EMPTY,
decisions: Vec::new(),
actions: vec![ParserAction::new(1, 0, 0, None)],
nodes: NodeSeqId::EMPTY,
};
let second = RecognizeOutcome {
actions: vec![ParserAction::new(2, 0, 0, None)],Found a 17 line (107 tokens) duplication in the following files:
let report_unrecovered_error = self.is_top_level_entry();
let start_state = atn.rule_to_start_state().get(rule_index).ok_or_else(|| {
AntlrError::Unsupported(format!("rule {rule_index} has no start state"))
})?;
let stop_state = atn
.rule_to_stop_state()
.get(rule_index)
.filter(|state| *state != usize::MAX)
.ok_or_else(|| {
AntlrError::Unsupported(format!("rule {rule_index} has no stop state"))
})?;
let start_index = self.current_visible_index();
self.clear_prediction_diagnostics();
self.reset_per_parse_caches();
self.reset_recognition_arena();
let caller_follow_state = self.pending_invoking_follow_state(atn);
```rust
---
Found a 17 line (105 tokens) duplication in the following files:
* Starting at line 180 of crates/antlr-rust-runtime/src/generated.rs
* Starting at line 277 of crates/antlr-rust-runtime/src/generated.rs
```rust
fn __from_node_with_invocation_states(
node: $crate::RuleNodeView<'a>,
invocation_states: Option<Vec<isize>>,
) -> Self {
$(
let __default = <$attrs>::default();
let __attrs = node.generated_attrs::<$attrs>().unwrap_or(&__default);
)?
Self {
__node: __GeneratedRuleContext::Stored(node),
__invocation_states: invocation_states,
__state: std::marker::PhantomData,
$(
$($field: __attrs.$field.clone(),)+
)?
}
}Found a 25 line (104 tokens) duplication in the following files:
state: 1,
decision: 0,
track_alt_number: false,
allow_semantic_context: false,
force_context: false,
fast_path: None,
alts: &alts,
},
0,
GeneratedStepRenderContext {
current_rule_index: 0,
embedded: None,
portable_locals: None,
decision_routing: DecisionRoutingRender::default(),
inline_action_statements: &BTreeMap::new(),
track_alt_numbers: false,
track_context_alt_numbers: false,
direct_generated_rule_calls: &[],
atn_preferred_rule_calls: &[],
adaptive_atn_preferred_rule_slots: &[],
adaptive_atn_probe_rule_slots: &[],
},
);
assert!(rendered.contains("ll1_decision_prediction(atn(), 1)"));
```rust
---
Found a 13 line (104 tokens) duplication in the following files:
* Starting at line 15312 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 16716 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn labeled_left_recursive_operator_atn() -> Atn {
let mut atn = ParserAtnBuilder::new(4);
for (state, kind) in [
(0, AtnStateKind::RuleStart),
(1, AtnStateKind::BlockStart),
(2, AtnStateKind::StarLoopEntry),
(3, AtnStateKind::StarBlockStart),
(4, AtnStateKind::Basic),
(5, AtnStateKind::Basic),
(6, AtnStateKind::Basic),
(7, AtnStateKind::StarLoopBack),
(8, AtnStateKind::LoopEnd),
(9, AtnStateKind::RuleStop),Found a 28 line (102 tokens) duplication in the following files:
state: 1,
decision: 0,
track_alt_number: false,
allow_semantic_context: true,
force_context: false,
fast_path: None,
alts: &alts,
},
0,
GeneratedStepRenderContext {
current_rule_index: 0,
embedded: None,
portable_locals: None,
decision_routing: DecisionRoutingRender::default(),
inline_action_statements: &BTreeMap::new(),
track_alt_numbers: false,
track_context_alt_numbers: false,
direct_generated_rule_calls: &[],
atn_preferred_rule_calls: &[],
adaptive_atn_preferred_rule_slots: &[],
adaptive_atn_probe_rule_slots: &[],
},
);
// One decision renders into a fresh String; snapshot the whole emitted control flow (the
// semantic-context gate, both predicate probes, the alt rewrite, the no-viable fallback)
// instead of six positive probes plus one negative guard.
insta::assert_snapshot!(
```rust
---
Found a 16 line (101 tokens) duplication in the following files:
* Starting at line 4258 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 4410 of crates/antlr-rust-codegen/src/generator/tests.rs
```rust
atn.set_loop_back_state(3, 4).expect("loop back state");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 2 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 3 })
.expect("transition");
atn.add_transition(
2,
ParserTransitionSpec::Atom {
target: 4,
label: 1,
},
)
.expect("transition");
atn.add_transition(4, ParserTransitionSpec::Epsilon { target: 1 })Found a 13 line (100 tokens) duplication in the following files:
let mut expected = BTreeSet::new();
for index in (1..self.rule_context_stack.len()).rev() {
let invoking_state = self.rule_context_stack[index].invoking_state;
let Ok(state_number) = usize::try_from(invoking_state) else {
continue;
};
let Some(Transition::Rule { follow_state, .. }) = atn
.state(state_number)
.and_then(|state| state.transitions().first())
.map(ParserTransition::data)
else {
continue;
};
```rust |
Reviewing PR #332
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe runtime now owns shared parser-driver, parser-entry-point, adaptive-retry, and lexer helper infrastructure. Code generation emits wiring and grammar-specific configuration. Compatibility advances to generated-code revision 11. ChangesRuntime and parser code-generation centralization
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant GeneratedParser
participant RuntimeDriver
participant GeneratedFallback
participant Diagnostics
GeneratedParser->>RuntimeDriver: invoke parser driver macro
RuntimeDriver->>GeneratedFallback: parse generated or interpreted rule
GeneratedFallback-->>RuntimeDriver: parse result or GeneratedRuleError
RuntimeDriver->>Diagnostics: drain and order diagnostics
Diagnostics-->>GeneratedParser: return parse output or error
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/support.rs`:
- Around line 278-284: Restrict the `output` extraction in the API-surface
parsing logic to lines within the `__antlr4_rust_parser_entry_points!`
invocation block before calling `find_map`. Preserve the existing trimming and
`type {output}` insertion behavior, while ignoring matching text elsewhere in
the module.
In `@crates/antlr-rust-runtime/src/generated.rs`:
- Around line 2355-2431: Update parser_driver_entry_ordering_invariants to
define named constants for each searched macro-body token sequence, including
macro markers and drained-state/action calls, then reuse those constants in
find, contains, and related assertions. Keep the existing ordering checks
unchanged while making expected token strings centralized and easier to update.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 5b70c0dc-5375-4a7b-9427-a18dc7d84449
⛔ Files ignored due to path filters (25)
crates/antlr-rust-codegen/src/generator/snapshots/antlr_rust_codegen__generator__tests__compact_left_recursive_rule_lifecycle.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/src/generator/snapshots/antlr_rust_codegen__generator__tests__compact_ordinary_rule_lifecycle.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/src/generator/snapshots/antlr_rust_codegen__generator__tests__generated_module_file_header.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/src/generator/snapshots/antlr_rust_codegen__generator__tests__generated_parser_optional_state_facade.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/src/generator/snapshots/antlr_rust_codegen__generator__tests__generated_plain_recognizer_facades.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/src/generator/snapshots/antlr_rust_codegen__generator__tests__generated_recognizers_reuse_cached_static_metadata.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/src/generator/snapshots/antlr_rust_codegen__generator__tests__lexer_lex_convenience.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/src/generator/snapshots/antlr_rust_codegen__generator__tests__parser_parse_convenience.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__cli__generated_codegen_api_checks.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__cli__generated_codegen_api_mismatch_diagnostic.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__parser__inlined_token_accessors_generated_api.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__transforms__configured_mutual_entry_generated_api.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__transforms__configured_precedence_entry_generated_api.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__transforms__inferred_mutual_entries_generated_api.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__transforms__pruned_unreachable_rule_generated_api.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__transforms__recursive_eof_entry_component_generated_api.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__transforms__unreachable_rule_default_generated_api.snapis excluded by!**/*.snapcrates/antlr-rust-g4-parser/src/generated/antlr_v4_lexer.rsis excluded by!**/generated/**crates/antlr-rust-g4-parser/src/generated/antlr_v4_parser.rsis excluded by!**/generated/**crates/antlr-rust-rs-parser/src/generated/rust_lexer.rsis excluded by!**/generated/**crates/antlr-rust-rs-parser/src/generated/rust_parser.rsis excluded by!**/generated/**crates/antlr-rust-runtime/src/xpath/generated/x_path_lexer.rsis excluded by!**/generated/**crates/antlr-rust-toml-parser/src/generated/toml_lexer.rsis excluded by!**/generated/**crates/antlr-rust-toml-parser/src/generated/toml_parser.rsis excluded by!**/generated/**docs/migration.mdis excluded by!**/docs/**
📒 Files selected for processing (17)
README.mdcrates/antlr-rust-codegen/src/generator/tests.rscrates/antlr-rust-codegen/src/lexer/render.rscrates/antlr-rust-codegen/src/lexer/render_model.rscrates/antlr-rust-codegen/src/parser/render/fallback.rscrates/antlr-rust-codegen/src/parser/render/mod.rscrates/antlr-rust-codegen/src/parser/render/rules.rscrates/antlr-rust-codegen/src/parser/render_model.rscrates/antlr-rust-codegen/src/parser/routing.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/cli.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/optimizations.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/support.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/typed_tree.rscrates/antlr-rust-runtime/src/generated.rscrates/antlr-rust-runtime/src/lib.rscrates/antlr-rust-runtime/src/parser.rsthird_party/antlr-v4-grammar/self-hosted.sha256
📊 Source Code Metrics (this PR vs
|
| File | Cyclomatic | Cognitive | Functions | LLOC | MI |
|---|---|---|---|---|---|
| crates/antlr-rust-codegen/src/generator/tests.rs | 292 (main: 294) 🟢 | 48 ⚪ | 207 ⚪ | 1330 (main: 1353) 🟢 | 0 ⚪ |
| crates/antlr-rust-runtime/src/generated.rs | 96 (main: 75) 🔴 | 10 (main: 8) 🔴 | 50 (main: 40) 🔴 | 92 (main: 61) 🔴 | 0 ⚪ |
| crates/antlr-rust-codegen/src/parser/routing.rs | 34 (main: 37) 🟢 | 26 (main: 27) 🟢 | 10 (main: 12) 🟢 | 93 (main: 97) 🟢 | 9.57 (main: 7.79) 🟢 |
| crates/antlr-rust-codegen/src/lexer/render_model.rs | 20 ⚪ | 1 ⚪ | 9 ⚪ | 22 ⚪ | 25.50 (main: 22.98) 🟢 |
| crates/antlr-rust-codegen/src/parser/render/rules.rs | 45 (main: 46) 🟢 | 52 (main: 54) 🟢 | 5 ⚪ | 114 (main: 116) 🟢 | 7.82 (main: 7.45) 🟢 |
| crates/antlr-rust-codegen/src/parser/render/mod.rs | 43 ⚪ | 41 ⚪ | 3 ⚪ | 90 (main: 92) 🟢 | 9.38 (main: 6.99) 🟢 |
| crates/antlr-rust-codegen/src/parser/render/fallback.rs | 7 (main: 9) 🟢 | 6 (main: 8) 🟢 | 1 ⚪ | 9 (main: 14) 🟢 | 40.80 (main: 37.51) 🟢 |
| crates/antlr-rust-runtime/src/lib.rs | 4 ⚪ | 3 ⚪ | 1 ⚪ | 7 ⚪ | 28.88 (main: 28.97) 🔴 |
Generated by mehen v1.8.1 — the code quality watcher.
…acing behavior Address PR #332 review findings. The runtime `parser_driver_entry_ordering_invariants` test anchored its first and third assertions on `self.$base.take_unknown_semantic_error()`, whose first occurrence is the `let _ =` drain inside the Err-arm abort branch — not the top-level surfacing check — so deleting the surfacing block left the test (and the whole generator suite) passing. The test now whitespace-flattens the macro body and anchors on the binder forms that are unique to each site (`Some(error)` for the surfacing check, `Some(semantic_error)` and `Some(abort)` for the Err-arm drains), asserts the surfacing anchor is unique, and keeps every search string in a named constant so macro renames are one-line updates. It also re-pins the boundary-diagnostics dispatch the generator test dropped: the post-tree `report_generated_parser_diagnostics` between the interpreted fallback and the surfacing check, and `report_unrecovered_parser_error` between the Err-arm conversion and its return. Verified by mutation: removing the surfacing block from `__antlr4_rust_parser_driver!` now fails the test. A new behavioral CLI case, `public_entry_surfaces_recorded_semantic_miss_after_clean_parse`, pins the same invariant end to end where source-text matching cannot: a generated parser under `--sem-unknown hook` whose untranslated action records a fail-loud miss during a structurally clean parse must return `AntlrError::Unsupported` from the public entry (zero syntax errors), and a reused parser stays clean. The same mutation fails this test with the miss escaping as a recovered Ok tree. Also from review: the `generated_parser_api` scrape reads the ParseOutput alias name only from inside the `__antlr4_rust_parser_entry_points!` invocation block, so an `output: ` line elsewhere in a module cannot add a bogus API-surface entry; the rule macro's `retry [$cond => $err]` arm is documented as back-compat-only for generated-code API revisions <= 10; `GeneratedParseOutput` documents when `validate()` is available; the shared `lex` rustdoc attributes `metadata()` to the generated module; and the README describes revision 10 in the past tense now that 11 is current.
Contributor
Author
|
@claude Thanks for the mutation-verified review — Finding 1 was a real gap. All four findings and both nits are addressed in f84fafb: Finding 1 (fixed, mutation-verified both ways). I also took the "stronger still" suggestion: a new behavioral CLI case Finding 2 (fixed). The runtime test now asserts Finding 3 (fixed). The Finding 4 + nits (fixed). The shared CodeRabbit's two inline comments (bounding the |
|
Claude finished @tinovyatkin's task in 1m 58s —— View job Verifying fixes in f84fafb
Verified against sourceI re-derived each claim directly against
Note: I wasn't able to execute All four findings and the two CodeRabbit nits are accurately and correctly addressed in f84fafb — nothing further to fix from my side. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/optimizations.rs`:
- Around line 535-541: Update the test around the failed SemanticMissLexer item
entry so clean accepts EOF, then invoke clean() immediately after the failed
item() call without resetting the parser via set_token_stream. Remove the
intervening token-stream replacement and retain the assertion that the clean
entry succeeds, ensuring parser state is validated in place.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: f5509106-f434-48e2-a715-64d177ee801c
📒 Files selected for processing (5)
README.mdcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/optimizations.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/support.rscrates/antlr-rust-runtime/src/generated.rscrates/antlr-rust-runtime/src/parser.rs
…case `set_token_stream` fully resets parser-owned state, so the reuse check in `public_entry_surfaces_recorded_semantic_miss_after_clean_parse` passed even if the failed entry had left the recorded semantic error behind. `clean` now matches `EOF` and runs directly after the failed `item()` entry on the same token stream (the cursor sits at EOF after the clean parse), so the assertion observes the drain itself rather than the reset.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #320.
Summary
Generated parsers no longer re-declare the 113-line parse-driver core and the 200–550-line entry-point block; each module now emits two thin runtime-macro invocations plus its grammar-specific knobs. The generated-code API revision is incremented to 11.
parse_rule,parse_rule_precedence,parse_rule_precedence_from_generated,parse_rule_precedence_inner,parse_interpreted_rule,parse_interpreted_rule_precedenceexpand from the new runtime__antlr4_rust_parser_driver!macro. Generated code supplies only afallback(parser, rule_index, precedence)binder block composing itsParserRuntimeOptions, and anadaptive_directflag.run_action(an empty method for grammars without action states), replacing the divergentfor action in actions { … }vslet _ = actions;template line.parse/parse_validated/parse_with_parser/parse_stream/parse_stream_validated/parse_stream_with_parser, the<Grammar>ParserParseOutputalias of the new runtimeGeneratedParseOutput<R, P>, and thevalidate()bridge (via the doc-hidden__GeneratedParserValidatetrait) expand from__antlr4_rust_parser_entry_points!. Entry-rule doc lists remain generated on the parser type rustdoc.GeneratedRuleErrormoves to the runtime withAdaptiveRetryalways present; the five-field state bundle becomesAdaptiveAtnRetryState<const RULES: usize>(RULES = 0without residual adaptive routing;retry_pending()constant-folds tofalse), and rule bodies carry a uniformretry [adaptive];clause handled by a new__antlr4_rust_generated_rule!arm.lex/lex_streamare runtime generic functions re-exported by generated lexer modules.Ok, diagnostics → abort → semantic-miss draining, sharedallow_generated_fallbackgate) previously pinned per grammar are now pinned once by a runtime test over the macro source; generator tests assert the module wiring.docs/migration.mdare updated; all checked-in recognizers are regenerated at revision 11.Generated-source size (lines / bytes)
Compile-time and binary-size deltas were not measured; the scaffolding was already generic over
L/H, so the wins are source size and single ownership of driver semantics.Tested
cargo test -p antlr-rust-codegen --libcargo test -p antlr-rust-codegen --tests(CLI integration, incl. revision handshake, frozen revision-9 fixture, adaptive-routing fixture compilingAdaptiveAtnRetryState<N>at N>0 end to end)parser_driver_entry_ordering_invariants)tests/kotlin-parity/run.sh)cargo clippy --locked --workspace --all-targets --all-features -- -D warningscargo fmt --check, doc-tests, pre-commit hk suiteReviewer notes
renders_parse_convenience_without_replacing_manual_constructor,generated_*and adaptive generator tests now assert macro invocations / new field paths; the ordering invariants moved to the runtime test named above.generated_parser_api(CLI test support) synthesizes the entry-points surface from the invocation the same way it already did for the facade macro, so the API-inventory snapshots keep listingparse*/validateand record the ParseOutput alias astypeinstead ofstruct.decisions.json(it was never checked in for that module).Summary by CodeRabbit