feat(codegen): table-drive generated rule dispatch - #338
Merged
Conversation
Replace one generated dispatch wrapper per parser rule with a function-pointer table and a single runtime-owned guard. Ordinary routing and direct subrule calls index the table while adaptive and interpreted exceptions retain explicit branches, preserving depth caps, listeners, stack growth, balanced exits, and left-recursive precedence. Make generated-code API revision 12 an intentional breaking boundary, remove the legacy dispatch and retry compatibility paths plus the revision-9 fixtures, and regenerate every checked-in recognizer including the XPath lexer. This removes 1,785 generated lines and 129,423 source bytes from the three checked-in parsers. The multi-parser generator binary shrinks by 757,088 bytes without measurable Kotlin parse regression. Fixes #322
Copy/Paste DetectionFound 36 duplication(s) across 9 changed non-generated Rust file(s) (threshold: 100 tokens). Show duplicationsFound a 21 line (226 tokens) duplication in the following files:
(9, AtnStateKind::RuleStop),
] {
assert_eq!(atn.add_state(kind, Some(0)).expect("state").index(), state);
}
atn.set_left_recursive_rule(0)
.expect("left-recursive rule start");
atn.set_precedence_rule_decision(2)
.expect("precedence decision");
atn.set_loop_back_state(8, 7).expect("loop-back state");
atn.set_rule_to_start_state(vec![0])
.expect("rule start states");
atn.set_rule_to_stop_state(vec![9])
.expect("rule stop states");
for state in [1, 2, 3] {
atn.add_decision_state(state).expect("decision state");
}
for (source, target) in [(0, 1), (2, 3), (2, 8), (7, 2), (8, 9)] {
atn.add_transition(source, ParserTransitionSpec::Epsilon { target })
.expect("epsilon transition");
}
for (source, target, label) in [(1, 2, 1), (1, 2, 2), (4, 6, 4), (5, 6, 3), (6, 7, 1)] {
```rust
---
Found a 44 line (215 tokens) duplication in the following files:
* Starting at line 4277 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 17033 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn plus_loop_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::PlusBlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))
.expect("state")
.index(),
3
);
assert_eq!(
atn.add_state(AtnStateKind::PlusLoopBack, Some(0))
.expect("state")
.index(),
4
);
assert_eq!(
atn.add_state(AtnStateKind::LoopEnd, Some(0))
.expect("state")
.index(),
5
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))
.expect("state")
.index(),
6
);Found a 25 line (193 tokens) duplication in the following files:
let mut atn = ParserAtnBuilder::new(1);
for (state_number, kind) in [
(0, AtnStateKind::RuleStart),
(1, AtnStateKind::StarLoopEntry),
(2, AtnStateKind::Basic),
(3, AtnStateKind::Basic),
(4, AtnStateKind::StarLoopBack),
(5, AtnStateKind::LoopEnd),
(6, AtnStateKind::RuleStop),
] {
assert_eq!(
atn.add_state(kind, Some(0)).expect("state").index(),
state_number
);
}
atn.set_rule_to_start_state(vec![0])
.expect("rule start states");
atn.set_rule_to_stop_state(vec![6])
.expect("rule stop states");
atn.add_decision_state(1).expect("decision state");
atn.set_loop_back_state(5, 4).expect("loop back state");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("entry transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 2 })
.expect("loop body");
```rust
---
Found a 39 line (188 tokens) duplication in the following files:
* Starting at line 4141 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 16741 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn block_decision_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::BlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
3
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))
.expect("state")
.index(),
4
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))
.expect("state")
.index(),
5
);
atn.set_end_state(1, 4).expect("block end state");Found a 27 line (145 tokens) duplication in the following files:
fn generated_match_token_recovers_missing_token_from_context_follow() {
let atn = generated_match_recovery_atn();
let data = RecognizerData::new(
"Mini.g4",
Vocabulary::new(
[None, Some("'X'"), Some("'Y'")],
[None, Some("X"), Some("Y")],
[None::<&str>, None, None],
),
);
let mut parser = BaseParser::new(
CommonTokenStream::new(Source {
tokens: vec![TestToken::eof("parser-test", 3, 1, 3)],
index: 0,
}),
data,
);
parser.rule_context_stack = vec![
RuleContextFrame {
rule_index: 0,
invoking_state: 0,
},
RuleContextFrame {
rule_index: 1,
invoking_state: 1,
},
];
```rust
---
Found a 26 line (142 tokens) duplication in the following files:
* Starting at line 4179 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 4402 of crates/antlr-rust-codegen/src/generator/tests.rs
```rust
atn.set_end_state(1, 4).expect("block end state");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 2 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 3 })
.expect("transition");
atn.add_transition(
2,
ParserTransitionSpec::Atom {
target: 4,
label: 1,
},
)
.expect("transition");
atn.add_transition(
3,
ParserTransitionSpec::Atom {
target: 4,
label: 2,
},
)
.expect("transition");
atn.add_transition(4, ParserTransitionSpec::Epsilon { target: 5 })
.expect("transition");
atn.add_decision_state(1).expect("decision state");Found a 26 line (128 tokens) duplication in the following files:
fn linear_rule_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))
.expect("state")
.index(),
3
);
```rust
---
Found a 18 line (128 tokens) duplication in the following files:
* Starting at line 16287 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 16312 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn epsilon_cycle_atn() -> Atn {
let mut atn = ParserAtnBuilder::new(1);
for (state_number, kind) in [
(0, AtnStateKind::RuleStart),
(1, AtnStateKind::Basic),
(2, AtnStateKind::RuleStop),
] {
assert_eq!(
atn.add_state(kind, Some(0)).expect("state").index(),
state_number
);
}
atn.set_rule_to_start_state(vec![0])
.expect("rule start states");
atn.set_rule_to_stop_state(vec![2])
.expect("rule stop states");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("transition");Found a 27 line (127 tokens) duplication in the following files:
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::BlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
3
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))
```rust
---
Found a 27 line (127 tokens) duplication in the following files:
* Starting at line 16742 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 16814 of crates/antlr-rust-runtime/src/parser.rs
```rust
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::BlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
3
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))Found a 26 line (125 tokens) duplication in the following files:
$input: $crate::char_stream::CharStream,
$hooks: $crate::parser::SemanticHooks,
{
pub fn metadata() -> &'static $crate::generated::GrammarMetadata {
$metadata()
}
/// Adds a listener for lexer diagnostics.
pub fn add_error_listener<T>(&mut self, listener: T)
where
T: for<'a> $crate::errors::ErrorListener<dyn $crate::recognizer::Recognizer + 'a>
+ ::core::marker::Send
+ 'static,
{
$crate::recognizer::Recognizer::add_error_listener(&mut self.$base, listener);
}
/// Removes every lexer error listener, including the default console listener.
pub fn remove_error_listeners(&mut self) {
$crate::recognizer::Recognizer::remove_error_listeners(&mut self.$base);
}
/// Routes every token through ATN interpretation instead of the compiled
/// lexer DFA, so the learned-DFA trace (`lexer_dfa_string`) observes each
/// match.
pub fn set_force_interpreted(&mut self, force_interpreted: bool) {
```rust
---
Found a 22 line (125 tokens) duplication in the following files:
* Starting at line 16768 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 16840 of crates/antlr-rust-runtime/src/parser.rs
```rust
atn.add_state(AtnStateKind::BlockEnd, Some(0))
.expect("state")
.index(),
4
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))
.expect("state")
.index(),
5
);
atn.set_rule_to_start_state(vec![0])
.expect("rule start states");
atn.set_rule_to_stop_state(vec![5])
.expect("rule stop states");
atn.add_decision_state(1).expect("decision state");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("transition");
atn.add_transition(
1,
ParserTransitionSpec::Atom {
target: 2,Found a 34 line (119 tokens) duplication in the following files:
outcomes.extend(
self.recognize_state(
atn,
RecognizeRequest {
state_number: *target,
stop_state,
index,
rule_start_index,
decision_start_index: next_decision_start_index,
init_action_rules,
predicates,
semantics,
rule_args,
member_actions,
return_actions,
local_int_arg,
member_values: member_values.clone(),
return_values: return_values.clone(),
rule_alt_number: next_alt_number,
track_alt_numbers,
consumed_eof,
committed_decision: transition_committed,
precedence,
depth: depth + 1,
recovery_symbols: epsilon_recovery_symbols.clone(),
recovery_state: epsilon_recovery_state,
},
visiting,
memo,
expected,
)
.into_iter()
.map(|mut outcome| {
prepend_decision(&mut outcome, decision);
```rust
---
Found a 13 line (117 tokens) duplication in the following files:
* Starting at line 143 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 170 of crates/antlr-rust-runtime/src/parser.rs
```rust
ordinary $parser:ident, $state:expr, $rule:expr, $allow_fallback:expr,
$atn:expr, $fatal:path;
retry [$($retry:tt)*];
bind ($ctx:ident, $rule_start:ident, $consumed_eof:ident, $sync_error:ident);
setup { $($setup:tt)* }
body { $($body:tt)* }
success { $($success:tt)* }
recovery { $($recovery:tt)* }
) => {
$crate::__antlr4_rust_generated_rule! {
@body
parser $parser;
enter $parser.base.enter_rule($state, $rule);Found a 25 line (115 tokens) duplication in the following files:
atn.add_state(AtnStateKind::BlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
3
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))
.expect("state")
.index(),
4
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))
```rust
---
Found a 15 line (113 tokens) duplication in the following files:
* Starting at line 18168 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 18446 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn generated_match_token_counts_single_token_deletion_recovery() {
let atn = generated_match_recovery_atn();
let data = RecognizerData::new(
"Mini.g4",
Vocabulary::new(
[None, Some("'X'"), Some("'Y'"), Some("'Z'")],
[None, Some("X"), Some("Y"), Some("Z")],
[None::<&str>, None, None, None],
),
);
let mut parser = BaseParser::new(
CommonTokenStream::new(Source {
tokens: vec![
TestToken::new(3).with_text("z"),
TestToken::new(2).with_text("y"),Found a 22 line (112 tokens) duplication in the following files:
fn plus_loop_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::PlusBlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))
```rust
---
Found a 18 line (112 tokens) duplication in the following files:
* Starting at line 15097 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 17514 of crates/antlr-rust-runtime/src/parser.rs
```rust
(4, AtnStateKind::Basic, 0),
(5, AtnStateKind::RuleStop, 0),
(6, AtnStateKind::RuleStart, 1),
(7, AtnStateKind::Basic, 1),
(8, AtnStateKind::RuleStop, 1),
] {
assert_eq!(
atn.add_state(kind, Some(rule_index))
.expect("state")
.index(),
state_number
);
}
atn.set_rule_to_start_state(vec![0, 6])
.expect("rule start states");
atn.set_rule_to_stop_state(vec![5, 8])
.expect("rule stop states");
atn.add_decision_state(2).expect("decision state");Found a 12 line (112 tokens) duplication in the following files:
let mut atn = ParserAtnBuilder::new(1);
for (state, kind, rule) in [
(0, AtnStateKind::RuleStart, 0),
(1, AtnStateKind::StarLoopEntry, 0),
(2, AtnStateKind::Basic, 0), // ops hub
(3, AtnStateKind::Basic, 0), // shift prec
(4, AtnStateKind::Basic, 0), // shift first >
(5, AtnStateKind::Basic, 0), // shift second >
(6, AtnStateKind::Basic, 0), // rel prec
(7, AtnStateKind::Basic, 0), // rel >
(8, AtnStateKind::LoopEnd, 0),
(9, AtnStateKind::RuleStop, 0),
```rust
---
Found a 22 line (112 tokens) duplication in the following files:
* Starting at line 17288 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 17705 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn predicate_after_token_atn() -> Atn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))Found a 22 line (111 tokens) duplication in the following files:
fn left_recursive_loop_with_caller_follow_atn(caller_symbol: i32) -> Atn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(1))
```rust
---
Found a 27 line (110 tokens) duplication in the following files:
* Starting at line 3504 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 3616 of crates/antlr-rust-codegen/src/generator/tests.rs
```rust
decision: 0,
alts: (1, 2),
track_alt_number: false,
allow_semantic_context: true,
force_context: false,
plus_loop: false,
fast_path: None,
body: &body,
},
0,
GeneratedStepRenderContext {
current_rule_index: 0,
embedded: None,
portable_locals: None,
decision_routing: DecisionRoutingRender::default(),
inline_action_statements: &BTreeMap::new(),
track_alt_numbers: false,
track_context_alt_numbers: false,
direct_generated_rule_calls: &[],
atn_preferred_rule_calls: &[],
adaptive_atn_preferred_rule_slots: &[],
adaptive_atn_probe_rule_slots: &[],
},
);
// The whole rendered star-loop captures the leading-predicate-to-exit-alt filtering.
insta::assert_snapshot!(Found a 14 line (110 tokens) duplication in the following files:
fn parser_matches_token_and_reports_mismatch() {
let source = Source {
tokens: vec![
TestToken::new(1).with_text("x"),
TestToken::eof("parser-test", 1, 1, 1),
],
index: 0,
};
let data = RecognizerData::new(
"Mini.g4",
Vocabulary::new([None, Some("'x'")], [None, Some("X")], [None::<&str>, None]),
);
let mut parser = BaseParser::new(CommonTokenStream::new(source), data);
let matched = parser.match_token(1).expect("token 1 should match");
```rust
---
Found a 13 line (109 tokens) duplication in the following files:
* Starting at line 17838 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 21808 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn parser_matches_token_and_reports_mismatch() {
let source = Source {
tokens: vec![
TestToken::new(1).with_text("x"),
TestToken::eof("parser-test", 1, 1, 1),
],
index: 0,
};
let data = RecognizerData::new(
"Mini.g4",
Vocabulary::new([None, Some("'x'")], [None, Some("X")], [None::<&str>, None]),
);
let mut parser = BaseParser::new(CommonTokenStream::new(source), data);Found a 22 line (108 tokens) duplication in the following files:
fn linear_rule_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))
```rust
---
Found a 22 line (108 tokens) duplication in the following files:
* Starting at line 4351 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 17033 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn plus_block_decision_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::PlusBlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))Found a 22 line (108 tokens) duplication in the following files:
) -> Option<RecognizeOutcome> {
let (error_index, message) = self.expected_error_message(rule_index, start_index, expected);
let diagnostic = diagnostic_for_token(self.token_at(error_index), message);
let mut next_index = error_index;
loop {
let symbol = self.token_type_at(next_index);
if sync_symbols.contains(&symbol) {
if next_index == error_index {
return None;
}
break;
}
if symbol == TOKEN_EOF {
break;
}
let after = self.consume_index(next_index, symbol);
if after == next_index {
break;
}
next_index = after;
}
let mut nodes = NodeSeqId::EMPTY;
```rust
---
Found a 15 line (108 tokens) duplication in the following files:
* Starting at line 22079 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 22103 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn outcome_ties_keep_later_non_recursive_alternative() {
let arena = RecognitionArena::default();
let first = RecognizeOutcome {
index: 1,
consumed_eof: false,
alt_number: 0,
member_values: MemberEnv::new(),
return_values: BTreeMap::new(),
diagnostics: DiagnosticSeqId::EMPTY,
decisions: Vec::new(),
actions: vec![ParserAction::new(1, 0, 0, None)],
nodes: NodeSeqId::EMPTY,
};
let second = RecognizeOutcome {
actions: vec![ParserAction::new(2, 0, 0, None)],Found a 17 line (107 tokens) duplication in the following files:
let report_unrecovered_error = self.is_top_level_entry();
let start_state = atn.rule_to_start_state().get(rule_index).ok_or_else(|| {
AntlrError::Unsupported(format!("rule {rule_index} has no start state"))
})?;
let stop_state = atn
.rule_to_stop_state()
.get(rule_index)
.filter(|state| *state != usize::MAX)
.ok_or_else(|| {
AntlrError::Unsupported(format!("rule {rule_index} has no stop state"))
})?;
let start_index = self.current_visible_index();
self.clear_prediction_diagnostics();
self.reset_per_parse_caches();
self.reset_recognition_arena();
let caller_follow_state = self.pending_invoking_follow_state(atn);
```rust
---
Found a 17 line (105 tokens) duplication in the following files:
* Starting at line 181 of crates/antlr-rust-runtime/src/generated.rs
* Starting at line 278 of crates/antlr-rust-runtime/src/generated.rs
```rust
fn __from_node_with_invocation_states(
node: $crate::RuleNodeView<'a>,
invocation_states: Option<Vec<isize>>,
) -> Self {
$(
let __default = <$attrs>::default();
let __attrs = node.generated_attrs::<$attrs>().unwrap_or(&__default);
)?
Self {
__node: __GeneratedRuleContext::Stored(node),
__invocation_states: invocation_states,
__state: std::marker::PhantomData,
$(
$($field: __attrs.$field.clone(),)+
)?
}
}Found a 25 line (104 tokens) duplication in the following files:
state: 1,
decision: 0,
track_alt_number: false,
allow_semantic_context: false,
force_context: false,
fast_path: None,
alts: &alts,
},
0,
GeneratedStepRenderContext {
current_rule_index: 0,
embedded: None,
portable_locals: None,
decision_routing: DecisionRoutingRender::default(),
inline_action_statements: &BTreeMap::new(),
track_alt_numbers: false,
track_context_alt_numbers: false,
direct_generated_rule_calls: &[],
atn_preferred_rule_calls: &[],
adaptive_atn_preferred_rule_slots: &[],
adaptive_atn_probe_rule_slots: &[],
},
);
assert!(rendered.contains("ll1_decision_prediction(atn(), 1)"));
```rust
---
Found a 13 line (104 tokens) duplication in the following files:
* Starting at line 15275 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 16679 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn labeled_left_recursive_operator_atn() -> Atn {
let mut atn = ParserAtnBuilder::new(4);
for (state, kind) in [
(0, AtnStateKind::RuleStart),
(1, AtnStateKind::BlockStart),
(2, AtnStateKind::StarLoopEntry),
(3, AtnStateKind::StarBlockStart),
(4, AtnStateKind::Basic),
(5, AtnStateKind::Basic),
(6, AtnStateKind::Basic),
(7, AtnStateKind::StarLoopBack),
(8, AtnStateKind::LoopEnd),
(9, AtnStateKind::RuleStop),Found a 28 line (102 tokens) duplication in the following files:
state: 1,
decision: 0,
track_alt_number: false,
allow_semantic_context: true,
force_context: false,
fast_path: None,
alts: &alts,
},
0,
GeneratedStepRenderContext {
current_rule_index: 0,
embedded: None,
portable_locals: None,
decision_routing: DecisionRoutingRender::default(),
inline_action_statements: &BTreeMap::new(),
track_alt_numbers: false,
track_context_alt_numbers: false,
direct_generated_rule_calls: &[],
atn_preferred_rule_calls: &[],
adaptive_atn_preferred_rule_slots: &[],
adaptive_atn_probe_rule_slots: &[],
},
);
// One decision renders into a fresh String; snapshot the whole emitted control flow (the
// semantic-context gate, both predicate probes, the alt rewrite, the no-viable fallback)
// instead of six positive probes plus one negative guard.
insta::assert_snapshot!(
```rust
---
Found a 17 line (102 tokens) duplication in the following files:
* Starting at line 781 of crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/parser.rs
* Starting at line 848 of crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/parser.rs
```rust
dir.join("L.g4").as_os_str(),
OsStr::new("--sem-patterns"),
dir.join("patterns.toml").as_os_str(),
OsStr::new("--sem-unknown"),
OsStr::new("error"),
OsStr::new("--require-full-semantics"),
OsStr::new("--out-dir"),
out.as_os_str(),
]);
assert!(
output.status.success(),
"stdout: {}\nstderr: {}",
utf8(&output.stdout),
utf8(&output.stderr)
);
let lexer = fs::read_to_string(out.join("l.rs")).expect("lexer should be emitted");Found a 16 line (101 tokens) duplication in the following files:
atn.set_loop_back_state(3, 4).expect("loop back state");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 2 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 3 })
.expect("transition");
atn.add_transition(
2,
ParserTransitionSpec::Atom {
target: 4,
label: 1,
},
)
.expect("transition");
atn.add_transition(4, ParserTransitionSpec::Epsilon { target: 1 })
```rust
---
Found a 13 line (100 tokens) duplication in the following files:
* Starting at line 7372 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 7396 of crates/antlr-rust-runtime/src/parser.rs
```rust
let mut expected = BTreeSet::new();
for index in (1..self.rule_context_stack.len()).rev() {
let invoking_state = self.rule_context_stack[index].invoking_state;
let Ok(state_number) = usize::try_from(invoking_state) else {
continue;
};
let Some(Transition::Rule { follow_state, .. }) = atn
.state(state_number)
.and_then(|state| state.transitions().first())
.map(ParserTransition::data)
else {
continue;
}; |
|
Warning Review limit reached
Next review available in: 52 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe PR centralizes generated-rule execution through a runtime dispatcher and generated function-pointer table. It removes per-rule dispatch wrappers, updates routing and integration tests, advances the compatibility contract to revision 12, and removes revision-9 compatibility fixtures. ChangesGenerated-rule dispatch revision
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant GeneratedParser
participant dispatch_generated_rule
participant BaseParser
GeneratedParser->>dispatch_generated_rule: pass rule index, precedence, fallback, and rule body
dispatch_generated_rule->>BaseParser: check depth cap and listener entry
dispatch_generated_rule->>GeneratedParser: invoke selected generated rule body
dispatch_generated_rule->>BaseParser: report listener exit
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Claude finished @tinovyatkin's task in 20m 49s —— View job Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. Reviewed at What I verifiedThe dispatcher is a faithful transcription of the deleted macro arm.
The router rewrite is behavior-preserving. The old The positional table can't misalign. Revision-12 boundary is complete and self-consistent — the CLAUDE.md checklist for
Build + test evidence (run locally on this branch)
The wildcard-only On the three CodeRabbit commentsAll three are already resolved on this head:
Non-blocking observations (no action needed to merge)
|
Contributor
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/antlr-rust-codegen/src/generator/tests.rs`:
- Around line 41-45: Add and accept the regenerated named snapshot for
compact_left_recursive_rule_lifecycle under the generator snapshots directory,
ensuring it matches the current rendered output used by
rendered_generated_rule_methods and is included in the repository.
In `@crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/parser.rs`:
- Around line 157-165: Strengthen the generated-parser assertion in the test
around the emitted parser content so it rejects every legacy
parse_generated_rule_<n>_dispatch wrapper, not only
parse_generated_rule_0_dispatch. Derive expected rule indexes from the available
rule table or use a generic per-rule pattern, while permitting the documented
adaptive and precedence exceptions; retain the positive dispatch_generated_rule
check.
In `@README.md`:
- Around line 173-178: Update the generator revision description in README.md to
explicitly state that generated code retains both ATN-preference and
adaptive-routing exceptions, rather than mentioning only ATN-preference routing.
Keep the surrounding dispatch-table and runtime-owned guard description
unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 69db191a-b8ea-43a6-aed4-873c564c0066
⛔ Files ignored due to path filters (13)
crates/antlr-rust-codegen/src/generator/snapshots/antlr_rust_codegen__generator__tests__compact_left_recursive_rule_lifecycle.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/src/generator/snapshots/antlr_rust_codegen__generator__tests__compact_ordinary_rule_lifecycle.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/src/generator/snapshots/antlr_rust_codegen__generator__tests__generated_module_file_header.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__cli__generated_codegen_api_checks.snapis excluded by!**/*.snapcrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/snapshots/antlr4_rust_gen_cli__cli__generated_codegen_api_mismatch_diagnostic.snapis excluded by!**/*.snapcrates/antlr-rust-g4-parser/src/generated/antlr_v4_lexer.rsis excluded by!**/generated/**crates/antlr-rust-g4-parser/src/generated/antlr_v4_parser.rsis excluded by!**/generated/**crates/antlr-rust-rs-parser/src/generated/rust_lexer.rsis excluded by!**/generated/**crates/antlr-rust-rs-parser/src/generated/rust_parser.rsis excluded by!**/generated/**crates/antlr-rust-runtime/src/xpath/generated/x_path_lexer.rsis excluded by!**/generated/**crates/antlr-rust-toml-parser/src/generated/toml_lexer.rsis excluded by!**/generated/**crates/antlr-rust-toml-parser/src/generated/toml_parser.rsis excluded by!**/generated/**docs/migration.mdis excluded by!**/docs/**
📒 Files selected for processing (13)
README.mdcrates/antlr-rust-codegen/src/generator/tests.rscrates/antlr-rust-codegen/src/parser/render/rules.rscrates/antlr-rust-codegen/src/parser/routing.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/cli.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/fixtures/revision9/codegen_api_lexer.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/fixtures/revision9/codegen_api_parser.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/parser.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/semantics.rscrates/antlr-rust-runtime/src/generated.rscrates/antlr-rust-runtime/src/lib.rscrates/antlr-rust-runtime/src/parser.rsthird_party/antlr-v4-grammar/self-hosted.sha256
💤 Files with no reviewable changes (2)
- crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/fixtures/revision9/codegen_api_lexer.rs
- crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/fixtures/revision9/codegen_api_parser.rs
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Strengthen the generated deep-nesting fixture to detect any numeric per-rule dispatch wrapper while still permitting adaptive dispatch helpers. Document that both ATN-preference and adaptive-routing exceptions remain explicit.
📊 Source Code Metrics (this PR vs
|
| File | Cyclomatic | Cognitive | Functions | LLOC | MI |
|---|---|---|---|---|---|
| crates/antlr-rust-codegen/src/generator/tests.rs | 292 ⚪ | 48 ⚪ | 207 ⚪ | 1332 (main: 1330) 🔴 | 0 ⚪ |
| crates/antlr-rust-runtime/src/generated.rs | 102 (main: 96) 🔴 | 14 (main: 10) 🔴 | 51 (main: 50) 🔴 | 101 (main: 92) 🔴 | 0 ⚪ |
| crates/antlr-rust-codegen/src/parser/routing.rs | 37 (main: 34) 🔴 | 25 (main: 26) 🟢 | 10 ⚪ | 94 (main: 93) 🔴 | 8.60 (main: 9.57) 🔴 |
| crates/antlr-rust-codegen/src/parser/render/rules.rs | 45 ⚪ | 52 ⚪ | 5 ⚪ | 113 (main: 114) 🟢 | 7.83 (main: 7.82) 🟢 |
| crates/antlr-rust-runtime/src/lib.rs | 4 ⚪ | 3 ⚪ | 1 ⚪ | 7 ⚪ | 29.69 (main: 28.88) 🟢 |
Generated by mehen v1.8.1 — the code quality watcher.
Contributor
Author
|
Claude review completed cleanly at |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
parse_generated_rule_N_dispatchwrapper with a generated function-pointer tableCompatibility
Revision 12 is now the only accepted generated-code API. The legacy dispatch/retry macro forms and frozen revision-9 compatibility fixtures are removed; older generated recognizers must be regenerated.
Measurements
The release
antlr4-rust-genbinary changed from 38,477,680 to 37,720,592 bytes: -757,088 bytes (-1.97%).Seven-round Kotlin parse-only mean averages remained within noise:
Validation
cargo test --locked --workspace --all-featurescargo clippy --locked --workspace --all-targets --all-features -- -D warningsFixes #322
Summary by CodeRabbit
Documentation
Breaking Changes
Improvements