chore: add SPDX license headers to all hand-written source files - #340
Merged
Conversation
Every .rs, .py, and .sh source file now opens with: // SPDX-License-Identifier: BSD-3-Clause // Copyright (c) 2026 Konstantin Vyatkin Generated files (*/generated/*.rs, *.inc.rs) are left untouched — they already carry a @generated marker and are regenerated from grammars, so a license header would be overwritten on the next regen.
|
Important Review skippedToo many files! This PR contains 186 files, which is 86 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (186)
You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Copy/Paste DetectionFound 101 duplication(s) across 167 changed non-generated Rust file(s) (threshold: 100 tokens). Show duplicationsFound a 132 line (611 tokens) duplication in the following files:
}
/// Splits mixed-case, snake-case, and punctuation-heavy grammar identifiers
/// into words for Rust identifier rendering.
pub(crate) fn split_identifier_words(name: &str) -> Vec<String> {
let mut words = Vec::new();
let mut current = String::new();
let chars: Vec<char> = name.chars().collect();
for (index, ch) in chars.iter().copied().enumerate() {
if !ch.is_ascii_alphanumeric() {
if !current.is_empty() {
words.push(ascii_lowercase(¤t));
current.clear();
}
continue;
}
let previous = index.checked_sub(1).and_then(|i| chars.get(i)).copied();
let next = chars.get(index + 1).copied();
let starts_new_word = !current.is_empty()
&& ch.is_ascii_uppercase()
&& (previous.is_some_and(|prev| prev.is_ascii_lowercase() || prev.is_ascii_digit())
|| (previous.is_some_and(|prev| prev.is_ascii_uppercase())
&& next.is_some_and(|next| next.is_ascii_lowercase())));
if starts_new_word {
words.push(ascii_lowercase(¤t));
current.clear();
}
current.push(ch);
}
if !current.is_empty() {
words.push(ascii_lowercase(¤t));
}
words
}
/// Produces a legal Rust identifier and leaves keyword handling to callers that
/// know whether raw identifiers are valid at the target position.
pub(crate) fn sanitize_identifier(value: &str) -> String {
let mut out = String::new();
for (index, ch) in value.chars().enumerate() {
if ch == '_' || ch.is_ascii_alphanumeric() {
if index == 0 && ch.is_ascii_digit() {
out.push('_');
}
out.push(ch);
} else {
out.push('_');
}
}
if out.is_empty() { "_".to_owned() } else { out }
}
/// Produces a legal Rust identifier, using raw syntax only for keywords that
/// permit it. Path keywords cannot be raw identifiers, so suffix those names.
pub(crate) fn rust_identifier(value: &str) -> String {
let identifier = sanitize_identifier(value);
if matches!(identifier.as_str(), "crate" | "self" | "Self" | "super") {
format!("{identifier}_")
} else if is_rust_keyword(&identifier) {
format!("r#{identifier}")
} else {
identifier
}
}
/// Returns true for Rust reserved and contextual keywords that cannot be used
/// directly as generated identifiers.
pub(crate) fn is_rust_keyword(value: &str) -> bool {
matches!(
value,
"as" | "async"
| "await"
| "break"
| "const"
| "continue"
| "crate"
| "dyn"
| "else"
| "enum"
| "extern"
| "false"
| "fn"
| "for"
| "gen"
| "if"
| "impl"
| "in"
| "let"
| "loop"
| "match"
| "mod"
| "move"
| "mut"
| "pub"
| "ref"
| "return"
| "Self"
| "self"
| "static"
| "struct"
| "super"
| "trait"
| "true"
| "type"
| "unsafe"
| "use"
| "where"
| "while"
| "abstract"
| "become"
| "box"
| "do"
| "final"
| "macro"
| "override"
| "priv"
| "try"
| "typeof"
| "unsized"
| "virtual"
| "yield"
)
}
/// Converts ASCII letters to lower case without using allocation-hiding string
/// case helpers disallowed by the strict Clippy policy.
fn ascii_lowercase(value: &str) -> String {
value.chars().map(|ch| ch.to_ascii_lowercase()).collect()
}
```rust
---
Found a 65 line (392 tokens) duplication in the following files:
* Starting at line 3 of crates/antlr-rust-codegen/src/rust_output.rs
* Starting at line 3 of tests/antlr-rust-runtime-testsuite/src/rust_names.rs
```rust
use icu_properties::{CodePointSetData, CodePointSetDataBorrowed, props};
/// Returns the byte end of a Rust identifier beginning at `start`.
#[allow(dead_code)] // This shared module is also compiled by the conformance harness.
pub(crate) fn rust_identifier_end(value: &str, start: usize) -> Option<usize> {
static XID_START: CodePointSetDataBorrowed<'static> =
CodePointSetData::new::<props::XidStart>();
static XID_CONTINUE: CodePointSetDataBorrowed<'static> =
CodePointSetData::new::<props::XidContinue>();
let mut chars = value.get(start..)?.char_indices();
let (_, first) = chars.next()?;
if first != '_' && !XID_START.contains(first) {
return None;
}
let mut end = start + first.len_utf8();
for (relative, ch) in chars {
if !XID_CONTINUE.contains(ch) {
break;
}
end = start + relative + ch.len_utf8();
}
Some(end)
}
/// Converts a grammar type name into a snake-case module file name.
pub(crate) fn module_name(name: &str) -> String {
split_identifier_words(name).join("_")
}
/// Converts an ANTLR grammar name into a Rust type name.
pub(crate) fn rust_type_name(name: &str) -> String {
split_identifier_words(name)
.into_iter()
.map(|part| {
let mut chars = part.chars();
chars.next().map_or_else(String::new, |first| {
let mut out = String::with_capacity(part.len());
out.push(first.to_ascii_uppercase());
out.push_str(chars.as_str());
out
})
})
.collect()
}
/// Converts an ANTLR rule name into a snake-case Rust method name.
pub(crate) fn rust_function_name(name: &str) -> String {
let words = split_identifier_words(name);
let ident = if words.is_empty() {
"rule".to_owned()
} else {
words.join("_")
};
rust_identifier(&ident)
}
/// Escapes a Rust string literal using explicit ASCII escape forms.
pub(crate) fn rust_string(value: &str) -> String {
value.escape_default().to_string()
}
/// Replaces every non-overlapping occurrence without relying on the
/// allocation-hiding `str::replace` helper prohibited by the workspace lints.
pub(crate) fn replace_all(text: &str, needle: &str, replacement: &str) -> String {Found a 66 line (387 tokens) duplication in the following files:
use javascript_parser_base::JavaScriptParserBase;
fn dump_tree<S: AsRef<str>>(
out: &mut dyn Write,
tree: Node<'_>,
rule_names: &[S],
depth: usize,
) -> io::Result<()> {
let pad = " ".repeat(depth);
match tree.kind() {
NodeKind::Rule => {
let rule = tree.as_rule().expect("rule node kind checked");
let name = rule_names
.get(rule.rule_index())
.map_or("<?>", AsRef::as_ref);
writeln!(
out,
"{pad}Rule({name}, children={})",
rule.child_count()
)?;
for child in rule.children() {
dump_tree(out, child, rule_names, depth + 1)?;
}
}
NodeKind::Terminal => writeln!(
out,
"{pad}Term({:?})",
tree.as_terminal().expect("terminal node kind checked").text()
)?,
NodeKind::Error => writeln!(
out,
"{pad}Err({:?})",
tree.as_error().expect("error node kind checked").text()
)?,
}
Ok(())
}
fn main() -> ExitCode {
let mut args = env::args().skip(1);
let mut input: Option<PathBuf> = None;
let mut tokens_only = false;
while let Some(arg) = args.next() {
match arg.as_str() {
"--input" => input = args.next().map(PathBuf::from),
"--tokens" => tokens_only = true,
other => {
eprintln!("unknown argument: {other}");
return ExitCode::from(2);
}
}
}
let Some(input) = input else {
eprintln!("missing --input <path>");
return ExitCode::from(2);
};
let source = match fs::read_to_string(&input) {
Ok(source) => source,
Err(error) => {
eprintln!("failed to read {}: {error}", input.display());
return ExitCode::FAILURE;
}
};
if tokens_only {
let lexer = JavaScriptLexer::with_typed_hooks(
```rust
---
Found a 54 line (320 tokens) duplication in the following files:
* Starting at line 608 of crates/antlr-rust-runtime/src/atn/lexer.rs
* Starting at line 679 of crates/antlr-rust-runtime/src/atn/lexer.rs
```rust
atn: &LexerAtn,
hooks: &mut H,
mut generated_action: A,
mut generated_predicate: P,
unknown_policy: UnknownSemanticPolicy,
mut accept_adjuster: E,
) -> Result<TokenId, TokenStoreError>
where
I: CharStream,
H: SemanticHooks,
A: FnMut(&mut BaseLexer<I>, LexerCustomAction) -> bool,
P: FnMut(&BaseLexer<I>, LexerPredicate) -> Option<bool>,
E: FnMut(&mut BaseLexer<I>, i32, usize),
{
let hooks = RefCell::new(hooks);
let token = next_token_with_hooks_impl(
lexer,
sink,
atn,
&mut |lexer, action| {
if !generated_action(lexer, action)
&& !dispatch_lexer_action_hook(&hooks, lexer, action)
&& unknown_policy == UnknownSemanticPolicy::Error
&& let (Ok(rule), Ok(index)) = (
usize::try_from(action.rule_index()),
usize::try_from(action.action_index()),
)
{
lexer.record_semantic_error(true, rule, index);
}
},
&mut |lexer, predicate| {
generated_predicate(lexer, predicate)
.or_else(|| dispatch_lexer_predicate_hook(&hooks, lexer, predicate))
.unwrap_or_else(|| match unknown_policy {
UnknownSemanticPolicy::AssumeTrue => true,
UnknownSemanticPolicy::AssumeFalse => false,
UnknownSemanticPolicy::Error => {
lexer.record_semantic_error(
false,
predicate.rule_index(),
predicate.pred_index(),
);
false
}
})
},
&mut |lexer| dispatch_lexer_before_token_hook(&hooks, lexer),
&mut accept_adjuster,
&mut |lexer, accept_position| {
dispatch_lexer_after_accept_hook(&hooks, lexer, accept_position);
},
LexerMatchStrategy {
compiled: None,Found a 44 line (283 tokens) duplication in the following files:
}
fn nullable_rule_indices(graph: &FinalizedAtnGraph) -> BTreeSet<usize> {
let transitions = transitions_by_id(graph);
let mut nullable = BTreeSet::new();
loop {
let previous = nullable.len();
for (rule, (&start, &stop)) in graph.rule_starts.iter().zip(&graph.rule_stops).enumerate() {
if epsilon_reaches(graph, &transitions, start, stop, &nullable) {
nullable.insert(rule);
}
}
if nullable.len() == previous {
return nullable;
}
}
}
fn epsilon_reaches(
graph: &FinalizedAtnGraph,
transitions: &BTreeMap<super::super::model::BuildTransitionId, &FinalizedTransition>,
start: usize,
stop: usize,
nullable_rules: &BTreeSet<usize>,
) -> bool {
let mut pending = vec![start];
let mut visited = BTreeSet::new();
while let Some(state) = pending.pop() {
if state == stop {
return true;
}
if !visited.insert(state) {
continue;
}
for transition in graph.states[state]
.transitions
.iter()
.filter_map(|transition| transitions.get(transition).copied())
{
match &transition.kind {
FinalizedTransitionKind::Rule {
rule_index, follow, ..
} if nullable_rules.contains(rule_index) => pending.push(*follow),
kind if kind.is_epsilon() => pending.push(transition.target),
```rust
---
Found a 44 line (265 tokens) duplication in the following files:
* Starting at line 13 of tests/javascript-parity/dumper/src/javascript_parser_base.rs
* Starting at line 13 of tests/typescript-parity/dumper/src/typescript_parser_base.rs
```rust
impl JavaScriptParserBase {
fn raw_token<S>(ctx: &mut ParserSemCtx<'_, S>, index: usize) -> Option<(i32, i32, String)>
where
S: TokenSource,
{
ctx.token_at(index).map(|token| {
(
token.channel(),
token.token_type(),
token.text_or_empty().to_owned(),
)
})
}
fn has_line_terminator_ahead<S>(ctx: &mut ParserSemCtx<'_, S>) -> bool
where
S: TokenSource,
{
let current = ctx.input_index();
let Some(previous) = current.checked_sub(1) else {
return false;
};
let Some((channel, mut token_type, mut text)) = Self::raw_token(ctx, previous) else {
return false;
};
if channel != HIDDEN_CHANNEL {
return false;
}
if token_type == LINE_TERMINATOR {
return true;
}
if token_type == WHITE_SPACES {
let Some(before_whitespace) = previous.checked_sub(1) else {
return false;
};
let Some((_, next_type, next_text)) = Self::raw_token(ctx, before_whitespace) else {
return false;
};
token_type = next_type;
text = next_text;
}
token_type == LINE_TERMINATOR
|| (token_type == MULTI_LINE_COMMENT && (text.contains('\r') || text.contains('\n')))
}Found a 41 line (250 tokens) duplication in the following files:
offset = skip_while(bytes, offset, is_ascii_whitespace);
RustLexemeKind::Trivia
} else if body[offset..].starts_with("//") {
offset = body[offset..]
.find('\n')
.map_or(body.len(), |newline| offset + newline);
RustLexemeKind::Trivia
} else if body[offset..].starts_with("/*") {
offset = block_comment_end(body, offset);
RustLexemeKind::Trivia
} else if let Some(end) = raw_literal_end(body, offset) {
offset = end;
RustLexemeKind::Literal
} else if let Some(end) = quoted_literal_end(body, offset) {
offset = end;
RustLexemeKind::Literal
} else if let Some(end) = raw_identifier_end(body, offset) {
offset = end;
RustLexemeKind::Identifier
} else if let Some(end) = rust_identifier_end(body, offset) {
offset = end;
RustLexemeKind::Identifier
} else if bytes[offset].is_ascii_punctuation() {
offset += 1;
RustLexemeKind::Punctuation(bytes[offset - 1])
} else {
offset += body[offset..]
.chars()
.next()
.expect("offset is within the string")
.len_utf8();
RustLexemeKind::Other
};
lexemes.push(RustLexeme {
kind,
start,
end: offset,
});
}
lexemes
}
```rust
---
Found a 45 line (233 tokens) duplication in the following files:
* Starting at line 910 of crates/antlr-rust-codegen/src/grammar/atn/lexer.rs
* Starting at line 488 of crates/antlr-rust-codegen/src/grammar/atn/parser.rs
```rust
let (rule, _) = self.current_rule.expect("building a lexer rule");
self.epsilon_closures.push((rule.id, pair.left, pair.right));
let entry = self.synthetic_state(
AtnStateKind::StarLoopEntry,
SyntheticReason::LoopBoundary,
owner,
);
self.graph.state_mut(entry).non_greedy = !greedy;
self.graph.add_decision(entry);
let end = self.synthetic_state(AtnStateKind::LoopEnd, SyntheticReason::LoopBoundary, owner);
let loop_state = self.synthetic_state(
AtnStateKind::StarLoopBack,
SyntheticReason::LoopBoundary,
owner,
);
self.graph.state_mut(end).loop_back_state = Some(loop_state);
for target in if greedy {
[pair.left, end]
} else {
[end, pair.left]
} {
self.synthetic_epsilon(entry, target, SyntheticReason::LoopBoundary, owner, false);
}
self.synthetic_epsilon(
pair.right,
loop_state,
SyntheticReason::LoopBoundary,
owner,
false,
);
self.synthetic_epsilon(
loop_state,
entry,
SyntheticReason::LoopBoundary,
owner,
false,
);
StatePair {
left: entry,
right: end,
}
}
fn element_list(&mut self, elements: &[StatePair], owner: ModelNodeId) -> StatePair {
for pair in elements.windows(2) {Found a 21 line (226 tokens) duplication in the following files:
(9, AtnStateKind::RuleStop),
] {
assert_eq!(atn.add_state(kind, Some(0)).expect("state").index(), state);
}
atn.set_left_recursive_rule(0)
.expect("left-recursive rule start");
atn.set_precedence_rule_decision(2)
.expect("precedence decision");
atn.set_loop_back_state(8, 7).expect("loop-back state");
atn.set_rule_to_start_state(vec![0])
.expect("rule start states");
atn.set_rule_to_stop_state(vec![9])
.expect("rule stop states");
for state in [1, 2, 3] {
atn.add_decision_state(state).expect("decision state");
}
for (source, target) in [(0, 1), (2, 3), (2, 8), (7, 2), (8, 9)] {
atn.add_transition(source, ParserTransitionSpec::Epsilon { target })
.expect("epsilon transition");
}
for (source, target, label) in [(1, 2, 1), (1, 2, 2), (4, 6, 4), (5, 6, 3), (6, 7, 1)] {
```rust
---
Found a 37 line (221 tokens) duplication in the following files:
* Starting at line 1237 of crates/antlr-rust-codegen/src/grammar/mutual_recursion.rs
* Starting at line 132 of crates/antlr-rust-codegen/src/grammar/transform/analysis.rs
```rust
let rules = rules_by_id(unit);
let mut nullable = BTreeSet::new();
loop {
let previous = nullable.len();
for (id, rule) in &rules {
if rule.block.alternatives.iter().any(|alternative| {
alternative
.elements
.iter()
.all(|element| element_nullable(element, names, &nullable))
}) {
nullable.insert(*id);
}
}
if nullable.len() == previous {
return nullable;
}
}
}
fn element_nullable(
element: &Element,
names: &BTreeMap<String, RuleId>,
nullable: &BTreeSet<RuleId>,
) -> bool {
if matches!(
element.quantifier,
Quantifier::Optional { .. } | Quantifier::ZeroOrMore { .. }
) {
return true;
}
match &element.kind {
ElementKind::Epsilon | ElementKind::Action { .. } | ElementKind::Predicate { .. } => true,
ElementKind::RuleCall(call) => names
.get(&call.name)
.is_some_and(|target| nullable.contains(target)),
ElementKind::Block(block) => block_is_nullable(block, names, nullable),Found a 44 line (215 tokens) duplication in the following files:
fn plus_loop_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::PlusBlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))
.expect("state")
.index(),
3
);
assert_eq!(
atn.add_state(AtnStateKind::PlusLoopBack, Some(0))
.expect("state")
.index(),
4
);
assert_eq!(
atn.add_state(AtnStateKind::LoopEnd, Some(0))
.expect("state")
.index(),
5
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))
.expect("state")
.index(),
6
);
```rust
---
Found a 27 line (212 tokens) duplication in the following files:
* Starting at line 1331 of crates/antlr-rust-codegen/src/embedded/translate.rs
* Starting at line 312 of crates/antlr-rust-rs-parser/src/cfg_syntax.rs
```rust
pub(crate) fn quoted_literal_end(body: &str, start: usize) -> Option<usize> {
let bytes = body.as_bytes();
let (quote, content) = match bytes.get(start..start + 2) {
Some([b'b' | b'c', b'"']) => (b'"', start + 2),
Some([b'b', b'\'']) => (b'\'', start + 2),
_ if bytes[start] == b'"' => (b'"', start + 1),
_ if bytes[start] == b'\'' => (b'\'', start + 1),
_ => return None,
};
if quote == b'\'' {
return char_literal_end(body, content);
}
let mut offset = content;
let mut escaped = false;
while offset < bytes.len() {
if escaped {
escaped = false;
} else if bytes[offset] == b'\\' {
escaped = true;
} else if bytes[offset] == quote {
return Some(offset + 1);
}
offset += 1;
}
Some(body.len())
}Found a 42 line (210 tokens) duplication in the following files:
.map(|alternative| self.alternative(alternative, 0))
.collect(),
options: block.options.clone(),
syntax: block.syntax,
span: block.span.clone(),
}),
ElementKind::Action { id, body } => {
let cloned_id = self.ids.action();
self.record(ModelNodeId::Action(cloned_id), ModelNodeId::Action(*id));
ElementKind::Action {
id: cloned_id,
body: body.clone(),
}
}
ElementKind::Predicate {
id,
body,
fail,
precedence,
} => {
let cloned_id = self.ids.predicate();
self.record(
ModelNodeId::Predicate(cloned_id),
ModelNodeId::Predicate(*id),
);
ElementKind::Predicate {
id: cloned_id,
body: body.clone(),
fail: fail.clone(),
precedence: *precedence,
}
}
kind => kind.clone(),
};
self.record(
ModelNodeId::Element(cloned.id),
ModelNodeId::Element(source.id),
);
cloned
}
fn record(&mut self, destination: ModelNodeId, source: ModelNodeId) {
```rust
---
Found a 25 line (193 tokens) duplication in the following files:
* Starting at line 16623 of crates/antlr-rust-runtime/src/parser.rs
* Starting at line 17590 of crates/antlr-rust-runtime/src/parser.rs
```rust
let mut atn = ParserAtnBuilder::new(1);
for (state_number, kind) in [
(0, AtnStateKind::RuleStart),
(1, AtnStateKind::StarLoopEntry),
(2, AtnStateKind::Basic),
(3, AtnStateKind::Basic),
(4, AtnStateKind::StarLoopBack),
(5, AtnStateKind::LoopEnd),
(6, AtnStateKind::RuleStop),
] {
assert_eq!(
atn.add_state(kind, Some(0)).expect("state").index(),
state_number
);
}
atn.set_rule_to_start_state(vec![0])
.expect("rule start states");
atn.set_rule_to_stop_state(vec![6])
.expect("rule stop states");
atn.add_decision_state(1).expect("decision state");
atn.set_loop_back_state(5, 4).expect("loop back state");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("entry transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 2 })
.expect("loop body");Found a 21 line (188 tokens) duplication in the following files:
pub(crate) fn raw_literal_end(body: &str, start: usize) -> Option<usize> {
let rest = &body[start..];
let prefix = ["br", "cr", "r"]
.into_iter()
.find(|prefix| rest.starts_with(prefix))?;
let mut quote = start + prefix.len();
while body.as_bytes().get(quote) == Some(&b'#') {
quote += 1;
}
if body.as_bytes().get(quote) != Some(&b'"') {
return None;
}
let hashes = quote - start - prefix.len();
let closing = format!("\"{}", "#".repeat(hashes));
let content = quote + 1;
Some(
body[content..]
.find(&closing)
.map_or(body.len(), |end| content + end + closing.len()),
)
}
```rust
---
Found a 39 line (188 tokens) duplication in the following files:
* Starting at line 4143 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 16743 of crates/antlr-rust-runtime/src/parser.rs
```rust
fn block_decision_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::BlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
3
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))
.expect("state")
.index(),
4
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))
.expect("state")
.index(),
5
);
atn.set_end_state(1, 4).expect("block end state");Found a 20 line (186 tokens) duplication in the following files:
pub(crate) fn char_literal_end(body: &str, content: usize) -> Option<usize> {
let bytes = body.as_bytes();
let end = if bytes.get(content) == Some(&b'\\') {
match bytes.get(content + 1).copied()? {
b'x' => content.checked_add(4)?,
b'u' if bytes.get(content + 2) == Some(&b'{') => {
content + 3 + body[content + 3..].find('}')? + 1
}
_ => content.checked_add(2)?,
}
} else {
content
+ body[content..]
.chars()
.next()
.filter(|ch| *ch != '\'' && *ch != '\n' && *ch != '\r')?
.len_utf8()
};
(bytes.get(end) == Some(&b'\'')).then_some(end + 1)
}
```rust
---
Found a 25 line (182 tokens) duplication in the following files:
* Starting at line 576 of crates/antlr-rust-codegen/src/embedded/antlr4rust/scopes.rs
* Starting at line 27 of crates/antlr-rust-rs-parser/src/cfg_syntax.rs
```rust
pub(crate) fn new(lexemes: &[RustLexeme]) -> Self {
let mut pairs = vec![None; lexemes.len()];
let mut stack = Vec::new();
for (position, lexeme) in lexemes.iter().enumerate() {
match lexeme.kind {
RustLexemeKind::Punctuation(open @ (b'(' | b'[' | b'{')) => {
let close = match open {
b'(' => b')',
b'[' => b']',
b'{' => b'}',
_ => unreachable!("matched opening delimiter"),
};
stack.push((position, close));
}
RustLexemeKind::Punctuation(close @ (b')' | b']' | b'}')) => {
if let Some((open, expected)) = stack.pop()
&& close == expected
{
pairs[open] = Some(position);
pairs[position] = Some(open);
}
}
_ => {}
}
}Found a 37 line (176 tokens) duplication in the following files:
ElementKind::Block(block) => ElementKind::Block(self.block(block)),
ElementKind::Action { id, body } => {
let cloned_id = self.ids.action();
self.record(ModelNodeId::Action(cloned_id), ModelNodeId::Action(*id));
ElementKind::Action {
id: cloned_id,
body: body.clone(),
}
}
ElementKind::Predicate {
id,
body,
fail,
precedence,
} => {
let cloned_id = self.ids.predicate();
self.record(
ModelNodeId::Predicate(cloned_id),
ModelNodeId::Predicate(*id),
);
ElementKind::Predicate {
id: cloned_id,
body: body.clone(),
fail: fail.clone(),
precedence: *precedence,
}
}
kind => kind.clone(),
};
self.record(
ModelNodeId::Element(cloned.id),
ModelNodeId::Element(source.id),
);
cloned
}
fn label(&mut self, source: &Label) -> Label {
```rust
---
Found a 17 line (163 tokens) duplication in the following files:
* Starting at line 956 of crates/antlr-rust-codegen/src/grammar/atn/lexer.rs
* Starting at line 534 of crates/antlr-rust-codegen/src/grammar/atn/parser.rs
```rust
let next = pair[1];
let state = self.graph.state(element.left);
let transition = (state.kind == AtnStateKind::Basic
&& self.graph.state(element.right).kind == AtnStateKind::Basic
&& state.transitions.len() == 1)
.then(|| state.transitions[0]);
let can_inline = transition.is_some_and(|transition| {
let transition = self.graph.transition(transition);
match &transition.kind {
BuildTransitionKind::Rule { follow, .. } => *follow == element.right,
_ => transition.target == element.right,
}
});
if can_inline {
let transition = transition.expect("checked above");
match &mut self.graph.transition_mut(transition).kind {
BuildTransitionKind::Rule { follow, .. } => *follow = next.left,Found a 26 line (160 tokens) duplication in the following files:
};
}
}
}
// Deepest choices first, so an inner result rolls up into its parent branch.
// The list is rebuilt from `per_branch` each pass, because folding an inner
// choice *creates* an entry for its parent that must then fold in turn.
let mut processed: BTreeSet<usize> = BTreeSet::new();
// Deepest unprocessed choice still holding entries. Folding one creates an
// entry for its parent, so the candidate set is re-examined every pass.
while let Some(choice) = per_branch
.keys()
.map(|(choice, _)| *choice)
.filter(|choice| !processed.contains(choice))
.max_by_key(|choice| depth_of_choice.get(choice).copied().unwrap_or(0))
{
processed.insert(choice);
let counts = per_branch
.iter()
.filter(|((candidate, _), _)| *candidate == choice)
.map(|((_, branch), count)| (*branch, *count))
.collect::<Vec<_>>();
if counts.is_empty() {
continue;
}
let agreed = if restricted_to_one_path {
```rust
---
Found a 28 line (160 tokens) duplication in the following files:
* Starting at line 993 of crates/antlr-rust-codegen/src/grammar/atn/lexer.rs
* Starting at line 641 of crates/antlr-rust-codegen/src/grammar/atn/parser.rs
```rust
self.current_rule.expect("building a lexer rule").1
}
fn basic_pair(&mut self, owner: ModelNodeId) -> StatePair {
StatePair {
left: self.authored_state(AtnStateKind::Basic, owner),
right: self.authored_state(AtnStateKind::Basic, owner),
}
}
fn epsilon_pair(&mut self, owner: ModelNodeId) -> StatePair {
let pair = self.basic_pair(owner);
self.authored_transition(pair.left, pair.right, BuildTransitionKind::Epsilon, owner);
pair
}
fn atom_pair(&mut self, owner: ModelNodeId, label: i32) -> StatePair {
let pair = self.basic_pair(owner);
self.authored_transition(
pair.left,
pair.right,
BuildTransitionKind::Atom(label),
owner,
);
pair
}
fn authored_state(&mut self, kind: AtnStateKind, owner: ModelNodeId) -> BuildStateId {Found a 26 line (153 tokens) duplication in the following files:
pub fn next_token_with_hooks<I, A, P, E>(
lexer: &mut BaseLexer<I>,
sink: &mut TokenSink<'_>,
atn: &LexerAtn,
mut custom_action: A,
mut semantic_predicate: P,
mut accept_adjuster: E,
) -> Result<TokenId, TokenStoreError>
where
I: CharStream,
A: FnMut(&mut BaseLexer<I>, LexerCustomAction),
P: FnMut(&BaseLexer<I>, LexerPredicate) -> bool,
E: FnMut(&mut BaseLexer<I>, i32, usize),
{
next_token_with_hooks_impl(
lexer,
sink,
atn,
&mut custom_action,
&mut semantic_predicate,
&mut |_| {},
&mut accept_adjuster,
&mut |_, _| {},
LexerMatchStrategy {
compiled: None,
use_cache: false,
```rust
---
Found a 22 line (149 tokens) duplication in the following files:
* Starting at line 156 of crates/antlr-rust-codegen/src/grammar/atn/lexer.rs
* Starting at line 107 of crates/antlr-rust-codegen/src/grammar/atn/parser.rs
```rust
self.create_rule_boundaries();
for (rule_index, rule) in self.grammar.unit.rules.iter().enumerate() {
self.current_rule = Some((rule, rule_index));
let pair = self.block(&rule.block, Quantifier::One, ModelNodeId::Rule(rule.id));
let start = self.graph.rule_starts[rule_index];
let stop = self.graph.rule_stops[rule_index];
self.synthetic_epsilon(
start,
pair.left,
SyntheticReason::RuleBoundary,
ModelNodeId::Rule(rule.id),
false,
);
self.synthetic_epsilon(
pair.right,
stop,
SyntheticReason::RuleBoundary,
ModelNodeId::Rule(rule.id),
false,
);
}
self.current_rule = None;Found a 23 line (148 tokens) duplication in the following files:
fn unscoped_reads_reject_alternatives_that_would_satisfy_them_unbound() {
let token_ref = |label: Option<&str>, target: &str, token_type| ElementRef {
label: label.map(ToOwned::to_owned),
target: target.to_owned(),
token_types: vec![token_type],
is_block: false,
is_list: false,
cardinality: ChildCardinality {
min: 1,
max: Some(1),
},
stable_accessor: true,
choice_branch: Vec::new(),
choice_arity: Vec::new(),
choice_spans: Vec::new(),
group_spans: Vec::new(),
branch_spans: Vec::new(),
leading_terminal: true,
span: None,
branch_local_cardinality: ChildCardinality::ONE,
group_local_cardinality: ChildCardinality::ONE,
};
let translate = |second: ElementRef| {
```rust
---
Found a 21 line (146 tokens) duplication in the following files:
* Starting at line 256 of crates/antlr-rust-codegen/src/grammar/atn/lexer.rs
* Starting at line 152 of crates/antlr-rust-codegen/src/grammar/atn/parser.rs
```rust
);
}
}
fn block(&mut self, block: &Block, quantifier: Quantifier, owner: ModelNodeId) -> StatePair {
let alternatives = block
.alternatives
.iter()
.map(|alternative| self.alternative(alternative))
.collect::<Vec<_>>();
if quantifier == Quantifier::One && alternatives.len() == 1 {
return alternatives[0];
}
let start_kind = match quantifier {
Quantifier::One | Quantifier::Optional { .. } => AtnStateKind::BlockStart,
Quantifier::ZeroOrMore { .. } => AtnStateKind::StarBlockStart,
Quantifier::OneOrMore { .. } => AtnStateKind::PlusBlockStart,
};
let start = self.synthetic_state(start_kind, SyntheticReason::BlockBoundary, owner);
if (quantifier == Quantifier::One && alternatives.len() > 1)Found a 27 line (145 tokens) duplication in the following files:
fn generated_match_token_recovers_missing_token_from_context_follow() {
let atn = generated_match_recovery_atn();
let data = RecognizerData::new(
"Mini.g4",
Vocabulary::new(
[None, Some("'X'"), Some("'Y'")],
[None, Some("X"), Some("Y")],
[None::<&str>, None, None],
),
);
let mut parser = BaseParser::new(
CommonTokenStream::new(Source {
tokens: vec![TestToken::eof("parser-test", 3, 1, 3)],
index: 0,
}),
data,
);
parser.rule_context_stack = vec![
RuleContextFrame {
rule_index: 0,
invoking_state: 0,
},
RuleContextFrame {
rule_index: 1,
invoking_state: 1,
},
];
```rust
---
Found a 26 line (142 tokens) duplication in the following files:
* Starting at line 4181 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 4404 of crates/antlr-rust-codegen/src/generator/tests.rs
```rust
atn.set_end_state(1, 4).expect("block end state");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 2 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 3 })
.expect("transition");
atn.add_transition(
2,
ParserTransitionSpec::Atom {
target: 4,
label: 1,
},
)
.expect("transition");
atn.add_transition(
3,
ParserTransitionSpec::Atom {
target: 4,
label: 2,
},
)
.expect("transition");
atn.add_transition(4, ParserTransitionSpec::Epsilon { target: 5 })
.expect("transition");
atn.add_decision_state(1).expect("decision state");Found a 25 line (142 tokens) duplication in the following files:
atn: &LexerAtn,
hooks: &mut H,
) -> Result<TokenId, TokenStoreError>
where
I: CharStream,
H: SemanticHooks,
{
let hooks = RefCell::new(hooks);
let token = next_token_with_hooks_impl(
lexer,
sink,
atn,
&mut |lexer, action| {
let _ = dispatch_lexer_action_hook(&hooks, lexer, action);
},
&mut |lexer, predicate| {
dispatch_lexer_predicate_hook(&hooks, lexer, predicate).unwrap_or(true)
},
&mut |lexer| dispatch_lexer_before_token_hook(&hooks, lexer),
&mut |_, _, _| {},
&mut |lexer, accept_position| {
dispatch_lexer_after_accept_hook(&hooks, lexer, accept_position);
},
LexerMatchStrategy {
compiled: None,
```rust
---
Found a 23 line (139 tokens) duplication in the following files:
* Starting at line 1027 of crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/compatibility.rs
* Starting at line 1086 of crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/compatibility.rs
```rust
ID: [a-z]+;\n",
)
.expect("delegate grammar should be writable");
let output = run_antlr4_rust_gen(&[
root.as_os_str(),
OsStr::new("-I"),
temp.path().as_os_str(),
OsStr::new("--actions"),
OsStr::new("embedded"),
OsStr::new("--sem-unknown"),
OsStr::new("error"),
OsStr::new("--require-full-semantics"),
OsStr::new("--out-dir"),
out.as_os_str(),
]);
assert!(
output.status.success(),
"stdout: {}\nstderr: {}",
utf8(&output.stdout),
utf8(&output.stderr)
);
let parser = fs::read_to_string(out.join("root_parser.rs")).expect("parser should be emitted");Found a 33 line (137 tokens) duplication in the following files:
alternatives: &[StatePair],
owner: ModelNodeId,
) -> StatePair {
let end = self.synthetic_state(
AtnStateKind::BlockEnd,
SyntheticReason::BlockBoundary,
owner,
);
self.graph.state_mut(start).end_state = Some(end);
for alternative in alternatives {
self.synthetic_epsilon(
start,
alternative.left,
SyntheticReason::BlockBoundary,
owner,
false,
);
self.synthetic_epsilon(
alternative.right,
end,
SyntheticReason::BlockBoundary,
owner,
false,
);
remove_tail_epsilons(&mut self.graph, alternative.left);
}
StatePair {
left: start,
right: end,
}
}
fn optional(&mut self, pair: StatePair, greedy: bool, owner: ModelNodeId) -> StatePair {
```rust
---
Found a 19 line (137 tokens) duplication in the following files:
* Starting at line 871 of crates/antlr-rust-codegen/src/grammar/atn/lexer.rs
* Starting at line 448 of crates/antlr-rust-codegen/src/grammar/atn/parser.rs
```rust
let (rule, _) = self.current_rule.expect("building a lexer rule");
self.epsilon_closures.push((rule.id, pair.left, pair.right));
let loop_state = self.synthetic_state(
AtnStateKind::PlusLoopBack,
SyntheticReason::LoopBoundary,
owner,
);
self.graph.state_mut(loop_state).non_greedy = !greedy;
self.graph.add_decision(loop_state);
let end = self.synthetic_state(AtnStateKind::LoopEnd, SyntheticReason::LoopBoundary, owner);
self.graph.state_mut(pair.left).loop_back_state = Some(loop_state);
self.graph.state_mut(end).loop_back_st(report preview truncated from 124640 bytes) |
|
Claude finished @tinovyatkin's task in 22m 11s —— View job Code review: SPDX license headers
No merge blockers. All three Codex findings are resolved by Verification of the prior review round
Test evidence:
Mechanical correctness (verified file-by-file, not spot-checked)
Findings1. The invariant has no enforcement and its exclusions live only in a commit message · P1
Neither 2. Grammars-v4-derived
|
Contributor
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c104b0f49b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -1,3 +1,5 @@ | |||
| // SPDX-License-Identifier: BSD-3-Clause | |||
There was a problem hiding this comment.
When antlr-rust-toml-parser is published or scanned for license compliance, this per-file BSD-3-Clause identifier conflicts with the crate's license = "Apache-2.0" metadata and its packaged Apache LICENSE/NOTICE. The same incorrect header was added to ast.rs, decode.rs, and string.rs; use an Apache-2.0 header for these files, or deliberately update the package metadata and license materials if they are intended to be separately licensed.
Useful? React with 👍 / 👎.
Contributor
Author
There was a problem hiding this comment.
Fixed in ac76faf — lib.rs, ast.rs, decode.rs, and string.rs now carry SPDX-License-Identifier: Apache-2.0, matching the crate metadata and its packaged LICENSE/NOTICE.
Took the first option (match the declared license) rather than re-licensing the package: the crate is Apache-2.0 precisely because its recognizers derive from the ASF-licensed grammars-v4 TOML grammar, and include ships /LICENSE + /NOTICE alongside /src/**, so the per-file identifier has to agree with them.
Verified this is the only crate in the workspace that diverges — every other member takes license.workspace = true (BSD-3-Clause).
| @@ -1,3 +1,5 @@ | |||
| # SPDX-License-Identifier: BSD-3-Clause | |||
There was a problem hiding this comment.
When the noninteractive_bundle_requires_its_exact_fingerprint integration test runs, this comment changes the Java support bundle fingerprint because fingerprint_directory hashes every file byte, from the snapshotted sha256:38f5... to sha256:6909.... The corresponding untrusted_rust_support_diagnostic snapshot was not updated, so the workspace test suite will fail until the new diagnostic is reviewed and accepted.
AGENTS.md reference: AGENTS.md:L251-L253
Useful? React with 👍 / 👎.
Contributor
Author
There was a problem hiding this comment.
Fixed in ac76faf — but by dropping the headers rather than re-recording the fingerprint.
The snapshot is doing its job here: this tree is a pinned byte-level reproduction of grammars-v4 (each file documents its upstream commit and blob, e.g. Source blob: 9547fea6...), so accepting a new fingerprint would have silently normalized exactly the drift it exists to catch. A sole-copyright notice also misstates the provenance of files that are deliberately not ours. The whole rust-support/ tree is now reverted to its base content and mode.
rust_support::* is green locally (6/6), and fingerprint_directory has no other call site over a checked-in tree.
| @@ -1,3 +1,5 @@ | |||
| # SPDX-License-Identifier: BSD-3-Clause | |||
There was a problem hiding this comment.
When this newly executable test module is invoked as ./tools/parse-bench/test_run.py, it still begins with a comment rather than a shebang, so execve cannot select Python and shells may instead try to interpret the Python source as shell commands. This file was 100644 before the commit; either retain that mode or add a valid Python shebang if direct execution is intended.
Useful? React with 👍 / 👎.
Contributor
Author
There was a problem hiding this comment.
Fixed in ac76faf — restored to 100644.
Correct diagnosis: the pre-commit hook stripped the executable bit from every script it rewrote, and my restore pass ran a blanket chmod +x instead of replaying the original modes. test_run.py is an importable pytest module, so shebang-less and non-executable is right.
Two other files were caught by the same over-correction and are also back to 100644: tests/kotlin-parity/dump_python.py and tools/fixed-lookahead-bench/bench.py. Every mode in the PR is now diffed against the base commit and matches.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Three corrections from review of the initial header sweep: - Drop headers from the rust-support fixture tree. Those files are pinned byte-level reproductions of grammars-v4 (each documents its upstream commit and blob), so a sole-copyright notice misstates their provenance. The bundle is sha256-fingerprinted precisely to catch content drift, and the headers broke noninteractive_bundle_requires_its_exact_fingerprint. - antlr-rust-toml-parser publishes as Apache-2.0 (its recognizers derive from the ASF-licensed grammars-v4 TOML grammar), so its sources carry an Apache-2.0 identifier. A BSD-3-Clause header there would contradict the crate's own metadata and packaged LICENSE/NOTICE. - Restore 100644 on dump_python.py, bench.py, and test_run.py. Re-adding the executable bit stripped by the pre-commit hook over-corrected these three, which were never executable; test_run.py in particular is an importable pytest module with no shebang.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a two-line license header to every hand-written source file (
.rs,.py,.sh):Scope was narrowed during review — the header is only applied to files this project actually owns:
*/generated/*.rs,*.inc.rs// @generated; regenerated from grammars, so a header would be overwrittencrates/antlr-rust-codegen/tests/fixtures/.../rust-support/**third_party/**crates/antlr-rust-toml-parser/src/*.rsusesSPDX-License-Identifier: Apache-2.0instead — that crate publishes as Apache-2.0 because its recognizers derive from the ASF-licensed grammars-v4 TOML grammar, and it ships its own LICENSE/NOTICE. Every other workspace member takeslicense.workspace = true(BSD-3-Clause).Shell and Python files place the header after the shebang so
execvestill resolves the interpreter.Review fixes (ac76faf)
All three Codex findings addressed:
noninteractive_bundle_requires_its_exact_fingerprint. Fixed by reverting the fixture tree, not by re-recording the fingerprint: accepting a new hash would normalize away exactly the drift the snapshot exists to detect.antlr-rust-toml-parser's Apache-2.0 metadata. Now Apache-2.0.chmod +xand over-corrected three never-executable modules (test_run.py,dump_python.py,bench.py). All modes now diffed against the base commit and matching.Test plan
rust_support::*green locally (6/6) — the fingerprint suite that caught the P19d2a8c11; no unintended mode changesthird_party/file touchedReviewer note
CodeRabbit skipped this PR (186 files vs. its 100-file limit, plus a credits/capacity message), so it has not reviewed the diff. Splitting the sweep would add churn without much benefit for a uniform two-line header change — but happy to split if a CodeRabbit pass is wanted.