expr: replace onig with fancy-regex by wtcpython · Pull Request #14329 · uutils/coreutils

wtcpython · 2026-08-31T12:07:58Z

Replace the onig C-dependency in expr with the pure-Rust fancy-regex crate.
This removes the onig_sys C library dependency while preserving full POSIX BRE,
backreference, multibyte UTF-8, and C locale compatibility.

Closes #1145

sylvestre · 2026-08-31T12:10:41Z

-                if is_start_of_expression {
+            re_string.push(curr);
+        } else {
+            match curr {


maybe move this into a new function?

github-actions · 2026-08-31T12:41:00Z

Binary size comparison:

Individual binary size comparison VS main (threshold: >=5% AND >=4 KB).

Total size of compared binaries: 153.88 MB (+1.68 MB, +1.10%)

Significant per-binary changes:
  expr       2.76 MB ->    3.77 MB  (+1.01 MB, +36.49%)
  ptx        2.42 MB ->    2.56 MB  (+140 KB, +5.65%)
  csplit     2.38 MB ->    2.51 MB  (+136 KB, +5.59%)
  nl         2.35 MB ->    2.48 MB  (+136 KB, +5.66%)
  tac        2.34 MB ->    2.48 MB  (+136 KB, +5.67%)

sylvestre · 2026-08-31T12:52:40Z

Binary size comparison:

Individual binary size comparison VS main (threshold: >=5% AND >=4 KB).

Total size of compared binaries: 153.89 MB (+1.70 MB, +1.12%)

Significant per-binary changes:
  expr       2.76 MB ->    3.79 MB  (+1.03 MB, +37.20%)
  ptx        2.42 MB ->    2.56 MB  (+140 KB, +5.65%)
  csplit     2.38 MB ->    2.51 MB  (+136 KB, +5.59%)
  nl         2.35 MB ->    2.48 MB  (+136 KB, +5.66%)
  tac        2.34 MB ->    2.48 MB  (+136 KB, +5.67%)

https://docs.rs/fancy-regex/latest/fancy_regex/#features

maybe we can disable some features

Sign in to view

+                        && !matches!(
+                            curr,
+                            '1'..='9'
+                                | '.'
+                                | '*'
+                                | '^'
+                                | '$'
+                                | '['
+                                | ']'
+                                | '\\'
+                                | 'w'
+                                | 'W'
+                                | 's'
+                                | 'S'
+                                | 'b'
+                                | 'B'
+                        )


github-actions · 2026-08-31T13:55:23Z

GNU testsuite comparison:

Skipping an intermittent issue tests/date/date-locale-hour (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/pr/bounded-memory (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/tail/follow-name (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/tail/tail-n0f (passes in this run but fails in the 'main' branch)
Congrats! The gnu test tests/cut/bounded-memory is now passing!
Congrats! The gnu test tests/tail/pipe-f is now passing!

Copilot

Pull request overview

This PR replaces expr’s onig (Oniguruma) dependency with the pure-Rust fancy-regex crate, updating the BRE-to-ERE transpilation and locale-aware matching logic to avoid a C library dependency and simplify cross-compilation (notably for WASI).

Changes:

Switch expr regex engine from onig to fancy-regex, including a new BRE→ERE transpiler and updated match evaluation logic.
Update workspace and crate dependencies/locks to remove onig and add fancy-regex.
Remove now-unneeded WASI sysroot installation steps from CI workflows and update documentation references.

Reviewed changes

Copilot reviewed 7 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/uu/expr/src/syntax_tree.rs	Replaces Oniguruma usage with `fancy-regex`, rewrites BRE→ERE transpilation, updates locale-aware match behavior, and adds unit tests.
src/uu/expr/Cargo.toml	Swaps `onig` dependency for `fancy-regex`.
Cargo.toml	Removes workspace `onig` dependency and adds `fancy-regex` workspace dependency configuration.
Cargo.lock	Lockfile updates reflecting removal of `onig` and addition of `fancy-regex` and its deps.
fuzz/Cargo.lock	Same lockfile updates for fuzz workspace.
README.md	Removes outdated mention of `expr` linking Oniguruma.
.github/workflows/wasi.yml	Removes WASI sysroot install step previously needed for C dependency.
.github/workflows/code-quality.yml	Removes WASI prerequisites step tied to Oniguruma.
.github/workflows/CICD.yml	Removes WASI sysroot setup tied to Oniguruma.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Pull request overview

Copilot reviewed 7 out of 9 changed files in this pull request and generated 1 comment.

+    let re_string = transpile_bre_to_ere(&pattern_str)?;
+
+    RegexBuilder::new(&format!("(?s){re_string}"))
+        .oniguruma_mode(true)
+        .build()
+        .map_err(|_| ExprError::InvalidRegexExpression)


sylvestre · 2026-09-01T09:16:15Z

Did you run benchmarks on it ? (hyperfine)
thanks

wtcpython · 2026-09-01T11:02:57Z

pattern: .*
Benchmark 1: main
  Time (mean ± σ):      11.1 ms ±   0.8 ms    [User: 8.3 ms, System: 2.0 ms]
  Range (min … max):     9.5 ms …  12.7 ms    200 runs
 
Benchmark 2: PR
  Time (mean ± σ):       9.7 ms ±   1.7 ms    [User: 7.7 ms, System: 1.3 ms]
  Range (min … max):     9.0 ms …  32.0 ms    200 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without anyinterferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Summary
  PR ran
    1.13 ± 0.21 times faster than main
pattern: \(.*\)
Benchmark 1: main
  Time (mean ± σ):       9.6 ms ±   0.4 ms    [User: 7.7 ms, System: 1.4 ms]
  Range (min … max):     9.0 ms …  10.5 ms    200 runs
 
Benchmark 2: PR
  Time (mean ± σ):       9.5 ms ±   0.4 ms    [User: 8.1 ms, System: 1.0 ms]
  Range (min … max):     9.0 ms …  10.9 ms    200 runs
 
Summary
  PR ran
    1.01 ± 0.06 times faster than main
pattern: [A]*
Benchmark 1: main
  Time (mean ± σ):      10.3 ms ±   1.1 ms    [User: 8.0 ms, System: 1.6 ms]
  Range (min … max):     9.6 ms …  23.6 ms    200 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without anyinterferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Benchmark 2: PR
  Time (mean ± σ):       9.4 ms ±   0.5 ms    [User: 7.6 ms, System: 1.2 ms]
  Range (min … max):     8.6 ms …  10.3 ms    200 runs
 
Summary
  PR ran
    1.10 ± 0.13 times faster than main
pattern: \(A*\)\1
Benchmark 1: main
  Time (mean ± σ):      10.8 ms ±   0.4 ms    [User: 8.7 ms, System: 1.5 ms]
  Range (min … max):    10.3 ms …  11.9 ms    200 runs
 
Benchmark 2: PR
  Time (mean ± σ):      11.3 ms ±   0.1 ms    [User: 9.3 ms, System: 1.4 ms]
  Range (min … max):    10.8 ms …  11.9 ms    200 runs
 
Summary
  main ran
    1.04 ± 0.04 times faster than PR

Copilot

🟡 Changes recommended

The new invalid-UTF-8 matching path can compute incorrect match byte ranges (and potentially panic) due to mixing String byte offsets with original byte indices.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Files reviewed: 7/9 changed files
Comments generated: 1
Review effort level: Lite

Copilot

🟡 Changes recommended

The new regex implementation appears to miss prior multiline matching behavior and has a bracket-mode parsing path that can skip the intended TrailingBackslash error handling.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

src/uu/expr/src/syntax_tree.rs:431

The previous onig-based implementation enabled both SINGLELINE and MULTILINE; the new fancy-regex pattern only sets dot-all via (?s). Without multiline mode, $/^ behavior can change (e.g., $ won’t match before a trailing newline), which risks breaking regex compatibility.

    RegexBuilder::new(&format!("(?s){re_string}"))
        .oniguruma_mode(true)
        .build()
        .map_err(|_| ExprError::InvalidRegexExpression)

Files reviewed: 7/9 changed files
Comments generated: 1
Review effort level: Lite

+        if in_bracket {
+            let is_first = re_string.len() == bracket_start_idx + 1
+                || (re_string.ends_with("[^") && re_string.len() == bracket_start_idx + 2);
+            if curr == ']' && !curr_is_escaped && !is_first {
+                in_bracket = false;


Replace the onig C-dependency in expr with the pure-Rust fancy-regex crate. This removes the onig_sys C library dependency while preserving full POSIX BRE, backreference, multibyte UTF-8, and C locale compatibility.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

oech3 · 2026-09-02T06:34:43Z

I think expr +1.01 MB is acceptible to remove C dep, but why are ptx, csplit, nl and tac affected?

sylvestre reviewed Aug 31, 2026

View reviewed changes

wtcpython force-pushed the expr-replace-onig branch from d49f620 to 4bdf740 Compare August 31, 2026 12:19

wtcpython force-pushed the expr-replace-onig branch 3 times, most recently from ccb43d7 to 734f140 Compare August 31, 2026 13:02

This comment was marked as resolved.

Sign in to view

oech3 reviewed Aug 31, 2026

View reviewed changes

wtcpython force-pushed the expr-replace-onig branch from 734f140 to 4c8805d Compare August 31, 2026 13:15

This comment was marked as resolved.

Sign in to view

Copilot AI lite review requested due to automatic review settings September 1, 2026 08:52

wtcpython force-pushed the expr-replace-onig branch from 4c8805d to dab85cb Compare September 1, 2026 08:52

Copilot started reviewing on behalf of wtcpython September 1, 2026 08:57 View session

wtcpython force-pushed the expr-replace-onig branch from dab85cb to ff7641e Compare September 1, 2026 09:01

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Comment thread src/uu/expr/src/syntax_tree.rs

Copilot AI review requested due to automatic review settings September 1, 2026 09:02

Copilot started reviewing on behalf of wtcpython September 1, 2026 09:05 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Comment thread src/uu/expr/src/syntax_tree.rs

Comment on lines +426 to +431

let re_string = transpile_bre_to_ere(&pattern_str)?;

RegexBuilder::new(&format!("(?s){re_string}"))

.oniguruma_mode(true)

.build()

.map_err(|_| ExprError::InvalidRegexExpression)

Copilot AI review requested due to automatic review settings September 2, 2026 05:17

wtcpython force-pushed the expr-replace-onig branch from ff7641e to 17f1407 Compare September 2, 2026 05:17

Copilot started reviewing on behalf of wtcpython September 2, 2026 05:17 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

Comment thread src/uu/expr/src/syntax_tree.rs Outdated

Copilot AI review requested due to automatic review settings September 2, 2026 05:36

wtcpython force-pushed the expr-replace-onig branch from 17f1407 to b0c5c1c Compare September 2, 2026 05:36

Copilot started reviewing on behalf of wtcpython September 2, 2026 05:38 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

Comment thread src/uu/expr/src/syntax_tree.rs

Comment on lines +288 to +292

if in_bracket {

let is_first = re_string.len() == bracket_start_idx + 1

|| (re_string.ends_with("[^") && re_string.len() == bracket_start_idx + 2);

if curr == ']' && !curr_is_escaped && !is_first {

in_bracket = false;

expr: replace onig with fancy-regex

92e9cad

Replace the onig C-dependency in expr with the pure-Rust fancy-regex crate. This removes the onig_sys C library dependency while preserving full POSIX BRE, backreference, multibyte UTF-8, and C locale compatibility.

wtcpython force-pushed the expr-replace-onig branch from b0c5c1c to 92e9cad Compare September 2, 2026 05:45

Copilot AI review requested due to automatic review settings September 2, 2026 05:45

Copilot AI reviewed Sep 2, 2026

Uh oh!

Uh oh!

Conversation

wtcpython commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sylvestre Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sylvestre commented Aug 31, 2026

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

github-actions Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

sylvestre commented Sep 1, 2026

Uh oh!

wtcpython commented Sep 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

oech3 commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wtcpython commented Aug 31, 2026 •

edited

Loading

github-actions Bot commented Aug 31, 2026 •

edited

Loading

github-actions Bot commented Aug 31, 2026 •

edited

Loading