Repository navigation
clean: replace regular expressions with structured parsing #66
Description
Activity
Dry-run releases of the new pin step ran on both variants of
release.ymland succeeded (run 37202601293 onrustas 0.6.1, run 37202603083 onkotlinas 0.4.1; nothing was committed, tagged or published).- rust (one tool): only
_VERSIONand the four hash lines ofplease_rustchanged. - kotlin (two tools):
_VERSION, and the four hashes of each ofplease_kotlinandplease_kotlin_wasmchanged, with different values per tool, and nothing else in the file.
That completes item 1 and its acceptance criterion (
dry_runrelease onrustandkotlin). Remaining items 2 to 10 are untouched.- rust (one tool): only
Item 2 (Kotlin JUnit) is in #67. Reading the code before changing it showed the situation was a little different from the inventory: the runner already preferred the launcher's XML, but it copied only the first
.xmlin the reports directory, while the launcher writes one report per engine (Jupiter, Vintage, Platform Suite). Results of any other engine were silently lost,<error>and<skipped>were not modelled, and the regex fallback never matched the launcher's real tree-style output (its tests only used synthetic lines). #67 reads every report withencoding/xml, recounts from the test cases, and drops the regex. Checked end to end through Please with a passing, a failing and a skipped test.Item 4 (CI assertions) is done:
lcovcheck(feat(ci): add lcovcheck and use case globs in the branch guard #68,tools/common/lcovcheck) asserts line statuses and function hits of a raw.lcovwith the shared parser, and per-file coverage in Please'scoverage.json, so nojqand no grep over text. Documented indocs/coverage-lcov.md.- The branch guard in
ci-main.ymlusescaseglobs (feat(ci): add lcovcheck and use case globs in the branch guard #68). - The coverage smoke steps of every language use it: ts ci(ts): assert coverage with lcovcheck instead of grep #71, kotlin ci(kotlin): assert coverage with lcovcheck instead of grep #70, swift ci(swift): assert coverage with lcovcheck instead of grep #72, rust ci(rust): assert coverage with lcovcheck instead of grep #69. They passed on the real runners.
Items done so far: 1 (release pins, #63 to #65 and the dry runs), 2 (Kotlin JUnit XML, #67), 4 (CI assertions). Next in the plan: 3 (WIT and Kotlin wasm onto
ast), 5 (release notes scanner, Vitest plugin), then 6 to 9.Item 3 (WIT and Kotlin wasm regexes) is done:
- The WIT parser is shared on
main(tools/common/wit/ast, feat(common): move the WIT ast parser to tools/common and add a lenient mode #73), with a lenient mode for callers that only need names. The parser kept only the non-standardstatic name: func(); the spec'sname: static func()failed, fixed in fix(common): parse static resource functions as the WIT spec writes them #75 (both spellings parse). please_witdiscovers worlds and package with it (refactor(wit): discover worlds and package with the AST parser #74); this also fixespackage a:b@1.0;, which the regex never matched. Its own copy of the parser is removed.please_kotlin_wasmparses WIT with it and maps types structurally (refactor(kotlin-wasm): read WIT with the shared AST parser instead of regexes #76): multi-line functions, comments, resource methods no longer listed as interface functions, packages spanning a directory.tupleis now an error instead of a bogus generated name.
Left from #66: 5 (release notes line scanner, Vitest alias plugin), 6 to 9 (Swift fallback, Rust cargo-test parser, deps.go, TOML).
DetectPackage/hasClassInSourcesin kotlin_wasm use regexes on Kotlin source (fits with the Kotlin items).- The WIT parser is shared on
Summary
Regular expressions are used in four layers of this repository: the Go parsers of each language's helper tool, one Python script, the release job, and the CI checks. Several of them parse something that has a structured form (JUnit XML, a WIT or TOML file, Please's coverage JSON, the
.lcovformat the shared package already parses), or edit a file that a release depends on. A regex there fails silently when the format drifts, or matches more than intended. This issue lists every use, ranks the risk, and proposes a structured replacement for each, to be done as small PRs.It came out of a bug we hit: a regex-based edit of test code silently dropped arguments, and the release job rewrites
tools/BUILDwithre.sub, which does nothing if the file's format changes.Inventory
release.yml(all language branches)re.subrewrites_VERSIONand thehashesoftools/BUILD; the hashes pattern replaces everyhashesblock it finds (Kotlin needed a lazy cross-block pattern per tool)scripts/update_release_pins.py(#63, synced by #64 forswiftand #65 forkotlin)tools/please_kotlin/testrunner/junit.go--reports-dir, so the launcher writes JUnit XML that is ignoredencoding/xmltools/please_wit/generate/generate.go(worldRegex,packageRegex); Kotlintools/please_kotlin_wasm/compile.go(funcRegex,packageRegex,ifaceRegex,resourceRegex)package,world,interface,func(...)over WIT sourcetools/please_wit/ast) andgenerate.goalready callsast.ParsePathelsewhereast. Kotlin cannot import it from thewitbranch, so move it totools/commononmain. The Kotlinpackageandclasslines can use a plain line scannerci-main.ymlbranch guard; the coverage smoke steps of each languagegrep -Eon file lists, on Please's console coverage table, and on.lcovtextcaseglobs for the guard;jqonplz-out/log/coverage.jsonfor line coverage; a smalllcovcheckcommand on the sharedtools/common/lcovparser for the.lcovassertions (statuses, functions, branches)scripts/generate_release_notes.pyre.search(rf"## \[v?{re.escape(version)}\]...", re.DOTALL); a list-marker regex for unwrappingDOTALLmatching over markdown## [version]heading, collect lines to the next## [; string methods for list markerstools/please_rust/download/metadata.goCargo.tomlforeditionand[lib] pathBurntSushi/toml)tools/please_swift/testrunner/runner.gotools/please_rust/testrunner/junit.gocargo testtextstrings.Cut/strings.Fieldson the documentedtest x ... oklinestools/please_rust/compile/deps.go-[0-9a-f]{16}$to strip a crate hash suffix-, hex checktools/please_ts/testrunner/vitest.goregexp.QuoteMetato build exact-match alias regexes for the generated Vite configNot a problem: the Starlark rules only use
startswith,mainhas no regex in Go, and the ANSI-strip pattern in Swift is a standard idiom (keep it if the fallback stays).Plan
Order by risk, one small PR per item on the branch that owns the code:
Release pins (docs: replace rust_rules_plugin_plan with comprehensive documentation #1)done; remaining: adry_runrelease dispatch to validate the new step end to end.lcovcheck,jq,caseglobs.ast(feat: Implement hermetic rust_toolchain rule #3).Rules for the replacements
strings.Cut,Fields) with tests beats a pattern.Acceptance criteria
dry_runrelease has run the pin step successfully onrustandkotlin.