Repository navigation
Coverage: make plz cover work and be documented for every language rule set #41
Description
Summary
plz cover does not reliably produce usable coverage across the language rule sets. Depending on the language, it fails outright, reports "No data", or reports numbers for the wrong files, and the requirements are only documented for Rust. Consumers of coverage reports (CI gates, dashboards such as Veritas) cannot trust the output today.
This was found while trying to ingest Please coverage into a quality dashboard (see the downstream issue linked below), but the problems are in the rule sets and their documentation and affect any user of plz cover.
Findings
Verified with ./pleasew cover on this repository's own test fixtures (test/rust/lib, test/ts/lib):
| Rule set | Result | Cause |
|---|---|---|
| Rust | Works: lib.rs 3/3 lines, 100.0% |
Needs [cover] fileextension = .rs in .plzconfig (documented in docs/rust/usage.md). Without it the report is silently Total coverage: No data. |
| TypeScript, Deno runner | Works: calculator.ts 6/6 lines, 100.0% |
Needs .ts under [cover] fileextension, which the ts branch's .plzconfig does not set and docs/ts does not mention. Without it the report lists other languages' files instead (tools/please_ts/commands.go 0/251, total 0.2%), which is misleading. |
| TypeScript, Vitest runner | Fails: the ts_test passes under plz test but fails under plz cover |
Startup Error: You cannot set "coverage.reportsDirectory" as <test run dir>. Vitest needs to be able to remove this directory before test run. The runner passes --coverage.reportsDirectory=<dir of the coverage file> (tools/please_ts/testrunner/vitest.go), and that directory is the test's working directory. The reported coverage is then 0/7 lines, 0.0%. |
| Kotlin, Swift, WIT | Not verified |
Reproduction (from a checkout of the ts branch):
./pleasew cover //test/ts/lib:calculator_test -o cover.fileextension:.ts # works
./pleasew cover //test/ts/lib:calculator_vitest_test -o cover.fileextension:.ts --show_all_output # failsNot covered here: the Go rules come from the upstream please-build/go-rules plugin, not this repository.
Proposed work
- TypeScript, Vitest runner: point
--coverage.reportsDirectoryat a subdirectory (for example<dir>/coverage), then copy or merge the generated lcov to the file Please expects, with a unit test invitest_test.go. Check that a coverage provider (@vitest/coverage-v8or istanbul) is available in the hermetic, offline vitest cache; if not, add it to thevitest_toolchainand document it. - Make a missing
[cover] fileextensionvisible: document the required setting for every rule set (ts,kotlin,swift, andwitif applicable) as is done for Rust, and consider a plugin-level check or warning when a language's test targets run undercoverwithout its extension configured, instead of a silent "No data" or a report of unrelated files. - Verify and fix Kotlin, Swift and WIT, and record the result for each.
- Add a coverage smoke test per language (a fixture whose coverage must come out non-zero under
plz cover) to the test suite or CI, so regressions like the Vitest one are caught. - Document the output of each rule set: where the report is written (
plz-out/log/coverage.xmlandcoverage.json), its format (Cobertura or lcov), and what it contains (line coverage, branch coverage, and per-function data where available; per-function data is what lets consumers join coverage with complexity to find untested complex code).
Acceptance criteria
-
./pleasew coveron each language's fixture reports plausible, non-zero coverage for the right files, includingts_testwithrunner = "vitest" - The
[cover]requirement is documented for every rule set, and a missing setting is no longer silent - A coverage smoke test per language guards against regressions
- Each rule set documents its report location, format and contents
Related
- Downstream consumer: walterjgsp/veritas#64 (coverage prerequisite) and walterjgsp/veritas#20 (coverage tab)
Activity
fix(swift): link and cover transitive library dependencies
ci(rust): capture stderr in the coverage smoke test
ci(ts): capture stderr in the coverage smoke tests
ci(kotlin): capture stderr in the coverage smoke test
ci(swift): capture stderr in the coverage smoke tests
fix(rust): pass the configured llvm tools to rust_test
walterjgsp commentedon Oct 3, 2026
Done in #42 (ts), #43 (kotlin), #44 (swift) and #45 (rust), all merged into their language branches.
Fixed
plz covernow works with the Vitest runner, and[cover] fileextension = .tsis set.rust_testignored its configured LLVM tools and used the host's, so coverage was 0% on a runner with an older LLVM; fixed.[cover]setting, the report location and format.Verified for upstream rules
go-rules): works with.goin[cover] fileextension.python-rules): coverage support exists upstream; not run here.Not done
[cover] fileextension: Please has no hook for it, so it is documented instead.Releases (ts 0.5.3, kotlin 0.3.6, swift 0.2.1) are not cut yet; Rust is under Unreleased.