Skip to content

fix(qa): score zio and kyo on the QA page, and unblock zio's stryker run - #113

Merged
kryptt merged 2 commits into
mainfrom
fix/qa-report-zio-kyo-modules
Sep 18, 2026
Merged

kryptt merged 2 commits into
mainfrom
fix/qa-report-zio-kyo-modules

Conversation

@kryptt

@kryptt kryptt commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Rebased onto main (ad30b87). Four PRs merged while this was open, and two
of them changed what this PR is even about. The corrections are called out inline;
the short version is that zio no longer renders n/a, because it no longer has
zero mutants.

The bug

site/tools/gen-qa-report.py builds the mutation table from a hard-coded
MUTATION_MODULES list, and that list stopped at jsoniter. zioIntegration
and kyoIntegration have been in the mutationAll alias for a while, so stryker
has been running over both — but the generator never looked for their reports, so
neither module has ever had a row on
quality-assurance.md. No error, no empty row,
just absence.

The two lists use different identifiers, and that is the trap the bug is made
of: quality.yml's loop takes sbt project ids (zioIntegration,
schemesLaws), MUTATION_MODULES takes on-disk directories (zio,
schemes-laws, which latest_report globs). Getting it backwards silently finds
nothing — exactly the failure being fixed — so both lists now carry a comment
saying so and pointing at each other.

A third instance of the same drift, fixed here: quality.yml's per-module loop
was generics schemes circeIntegration avroIntegration jsoniterIntegration —
missing schemesLaws as well as both integrations. A module absent from that
loop produces no report.json on the release run, so its row falls back to
dashes, or keeps whatever a developer's local run left on disk. The loop now
matches the mutationAll alias.

What the rebase changed

zio no longer scores n/a. When this PR was written, zio/ was a 98-line
DI integration — ZEnvironment / ZLayer / Ref wiring delegating straight into
ZIO's API — and stryker generated zero mutants for it, so n/a was the honest
row. #92 then landed the ecosystem optics (zio-schema DynamicValue kit, zio-json
AST + JsonCursor, STM focus ops, Chunks) and with them a lot of genuinely
mutatable code.

On the rebased tree project zioIntegration; stryker does not print n/a. It
aborts, with the same exception #109 fixed for kyo, now in
zio/json/JsonOptics.scala:

stryker4s.exception.UnableToFixCompilerErrorsException: Unable to remove non-compiling mutants
  'Extension without extension methods'
  'Not found: type To' / 'Not found: self' / 'Not found: type A'

Same mechanism, same fix, so the second commit applies it: stryker4s 0.20.3
re-prints every mutated file through scalameta, and a single-method
significant-indentation extension clause is not a Term.Block, so it comes back
as the one-line form with the method's leading Scaladoc replayed between the two —
the forced newline lands def in column 0. Four such clauses exist in the module:
JsonCursor.optional and JsonCodec.stringPrism in json/JsonOptics.scala,
Schema.dynamicPrism and BinaryCodec.prism in schema/SchemaOptics.scala.
Bracing their bodies makes the printer emit the braced form and the newline
harmless. ZioOptics's single-method clause already hoists its Scaladoc above
extension, which dodges the same trap, so it is untouched.

Verified rather than asserted: javap -p -c over JsonOptics$package$ and
SchemaOptics$package$ is byte-for-byte identical before and after the
bracing, and zioIntegration/test passes unchanged (268 examples).

kyo's numbers moved too, because #111's survivor-killing pass landed in
between: 20 K / 4 S / 38.5% became 30 K / 5 S / 47.6%.

Evidence: the generated rows

Produced by the patched generator reading the real report.json of a run on
this tree, not written by hand:

| `zio` | 37 | 0 | 6 | 0 | 0 | 86.0% | 86.0% |  |
| `kyo` | 30 | 0 | 5 | 28 | 0 | 47.6% | 85.7% | The no-coverage block is all `RecordIsoMacro`: quoted-macro code that expands at compile time, so like `generics` its mutants leave no runtime footprint. The covered score is the one that reads the hand-written optics. |
  • zio is a clean, high-signal row: 37 killed, 6 survived, and no
    no-coverage block at all. Everything stryker can mutate — the DynamicValue
    and zio.json tree navigation, the JsonCursor write descent, the Chunk
    index guards — is genuinely executed by the suite. It needs no Notes caveat,
    so it has none.
  • kyo's low total score is a macro artefact, not a coverage hole. All 28
    no-coverage mutants are in RecordIsoMacro — quoted-macro code that expands at
    compile time and leaves no runtime footprint for a test run to cover, the
    same structural reason generics scores 0%. The covered column (85.7%) is
    the one that describes the hand-written optics, and the Notes column says so.

Only the mutation table's two new rows are touched; the coverage table and the
composition matrix are left byte-identical, because a full in-place generator run
in a clone without a scoverage aggregate would have replaced the real coverage
numbers with "No scoverage aggregate report found".

The n/a branch stays, without zio as its example

Nothing hits it today. It is kept anyway: the old code rendered a zero-mutant
module as the same em-dash it uses for "no report at all", conflating "nothing
to score" with "nothing ran", and printing 0.0% would read as a test-quality
failure for a module that offers a mutator no purchase. n/a is also what
stryker's own console prints. The page now states the distinction as a general
rule instead of pointing at zio.

The all-NoCoverage shape is deliberately left alone: generics keeps 0.0% total
with an em-dash covered score, because those mutants do exist — they are simply
never executed.

report shape total covered
no mutants at all n/a n/a
only Ignored (StringLiteral-excluded) mutants n/a n/a
all NoCoverage (the generics shape) 0.0% —
ordinary kill/survive 86.0% 86.0%
no report.json — — (+ the "run sbt mutationAll" note)

Conflict resolution against main

Both conflicting hunks were keep-both — main and this branch edited the same
regions for unrelated reasons.

Nothing from either side was dropped. #111's Known equivalent mutants
catalogue is untouched.

Gates

Clone on Temurin 25, never the primary repo. kyo confirmed in the aggregate.

  • sbt "++ 3" compile test — success, 0 errors; kyo compiled and its four
    suites ran.
  • sbt benchmarks/compile — success (outside the root aggregate, breaks
    silently).
  • sbt mimaReportBinaryIssues — success.
  • sbt "docs/mdoc; docs/laikaSite" — success; rendered quality-assurance.html
    carries both new rows.
  • sbt scalafmtCheckAll "scalafixAll --check" — success, run last, after
    purging the target/streams/*/scalafmt* and *scalafix* caches; it genuinely
    re-read 20 source groups rather than reporting a cached pass.
  • zioIntegration/test — 268 examples, 0 failures, with the braces applied.
  • project zioIntegration; stryker / project kyoIntegration; stryker, in the
    project-switch form — never <module>/stryker, which reports 100% NoCoverage.
  • javap -p -c diff over the two affected package objects — identical.

🤖 Generated with Claude Code

https://claude.ai/code/session_0194EHFR4NamCpTHiqy7B74V

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

🚀 Cloudflare Pages preview for fix/qa-report-zio-kyo-modules is live:

https://43c69ef7.cats-eo-docs.pages.dev

Branch alias: https://fix-qa-report-zio-kyo-module.cats-eo-docs.pages.dev

Built from commit eee9a1272fb81dd638c0c6ccfc0c8eead34a3c1d · updated on every push.

kryptt and others added 2 commits September 18, 2026 18:15
`site/tools/gen-qa-report.py`'s `MUTATION_MODULES` stopped at `jsoniter`, so
the two effect-system integrations never had a row on the quality-assurance
page even though `mutationAll` has been running stryker over both. The
generator looked for reports it was never told to look for, which is the
quietest possible failure: no error, no empty row, just absence.

`.github/workflows/quality.yml`'s per-module loop had drifted the same way,
and further: it was missing `schemesLaws` too. A module absent from that loop
produces no report.json on the release-tag run, so its row falls back to
dashes — or, worse, keeps whatever numbers a developer's local run happened to
leave on disk. The loop now matches the `mutationAll` alias, and both lists
carry a comment saying they must stay in step. The loop takes sbt PROJECT IDS
(`zioIntegration`), `MUTATION_MODULES` takes DIRECTORIES (`zio`); getting that
backwards is what silently finds nothing, so it is written down.

Two honest edge cases the rows had to render correctly:

  - `zio` generates ZERO mutants on main — it is `ZEnvironment` / `ZLayer` /
    `Ref` wiring that delegates straight into ZIO's API, with no operator,
    literal or branch for a mutator to change. That is `n/a`, NOT 0%: a module
    with nothing to mutate has not failed a test-quality bar, it has no bar to
    fail. The old code divided only when the denominator was non-zero, but it
    rendered the empty case as the same em-dash it uses for "no report at all",
    conflating "nothing to score" with "nothing ran". `n/a` is also what
    stryker's own console prints here.
  - `kyo` scores 38.5% total / 83.3% covered, and the entire no-coverage block
    is `RecordIsoMacro` — quoted-macro code that expands at compile time, the
    same structural reason `generics` scores 0%. The Notes column now says so,
    so the total score does not read as a coverage hole.

The all-NoCoverage shape (`generics`: 0.0% total, covered undefined) is
deliberately left alone — those mutants exist, they are simply never executed.

Rows verified end to end: stryker was run per-module in a clone
(`project zioIntegration; stryker`, same for kyo — never `<module>/stryker`)
and the generator read the resulting report.json for both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194EHFR4NamCpTHiqy7B74V
The rebase onto main invalidated this branch's own premise. When it was
written, `zio` was `ZEnvironment` / `ZLayer` / `Ref` wiring that delegated
straight into ZIO's API, so stryker generated ZERO mutants for it and the
honest row was `n/a`. #92 then landed the ecosystem optics — the zio-schema
`DynamicValue` kit, the zio-json AST and `JsonCursor` bridge, the STM focus
ops, `Chunks` — and with them a great deal of genuinely mutatable code.

`project zioIntegration; stryker` on the rebased tree does not print `n/a`.
It aborts, with the same UnableToFixCompilerErrorsException that #109 fixed
for kyo, on `json/JsonOptics.scala`:

    'Extension without extension methods'
    'Not found: type To' / 'Not found: self' / 'Not found: type A'

Same mechanism, same fix. stryker4s 0.20.3 re-prints every mutated file
through scalameta; a SINGLE-METHOD significant-indentation `extension` clause
is not a `Term.Block`, so it comes back as the one-line form with the method's
leading Scaladoc replayed between the two, and the forced newline lands `def`
in column 0. Four such clauses exist in the module — two in `json/JsonOptics`
(`JsonCursor.optional`, `JsonCodec.stringPrism`) and two in
`schema/SchemaOptics` (`Schema.dynamicPrism`, `BinaryCodec.prism`). Bracing
their bodies makes the printer emit the braced form and the newline harmless.
`ZioOptics`'s single-method clause already hoists its Scaladoc above
`extension`, which dodges the same trap, so it is left alone.

Verified, not asserted: `javap -p -c` over `JsonOptics$package$` and
`SchemaOptics$package$` is byte-for-byte identical before and after, and
`zioIntegration/test` passes unchanged (268 examples).

With that, both effect-system rows are real measurements rather than
placeholders, regenerated by `site/tools/gen-qa-report.py` from the actual
report.json:

  | `zio` | 37 | 0 | 6 | 0 | 0 | 86.0% | 86.0% |
  | `kyo` | 30 | 0 | 5 | 28 | 0 | 47.6% | 85.7% |

zio is a clean high-signal row: 37 killed, 6 survived, and no no-coverage
block at all. kyo moved off #109's 20/4/28 (38.5% / 83.3%) because #111's
survivor-killing pass landed in between; its 28 no-coverage mutants are still
all `RecordIsoMacro`, so the note stands.

The generator's `n/a` branch stays. Nothing hits it today, but rendering a
zero-mutant module as the same em-dash it uses for "no report at all"
conflates "nothing to score" with "nothing ran", and printing 0.0% would read
as a test-quality failure where there is no bar to fail. The page now says so
explicitly rather than pointing at zio as the example.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194EHFR4NamCpTHiqy7B74V
@kryptt
kryptt force-pushed the fix/qa-report-zio-kyo-modules branch from 62b097b to eee9a12 Compare September 18, 2026 16:30
@kryptt kryptt changed the title fix(qa): score zio and kyo on the QA page fix(qa): score zio and kyo on the QA page, and unblock zio's stryker run Sep 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Benchmark A/B

Allocation (B/op) — authoritative

Benchmark params base head Δ
ZioDiBench.eoDrillGet - 0.0 0.0 -1.2%
ZioDiBench.eoServiceGet - 0.0 0.0 -0.0%
ZioDiBench.eoDrillModify - 528.0 528.0 +0.0%
ZioDiBench.eoServiceReplace - 456.0 456.0 -0.0%
ZioDiBench.handDrillModify - 648.0 648.0 +0.0%
ZioDiBench.handServiceGet - 120.0 120.0 -0.0%
ZioDiBench.handServiceReplace - 576.0 576.0 -0.0%
ZioDiBench.handDrillGet - 120.0 120.0 +0.0%
Timing (ns/op) — directional only, same-VM but shared runner
Benchmark params base head Δ
ZioDiBench.eoDrillModify - 78.3 84.0 +7.2%
ZioDiBench.eoServiceReplace - 62.6 60.5 -3.3%
ZioDiBench.handDrillModify - 71.0 72.2 +1.7%
ZioDiBench.eoDrillGet - 6.8 6.8 -1.1%
ZioDiBench.handServiceGet - 8.6 8.5 -1.0%
ZioDiBench.handServiceReplace - 64.4 64.1 -0.5%
ZioDiBench.handDrillGet - 8.4 8.4 +0.5%
ZioDiBench.eoServiceGet - 2.9 2.9 -0.4%

base_sha: ad30b87ce2171fdc815fc72e8cb6e4eba6c988d7 · head_sha: eee9a1272fb81dd638c0c6ccfc0c8eead34a3c1d · jdk: temurin-21 · runner: ubuntu-22.04 · jmh_params: -i 3 -wi 2 -f 1 -t 1 -foe true -prof gc -rf json · profile: pr:-i3-wi2-f1-t1-gc

@kryptt
kryptt merged commit 2544574 into main Sep 18, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant