test(semantic): raise scoring.py mutation kill rate 53.4% -> 98.3%

[kevincostner17]

What

Mutation testing of semantic/scoring.py killed only 31 of 58 mutants. This adds 45 tests that kill 26 of the 27 survivors; the last is proven equivalent, not left unexplained.

Run	Killed	Survived	Kill rate
Before	31/58	27	53.4%
After	57/58	1 (equivalent)	98.3%

All 58 mutation sites in the module, not applied: 0. Mutation classes are the ones the brief names: >=→>, <→<=, and→or, boolean-constant flips, numeric-threshold shifts.

Why the survivors mattered

Group 1 — risk-tier boundaries. confidence < 0.70 could become <=; >= 0.85 and >= 0.90 could become >; the literals 0.70 and 0.85 could be moved outright. Not one test noticed.

risk_for decides what the policy gate applies — under semantic_mode="auto" a proposal is applied without review only when risk != "high". So every one of these mutants silently changes which repairs auto-apply. Each boundary is now pinned at the value, just below it and just above it.

Group 2 — the audit feature record (calibration_features). The confidence=/margin= evidence-string split indices, the evidence-kind guards, the 1.0/0.0 role-confidence pair, round(coverage, 6), and the sum(1 for …) memory-support increment.

These change no cleaning decision — the record is hashed into ActionConfidence.features_hash and never reaches the gate. They are still real: the docstring promises that "a report consumer can verify two actions were scored from identical evidence", and a silently wrong feature breaks precisely that, letting two actions scored from different evidence hash identically. Pinned field by field, including an explicit non-collision test.

Group 3 — the isotonic calibration table. Both end clamps (raw <= xs[0], raw >= xs[-1]), the interpolation, the [0.0, 1.0] output clamp, the 4-dp rounding, the table is None or curve is None fallback, and the once-only "calibration table missing" warning.

All of it was uncovered, because calibration ships as identity — without an installed table the version string is literally "uncalibrated". It is nonetheless the code that sets the confidence the gate reads for anyone who installs a table. Worth noting raw >= xs[-1] → > does not merely return a wrong number: it raises IndexError.

The one surviving mutant is equivalent

frac = (raw - xs[lo]) / span if span else 0.0 → 0.05 survives, and no test can kill it. The line is reached only when xs[0] < raw < xs[-1]; hi = bisect_right(xs, raw) is the first index with xs[hi] > raw strictly, and xs[lo] = xs[hi-1] <= raw. So span > 0 for every curve, including the repeated-x curves from_json accepts — the guard is dead code.

Asserted over five awkward curves in test_the_zero_span_guard_in_the_interpolator_is_unreachable rather than argued only in prose.

A fragility recorded rather than hidden

calibration_features recovers two numeric features by string-scraping the human-readable evidence prose:

float(e.detail.split("confidence=")[1].split()[0])   # scoring.py:146

Both parses are wrapped in except (IndexError, ValueError) that degrades to None, so rewording an evidence sentence in experts.py would silently change the feature record — and therefore features_hash — with nothing failing and nothing raised. The tests now pin the current parse (first marker, next token), the wrong-evidence-kind rejection, and the malformed-input None degradation. The evidence-detail format is a load-bearing interface with no declared schema.

Scope

Tests only — no src/ change. No behaviour changes, so no changelog entry and no compatibility impact.

Verification

• tests/test_scoring_boundaries.py — 45 passed
• Full suite, py3.12 — 7203 passed, 0 failed, 0 errors
• ruff check + ruff format --check clean
• Mutation re-run: 57/58 killed, 1 proven-equivalent survivor

Two harness defects found and fixed during this work

Reported in the interest of not overstating the evidence — earlier figures from this harness are withdrawn.

1. The mutated file was not the imported file. The venv had freshdata installed editable against a different worktree, and with a src/ layout pytest's pythonpath = ["."] does not shadow it — so the harness mutated a file nothing imported and every mutant trivially "survived". Fixed by forcing PYTHONPATH=<root>/src into the pytest subprocess.

2. Mutate._hit shared one counter across all four mutation kinds while the driver indexed per kind. Most (kind, index) pairs therefore matched no node and were silently skipped: only 20 of this module's 58 sites ever ran. Fixed with per-kind counters and pre-order traversal matching the collector; the driver now prints NOT APPLIED and the summary reports a not applied: count, so the failure cannot be silent again.

Both baselines above were re-measured with the corrected harness over all 58 sites.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: a3c81f0e-9bdb-45f9-a3b1-153de8a9a4b0

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}