Report: interval floor of 10 paired cases, rank only interval-backed variants

[cirsteve]

Stacked on #19 (both re-render evidence/paa/reference/); retarget to main once #19 merges.

Summary

Addresses review items B2 and B3 on src/scout/replay/reporting.py, and records the family size for B4.

• B2 BOOTSTRAP_MIN_PAIRED_CASES 2 → 10. Below the floor the mean is still reported beside common_case_count, but interval_available is false and no bound is published. The Markdown says unavailable (< 10 paired cases).
• B3 A segment's ranking now contains only variants whose 95% interval excludes zero, ascending by mean delta. Every other variant with a mean goes into indistinguishable_from_baseline, an unordered set the Markdown renders as a set. Each variant carries interval_excludes_zero (null when no interval).
• B4 Each segment carries interval_family_size, the number of intervals it holds at the unadjusted 95% level. No correction is applied; the size is recorded so a reader can apply one. The Markdown states this per segment.
• REPORT_SCHEMA_VERSION 2 → 4. It skips 3 because the relevance report already uses 3 and render_markdown was dispatching on that number; it now dispatches on the presence of the relevance task document.
• Re-render evidence/paa/reference/ and the golden fixture (the fixture batch has two paired cases, now below the floor: empty ranking, one indistinguishable variant). Operator doc updated in docs/operations/offline-replay.md.

Not in this PR: B1 (repeats per case) moves the canonical plan hash and gets its own PR.

Test plan

• uv run pytest (2441 passed, 11 skipped) including new TestIntervalPartition unit tests, a 10-case sweep where both variants rank, and a 3-case sweep where neither does
• uv run ruff check ., uv run mypy .

🤖 Generated with Claude Code

https://claude.ai/code/session_01RzSnFgZ2rHydpunbz2b1V9

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Essentials

Run ID: 13ee27e7-93e2-4268-865f-dc8f90c59105

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

🟢 Approval recommended

The reporting behavior changes are consistently implemented across JSON/Markdown, reflected in updated fixtures, and covered by targeted new/updated tests for both above-floor and below-floor scenarios.

Pull request overview

This PR updates Scout’s batch/sweep replay reporting to (1) require a higher minimum paired-case count before publishing bootstrap confidence intervals, and (2) restrict segment “ranking” to interval-backed variants whose 95% interval excludes zero, while tracking and reporting the interval family size for auditability. It also bumps the report schema version accordingly, refreshes golden/reference evidence, and updates the pinned jig revision.

Changes:

• Raise BOOTSTRAP_MIN_PAIRED_CASES from 2 → 10; below the floor, report the mean/common-case count but omit CI bounds and mark interval_available: false.
• Partition variants into ranking (interval excludes zero, sorted by mean delta) vs indistinguishable_from_baseline (unordered set semantics; deterministic name-sorted representation), and add interval_family_size per segment.
• Bump schema version to 4, update markdown rendering, update jig pin, and re-render reference evidence / fixtures and docs.

File summaries

File	Description
src/scout/replay/reporting.py	Implements the new interval floor, interval-excludes-zero partitioning, interval family size, schema bump, and updated Markdown rendering/dispatch.
tests/test_replay_reporting.py	Updates existing assertions and adds focused unit tests for interval partitioning and floor behavior.
tests/test_replay_cli.py	Aligns CLI JSON report version assertion with the new schema version constant.
docs/operations/offline-replay.md	Documents the new interval floor, ranking semantics, and interval family size rationale.
src/scout/replay/experiments.py	Updates pinned JIG_REVISION.
pyproject.toml	Updates the jig git dependency revision.
uv.lock	Updates locked jig source revision to match the pin.
tests/test_jig_contract.py	Updates the contract-test docstring to reflect the new jig revision context.
web/tests/fixtures/replay-worker-configuration.json	Updates the jig revision in the web test fixture.
src/scout/paa/reference_evidence.py	Fixes the reference evidence regeneration command path in the error message.
evidence/paa/reference/reference-manifest.json	Re-rendered reference evidence manifest consistent with updated reporting outputs.
evidence/paa/reference/experiment-summary.json	Re-rendered reference experiment summary reflecting new report schema/fields.
tests/fixtures/paa_reference/expected/reference-manifest.json	Updates golden manifest fixture to match regenerated artifacts.
tests/fixtures/paa_reference/expected/experiment-summary.json	Updates golden experiment summary fixture for schema v4 and new segment fields.

Review details

• Files reviewed: 8/8 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.