Guard patient partitions and IRLS convergence; add reproducible comparator benchmark

.gitignore

@@ -34,3 +34,8 @@ validation/predrupdate/gusto_estimates.csv
 examples/improve/cohort.parquet
 examples/improve/predictions.parquet
 examples/improve/out/
+
+# Isolated benchmark environments and scratch reruns
+.venv-clean/
+.venv-benchmark/
+benchmarks/alternatives/results/run*/

README.md

@@ -8,6 +8,120 @@ prespecified subgroup behaviour, and a recalibration ladder that reports how muc
 miscalibration is repairable without touching the model, and discloses the ladder's own
 optimism rather than reporting only its best face.
 
+## Quickstart with your own predictions
+
+No PCam files or model weights are needed. From a checkout, run `uv sync --locked`, then save
+these three files in a new `my-cohort/` directory. This tiny, fully synthetic CSV example is
+public demonstration data, not patient data or a statistical validation benchmark.
+
+`cohort.csv` (one row per observation):
+
+```csv
+subject_id,patient_id,label
+v01,p01,0
+v02,p01,1
+v03,p02,1
+v04,p02,0
+v05,p03,0
+v06,p03,1
+v07,p04,1
+v08,p04,0
+v09,p05,0
+v10,p05,1
+v11,p06,1
+v12,p06,0
+```
+
+`predictions.csv` (one row per scored observation and model):
+
+```csv
+subject_id,model_id,predicted
+v01,my_model,0.20
+v02,my_model,0.65
+v03,my_model,0.35
+v04,my_model,0.55
+v05,my_model,0.30
+v06,my_model,0.75
+v07,my_model,0.60
+v08,my_model,0.40
+v09,my_model,0.70
+v10,my_model,0.45
+v11,my_model,0.80
+v12,my_model,0.25
+```
+
+`cohort.yaml` (paths are relative to this file):
+
+```yaml
+cohort_id: synthetic-own-data-demo
+version: 0
+subject_key: subject_id
+data: cohort.csv
+unit_noun: visit
+outcome:
+  type: binary
+  field: label
+  positive_label: 1
+clustering:
+  field: patient_id
+  name: patient
+coverage:
+  min_fraction: 1.0
+  compare_on: both
+completeness:
+  require_outcome: true
+  on_violation: fail
+expected_models: [my_model]
+on_missing_model: fail
+thresholds: [0.5]
+uncertainty:
+  n_boot: 20  # Fast plumbing check only; the library default is 2000.
+  seed: 1337
+```
+
+From the checkout, run the public API (CSV and Parquet are both supported):
+
+```bash
+uv run python - <<'PY'
+from pathlib import Path
+from predval import evaluate, load_cohort, load_predictions, write_findings, write_report
+
+root = Path("my-cohort")
+result = evaluate(load_cohort(root / "cohort.yaml"), load_predictions(root / "predictions.csv"))
+out = root / "results"
+result.write(out)
+write_report(result, out / "report.html")
+write_findings(result, out / "findings.json")
+print(out / "report.html")
+PY
+```
+
+Open `my-cohort/results/report.html`; `findings.json` is the machine-readable companion, and
+`result.write` saves the metric tables and provenance manifest. With only six independent
+patients, warnings or unavailable recalibration rungs are expected, not a reason to hide rows.
+
+To replace the toy data:
+- Keep `subject_id` unique for each observation and identical across the two files. With one row
+  per patient it can identify that patient; repeated visits need unique visit IDs plus a separate
+  patient clustering column. Use pseudonymous IDs, not names or medical record numbers.
+- Declare the actual independence unit in `clustering.field` (patient, site, slide, etc.).
+  Omitting it treats observations as independent; a unique row ID is not a substitute for a
+  shared cluster ID. Whether clusters are independent is a study-design judgment.
+- List all intended model IDs in `expected_models`; this example fails if an entire model is
+  missing. `predicted` must be the probability of `outcome == positive_label`, supplied as a
+  finite number in `[0, 1]` (not the probability of the other class). An unscored observation has no prediction
+  row, never a zero placeholder. Set the coverage floor before analysis; `1.0` here requires
+  every usable observation to be scored by each model. `compare_on: both` also evaluates the
+  common scored subset. Review missing outcomes and coverage before interpreting comparisons.
+- Prespecify the outcome, thresholds, subgroups, model roster and analysis plan. Replace the
+  demonstration bootstrap budget with a justified analysis budget; more resamples cannot fix
+  too few independent clusters or data-adaptive selection.
+
+A completed run checks these inputs and reports performance on this cohort. It does not prove
+clinical utility, transportability, calibrated probabilities, or that a selected recalibration
+will improve a new cohort. Gain intervals resample fixed cross-fitted predictions; neither
+conditional coverage nor full fitting-and-selection coverage is established by that choice.
+
 ## The pitch, narrowed
 
 Most tools in this space either train something (a new calibration layer, a monitoring model)
@@ -22,10 +136,14 @@ Three things follow from that scope, and they are what predval spends its effort
   flatters the model.
 - **The unit of analysis and the unit of independence are usually not the same file column.**
   When predictions come from grouped data — patients with repeat visits, patches from the same
-  slide, images from the same device — per-row confidence intervals are not slightly optimistic,
-  they are wrong by roughly the square root of the cluster size. predval takes a declared
-  clustering unit and resamples whole clusters everywhere it computes uncertainty, and renders
-  the naive-vs-cluster comparison once so the gap is a number, not a caveat in a footnote.
+  slide, images from the same device — ignoring positive within-cluster dependence can seriously
+  understate uncertainty. For an illustrative mean with independent clusters of equal size `m`
+  and exchangeable within-cluster correlation `ICC`, the standard-error inflation is
+  `sqrt(1 + (m - 1) * ICC)`;
+  `sqrt(m)` is the special case `ICC = 1`, not a universal factor for every metric. predval takes
+  a declared clustering unit and resamples whole clusters everywhere it computes uncertainty,
+  and renders the naive-vs-cluster comparison once so the gap is a number, not a caveat in a
+  footnote.
 - **A recalibration that improves the numbers is a finding about the ladder's own fit, not a
   promotion of the model, until it survives being scored on clusters it never saw.** Every rung of
   predval's ladder — intercept, intercept+slope, spline — is fit twice: once on the rows it will
@@ -34,15 +152,20 @@ Three things follow from that scope, and they are what predval spends its effort
   validated model, and — this is the part most recalibration tooling skips — refuses to
   manufacture a gain where none exists: on the PCam fixture below, nine of the fifteen members
   carry a detected miscalibration, ten have the repair's benefit left honestly unproven at this
-  cohort's small cluster count, and exactly one — `p4m_seed7` at rung2 — gets a real,
-  interval-backed repair. Five members carry at least one rung whose recalibration *reliably
-  harms* them, and `p4m_seed7` is one of the five: its rung2 repair is demonstrated while its
-  rung1 is harmful, which is why harm is reported independently of whether a repair was found.
-  Two of those five, `e2cnn_s21` and `e2cnn_s99`, are harmed at every rung and have no admissible
-  repair at all. The remaining six show no detected fault, which the report states as a null
+  cohort's small cluster count, and exactly one — `p4m_seed7` at rung2 — has a positive
+  estimated gain whose interval excludes zero under the configured fixed-prediction resampling.
+  Five members have at least one rung with a gain interval entirely below zero, including
+  `p4m_seed7` at rung1: estimated harm is reported independently of estimated benefit at rung2.
+  Two of those five, `e2cnn_s21` and `e2cnn_s99`, have negative gain intervals at every rung
+  and no admissible repair under the configured criterion. The remaining six show no detected fault, which the report states as a null
   result and never as a clean bill of health: this harness's calibration test can fail to reject
   "calibrated", but it can never establish it.
 
+The paired gain intervals hold fitted cross-fold predictions fixed. They do not establish
+nominal coverage for the full fitting-and-selection procedure, or conditional coverage merely
+because fitting is held fixed during resampling. Limited independent clusters and data-adaptive
+selection of models or rungs warrant caution when interpreting an interval that excludes zero. See [the interval limitations](docs/spec.md#45-rung-intervals-the-paired-cross-fit-gain-carries-one-the-per-rung-refit-is-deferred).
+
 ## What predval deliberately does not do
 
 - **It never loads model weights.** No `torch`, no `tensorflow`, no checkpoints. Inputs are a
@@ -253,9 +376,11 @@ matching PCam's — the full B=2000 run is frozen and regression-checked by
 headline numbers fails loudly.
 
 **Honest read of the verdict:** AUROC 0.791, comfortably inside the literature's expected
-0.72–0.82 range on the first attempt. Calibration slope 0.835 shows real, if modest, displacement
-scoring off the development region — the recalibration ladder finds a small but statistically
-real gain at rungs 2–3 (both exclude zero), while rung 1 alone is actually slightly negative. This
+0.72–0.82 range on the first attempt. Calibration slope 0.835 indicates modest displacement
+scoring off the development region. Rungs 2–3 have small positive estimated gains whose
+intervals exclude zero under the configured fixed-prediction resampling; rung 1 has a slightly
+negative point estimate. Limited independent clusters and data-adaptive selection limit what
+these intervals establish; they do not validate the full fitting-and-selection procedure. This
 is a much milder miscalibration story than PCam's two badly-miscalibrated members, which is an
 honest property of this cohort rather than something to chase. The real finding of this example
 isn't the numbers — it's that predval's ingestion, coverage, roster, clustering, and ladder code

benchmarks/alternatives/.gitignore

@@ -0,0 +1,10 @@
+# Generated row-level fixtures and predictions regenerate from the locked script.
+# Commit compact numerical summaries and hashes, not bulk row-level inputs.
+results/python/coverage_inputs/
+results/python/wdbc_heldout.csv
+results/python/wdbc_heldout_corrected.csv
+results/python/synthetic_clustered.csv
+results/python/synthetic_clustered_corrected.csv
+results/r/*_r_corrected.csv
+results/r/*_rsample_gains.csv
+results/r-repeat*/

benchmarks/alternatives/FIXTURE_LICENSE.md

@@ -0,0 +1,22 @@
+# Fixture attribution
+
+Wisconsin Diagnostic Breast Cancer is provided by the UCI Machine Learning Repository
+under Creative Commons Attribution 4.0 International:
+https://creativecommons.org/licenses/by/4.0/
+
+Wolberg, W., Mangasarian, O., Street, N., & Street, W. (1993).
+Breast Cancer Wisconsin (Diagnostic). UCI Machine Learning Repository.
+https://doi.org/10.24432/C5DW2B
+
+Source/license: https://archive.ics.uci.edu/dataset/17/breast+cancer+wisconsin+diagnostic
+
+The benchmark uses scikit-learn's packaged copy. It derives a fixed held-out subset,
+probabilities from a model fitted on separate training rows, deliberately distorted
+probabilities, and row-derived identifiers. These are alterations; no creator endorsement
+is implied. Generated row-level data are not committed or included in the compact bundle.
+
+The repeated-patient fixture is wholly synthetic and generated by the repository's own
+benchmark code. It is not clinical validation and contains no real patient records.
+
+The benchmark's original code is covered by the repository's Apache-2.0 LICENSE.
+No source-only competitor package is redistributed in the compact review bundle.

benchmarks/alternatives/INTEGRATION.md

@@ -0,0 +1,112 @@
+# Reviewed-change integration: 2026-10-10
+
+## Result
+
+The reviewed patient-partition guard, alternatives comparator, and IRLS iteration-limit
+fix are combined on local branch `integrate/reviewed-predval-20261010`.
+The combined suite passes **282 tests**, with **20 optional fixture-dependent skips**,
+**zero failures**, and **zero expected failures**. The comparator contracts also pass
+separately: **6/6**, including the formerly expected-failure quasi-separation case.
+Ruff lint, Ruff formatting (63 files), and `git diff --check` pass.
+
+Both Python comparator reruns reproduce **all 13 stable input/result hashes exactly**
+from the original benchmark. Thus the two main fixtures' point estimates, corrections,
+intervals, coverage flags, and suppression summaries have not changed. This does not imply
+that every possible input is unchanged: exhausted IRLS fits now become unavailable rather
+than being reported as converged finite fits.
+
+## Source and evidence provenance
+
+- Patient-partition guard: `99ca3d72f1d4f8340df9a59de3616277e9ebdb7a`.
+- Original comparator: `868276ef960f0bc7f2bc75cc79926ff7b3c2e1f1`; its original
+  preregistration commits `dae6e1a` and `756e5a1` remain in the branch history.
+- Reviewed convergence fix: `62c8af368824d75e1dd915e78c131f46e333a340`, cherry-picked
+  without conflicts as `617ddac480849ac48de41c1c1b1d050ddc427492`.
+- Integration-specific source edit: remove the strict xfail marker from
+  `tests/test_benchmark.py` and update its explanatory comment. The core implementation,
+  benchmark runner, R adapter, and frozen benchmark protocol are unchanged beyond the
+  reviewed guard. README wording now distinguishes historical and integrated checks.
+- Package metadata still says PREDVAL 0.1.0. The source commits and file hashes in
+  [`verification/integrated/status.json`](verification/integrated/status.json), rather
+  than the package version alone, identify the code tested here.
+
+Original compact results and verification logs remain byte-for-byte unchanged. Their old
+269-pass/5-pass/1-xfail counts describe the original comparator, not this integrated source.
+New integration logs and the hash-comparison record live separately under
+`verification/integrated/`. The independent statistical-stress study is **not integrated**;
+its frozen baseline and worktree were not changed by this task.
+
+## What was rerun
+
+The full combined Python suite ran against this worktree. The comparator ran twice with
+`PYTHONPATH` explicitly selecting this worktree's `src`; source imports were checked in
+both interpreters. These used two existing isolated Python 3.12.14 environments originally
+created from the benchmark's complete requirements lock, not two new installations during
+integration. Each run verified 13 stable file hashes, for 26 new file-hash checks. Both
+manifests equal the historical manifest, SHA-256:
+
+`a2a541ec30fbffe83103e04d2c85a5e83114a671e9818488e896a2f34eaf510d`
+
+The R adapter consumes the exported fixtures and corrected predictions, all byte-identical
+here, and its source is unchanged. All seven retained R stable files were checked against
+the original recorded hashes. **R computation was not repeated for integration.** The
+original repeated R computation is reused evidence, not a newly executed R result.
+
+The benchmark-body Python wall times were 3.86 and 5.47 seconds. As in the original report,
+these exclude imports/installation, reflect concurrent machine load, and do not compare
+individual tools' speed. Exact resource records are in the integration JSON.
+
+## Reproduce the integration checks
+
+Use the locked Python setup in the main benchmark README. From the repository root:
+
+```sh
+export PYTHONPATH="$PWD/src"
+export MPLCONFIGDIR=/tmp/predval-mpl XDG_CACHE_HOME=/tmp/predval-cache
+export OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1
+.venv-benchmark/bin/python -m pytest tests benchmarks/alternatives/tests -ra
+.venv-benchmark/bin/ruff check .
+.venv-benchmark/bin/ruff format --check .
+git diff --check
+.venv-benchmark/bin/python benchmarks/alternatives/run_benchmark.py --out /tmp/predval-integrated-a
+.venv-benchmark/bin/python benchmarks/alternatives/run_benchmark.py --out /tmp/predval-integrated-b
+cmp benchmarks/alternatives/results/python/manifest.json /tmp/predval-integrated-a/manifest.json
+cmp /tmp/predval-integrated-a/manifest.json /tmp/predval-integrated-b/manifest.json
+```
+
+The output directories must not already exist. To check the contents as well as the manifest:
+
+```sh
+.venv-benchmark/bin/python - <<'PY'
+import hashlib, json
+from pathlib import Path
+baseline = Path('benchmarks/alternatives/results/python/manifest.json')
+expected = json.loads(baseline.read_text())['files']
+for directory in ('/tmp/predval-integrated-a', '/tmp/predval-integrated-b'):
+    root = Path(directory)
+    assert (root / 'manifest.json').read_bytes() == baseline.read_bytes()
+    for relative, digest in expected.items():
+        assert hashlib.sha256((root / relative).read_bytes()).hexdigest() == digest
+print(f'Passed: {2 * len(expected)} file-hash checks and both baseline-manifest comparisons')
+PY
+```
+
+## Remaining limits
+
+- Twenty tests did not run because optional PCam, GUSTO, and IMPROVE example/golden fixtures
+  were unavailable. No full golden-output revalidation is claimed.
+- The guard retains the same estimator, tolerance, and iteration limit. A valid finite-MLE
+  fit can be unavailable if it needs more iterations. This change does not prove every
+  unavailable-reason label is correct or detect every possible separation condition.
+  Because the existing half-pair guard withholds a rung if its apparent fit or any cross-fit
+  fold is unavailable, one iteration-exhausted fold can now withhold the entire rung for a
+  cohort that previously received a finite last-iterate result.
+- The two comparator gain intervals still cross zero. They condition on fixed held-out
+  predictions and do not measure repeated-sampling coverage or correction-training uncertainty.
+  Shared fitting and overlapping training folds also mean that holding predictions fixed is
+  insufficient to establish conditional coverage. No replacement interval is introduced here.
+- The original isitfair adapter remains unexecuted; source review is not an execution result.
+- These are bounded engineering/method checks, not clinical validation or a superiority claim.
+- At the time of the original integration checks, all work was local; no push, pull request,
+  merge, or deployment had been performed. The reviewed tree was subsequently published in
+  [draft PR #6](https://github.com/aRealGem/predval/pull/6), at published head `4829570`.