Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6,593 changes: 6,593 additions & 0 deletions docs/reports/improve.html

Large diffs are not rendered by default.

45 changes: 45 additions & 0 deletions tests/golden/improve/GOLDEN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# IMPROVE golden baseline

Frozen regression baseline for the IMPROVE neoepitope-immunogenicity example. Generated by:

```bash
uv run python examples/improve/build_fixture.py --src 'improve_results/results copy'
uv run python examples/improve/run_evaluation.py
```

(default config: `B=2000`, `seed=1337`, from `examples/improve/cohort.yaml`; 7 models, 17,520
peptides, G=70 patient clusters, 467 immunogenic = 2.666% prevalence)

| | |
|---|---|
| generated | 2026-10-07 |
| predval commit | `4f9fbd0` (predval 0.1.0) |
| source repo | `https://github.com/SRHgroup/IMPROVE_paper` at `c67094276b52358b5202ad9e5597acd87888ea1d` |
| `results.zip` sha256 | `26897006104ef0e57aeedc003804c9a7e43d3735d9df157d651516f2739e00f9` |
| wall time | ~12.5 min on a 4-core container |

`findings.json` keeps its `provenance.git_commit` as generated; `tests/test_golden_improve.py`
normalizes that one field out of the comparison, as the GUSTO and PCam goldens do.

Reproducibility was verified at freeze time: a second independent run at the same commit was
compared against these files by `tests/test_golden_improve.py`.

## Headline numbers this baseline pins

- **Prevalence anchoring.** The six IMPROVE random-forest configurations predict a mean
probability of 0.39-0.40 against an observed rate of 0.0267, i.e. 14.6-15.1x too high. The
NNAlign comparator is anchored (1.1x).
- **Discrimination.** AUROC 0.63-0.65 for the IMPROVE models, 0.61 for NNAlign
(patient-clustered 95% CIs roughly +/-0.04).
- **Calibration.** Intercepts about -3.25 (CI excludes 0) with slopes about 1.1 (CI includes 1):
the IMPROVE scores are a ranking with the right spread and the wrong level. NNAlign:
intercept -0.09 [-0.33, 0.15], slope 0.97.
- **Recalibration ladder.** Cross-fitted rung1 (intercept only, fitted on held-out patients)
cuts Brier from ~0.17 to 0.0258 for every IMPROVE model, a demonstrated repair. That repaired
Brier equals the prevalence-only constant predictor (0.0259): once the level is fixed, the
probabilities carry almost no Brier-scale skill beyond the base rate. NNAlign gains nothing
from recalibration.
- **Unit of analysis.** Patient-clustered AUROC intervals are 1.3-1.7x wider than naive
per-peptide ones.
- **Fragility.** Dropping patient RH-31 moves AUROC the most for six of seven models
(largest 0.013).
Binary file added tests/golden/improve/calibration.parquet
Binary file not shown.
Binary file added tests/golden/improve/coverage.parquet
Binary file not shown.
Loading