Repository navigation
Conversation
…verage-delta column it broke Before: predval's neoepitope evidence was one model. examples/improve/ showed that IMPROVE's own random forest ships probabilities anchored ~15x above the 2.67% base rate, with the whole fault in the intercept. Read strictly, that is a statement about one training pipeline. The report's per-model coverage-delta column, meanwhile, had never executed on any fixture: GUSTO, PCam and examples/improve all had uniform coverage across their rosters, and the column renders only when coverage differs. After: examples/improve_panel/ holds the patients, peptides, labels, clustering and both prespecified subgroups fixed and swaps in four immunogenicity predictors published by four other groups (DeepImmuno, DeepHLApan, DeepNetBim, iTTCA-RF), with IMPROVE's best variant kept as the in-house reference. One of them scores only 9-mers, so this is also the first fixture in the repo with non-uniform coverage -- which crashed the report renderer, and that is fixed here with regression tests covering the branch. The fault was in coverage_delta's pivot: dropna=False fills the full cross product of its index levels, inventing cells no stratum ever produced (stratum_kind="overall" crossed with a subgroup level), so a per-model lookup returned a Series rather than a scalar and np.isfinite raised. Those invented cells are now dropped at the source, since they carry neither subset, and the report pins all five keys when it reads the primary table's own cell. A missing delta now renders "n/a" rather than "0.0000", because the two are different statements. How: the fixture joins the release's benchmark_comparison table to its own patient-partitioned cross-validation output. Mut_peptide + HLA_allele is not a unique key -- 167 peptide-HLA pairs recur across patients and joining on them silently expands 17,520 rows to 17,884 with spurious label disagreements. Adding Expression, which is measured per patient, makes it exact, and build_fixture.py asserts uniqueness, the row count, label agreement and the [0, 1] bound rather than trusting any of them. Predictors whose published output is a binding or presentation score rather than an immunogenicity probability are excluded by name in cohort.yaml, as part of the prespecification; holding a presentation probability to the immunogenicity base rate would manufacture a miscalibration the model never asserted. skill_ladder.py is a sidecar, outside predval's contract like budget_yield.py: the report quotes Brier skill at the admissible rung, which is parsimony-first by design, so this scores every rung the ladder can fit and lets a reader check whether a skill result is a property of the scores or of the selection rule. It reports three outcomes rather than two, because "no rung reaches reliably positive skill" is a weaker claim than "every rung is reliably negative". No library behaviour changes for a uniform-coverage roster: the frozen GUSTO and PCam golden files still match byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138yQfFhvnVRHd1XhTEz54w
… B=2000 run found it Three of five models are reliably negative at the admissible rung and a fourth straddles zero; the docstring had said four of five were negative. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138yQfFhvnVRHd1XhTEz54w
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before: predval's neoepitope evidence covered one model family.
examples/improve/showed that IMPROVE's own random forest gives probabilities about 15x above the 2.67% base rate, but strictly that only describes one training pipeline. Separately, the report's per-model coverage-delta column had never actually run. Every fixture so far (GUSTO, PCam,examples/improve/) had uniform coverage, and the column only renders when coverage differs across models.After:
examples/improve_panel/keeps the same 70 patients, 17,520 peptides, labels, clustering and prespecified subgroups, and swaps in four immunogenicity predictors published by four other groups: DeepImmuno, DeepHLApan, DeepNetBim and iTTCA-RF. IMPROVE's best variant stays in as an in-house reference. DeepNetBim scores only 9-mers, which makes this the repo's first fixture with uneven coverage. That crashed the report renderer, and this PR fixes it.What changes in the library.
coverage_deltausespivot_table(..., dropna=False), which fills in every combination of its index levels, including cells no stratum ever produced (for example, the "overall" stratum paired with a subgroup level). So the report's per-model lookup got a Series back instead of one number, andnp.isfiniteraised an error. This PR:Two regression tests cover uneven coverage. Rosters with uniform coverage are unaffected: the frozen GUSTO and PCam golden files still match byte for byte.
What the panel shows (B=2000 resamples clustered by patient, seed 1337):
How. The panel table joins to the release's patient-partitioned CV output on peptide + HLA + Expression. Peptide + HLA alone is not unique: 167 pairs repeat across patients, and joining on them silently grows 17,520 rows to 17,884.
build_fixture.pyasserts that the key is unique, that the row count holds, that labels agree, and that every score is in [0, 1]. Rebuilding from the release reproduces both parquets byte for byte. Binding and presentation predictors are excluded by name incohort.yamlas part of the prespecification, because holding them to the immunogenicity base rate would invent a miscalibration they never claimed.skill_ladder.pyis a sidecar outside predval's contract. The report quotes skill only at the admissible rung, which is a selection rule. The sidecar scores every rung, so a reader can check that the negative-skill result does not depend on that rule.Not independent replication. The patients, labels and assay are the same as in
examples/improve/. This tests breadth across predictors, not across cohorts.🤖 Generated with Claude Code
https://claude.ai/code/session_0138yQfFhvnVRHd1XhTEz54w
Generated by Claude Code