diff --git a/.gitignore b/.gitignore index fb80953..c3b7307 100644 --- a/.gitignore +++ b/.gitignore @@ -34,3 +34,8 @@ validation/predrupdate/gusto_estimates.csv examples/improve/cohort.parquet examples/improve/predictions.parquet examples/improve/out/ + +# Isolated benchmark environments and scratch reruns +.venv-clean/ +.venv-benchmark/ +benchmarks/alternatives/results/run*/ diff --git a/README.md b/README.md index 43b3355..cdcad27 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,120 @@ prespecified subgroup behaviour, and a recalibration ladder that reports how muc miscalibration is repairable without touching the model, and discloses the ladder's own optimism rather than reporting only its best face. +## Quickstart with your own predictions + +No PCam files or model weights are needed. From a checkout, run `uv sync --locked`, then save +these three files in a new `my-cohort/` directory. This tiny, fully synthetic CSV example is +public demonstration data, not patient data or a statistical validation benchmark. + +`cohort.csv` (one row per observation): + +```csv +subject_id,patient_id,label +v01,p01,0 +v02,p01,1 +v03,p02,1 +v04,p02,0 +v05,p03,0 +v06,p03,1 +v07,p04,1 +v08,p04,0 +v09,p05,0 +v10,p05,1 +v11,p06,1 +v12,p06,0 +``` + +`predictions.csv` (one row per scored observation and model): + +```csv +subject_id,model_id,predicted +v01,my_model,0.20 +v02,my_model,0.65 +v03,my_model,0.35 +v04,my_model,0.55 +v05,my_model,0.30 +v06,my_model,0.75 +v07,my_model,0.60 +v08,my_model,0.40 +v09,my_model,0.70 +v10,my_model,0.45 +v11,my_model,0.80 +v12,my_model,0.25 +``` + +`cohort.yaml` (paths are relative to this file): + +```yaml +cohort_id: synthetic-own-data-demo +version: 0 +subject_key: subject_id +data: cohort.csv +unit_noun: visit +outcome: + type: binary + field: label + positive_label: 1 +clustering: + field: patient_id + name: patient +coverage: + min_fraction: 1.0 + compare_on: both +completeness: + require_outcome: true + on_violation: fail +expected_models: [my_model] +on_missing_model: fail +thresholds: [0.5] +uncertainty: + n_boot: 20 # Fast plumbing check only; the library default is 2000. + seed: 1337 +``` + +From the checkout, run the public API (CSV and Parquet are both supported): + +```bash +uv run python - <<'PY' +from pathlib import Path +from predval import evaluate, load_cohort, load_predictions, write_findings, write_report + +root = Path("my-cohort") +result = evaluate(load_cohort(root / "cohort.yaml"), load_predictions(root / "predictions.csv")) +out = root / "results" +result.write(out) +write_report(result, out / "report.html") +write_findings(result, out / "findings.json") +print(out / "report.html") +PY +``` + +Open `my-cohort/results/report.html`; `findings.json` is the machine-readable companion, and +`result.write` saves the metric tables and provenance manifest. With only six independent +patients, warnings or unavailable recalibration rungs are expected, not a reason to hide rows. + +To replace the toy data: +- Keep `subject_id` unique for each observation and identical across the two files. With one row + per patient it can identify that patient; repeated visits need unique visit IDs plus a separate + patient clustering column. Use pseudonymous IDs, not names or medical record numbers. +- Declare the actual independence unit in `clustering.field` (patient, site, slide, etc.). + Omitting it treats observations as independent; a unique row ID is not a substitute for a + shared cluster ID. Whether clusters are independent is a study-design judgment. +- List all intended model IDs in `expected_models`; this example fails if an entire model is + missing. `predicted` must be the probability of `outcome == positive_label`, supplied as a + finite number in `[0, 1]` (not the probability of the other class). An unscored observation has no prediction + row, never a zero placeholder. Set the coverage floor before analysis; `1.0` here requires + every usable observation to be scored by each model. `compare_on: both` also evaluates the + common scored subset. Review missing outcomes and coverage before interpreting comparisons. +- Prespecify the outcome, thresholds, subgroups, model roster and analysis plan. Replace the + demonstration bootstrap budget with a justified analysis budget; more resamples cannot fix + too few independent clusters or data-adaptive selection. + +A completed run checks these inputs and reports performance on this cohort. It does not prove +clinical utility, transportability, calibrated probabilities, or that a selected recalibration +will improve a new cohort. Gain intervals resample fixed cross-fitted predictions; neither +conditional coverage nor full fitting-and-selection coverage is established by that choice. + ## The pitch, narrowed Most tools in this space either train something (a new calibration layer, a monitoring model) @@ -22,10 +136,14 @@ Three things follow from that scope, and they are what predval spends its effort flatters the model. - **The unit of analysis and the unit of independence are usually not the same file column.** When predictions come from grouped data — patients with repeat visits, patches from the same - slide, images from the same device — per-row confidence intervals are not slightly optimistic, - they are wrong by roughly the square root of the cluster size. predval takes a declared - clustering unit and resamples whole clusters everywhere it computes uncertainty, and renders - the naive-vs-cluster comparison once so the gap is a number, not a caveat in a footnote. + slide, images from the same device — ignoring positive within-cluster dependence can seriously + understate uncertainty. For an illustrative mean with independent clusters of equal size `m` + and exchangeable within-cluster correlation `ICC`, the standard-error inflation is + `sqrt(1 + (m - 1) * ICC)`; + `sqrt(m)` is the special case `ICC = 1`, not a universal factor for every metric. predval takes + a declared clustering unit and resamples whole clusters everywhere it computes uncertainty, + and renders the naive-vs-cluster comparison once so the gap is a number, not a caveat in a + footnote. - **A recalibration that improves the numbers is a finding about the ladder's own fit, not a promotion of the model, until it survives being scored on clusters it never saw.** Every rung of predval's ladder — intercept, intercept+slope, spline — is fit twice: once on the rows it will @@ -34,15 +152,20 @@ Three things follow from that scope, and they are what predval spends its effort validated model, and — this is the part most recalibration tooling skips — refuses to manufacture a gain where none exists: on the PCam fixture below, nine of the fifteen members carry a detected miscalibration, ten have the repair's benefit left honestly unproven at this - cohort's small cluster count, and exactly one — `p4m_seed7` at rung2 — gets a real, - interval-backed repair. Five members carry at least one rung whose recalibration *reliably - harms* them, and `p4m_seed7` is one of the five: its rung2 repair is demonstrated while its - rung1 is harmful, which is why harm is reported independently of whether a repair was found. - Two of those five, `e2cnn_s21` and `e2cnn_s99`, are harmed at every rung and have no admissible - repair at all. The remaining six show no detected fault, which the report states as a null + cohort's small cluster count, and exactly one — `p4m_seed7` at rung2 — has a positive + estimated gain whose interval excludes zero under the configured fixed-prediction resampling. + Five members have at least one rung with a gain interval entirely below zero, including + `p4m_seed7` at rung1: estimated harm is reported independently of estimated benefit at rung2. + Two of those five, `e2cnn_s21` and `e2cnn_s99`, have negative gain intervals at every rung + and no admissible repair under the configured criterion. The remaining six show no detected fault, which the report states as a null result and never as a clean bill of health: this harness's calibration test can fail to reject "calibrated", but it can never establish it. +The paired gain intervals hold fitted cross-fold predictions fixed. They do not establish +nominal coverage for the full fitting-and-selection procedure, or conditional coverage merely +because fitting is held fixed during resampling. Limited independent clusters and data-adaptive +selection of models or rungs warrant caution when interpreting an interval that excludes zero. See [the interval limitations](docs/spec.md#45-rung-intervals-the-paired-cross-fit-gain-carries-one-the-per-rung-refit-is-deferred). + ## What predval deliberately does not do - **It never loads model weights.** No `torch`, no `tensorflow`, no checkpoints. Inputs are a @@ -253,9 +376,11 @@ matching PCam's — the full B=2000 run is frozen and regression-checked by headline numbers fails loudly. **Honest read of the verdict:** AUROC 0.791, comfortably inside the literature's expected -0.72–0.82 range on the first attempt. Calibration slope 0.835 shows real, if modest, displacement -scoring off the development region — the recalibration ladder finds a small but statistically -real gain at rungs 2–3 (both exclude zero), while rung 1 alone is actually slightly negative. This +0.72–0.82 range on the first attempt. Calibration slope 0.835 indicates modest displacement +scoring off the development region. Rungs 2–3 have small positive estimated gains whose +intervals exclude zero under the configured fixed-prediction resampling; rung 1 has a slightly +negative point estimate. Limited independent clusters and data-adaptive selection limit what +these intervals establish; they do not validate the full fitting-and-selection procedure. This is a much milder miscalibration story than PCam's two badly-miscalibrated members, which is an honest property of this cohort rather than something to chase. The real finding of this example isn't the numbers — it's that predval's ingestion, coverage, roster, clustering, and ladder code diff --git a/benchmarks/alternatives/.gitignore b/benchmarks/alternatives/.gitignore new file mode 100644 index 0000000..33a3cf8 --- /dev/null +++ b/benchmarks/alternatives/.gitignore @@ -0,0 +1,10 @@ +# Generated row-level fixtures and predictions regenerate from the locked script. +# Commit compact numerical summaries and hashes, not bulk row-level inputs. +results/python/coverage_inputs/ +results/python/wdbc_heldout.csv +results/python/wdbc_heldout_corrected.csv +results/python/synthetic_clustered.csv +results/python/synthetic_clustered_corrected.csv +results/r/*_r_corrected.csv +results/r/*_rsample_gains.csv +results/r-repeat*/ diff --git a/benchmarks/alternatives/FIXTURE_LICENSE.md b/benchmarks/alternatives/FIXTURE_LICENSE.md new file mode 100644 index 0000000..705422d --- /dev/null +++ b/benchmarks/alternatives/FIXTURE_LICENSE.md @@ -0,0 +1,22 @@ +# Fixture attribution + +Wisconsin Diagnostic Breast Cancer is provided by the UCI Machine Learning Repository +under Creative Commons Attribution 4.0 International: +https://creativecommons.org/licenses/by/4.0/ + +Wolberg, W., Mangasarian, O., Street, N., & Street, W. (1993). +Breast Cancer Wisconsin (Diagnostic). UCI Machine Learning Repository. +https://doi.org/10.24432/C5DW2B + +Source/license: https://archive.ics.uci.edu/dataset/17/breast+cancer+wisconsin+diagnostic + +The benchmark uses scikit-learn's packaged copy. It derives a fixed held-out subset, +probabilities from a model fitted on separate training rows, deliberately distorted +probabilities, and row-derived identifiers. These are alterations; no creator endorsement +is implied. Generated row-level data are not committed or included in the compact bundle. + +The repeated-patient fixture is wholly synthetic and generated by the repository's own +benchmark code. It is not clinical validation and contains no real patient records. + +The benchmark's original code is covered by the repository's Apache-2.0 LICENSE. +No source-only competitor package is redistributed in the compact review bundle. diff --git a/benchmarks/alternatives/INTEGRATION.md b/benchmarks/alternatives/INTEGRATION.md new file mode 100644 index 0000000..425a161 --- /dev/null +++ b/benchmarks/alternatives/INTEGRATION.md @@ -0,0 +1,112 @@ +# Reviewed-change integration: 2026-10-10 + +## Result + +The reviewed patient-partition guard, alternatives comparator, and IRLS iteration-limit +fix are combined on local branch `integrate/reviewed-predval-20261010`. +The combined suite passes **282 tests**, with **20 optional fixture-dependent skips**, +**zero failures**, and **zero expected failures**. The comparator contracts also pass +separately: **6/6**, including the formerly expected-failure quasi-separation case. +Ruff lint, Ruff formatting (63 files), and `git diff --check` pass. + +Both Python comparator reruns reproduce **all 13 stable input/result hashes exactly** +from the original benchmark. Thus the two main fixtures' point estimates, corrections, +intervals, coverage flags, and suppression summaries have not changed. This does not imply +that every possible input is unchanged: exhausted IRLS fits now become unavailable rather +than being reported as converged finite fits. + +## Source and evidence provenance + +- Patient-partition guard: `99ca3d72f1d4f8340df9a59de3616277e9ebdb7a`. +- Original comparator: `868276ef960f0bc7f2bc75cc79926ff7b3c2e1f1`; its original + preregistration commits `dae6e1a` and `756e5a1` remain in the branch history. +- Reviewed convergence fix: `62c8af368824d75e1dd915e78c131f46e333a340`, cherry-picked + without conflicts as `617ddac480849ac48de41c1c1b1d050ddc427492`. +- Integration-specific source edit: remove the strict xfail marker from + `tests/test_benchmark.py` and update its explanatory comment. The core implementation, + benchmark runner, R adapter, and frozen benchmark protocol are unchanged beyond the + reviewed guard. README wording now distinguishes historical and integrated checks. +- Package metadata still says PREDVAL 0.1.0. The source commits and file hashes in + [`verification/integrated/status.json`](verification/integrated/status.json), rather + than the package version alone, identify the code tested here. + +Original compact results and verification logs remain byte-for-byte unchanged. Their old +269-pass/5-pass/1-xfail counts describe the original comparator, not this integrated source. +New integration logs and the hash-comparison record live separately under +`verification/integrated/`. The independent statistical-stress study is **not integrated**; +its frozen baseline and worktree were not changed by this task. + +## What was rerun + +The full combined Python suite ran against this worktree. The comparator ran twice with +`PYTHONPATH` explicitly selecting this worktree's `src`; source imports were checked in +both interpreters. These used two existing isolated Python 3.12.14 environments originally +created from the benchmark's complete requirements lock, not two new installations during +integration. Each run verified 13 stable file hashes, for 26 new file-hash checks. Both +manifests equal the historical manifest, SHA-256: + +`a2a541ec30fbffe83103e04d2c85a5e83114a671e9818488e896a2f34eaf510d` + +The R adapter consumes the exported fixtures and corrected predictions, all byte-identical +here, and its source is unchanged. All seven retained R stable files were checked against +the original recorded hashes. **R computation was not repeated for integration.** The +original repeated R computation is reused evidence, not a newly executed R result. + +The benchmark-body Python wall times were 3.86 and 5.47 seconds. As in the original report, +these exclude imports/installation, reflect concurrent machine load, and do not compare +individual tools' speed. Exact resource records are in the integration JSON. + +## Reproduce the integration checks + +Use the locked Python setup in the main benchmark README. From the repository root: + +```sh +export PYTHONPATH="$PWD/src" +export MPLCONFIGDIR=/tmp/predval-mpl XDG_CACHE_HOME=/tmp/predval-cache +export OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 +.venv-benchmark/bin/python -m pytest tests benchmarks/alternatives/tests -ra +.venv-benchmark/bin/ruff check . +.venv-benchmark/bin/ruff format --check . +git diff --check +.venv-benchmark/bin/python benchmarks/alternatives/run_benchmark.py --out /tmp/predval-integrated-a +.venv-benchmark/bin/python benchmarks/alternatives/run_benchmark.py --out /tmp/predval-integrated-b +cmp benchmarks/alternatives/results/python/manifest.json /tmp/predval-integrated-a/manifest.json +cmp /tmp/predval-integrated-a/manifest.json /tmp/predval-integrated-b/manifest.json +``` + +The output directories must not already exist. To check the contents as well as the manifest: + +```sh +.venv-benchmark/bin/python - <<'PY' +import hashlib, json +from pathlib import Path +baseline = Path('benchmarks/alternatives/results/python/manifest.json') +expected = json.loads(baseline.read_text())['files'] +for directory in ('/tmp/predval-integrated-a', '/tmp/predval-integrated-b'): + root = Path(directory) + assert (root / 'manifest.json').read_bytes() == baseline.read_bytes() + for relative, digest in expected.items(): + assert hashlib.sha256((root / relative).read_bytes()).hexdigest() == digest +print(f'Passed: {2 * len(expected)} file-hash checks and both baseline-manifest comparisons') +PY +``` + +## Remaining limits + +- Twenty tests did not run because optional PCam, GUSTO, and IMPROVE example/golden fixtures + were unavailable. No full golden-output revalidation is claimed. +- The guard retains the same estimator, tolerance, and iteration limit. A valid finite-MLE + fit can be unavailable if it needs more iterations. This change does not prove every + unavailable-reason label is correct or detect every possible separation condition. + Because the existing half-pair guard withholds a rung if its apparent fit or any cross-fit + fold is unavailable, one iteration-exhausted fold can now withhold the entire rung for a + cohort that previously received a finite last-iterate result. +- The two comparator gain intervals still cross zero. They condition on fixed held-out + predictions and do not measure repeated-sampling coverage or correction-training uncertainty. + Shared fitting and overlapping training folds also mean that holding predictions fixed is + insufficient to establish conditional coverage. No replacement interval is introduced here. +- The original isitfair adapter remains unexecuted; source review is not an execution result. +- These are bounded engineering/method checks, not clinical validation or a superiority claim. +- At the time of the original integration checks, all work was local; no push, pull request, + merge, or deployment had been performed. The reviewed tree was subsequently published in + [draft PR #6](https://github.com/aRealGem/predval/pull/6), at published head `4829570`. diff --git a/benchmarks/alternatives/PROTOCOL.md b/benchmarks/alternatives/PROTOCOL.md new file mode 100644 index 0000000..054b05a --- /dev/null +++ b/benchmarks/alternatives/PROTOCOL.md @@ -0,0 +1,78 @@ +# Alternatives benchmark protocol (registered before benchmark execution) + +Date: 2026-10-10. PREDVAL baseline: `99ca3d72f1d4f8340df9a59de3616277e9ebdb7a`. +This is an engineering/method-comparison benchmark, not clinical validation or a claim +that a fitted model is suitable for patient care. No runtime, accuracy, or novelty winner +will be inferred from unavailable software. + +## Fixed scope and inputs + +1. Independent binary rows: Wisconsin Diagnostic Breast Cancer, bundled with scikit-learn, + UCI dataset 17 (CC BY 4.0). Use its original measurements to create strictly held-out + predictions from an explicitly specified logistic regression pipeline. This is a real, + redistributable methods fixture, not an external clinical validation study. +2. Synthetic clustered patients: seed 20261010; 120 patients, unequal 4–20 repeated rows, + patient random effect and patient-correlated binary labels. Published predictions are + deliberately distorted. This controlled fixture exercises clustering, correction, and + missing-coverage failure paths, and is explicitly synthetic. +3. Missing-model-coverage variants: remove predictions from patients in the upper 30% of + mean predicted risk, and separately remove an entire declared model. The cohort and + model roster stay fixed. Compare full available rows and the common intersection. + +## Matched estimands and configuration + +- Positive outcome is coded 1 throughout. Brier is mean squared error over rows, not a + patient-weighted mean. AUROC uses tied-score midranks. Point-metric tolerance: 1e-10. +- Calibration slope is the slope from unpenalized logistic `y ~ 1 + logit(p)`. + Calibration-in-the-large is a separate intercept with logit(p) as an offset (slope 1). + Do not equate an intercept from a free-slope fit with calibration-in-the-large. +- Matched correction: unpenalized logistic intercept+slope on clipped logit probability + (epsilon 1e-6), five disjoint patient folds. Use PREDVAL's fixed sorted-patient round-robin + assignment as an exported common split, and give competitors those exact folds when + their public API supports it. Compare pooled held-out Brier and AUROC, not unweighted + fold-mean metrics. Label non-equivalent fits separately rather than declaring disagreement. +- Intervals: 1,000 paired whole-patient percentile bootstrap replicates; seed 20261011; + 95% nominal intervals. Conditional on the fixed held-out predictions; no claim of full + correction-training uncertainty. Use the exact same resample indices for custom adapters. + Compare row-bootstrap intervals separately as different assumptions, not matched inference. +- Independent-row fixture: stratified 50% held-out split, seed 20261010; StandardScaler plus + LogisticRegression(C=1, max_iter=2000, solver='lbfgs'); no hyperparameter tuning. The positive + class is malignant. A fixed `sigmoid(-0.4 + 1.4*logit(p))` distortion supplies a correction case. + +## Competitors and fair interpretation + +- PyCalEva: installed registry release, public CalibrationEvaluator for available metrics. + It is primarily a calibration evaluator, so missing end-to-end cohort machinery is not + a numerical defect. Document required adapter work rather than handicap it. +- probably + rsample: public `cal_validate_logistic` accepts an rset; grouped rsets and group + bootstraps are supported. Export matched folds and provide a runnable R adapter. If R is + absent or cannot be installed safely, mark execution unavailable, never failed metrics. +- isitfair: use official registry release if available, otherwise inspect official source + without running unrecognized code. Respect explicit folds on its cross-fitted ladder and + its one-row-per-patient convention. Do not call a row bootstrap a grouped-patient bootstrap. +- PREDVAL is not privileged as the numerical oracle: independently calculate metrics with + scikit-learn/statsmodels and a small shared resampling reference. + +## Deliverables and stopping rule + +Commit protocol before results; record exact software versions, input hashes, outputs, +commands, convergence/failure information, supported/configurable/unavailable capabilities, +and adapter source line counts (not invented developer-time savings). Run each executable +path twice and compare stable output hashes. Deliver CSV/JSON summaries, reproducible scripts, +and a bounded findings report. Stop after both fixtures, coverage variants, and clean repeat +have completed, or document specific dependency/access blockers. No paid compute, private +clinical data, data-use agreements, outreach, push, PR, merge, or deployment. + +## Source-review amendment (before R or isitfair execution) + +Official source inspection found that probably's `smooth=FALSE` fits `y ~ p`, while +PREDVAL rung2 fits `y ~ logit(p)`. These are different correction families. Report probably's +native result separately and cross-check it against a raw-probability statsmodels GLM; +retain a separate R `glm(y ~ logit(p))` adapter as the exactly matched control. Do not feed +logits into a probability API merely to force equality. Isitfair's native Platt fit uses +L2-regularized sklearn logistic regression and epsilon 1e-10, so it too is a separate native +configuration, not an exact unpenalized rung2 equivalent. No executed isitfair result will +be claimed without an authorized executable package. + +An isolated conda-forge R setup may be attempted for at most 15 minutes and less than 1 GB +of planned package downloads. No system-level R installation or security changes. diff --git a/benchmarks/alternatives/README.md b/benchmarks/alternatives/README.md new file mode 100644 index 0000000..44b3e39 --- /dev/null +++ b/benchmarks/alternatives/README.md @@ -0,0 +1,255 @@ +# Bounded comparison: PREDVAL, PyCalEva, probably + rsample, and isitfair + +This is a reproducible **methods/engineering comparison**, not clinical validation. +PREDVAL's added value here is explicit cohort/cluster contracts and connected diagnostics. +Its point metrics are not novel, and correct alternatives can reproduce its results. + +The protocol was committed before execution (`dae6e1a`), with a source-review amendment +(`756e5a1`) distinguishing correction families before any R or isitfair execution. +Both commits descend from patient-partition guard `99ca3d7`; the original benchmark +changed no core PREDVAL code. This integrated branch also includes the subsequently +reviewed IRLS iteration-limit guard (`62c8af3`, cherry-picked as `617ddac`). Original +results and verification records below remain unchanged and describe the historical +baseline. Separately labeled current-branch checks are in [INTEGRATION.md](INTEGRATION.md). + +## What is and is not executed + +- **PREDVAL 0.1.0, PyCalEva 0.8.2:** executable numerical checks on both fixtures. +- **Independent sklearn/statsmodels control:** point metrics, unpenalized grouped correction, + and a separate custom patient bootstrap. PREDVAL is not its own numerical oracle. +- **probably 1.2.0 + rsample 1.3.2 (R 4.5.3):** native adapter executed on both fixtures, + including grouped cross-validation and 1,000 grouped bootstrap replicates. `cal_validate_logistic` accepts an rset, including + grouped rsample splits. It is not fair to describe this stack as lacking grouped validation. +- **isitfair:** official source reviewed at `70ffb8d21c3e24eddfee412220b8a7a701768937`. + No registry package was found; no source code was executed. An unexecuted public-API adapter + is included. Its existing integrated reports and cross-fitted recalibration ladder make it + a close competitor; this benchmark does not establish superiority over it. + +## Fixtures and estimands + +- `wdbc_heldout`: 285 held-out observations (106 malignant), from the 569-row Wisconsin + Diagnostic Breast Cancer dataset bundled with scikit-learn. Train/test separation is + fixed, stratified, and 50:50. Scaling and model fitting use only the training half. + Logistic predictions are deliberately distorted to exercise calibration. This is a + real-data test fixture, not an externally validated clinical model or deployment study. +- `synthetic_clustered`: 1,332 repeated rows in 120 synthetic patients, 560 events, unequal + cluster sizes and correlated outcomes. Its generator and all seeds are in the protocol. + It contains no real patient records and supplies no evidence of clinical effectiveness. +- Row-weighted Brier and AUROC are matched. Calibration intercept fixes slope at one; + calibration slope is from a separate joint intercept+slope logistic fit. +- Correction uses the same exported, disjoint, five-fold patient assignments. Pooled + held-out metrics are used, not averages of fold AUROCs. PREDVAL rung2 and the independent + GLM use unpenalized logit-scale regression, epsilon 1e-6. +- Gain intervals use 1,000 paired patient bootstrap replicates, conditional on fixed + held-out predictions. They do not include correction-training uncertainty. One fixture's + interval is not an empirical confidence-interval coverage study. + +## Python findings + +PREDVAL and PyCalEva agree exactly on the four checked AUROC/Brier point estimates. +Against independent statsmodels, PREDVAL's calibration intercept/slope differ by less than +2e-15. Grouped rung2 probabilities agree within 4e-14 on these fixtures. + +| Fixture | Published Brier | Grouped corrected Brier | Paired gain | 95% patient-bootstrap gain CI | +|---|---:|---:|---:|---:| +| WDBC held out | 0.031263 | 0.029256 | 0.002007 | [-0.005336, 0.009427] | +| Synthetic clustered | 0.181410 | 0.170541 | 0.010869 | [-0.001163, 0.022167] | + +Both gains are uncertain in these fixtures. The synthetic row-bootstrap interval is +[0.005338, 0.016573], which excludes zero; respecting patients gives an interval 2.08 times +as wide that crosses zero. For WDBC, every cluster is a singleton; small row/patient interval differences reflect +ordering and Monte Carlo draws, not clustering. For the synthetic fixture this is a concrete +consequence of changing the independence +assumption, not evidence that PyCalEva is numerically wrong. A PyCalEva-plus-custom-patient- +bootstrap adapter reproduces PREDVAL's paired interval to numerical precision. + +PREDVAL also withholds the WDBC spline rung after a non-convergence signal. This is reported, +not hidden to make the tool look better; this benchmark does not independently diagnose that +spline fit or establish that every PREDVAL unavailable-reason label is correct. + +### Coverage faults + +The synthetic partial model has **identical scores** to the full model where present; only +availability differs. Removing the top 30% of patients by mean predicted risk leaves +919/1,332 rows (68.99%). Thus a separate-model comparison on available rows changes the +cohort while appearing to compare models: + +| Available cohort | n | AUROC | Brier | +|---|---:|---:|---:| +| Full model's available rows | 1,332 | 0.821898 | 0.181410 | +| Shared/partial rows | 919 | 0.783196 | 0.177929 | + +PREDVAL rejects this at a declared 90% coverage floor, warns that the intersection excludes +31.0% of rows, and detects a completely absent declared model. The same checks are easy to +implement around array-based tools, but the caller must supply the full cohort and roster; +aligned y/p arrays alone cannot reveal missing rows or a missing model. These checks exercise +public I/O/contract functions, not a complete rendered-report/UI workflow. + +## Fair capability comparison + +| Capability | PREDVAL | PyCalEva | probably + rsample | isitfair | +|---|---|---|---|---| +| Binary point AUROC/Brier | Native, executed | Native, executed | Native yardstick, executed | Native, source reviewed | +| Patient-grouped correction folds | Native | Caller supplies correction | Configurable grouped rset or explicit splits | Configurable explicit fold vector | +| Correct repeated-patient bootstrap | Native | Custom loop | Native group_bootstraps plus metric mapping | Custom outer cluster loop needed; built-in patient resampling assumes one row per patient | +| Coverage denominator and declared roster | Native contract, executed | Caller glue | Caller glue | Caller glue in reviewed API | +| Integrated report | Native | Calibration PDF | Requires composition | Native integrated HTML/PDF | +| Correction-family equality to rung2 | Reference definition | No native correction in checked evaluator | Native smooth=FALSE uses y~p, a different family | Platt uses L2 regularization and epsilon 1e-10, different configuration | + +“Caller glue” is not “impossible.” No developer-time savings were measured. Code counts +are implementation-specific and not evidence of programmer effort or package quality. +PyCalEva's `metrics()` documentation calls Brier scaled, but the checked 0.8.2 implementation +and `.brier` property return ordinary mean squared error; direct numerical agreement confirms +which quantity was compared. + +## Reproduce Python without R + +From the repository root, using Python 3.12.14 (recorded lock): + +```sh +uv venv .venv-benchmark --python 3.12 +uv pip install --python .venv-benchmark/bin/python -r benchmarks/alternatives/requirements-lock.txt +uv pip install --python .venv-benchmark/bin/python --no-deps -e . +MPLCONFIGDIR=/tmp/predval-mpl XDG_CACHE_HOME=/tmp/predval-cache \ + OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 \ + .venv-benchmark/bin/python benchmarks/alternatives/run_benchmark.py --out /tmp/predval-run-a +MPLCONFIGDIR=/tmp/predval-mpl XDG_CACHE_HOME=/tmp/predval-cache \ + OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 \ + .venv-benchmark/bin/pytest benchmarks/alternatives/tests +``` + +Use a fresh output directory; the script refuses to overwrite one. Repeat in a second fresh +venv from the same lock and compare `manifest.json`: all stable data and result hashes must +match. `resource_usage.json` is intentionally excluded from that manifest because timing/RSS +are machine- and load-dependent. Its time covers the benchmark body after imports and excludes +installation; it is **not a between-tool speed benchmark**. + +Optional R command, once the registry packages listed in the adapter are installed: + +```sh +Rscript benchmarks/alternatives/probably_adapter.R /tmp/predval-run-a /tmp/predval-r-run-a +``` + +This gives native probably's raw-probability logistic correction the exact same folds and +checks it against R's raw-probability GLM. A separate R logit GLM checks the truly matched +PREDVAL rung2. Native grouped bootstrap uses R/rsample RNG draws, not identical NumPy draws, +so its Monte Carlo interval need not match the Python interval bit for bit. + +## Sources, redistribution, and versions + +The WDBC-derived fixture is redistributed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). +Credit: Wolberg, W., Mangasarian, O., Street, N., & Street, W. (1993), +*Breast Cancer Wisconsin (Diagnostic)*, UCI Machine Learning Repository, +[doi:10.24432/C5DW2B](https://doi.org/10.24432/C5DW2B). +Changes: selected a held-out subset, fitted logistic probabilities using separate training +rows, deliberately distorted probabilities, and replaced identifiers with row-derived labels. +No endorsement by the data creators is implied. + +Primary sources checked 2026-10-10: + +- [UCI dataset and license](https://archive.ics.uci.edu/dataset/17/breast+cancer+wisconsin+diagnostic) +- [scikit-learn bundled data](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html) +- [PyCalEva official repository](https://github.com/MartinWeigl/pycaleva) +- [probably resampling validation](https://probably.tidymodels.org/reference/cal_validate_logistic.html) +- [probably logistic estimation](https://probably.tidymodels.org/reference/cal_estimate_logistic.html) +- [rsample grouped CV](https://rsample.tidymodels.org/reference/group_vfold_cv.html) +- [rsample grouped bootstrap](https://rsample.tidymodels.org/reference/group_bootstraps.html) +- [isitfair pinned official source](https://github.com/HugoGuillen/isitfair/tree/70ffb8d21c3e24eddfee412220b8a7a701768937) + +Python numerical dependencies: NumPy 2.4.6, pandas 3.0.5, SciPy 1.17.1, scikit-learn 1.9.0, +statsmodels 0.14.6. Every transitive Python dependency is pinned in requirements-lock.txt. +The checked probably GitHub source is commit `974f0154cc7d44ae41d8072161996b23a19d3d2a`; +installed R package versions, when executed, take precedence over that source-only snapshot. + +## Historical IRLS defect and integrated guard + +In baseline `99ca3d7`, PREDVAL's IRLS implementation could return its last iterate after +hitting the iteration limit without convergence. On `y=[0,0,1,1]`, +`p=[0.2,0.5,0.5,0.8]`, the quasi-separated likelihood has no finite slope MLE, but the +baseline returned a finite slope near 22.288 and did not mark rung2 unavailable. +The original comparison recorded this as a strict expected failure. + +The integrated branch includes the reviewed iteration-limit guard: exhausted IRLS fits +now return `None`, and this regression is an ordinary passing test. This changes the +failure guard, not the estimator, convergence tolerance, or iteration budget. The two +main fixtures' matched rung2 estimates still agree with independently converged GLMs; +all 13 stable Python output/input hashes are unchanged in the integration reruns. +That bounded check does not establish universal non-convergence detection or statistical +validity. See [INTEGRATION.md](INTEGRATION.md) for current-source provenance and caveats. + +## Reproducibility and measured resource use + +Two isolated Python environments were created from the complete pinned package list. +After removing the shared PREDVAL transform from the independent GLM control, all stable +input/result file hashes matched across the two fresh-environment runs. Their manifest +SHA-256 is `a2a541ec30fbffe83103e04d2c85a5e83114a671e9818488e896a2f34eaf510d`. +Measured benchmark-body wall times were 1.96 and 2.38 seconds, with peak process RSS about +285 MiB. These exclude imports/installations, include the combined benchmark, and cannot +support a between-package speed claim. Exact records: `results/reproducibility.json`. + +`results/glue_inventory.json` counts physical lines for the custom correction, cluster +bootstrap, coverage wrapper, and language-specific adapters. Counts include comments and +validation/export scaffolding; they are not a minimum implementation size or measured time +saved. This deliberately avoids an invented developer-effort advantage. + +Row-level generated fixture/prediction files are not committed or bundled. Regenerate them +with the script; their hashes remain in the manifest for verification. Compact metric tables, +flags, seeds, software locks, and verification records are retained. + +## Executed R comparison + +R 4.5.3 with probably 1.2.0, rsample 1.3.2, and yardstick 1.4.0 ran successfully in an +isolated conda-forge environment. Resolver-planned downloads were 439,535,932 bytes across +227 packages, within the prespecified setup budget. No system-wide installation was used. +`conda-linux-64.lock` records exact official package URLs and checksums; use micromamba's +`create --file` with a fresh prefix to reproduce. Full package versions are in +`results/r/sessionInfo.txt`. + +| Fixture | probably native y~p Brier | PREDVAL y~logit(p) Brier | probably AUROC | PREDVAL AUROC | +|---|---:|---:|---:|---:| +| WDBC held out | 0.027718 | 0.029256 | 0.968747 | 0.984031 | +| Synthetic clustered | 0.172383 | 0.170541 | 0.818836 | 0.818551 | + +These are native different correction families on identical held-out folds, not a contest +with one universally better method. The explicit raw-probability statsmodels control checks +probably's family; the separate R logit GLM checks PREDVAL's family. Prediction-difference +checks and native grouped-bootstrap summaries are recorded under `results/r/`. + +R `group_vfold_cv` passed the no-patient-overlap check. `group_bootstraps` supplied 1,000 +native cluster resamples for each fixture. R's resamples differ from NumPy's; their intervals +are independently generated Monte Carlo results on fixed corrected predictions. + +## Original baseline verification status (archived) + +These counts refer to the original comparator at `868276e`, before integration of the +IRLS guard. Current combined-suite results are recorded in [INTEGRATION.md](INTEGRATION.md). + +- Existing PREDVAL suite: 269 passed, 20 skipped because optional example/golden fixtures + were unavailable. This does not claim those skipped integration cases passed. +- New benchmark contracts: 5 passed, 1 strict expected failure for quasi-separation. +- Ruff and `git diff --check`: passed. +- Two fresh Python environments: identical stable input/result manifests. +- isitfair: source-reviewed only; its adapter has not been executed or validated. + +The independent raw-probability statsmodels checks agree with native probably predictions +within 1.04e-8 (WDBC) and 9.71e-11 (synthetic). R's matched logit-GLM checks agree with PREDVAL +within 6.73e-12 and 5.78e-15. Default IRLS stopping tolerances differ; the benchmark assertions +are 1e-6 and 1e-8 respectively, specified before the final R run. + +For same-final-input repeats, the compact verification/summary command is: + +```sh +python benchmarks/alternatives/verify_results.py PY_RUN_A PY_RUN_B R_RUN_A R_RUN_B +``` + +It verifies every Python input/result hash, checks R metrics, predictions, bootstrap draws +and session information byte for byte, and writes a compact native-R bootstrap summary. + +Final same-input R repeats matched all seven stable files byte for byte (metrics, independent +agreement, both corrected vectors, both 1,000-draw bootstrap sequences, session information). +Native rsample 95% gain intervals were WDBC [-0.005282, 0.009331] and synthetic +[-0.001058, 0.022890], both crossing zero, consistent with the Python conclusion of uncertainty. +The R benchmark bodies took 142.26 and 187.48 seconds after imports; these include native +rsample bootstrap-object construction and are not workload-matched to the Python body. +R timing/session text has trailing whitespace stripped for repository hygiene; numerical +outputs are untouched. The pre-normalization repeats were also byte-identical. diff --git a/benchmarks/alternatives/conda-linux-64.lock b/benchmarks/alternatives/conda-linux-64.lock new file mode 100644 index 0000000..9e86869 --- /dev/null +++ b/benchmarks/alternatives/conda-linux-64.lock @@ -0,0 +1,229 @@ +# platform: linux-64 +@EXPLICIT +https://conda.anaconda.org/conda-forge/linux-64/libgomp-16.2.0-he0feb66_7.conda#d904c7ddd95e026254b4b80808ee9e12 +https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.3.2-h25fd6f3_3.conda#0de0122d9570a8ab637c6b73db268389 +https://conda.anaconda.org/conda-forge/linux-64/pandoc-3.12.1-ha770c72_0.conda#c426f0a6202837347387cc6fdd4c381e +https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-20_gnu.conda#a9f577daf3de00bca7c3c76c0ecbd1de +https://conda.anaconda.org/conda-forge/linux-64/zstd-1.5.7-hb78ec9c_7.conda#aa459086047c0e5e27023ab19f8cb86a +https://conda.anaconda.org/conda-forge/linux-64/libgcc-16.2.0-ha9f2e26_7.conda#735629b400b70eeab21a2ba8f5b1c2ce +https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.46.1-default_hbd61a6d_103.conda#2362c70e7e79827c32db35547d9c179b +https://conda.anaconda.org/conda-forge/linux-64/xorg-libxau-1.0.12-h7cc23a3_2.conda#da33efee1a93603d92e06d4a2b68def6 +https://conda.anaconda.org/conda-forge/linux-64/xorg-libxdmcp-1.1.5-h7cc23a3_2.conda#eabfc45786244de252090cf8d55128a8 +https://conda.anaconda.org/conda-forge/linux-64/xorg-libice-1.1.2-h280c20c_0.conda#85c9442aec283b4e464fa9ecc484a2f3 +https://conda.anaconda.org/conda-forge/linux-64/libffi-3.7.0-h81df57d_1.conda#6525a0b06aa4fd390795f0740636a9dd +https://conda.anaconda.org/conda-forge/linux-64/c-ares-1.34.8-hebe6cf0_2.conda#3ad2b403be8fc5612b9159e2ce621f69 +https://conda.anaconda.org/conda-forge/linux-64/libev-4.33-h280c20c_3.conda#7bc31538d6e3fb349e897353969237ec +https://conda.anaconda.org/conda-forge/linux-64/fribidi-1.0.17-h7cc23a3_0.conda#f9d5302bca196ee87e62f393f2626562 +https://conda.anaconda.org/conda-forge/linux-64/keyutils-1.6.3-h7cc23a3_1.conda#ba55d1b89fd7775e67de8291029b4059 +https://conda.anaconda.org/conda-forge/linux-64/pthread-stubs-0.4-h7cc23a3_1004.conda#bb66b610707b811e3c65181a6431e7a8 +https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.6-hdb14827_1.conda#ee6c0cd80a60961a1f48aa3e0b91f986 +https://conda.anaconda.org/conda-forge/linux-64/libwebp-base-1.6.0-hd42ef1d_1.conda#9332b53d0ea93c5d39e33be03a0c611a +https://conda.anaconda.org/conda-forge/linux-64/bzip2-1.0.8-hda65f42_10.conda#e675fabcf81499adc7edf58124fb1e01 +https://conda.anaconda.org/conda-forge/linux-64/libpng-1.6.59-h922cc85_0.conda#8c9caf742a485ac688ffa88bbadbf0be +https://conda.anaconda.org/conda-forge/linux-64/libiconv-1.18-h0cb94f2_3.conda#f92233bf33e24a25668bb2119e2c51f9 +https://conda.anaconda.org/conda-forge/linux-64/libdeflate-1.25-hd45a770_1.conda#40f9b31aa9cf007789867df0decd0492 +https://conda.anaconda.org/conda-forge/linux-64/libexpat-2.8.5-hd2095e1_0.conda#72cfd2b95b0540247220b42f58a148d7 +https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.42.4-hcfc3c73_0.conda#54a948337e66706380ff24d6b5fde63f +https://conda.anaconda.org/conda-forge/linux-64/libjpeg-turbo-3.2.0-hb03c661_1.conda#898d1c9793eaa52efc4727bd84d2e39a +https://conda.anaconda.org/conda-forge/linux-64/make-4.4.1-hb03c661_3.conda#b4a198dc40024de2a59c1343e4ae4144 +https://conda.anaconda.org/conda-forge/linux-64/sed-4.10-h19d0853_0.conda#e4a1ff2e59a5d9b38be71d15c93cf1bd +https://conda.anaconda.org/conda-forge/linux-64/liblzma-5.8.3-hb03c661_1.conda#1390b7c5ac0b1d8e447bc5efa6d3c8c2 +https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.13-noxft_h1df4ec4_4.conda#676a2f9b1c4fcf57b70aa5c65f6c60d0 +https://conda.anaconda.org/conda-forge/linux-64/libgfortran5-16.2.0-h6b99dfc_7.conda#7719ed7be38ad8c492c4bd5cc2bce0c4 +https://conda.anaconda.org/conda-forge/linux-64/libuv-1.52.1-h280c20c_1.conda#01c4ed87826af55996768af6dbc936b4 +https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-16.2.0-h934c35e_7.conda#9b6707e0caca917fa6ea9ee473f37a5d +https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-16.2.0-h69a702a_7.conda#2005515ac9340f816fb2ac7e83afe553 +https://conda.anaconda.org/conda-forge/linux-64/libxcb-1.17.0-hb83e432_2.conda#61f0ffba9161cbb3a77722869b21e4bb +https://conda.anaconda.org/conda-forge/linux-64/libedit-3.1.20260512-pl5321h373387f_0.conda#27dbd7e2984b8fe48e069379636ccfdd +https://conda.anaconda.org/conda-forge/linux-64/readline-8.3-hd6e31c0_1.conda#69c01c781e7c8190bbdaf79e3848c5ca +https://conda.anaconda.org/conda-forge/linux-64/pcre2-10.47-h8b3dc9c_1.conda#06f6113cf1ff4a54b65f87ead132a5f1 +https://conda.anaconda.org/conda-forge/linux-64/libfreetype6-2.14.3-h5e6c136_2.conda#b72a266a9317036fd0464cb98b027cd0 +https://conda.anaconda.org/conda-forge/linux-64/xorg-libsm-1.2.6-h0d788c3_1.conda#aa7459ed9ad086ba11dda843d764b33d +https://conda.anaconda.org/conda-forge/linux-64/bwidget-1.10.1-hfc2019e_3.conda#38d3e0382f3f10344ac30772dec58035 +https://conda.anaconda.org/conda-forge/linux-64/libgfortran-16.2.0-h69a702a_7.conda#3a8d285f5f22138ef826ad54cf267e73 +https://conda.anaconda.org/conda-forge/linux-64/graphite2-1.3.15-h54a6638_1.conda#f9fe2984587fa8235a6af6004760cd18 +https://conda.anaconda.org/conda-forge/linux-64/icu-78.3-py310h44b86e0_2.conda#72a381cbad04f24b1c2a43ef707f45b4 +https://conda.anaconda.org/conda-forge/linux-64/pixman-0.46.4-h54a6638_3.conda#0ee5bb30034b081a1386c1e2c98ab0a7 +https://conda.anaconda.org/conda-forge/linux-64/libsanitizer-16.2.0-h3048135_7.conda#9cdd9af432fa4eb62094aaf6211425bf +https://conda.anaconda.org/conda-forge/linux-64/lerc-4.2.0-hdb68285_0.conda#fb9d356b1a57d6d54768be7ebd5fce09 +https://conda.anaconda.org/conda-forge/linux-64/xorg-libx11-1.8.13-he1eb515_1.conda#8c282bbe4808a3cc80a5c98e9aec1cfc +https://conda.anaconda.org/conda-forge/linux-64/libglib-2.90.0-h569388d_0.conda#c85e9539551f1fd21104e518d940c244 +https://conda.anaconda.org/conda-forge/linux-64/libfreetype-2.14.3-ha770c72_2.conda#f8054e759d0ddddaf34a5c8fedc900a1 +https://conda.anaconda.org/conda-forge/linux-64/libopenblas-0.3.34-pthreads_hf13c14d_2.conda#7e19fcd73d7f0997d3cb3326d2586a86 +https://conda.anaconda.org/conda-forge/linux-64/libpsl-0.23.1-hd9e3e90_1.conda#77be60417aa572afcb48cbe0f967401d +https://conda.anaconda.org/conda-forge/linux-64/libtiff-4.7.2-hcc2c06a_1.conda#0907d876f460ebc941ae590338b6102f +https://conda.anaconda.org/conda-forge/linux-64/xorg-libxrender-0.9.12-hb03c661_1.conda#e470d224a7a5be1b1d021bded7abb536 +https://conda.anaconda.org/conda-forge/linux-64/xorg-libxext-1.3.7-h7cc23a3_1.conda#e5b6b28536b81b3f4cb20db4668a4642 +https://conda.anaconda.org/conda-forge/linux-64/xorg-libxt-1.3.1-h7cc23a3_1.conda#2121765de9c261e2a95ab9fc6efabefb +https://conda.anaconda.org/conda-forge/linux-64/tktable-2.10-h0eb2ad0_11.conda#25d73f1e53349312f316ad82b78b5a7f +https://conda.anaconda.org/conda-forge/linux-64/fontconfig-2.18.3-h4db4eae_1.conda#922776b528a470ab5afa81fd42abfa1d +https://conda.anaconda.org/conda-forge/linux-64/libblas-3.11.0-11_h4a7cf45_openblas.conda#b8cce1486f3c62f33291d6318c193d0d +https://conda.anaconda.org/conda-forge/linux-64/libcblas-3.11.0-11_h0358290_openblas.conda#73a248c30811075059b5aeda766c2624 +https://conda.anaconda.org/conda-forge/linux-64/liblapack-3.11.0-11_h47877c9_openblas.conda#5622a13917855ce4b953b9e655cf9262 +https://conda.anaconda.org/conda-forge/linux-64/gsl-2.7-he838d99_0.tar.bz2#fec079ba39c9cca093bf4c00001825de +https://conda.anaconda.org/conda-forge/noarch/ca-certificates-2026.7.22-hbd8a1cb_0.conda#0f51e2391ade309db462a55611263e9c +https://conda.anaconda.org/conda-forge/noarch/_r-mutex-1.0.1-anacondar_1.tar.bz2#19f9db5f4f1b7f5ef5f6d67207f25f38 +https://conda.anaconda.org/conda-forge/noarch/tzdata-2026c-h151e31d_0.conda#fcb489df604d100968b737f2cb6076c6 +https://conda.anaconda.org/conda-forge/noarch/libstdcxx-devel_linux-64-16.2.0-h86e191b_107.conda#a713d3c09518a9fc8f603ac0e157263c +https://conda.anaconda.org/conda-forge/noarch/libgcc-devel_linux-64-16.2.0-he3ce08f_107.conda#f689d67c3e828fe80ff290d15f361bc5 +https://conda.anaconda.org/conda-forge/noarch/kernel-headers_linux-64-6.12.0-he073ed8_6.conda#ad5a27ffc9350af2624927dd79edf2be +https://conda.anaconda.org/conda-forge/noarch/font-ttf-dejavu-sans-mono-2.37-hab24e00_0.tar.bz2#0c96522c6bdaed4b1566d11387caaf45 +https://conda.anaconda.org/conda-forge/noarch/font-ttf-inconsolata-3.000-h77eed37_0.tar.bz2#34893075a5c9e55cdafac56607368fc6 +https://conda.anaconda.org/conda-forge/noarch/font-ttf-source-code-pro-2.038-h77eed37_0.tar.bz2#4d59c254e01d9cde7957100457e2d5fb +https://conda.anaconda.org/conda-forge/noarch/font-ttf-ubuntu-0.83-h77eed37_3.conda#49023d73832ef61042f6a237cb2687e7 +https://conda.anaconda.org/conda-forge/noarch/sysroot_linux-64-2.39-hc4b9eeb_6.conda#bc073c887526f93b09b8ca490192790b +https://conda.anaconda.org/conda-forge/noarch/fonts-conda-forge-1-hc364b38_1.conda#a7970cd949a077b7cb9696379d338681 +https://conda.anaconda.org/conda-forge/noarch/fonts-conda-ecosystem-1-0.tar.bz2#fee5683a3f04bd15cbd8318b096a27ab +https://conda.anaconda.org/conda-forge/linux-64/openssl-3.6.5-h781a0a9_0.conda#5a51a2ea653f120240b1a9bf76e0aae9 +https://conda.anaconda.org/conda-forge/linux-64/binutils_impl_linux-64-2.46.1-default_hfdba357_103.conda#e86f9dc615d7533c5be8b15b3a1dee8d +https://conda.anaconda.org/conda-forge/linux-64/cairo-1.18.6-h3c89d7e_0.conda#7920701f9abcb488411495742cf365a7 +https://conda.anaconda.org/conda-forge/linux-64/libnghttp2-1.68.1-h74cf4be_1.conda#75d211f04ac87506f1f43420712a69fe +https://conda.anaconda.org/conda-forge/linux-64/libssh2-1.11.1-h6154650_1.conda#5c526561e1d2a2e071941174eae84557 +https://conda.anaconda.org/conda-forge/linux-64/krb5-1.22.2-hbc21106_2.conda#53318d715316929a574f83591308b1f8 +https://conda.anaconda.org/conda-forge/linux-64/binutils_linux-64-2.46.1-default_h4852527_103.conda#ca26f0a4f8d04fd83147af557fbd932c +https://conda.anaconda.org/conda-forge/linux-64/gcc_impl_linux-64-16.2.0-h9891192_7.conda#f548b09900fed2701931f8cf925d8b9b +https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-14.6.0-h23af247_1.conda#bd7ae146a629d6c17e4be9365d4ec811 +https://conda.anaconda.org/conda-forge/linux-64/libgit2-1.9.7-hf27fc33_1.conda#1cf3ef849e34bc3283641cfda2675281 +https://conda.anaconda.org/conda-forge/linux-64/libcurl-8.22.0-ha042cf0_0.conda#e617deee7af8eb0c04f6296d12b54157 +https://conda.anaconda.org/conda-forge/linux-64/gxx_impl_linux-64-16.2.0-h0d273dc_7.conda#ea011fbfe6ce5b3a74a8e34fc9c8481a +https://conda.anaconda.org/conda-forge/linux-64/gcc_linux-64-16.2.0-h0ab548f_2.conda#3928593d74a0b17f84fba5977cb7023d +https://conda.anaconda.org/conda-forge/linux-64/gfortran_impl_linux-64-16.2.0-hf930d89_7.conda#b8fa9e9457b9e75f2bf85ad8017fa1d7 +https://conda.anaconda.org/conda-forge/linux-64/pango-1.58.2-h7bb47b9_1.conda#0b889b1c6b472d215d6332c6b195589c +https://conda.anaconda.org/conda-forge/linux-64/curl-8.22.0-ha042cf0_0.conda#399a7c0b7ca511de34c5ee5762cee149 +https://conda.anaconda.org/conda-forge/linux-64/gxx_linux-64-16.2.0-h23aca61_2.conda#38b21695fc42aa9bf6d4883e45a80fd9 +https://conda.anaconda.org/conda-forge/linux-64/r-base-4.5.3-h502d0c9_3.conda#8e584ef995c58d546ca1b2642fa4deae +https://conda.anaconda.org/conda-forge/linux-64/r-bit-4.6.0-r45h54b55ab_1.conda#6b666bedcfbe9dee03efb5fb95ac18e1 +https://conda.anaconda.org/conda-forge/linux-64/r-base64enc-0.1_6-r45h54b55ab_0.conda#e573dc135976197e7f819eec202c93f1 +https://conda.anaconda.org/conda-forge/linux-64/r-mime-0.13-r45h54b55ab_1.conda#26aa2fa52d4caed58336f2b4887916d6 +https://conda.anaconda.org/conda-forge/linux-64/r-xfun-0.61-r45hddc5280_1.conda#fdc3b42e200db84ffe0636a6f0b96f6e +https://conda.anaconda.org/conda-forge/linux-64/r-digest-0.6.39-r45h3697838_0.conda#a0e856537aa7d62a6835e3a528d29517 +https://conda.anaconda.org/conda-forge/linux-64/r-parallelly-1.48.0-r45h54b55ab_0.conda#4db06e4d5b749678d35b346e442d17a5 +https://conda.anaconda.org/conda-forge/linux-64/r-stringi-1.8.9-r45h3d52c89_0.conda#8b4ab64c1ea52b50aded3186643a9f9f +https://conda.anaconda.org/conda-forge/linux-64/r-colorspace-2.1_3-r45h54b55ab_0.conda#472eadbb95c9103963bae1631ec23b0f +https://conda.anaconda.org/conda-forge/linux-64/r-fastmap-1.2.0-r45h3697838_2.conda#245526991ad3b8a1dc97f2dcb5031065 +https://conda.anaconda.org/conda-forge/linux-64/r-data.table-1.18.6.1-r45hbd9402d_0.conda#34e991670e902d15eaeb84790048974b +https://conda.anaconda.org/conda-forge/linux-64/r-kernsmooth-2.23_27-r45hf1ecf5c_0.conda#9c164aed0ddfd01f45273ba00f6f23b5 +https://conda.anaconda.org/conda-forge/linux-64/r-lattice-0.23_1-r45hddc5280_0.conda#902c7c607b9ab6500debac3acfc6b068 +https://conda.anaconda.org/conda-forge/linux-64/r-rpart-4.1.27-r45h54b55ab_0.conda#072cbea77e0705c0126f04a883ab912b +https://conda.anaconda.org/conda-forge/linux-64/r-mass-7.3_66-r45h54b55ab_0.conda#700d7bffd1c317597b574b218b567017 +https://conda.anaconda.org/conda-forge/linux-64/r-warp-0.2.3-r45h54b55ab_0.conda#8f6a66f0eb973fc76ce601063d147048 +https://conda.anaconda.org/conda-forge/linux-64/r-rcpp-1.1.2-r45h3697838_0.conda#1d6e2e4764de896f03fb73e28614022e +https://conda.anaconda.org/conda-forge/linux-64/r-s7-0.2.2-r45h54b55ab_0.conda#598ca5290f9be9b22804436295471201 +https://conda.anaconda.org/conda-forge/linux-64/r-rappdirs-0.3.4-r45h54b55ab_0.conda#9fa763ece102dca3e50892bad36896ff +https://conda.anaconda.org/conda-forge/linux-64/r-fs-2.1.0-r45h3697838_0.conda#67c48aad5ce921ad990b951a1f194f42 +https://conda.anaconda.org/conda-forge/linux-64/r-yaml-2.3.12-r45h54b55ab_0.conda#d0d07c6f14b640a98cf6a5e3e4e2e723 +https://conda.anaconda.org/conda-forge/linux-64/r-sys-3.4.3-r45h54b55ab_1.conda#f23bbac61ab0536f59f4caa4c2ebdf88 +https://conda.anaconda.org/conda-forge/linux-64/r-curl-8.0.0-r45h444973c_0.conda#8f92ea8dfec9d170d50d31797979695d +https://conda.anaconda.org/conda-forge/linux-64/r-dicedesign-1.10-r45h54b55ab_2.conda#8d008a251f7fa40644a71f3fa4a6da35 +https://conda.anaconda.org/conda-forge/linux-64/r-fansi-1.0.7-r45h54b55ab_0.conda#309740f1b6a2d4cb16d395be786c85c5 +https://conda.anaconda.org/conda-forge/linux-64/r-gower-1.0.2-r45h54b55ab_0.conda#48168b9d60326506af819549391b5b6c +https://conda.anaconda.org/conda-forge/linux-64/r-magrittr-2.0.5-r45h54b55ab_0.conda#2e26e018edc1f198aa72a8f2127fab00 +https://conda.anaconda.org/conda-forge/linux-64/r-jsonlite-2.0.0-r45h54b55ab_1.conda#026c72026f431daf8a5719e09e704faa +https://conda.anaconda.org/conda-forge/linux-64/r-utf8-1.2.6-r45h54b55ab_1.conda#75cb2540ac930ea879e92d99e00e97c3 +https://conda.anaconda.org/conda-forge/linux-64/r-farver-2.1.2-r45h3697838_2.conda#7896efcfd50f8c1f207acce5d4ab1cc0 +https://conda.anaconda.org/conda-forge/linux-64/r-cli-3.6.6-r45h3697838_0.conda#ed8dcbbe9d66e9093ba58891e3b6065a +https://conda.anaconda.org/conda-forge/linux-64/r-rlang-1.3.0-r45h3697838_0.conda#afa49304f86d922c157d08e9e9d9ceb8 +https://conda.anaconda.org/conda-forge/linux-64/r-glue-1.8.1-r45h54b55ab_0.conda#65911c2e9cac35ea0235d3d6d4aa9625 +https://conda.anaconda.org/conda-forge/linux-64/r-bit64-4.8.6-r45hddc5280_0.conda#5cea8ea9e1bf30dc0b863e2b2fcb972b +https://conda.anaconda.org/conda-forge/linux-64/r-matrix-1.7_6-r45h0e4624f_0.conda#4b74bbbc18d62c7c93a84c558372b983 +https://conda.anaconda.org/conda-forge/linux-64/r-nnet-7.3_21-r45hddc5280_0.conda#371b62191b9d87d9caead88954d1e772 +https://conda.anaconda.org/conda-forge/linux-64/r-class-7.3_24-r45h54b55ab_0.conda#7ea996d2b2617350c14416537e6a3d18 +https://conda.anaconda.org/conda-forge/linux-64/r-rcpparmadillo-15.6.0_1-r45h1fcd065_0.conda#909cfe430769dd82e7d12cc2b46b3aef +https://conda.anaconda.org/conda-forge/linux-64/r-lbfgs-1.2.1.2-r45h3697838_5.conda#96b87eea8a76283e2cda7468b34ae2e5 +https://conda.anaconda.org/conda-forge/linux-64/r-askpass-1.2.1-r45h54b55ab_1.conda#1ae3c72e90c6a3871c594448d3e152e4 +https://conda.anaconda.org/conda-forge/linux-64/r-zip-3.0.2-r45h54b55ab_0.conda#9998629d3f28b086ed3a69a564ec257f +https://conda.anaconda.org/conda-forge/linux-64/r-cachem-1.1.0-r45h54b55ab_2.conda#42617e710293f0b76ccc08855e0edbc4 +https://conda.anaconda.org/conda-forge/linux-64/r-ellipsis-0.3.3-r45h54b55ab_0.conda#e43b3e7101a90ac57f58a21da67c99a4 +https://conda.anaconda.org/conda-forge/linux-64/r-survival-3.8_12-r45hddc5280_0.conda#2165bf9ffcefc182e65ada290f3dbe29 +https://conda.anaconda.org/conda-forge/linux-64/r-openssl-2.4.2-r45h68c19f5_0.conda#6568a5797c9d74e89ef94b82724665aa +https://conda.anaconda.org/conda-forge/linux-64/r-htmltools-0.5.9-r45h3697838_0.conda#2a2297687ae137e7fa90d3c996895e61 +https://conda.anaconda.org/conda-forge/noarch/r-listenv-1.1.0-r45hacb3a79_0.conda#adc0aad6fbb0d8b89c760054d37dbb22 +https://conda.anaconda.org/conda-forge/noarch/r-squarem-2026.1-r45hc72bb7e_0.conda#b8a3906de179464adc9e2dcf005cfeaf +https://conda.anaconda.org/conda-forge/noarch/r-shape-1.4.6.1-r45ha770c72_2.conda#9f062131995e818b7bcf8fe65d3f4424 +https://conda.anaconda.org/conda-forge/noarch/r-cpp11-0.5.5-r45h785f33e_0.conda#71c68a3993a696b6b75f7a6550b3750f +https://conda.anaconda.org/conda-forge/noarch/r-evaluate-1.0.5-r45hc72bb7e_1.conda#4e0c71ab78d7292372a89d4daecb49af +https://conda.anaconda.org/conda-forge/noarch/r-gitcreds-0.1.2-r45hc72bb7e_4.conda#8864884cea757c51580b45be5c362068 +https://conda.anaconda.org/conda-forge/noarch/r-ini-0.3.1-r45hc72bb7e_1007.conda#e680ce2dae5ccc37ed94fc0696e6f10d +https://conda.anaconda.org/conda-forge/noarch/r-clipr-0.8.1-r45hc72bb7e_0.conda#6d3690eb86e7fdea974915a5ce963561 +https://conda.anaconda.org/conda-forge/noarch/r-rstudioapi-0.19.0-r45hc72bb7e_0.conda#5f1b6b899fa3204ab77403e75b9df5fd +https://conda.anaconda.org/conda-forge/noarch/r-codetools-0.2_20-r45hc72bb7e_2.conda#8e9e5b14f74a6040c77e0b9c8bfa84ca +https://conda.anaconda.org/conda-forge/noarch/r-viridislite-0.4.3-r45hc72bb7e_0.conda#bcadc0a3726e2191e51449f67fa5e0a9 +https://conda.anaconda.org/conda-forge/noarch/r-timedate-4052.112-r45hc72bb7e_0.conda#b9ce34da7db4d27de7a478d629b1b2b0 +https://conda.anaconda.org/conda-forge/noarch/r-assertthat-0.2.1-r45hc72bb7e_6.conda#b5291c60f52b43108a2973ba6f5885d1 +https://conda.anaconda.org/conda-forge/noarch/r-whisker-0.4.1-r45hc72bb7e_3.conda#5cd861608ecbeab0574d59a7754deba7 +https://conda.anaconda.org/conda-forge/noarch/r-rprojroot-2.1.1-r45hc72bb7e_1.conda#e1b7c51411ad3dd2a95aea6d466b12f7 +https://conda.anaconda.org/conda-forge/noarch/r-numderiv-2016.8_1.1-r45hc72bb7e_7.conda#4458f15e0f1edb2009e76322f666d0ec +https://conda.anaconda.org/conda-forge/noarch/r-withr-3.0.3-r45hc72bb7e_0.conda#388d25e67f93a5fec2aebe56d6a0a1d9 +https://conda.anaconda.org/conda-forge/noarch/r-labeling-0.4.3-r45hc72bb7e_2.conda#a41490fdf607381035aa0304c21407c9 +https://conda.anaconda.org/conda-forge/noarch/r-rcolorbrewer-1.1_3-r45h785f33e_4.conda#d8fa238420cb6de47d463b7345a761eb +https://conda.anaconda.org/conda-forge/noarch/r-pkgconfig-2.0.3-r45hc72bb7e_5.conda#40a5fdd06c7e7880758a021cf2df6c12 +https://conda.anaconda.org/conda-forge/noarch/r-crayon-1.5.3-r45hc72bb7e_2.conda#4f111ce078b9690abaad6248b831a370 +https://conda.anaconda.org/conda-forge/noarch/r-r6-2.6.1-r45hc72bb7e_1.conda#750802806d7d640c286ed8491bb395dc +https://conda.anaconda.org/conda-forge/noarch/r-generics-0.1.4-r45hc72bb7e_1.conda#f19c9493b80f63a41fa017ec3b27bc2e +https://conda.anaconda.org/conda-forge/noarch/r-highr-0.12-r45hc72bb7e_0.conda#0b5902d6af02a23bda1794d46090db42 +https://conda.anaconda.org/conda-forge/noarch/r-tinytex-0.60-r45hc72bb7e_0.conda#5ee226c1449c66d1cd2399d942faa04d +https://conda.anaconda.org/conda-forge/noarch/r-progressr-1.0.0-r45hc72bb7e_0.conda#386049538faa3b7679091494a9a1004f +https://conda.anaconda.org/conda-forge/noarch/r-munsell-0.5.1-r45hc72bb7e_2.conda#c78bd534986cde8fc0cb08cc9a1a2cc6 +https://conda.anaconda.org/conda-forge/noarch/r-lifecycle-1.0.5-r45hc72bb7e_0.conda#5f8369dfbdff08878e58bf15529fca3a +https://conda.anaconda.org/conda-forge/noarch/r-splitfngr-0.1.2-r45hc72bb7e_0.conda#35a5489e4d64d783a1fe578bc05c74dd +https://conda.anaconda.org/conda-forge/noarch/r-memoise-2.0.1-r45hc72bb7e_4.conda#2b04206ff6ea5a92e8e36bdaa5feb3cc +https://conda.anaconda.org/conda-forge/noarch/r-credentials-2.0.3-r45hc72bb7e_1.conda#a94730bc5fa7197875f58d4b092a541a +https://conda.anaconda.org/conda-forge/noarch/r-jquerylib-0.1.4-r45hc72bb7e_4.conda#49a9ed6ed01f4ae6067ead552795bfce +https://conda.anaconda.org/conda-forge/noarch/r-fontawesome-0.5.3-r45hc72bb7e_1.conda#e9fccb3617ec9776569c6496fa254e64 +https://conda.anaconda.org/conda-forge/noarch/r-diagram-1.6.5-r45ha770c72_4.conda#11676b1bcbee45cae2dffbb1ffaf0b7e +https://conda.anaconda.org/conda-forge/noarch/r-globals-0.19.1-r45hc72bb7e_0.conda#5bb935fc63812c4978f0ac443b8b3420 +https://conda.anaconda.org/conda-forge/noarch/r-prettyunits-1.2.0-r45hc72bb7e_2.conda#07465728b1fd99d28b286156dac895a3 +https://conda.anaconda.org/conda-forge/noarch/r-desc-1.4.3-r45hc72bb7e_2.conda#2428c825ae5b2fc162edaeb3fa76b486 +https://conda.anaconda.org/conda-forge/noarch/r-knitr-1.52-r45hacb3a79_0.conda#b127436178c659586a1ea6a9e63598da +https://conda.anaconda.org/conda-forge/noarch/r-gtable-0.3.6-r45hc72bb7e_1.conda#f686123cfba49e6299fae7e029a40266 +https://conda.anaconda.org/conda-forge/noarch/r-scales-1.4.0-r45hc72bb7e_1.conda#8bc81cc6fd130cef963bc9e082726a14 +https://conda.anaconda.org/conda-forge/noarch/r-future-1.76.0-r45hacb3a79_0.conda#7ec394598a99c71084f716f3e077847f +https://conda.anaconda.org/conda-forge/noarch/r-future.apply-1.20.2-r45hc72bb7e_0.conda#ec6b2f2d6089e26eef6452f11ee1a67e +https://conda.anaconda.org/conda-forge/noarch/r-lava-1.9.3-r45hacb3a79_0.conda#7bf82c8c677b3d23029f63ac5765d162 +https://conda.anaconda.org/conda-forge/linux-64/r-isoband-0.3.0-r45h3697838_0.conda#fb538d0b46d6a44aa1ff666b15422ba6 +https://conda.anaconda.org/conda-forge/linux-64/r-timechange-0.4.0-r45h3697838_0.conda#77b6b03b30183b189906b08ea0868b21 +https://conda.anaconda.org/conda-forge/linux-64/r-tzdb-0.5.0-r45h3697838_2.conda#54673c8b5186c85794f0bd40d7c49bb4 +https://conda.anaconda.org/conda-forge/linux-64/r-sass-0.4.10-r45h3697838_1.conda#a3ccf52eefa448628535ee758c5a8a37 +https://conda.anaconda.org/conda-forge/linux-64/r-vctrs-0.7.3-r45h3697838_0.conda#372cefc1b05a567d1fbe19633766ab0c +https://conda.anaconda.org/conda-forge/linux-64/r-gert-2.4.1-r45h7df5fc1_0.conda#acb7d0d15da0942696da919ef9181751 +https://conda.anaconda.org/conda-forge/linux-64/r-lobstr-1.2.2-r45hfa9885e_0.conda#d8bf9af57fc598a5b71b365ab30598f6 +https://conda.anaconda.org/conda-forge/linux-64/r-lubridate-1.9.5-r45h54b55ab_0.conda#cec48e713c16eb74b4984019282ad03a +https://conda.anaconda.org/conda-forge/linux-64/r-clock-0.7.4-r45h3697838_0.conda#b704af8c6d1f3c7241debde4c3c2c994 +https://conda.anaconda.org/conda-forge/linux-64/r-slider-0.3.3-r45h3697838_0.conda#d157647300bcd9dd638da5fdc001f173 +https://conda.anaconda.org/conda-forge/linux-64/r-sparsevctrs-0.3.6-r45h54b55ab_0.conda#81c927f4602a8f849d4117b0766f863a +https://conda.anaconda.org/conda-forge/linux-64/r-purrr-1.2.2-r45h54b55ab_0.conda#dc14c484a0415f43dc1b90b518beb41a +https://conda.anaconda.org/conda-forge/noarch/r-bslib-0.12.0-r45hc72bb7e_0.conda#62a97fa00e2f48b594a592dbcbfbe8a5 +https://conda.anaconda.org/conda-forge/noarch/r-httr2-1.3.0-r45hc72bb7e_0.conda#b240e3c0e61de35f1f43729f4ea2888e +https://conda.anaconda.org/conda-forge/noarch/r-hms-1.1.4-r45hc72bb7e_0.conda#4677c1ad37a9452e27e27d9ed1b8ae90 +https://conda.anaconda.org/conda-forge/noarch/r-stringr-1.6.0-r45h785f33e_0.conda#8db438d8aa370726ee1ce8bf458f2e6d +https://conda.anaconda.org/conda-forge/noarch/r-pillar-1.11.1-r45hc72bb7e_0.conda#807ef77a70fc5156f830d6c683d07a29 +https://conda.anaconda.org/conda-forge/noarch/r-ggplot2-4.0.3-r45h785f33e_0.conda#866fbbf3f356140922d618f26f2b889c +https://conda.anaconda.org/conda-forge/noarch/r-tidyselect-1.2.1-r45hc72bb7e_2.conda#15aa0f323385403fe182e46a1d095e1b +https://conda.anaconda.org/conda-forge/noarch/r-furrr-0.4.0-r45hc72bb7e_0.conda#2813f7fa1b9d4e249ee29ed520df180c +https://conda.anaconda.org/conda-forge/noarch/r-rmarkdown-2.32-r45hacb3a79_0.conda#cf4da65121b4a7a64888455ef67ddd53 +https://conda.anaconda.org/conda-forge/noarch/r-gh-1.6.1-r45hc72bb7e_0.conda#c26e14a9021dfadd58a31980087838a0 +https://conda.anaconda.org/conda-forge/noarch/r-progress-1.2.3-r45hc72bb7e_2.conda#619b691b0965c6894eb99d3851857df7 +https://conda.anaconda.org/conda-forge/noarch/r-usethis-3.2.1-r45hc72bb7e_1.conda#dcbfbae5c448a2e54f37457f16075468 +https://conda.anaconda.org/conda-forge/linux-64/r-tibble-3.3.1-r45h54b55ab_0.conda#ad28f67cbb0b10a5beaa7bf968761cad +https://conda.anaconda.org/conda-forge/linux-64/r-prodlim-2026.03.11-r45h3697838_0.conda#00689a494b747f40c7a760108acdb467 +https://conda.anaconda.org/conda-forge/linux-64/r-vroom-1.7.1-r45h3697838_0.conda#8e45deb7cb8502dda9073be1a5e664df +https://conda.anaconda.org/conda-forge/linux-64/r-dplyr-1.2.1-r45h3697838_0.conda#7c8e643f764769a184f0d0d06ac0e789 +https://conda.anaconda.org/conda-forge/linux-64/r-ipred-0.9_15-r45h54b55ab_2.conda#3026e00ad9eead1acf371d94a317496e +https://conda.anaconda.org/conda-forge/linux-64/r-readr-2.2.0-r45h3697838_0.conda#a5f0d7d99d912486b2d93a576fadb8e0 +https://conda.anaconda.org/conda-forge/linux-64/r-tidyr-1.3.2-r45h3697838_0.conda#e571d4d42233dd2aa3bdb0057ffbdc63 +https://conda.anaconda.org/conda-forge/noarch/r-sfd-0.1.0-r45hc72bb7e_2.conda#722c31e970ef82439c4e88449ad86188 +https://conda.anaconda.org/conda-forge/noarch/r-hardhat-1.4.3-r45hc72bb7e_0.conda#d1ddc05be5ae353ecf22442c76cd0bd4 +https://conda.anaconda.org/conda-forge/noarch/r-modelenv-0.2.0-r45hc72bb7e_1.conda#befcb47a71aeb4b07eeb563a45fb4228 +https://conda.anaconda.org/conda-forge/noarch/r-modeldata-1.6.0-r45hacb3a79_0.conda#52003943948b56084913e8551b7e8f9d +https://conda.anaconda.org/conda-forge/noarch/r-mixopt-0.1.3-r45hc72bb7e_0.conda#ee205159250d6501ade0ff1663923e36 +https://conda.anaconda.org/conda-forge/noarch/r-rsample-1.3.2-r45hc72bb7e_0.conda#a361947d7d35dd7806b98a177e79032a +https://conda.anaconda.org/conda-forge/noarch/r-dials-1.4.4-r45hc72bb7e_0.conda#8ad1b317a2795f05ed70a1a6eb3d92d3 +https://conda.anaconda.org/conda-forge/noarch/r-recipes-1.4.0-r45hacb3a79_0.conda#416c22c7a78347754eecf2bb29d3b8d0 +https://conda.anaconda.org/conda-forge/noarch/r-parsnip-1.6.1-r45hacb3a79_0.conda#ef23280eb12fb0838c4d4eb1c06cb48a +https://conda.anaconda.org/conda-forge/noarch/r-tailor-0.1.0-r45hc72bb7e_1.conda#2aa17f9b3dd7eede9d3cd5d1c29f2a45 +https://conda.anaconda.org/conda-forge/noarch/r-butcher-0.4.0-r45hc72bb7e_0.conda#80312754cab7696b126d3d8ccac81b49 +https://conda.anaconda.org/conda-forge/noarch/r-workflows-1.3.0-r45hc72bb7e_1.conda#0ebbb4ce53157cc0ceb23ee5c26e3846 +https://conda.anaconda.org/conda-forge/linux-64/r-yardstick-1.4.0-r45h54b55ab_0.conda#e7e3ee3b2f419f2da415bc02192567c8 +https://conda.anaconda.org/conda-forge/linux-64/r-gaupro-0.2.17-r45h3704496_0.conda#5b4c70799cb7128a5ffbbe7c51ac23a8 +https://conda.anaconda.org/conda-forge/noarch/r-tune-2.1.0-r45hc72bb7e_0.conda#dc4712e9c84d4e8be0554c1fe859e428 +https://conda.anaconda.org/conda-forge/noarch/r-probably-1.2.0-r45hc72bb7e_0.conda#a571554d3d721d9c259053744159ac95 diff --git a/benchmarks/alternatives/isitfair_adapter.py b/benchmarks/alternatives/isitfair_adapter.py new file mode 100644 index 0000000..556d206 --- /dev/null +++ b/benchmarks/alternatives/isitfair_adapter.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python3 +"""Prepared adapter only. No isitfair execution was authorized in this benchmark. + +After installing an approved pinned isitfair build: +python isitfair_adapter.py path/to/python-results output-dir +Native L2-regularized Platt is a DIFFERENT correction from PREDVAL's unpenalized rung2. +The scores API internally computes native intervals; this adapter discards them. +Those intervals assume one row per patient and are invalid for repeated-row data. +""" + +from __future__ import annotations + +import argparse +from pathlib import Path + +import numpy as np +import pandas as pd + + +def main(source: Path, out: Path): + from isitfair import FairnessAudit + from sklearn.metrics import brier_score_loss, roc_auc_score + + out.mkdir(parents=True, exist_ok=False) + rows = [] + for name in ("wdbc_heldout", "synthetic_clustered"): + df = pd.read_csv(source / f"{name}.csv") + # A single overall group requests the common correction; no claim of a + # clinically meaningful subgroup or fairness comparison is made here. + audit = FairnessAudit( + df.y.to_numpy(), + df.p.to_numpy(), + {"cohort": np.repeat("all", len(df))}, + threshold=0.5, + n_bootstrap=20, + random_state=20261010, + ) + corrected = audit.recalibration_ladder_scores( + by="cohort", method="platt", folds=df.fold.to_numpy() + )["common"] + for method, p in [("published", df.p.to_numpy()), ("native_platt", corrected)]: + rows.append( + { + "fixture": name, + "method": method, + "brier": brier_score_loss(df.y, p), + "auroc": roc_auc_score(df.y, p), + } + ) + df["isitfair_native_platt"] = corrected + df.to_csv(out / f"{name}_isitfair.csv", index=False) + pd.DataFrame(rows).to_csv(out / "isitfair_metrics.csv", index=False) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("source", type=Path) + parser.add_argument("out", type=Path) + args = parser.parse_args() + main(args.source, args.out) diff --git a/benchmarks/alternatives/probably_adapter.R b/benchmarks/alternatives/probably_adapter.R new file mode 100644 index 0000000..8399228 --- /dev/null +++ b/benchmarks/alternatives/probably_adapter.R @@ -0,0 +1,90 @@ +#!/usr/bin/env Rscript +# Native probably calibration and matched R GLM controls, with shared patient folds. +# Rscript benchmarks/alternatives/probably_adapter.R path/to/python-results output-dir +# Requires probably, rsample, yardstick, dplyr, readr, tidyr and purrr. +suppressPackageStartupMessages({ + library(probably); library(rsample); library(yardstick) + library(dplyr); library(readr); library(tidyr); library(purrr) +}) +started <- proc.time() +args <- commandArgs(trailingOnly = TRUE) +stopifnot(length(args) == 2) +dir.create(args[[2]], recursive = TRUE, showWarnings = FALSE) +results <- list() +agreements <- list() +for (fixture in c("wdbc_heldout", "synthetic_clustered")) { + d <- read_csv(file.path(args[[1]], paste0(fixture, ".csv")), show_col_types = FALSE) + d <- d |> mutate(truth = factor(if_else(y == 1, "event", "nonevent"), + levels = c("event", "nonevent")), + .pred_event = p, .pred_nonevent = 1 - p, .row = row_number()) + # Exported assignments are identical to PREDVAL, not a newly randomized split. + splits <- lapply(sort(unique(d$fold)), function(f) { + train <- which(d$fold != f); test <- which(d$fold == f) + stopifnot(length(intersect(d$patient_id[train], d$patient_id[test])) == 0) + make_splits(list(analysis = train, assessment = test), d) + }) + folds <- manual_rset(splits, paste0("Fold", seq_along(splits))) + validated <- cal_validate_logistic(folds, truth, c(.pred_event, .pred_nonevent), + smooth = FALSE, save_pred = TRUE, + metrics = metric_set(brier_class, roc_auc)) + # Pool held-out observations; do not average AUROC across unequal folds. + native <- bind_rows(validated$.predictions_cal) |> arrange(.row) + stopifnot(identical(native$.row, seq_len(nrow(d)))) + d$probably_raw_probability <- native$.pred_event + d$r_glm_logit <- NA_real_ + d$r_glm_raw_probability <- NA_real_ + for (f in sort(unique(d$fold))) { + train <- d$fold != f + d$z <- qlogis(pmin(pmax(d$p, 1e-6), 1 - 1e-6)) + logit_fit <- glm(y ~ z, data = d[train, ], family = binomial(), + control = glm.control(epsilon = 1e-11, maxit = 100)) + raw_fit <- glm(y ~ p, data = d[train, ], family = binomial(), + control = glm.control(epsilon = 1e-11, maxit = 100)) + stopifnot(logit_fit$converged, raw_fit$converged) + d$r_glm_logit[!train] <- predict(logit_fit, d[!train, ], type = "response") + d$r_glm_raw_probability[!train] <- predict(raw_fit, d[!train, ], type = "response") + } + stopifnot(max(abs(d$probably_raw_probability - d$r_glm_raw_probability)) < 1e-6) + py <- read_csv(file.path(args[[1]], paste0(fixture, "_corrected.csv")), + show_col_types = FALSE) + stopifnot(max(abs(py$predval_rung2 - d$r_glm_logit)) < 1e-8) + stopifnot(max(abs(py$statsmodels_raw_probability - d$probably_raw_probability)) < 1e-6) + agreements[[length(agreements) + 1]] <- tibble( + fixture = fixture, + max_abs_predval_vs_r_logit = max(abs(py$predval_rung2 - d$r_glm_logit)), + max_abs_statsmodels_vs_probably_raw = max(abs(py$statsmodels_raw_probability - + d$probably_raw_probability)) + ) + for (column in c("p", "probably_raw_probability", "r_glm_logit", "r_glm_raw_probability")) { + p <- d[[column]] + results[[length(results) + 1]] <- tibble( + fixture = fixture, tool = column, brier = mean((d$y - p)^2), + auroc = roc_auc_vec(d$truth, p, event_level = "first") + ) + } + write_csv(d |> select(subject_id, patient_id, fold, y, p, + probably_raw_probability, r_glm_logit, r_glm_raw_probability), + file.path(args[[2]], paste0(fixture, "_r_corrected.csv"))) + # Demonstrate rsample native grouped splitting and grouped bootstrap support. + # These random folds are a capability check, not the matched comparison partition. + set.seed(20261010) + grouped <- group_vfold_cv(d, patient_id, v = 5) + stopifnot(all(map_lgl(grouped$splits, function(s) { + length(intersect(analysis(s)$patient_id, assessment(s)$patient_id)) == 0 + }))) + set.seed(20261011) + boot <- group_bootstraps(d, patient_id, times = 1000) + # Conditional-on-predictions paired gain; whole patients remain intact. + gains <- map_dbl(boot$splits, function(s) { + b <- analysis(s) + mean((b$y - b$p)^2 - (b$y - b$r_glm_logit)^2) + }) + write_csv(tibble(fixture = fixture, replicate = seq_along(gains), gain = gains), + file.path(args[[2]], paste0(fixture, "_rsample_gains.csv"))) +} +write_csv(bind_rows(results), file.path(args[[2]], "r_metrics.csv")) +writeLines(trimws(capture.output(sessionInfo()), which = "right"), + file.path(args[[2]], "sessionInfo.txt")) +write_csv(bind_rows(agreements), file.path(args[[2]], "r_agreement.csv")) +writeLines(trimws(capture.output(proc.time() - started), which = "right"), + file.path(args[[2]], "resource_usage.txt")) diff --git a/benchmarks/alternatives/requirements-lock.txt b/benchmarks/alternatives/requirements-lock.txt new file mode 100644 index 0000000..baacec3 --- /dev/null +++ b/benchmarks/alternatives/requirements-lock.txt @@ -0,0 +1,53 @@ +annotated-types==0.8.0 +asttokens==3.0.2 +cloudpickle==3.1.2 +contourpy==1.4.0 +cycler==0.12.1 +defusedxml==0.7.1 +executing==2.2.1 +fonttools==4.66.1 +fpdf2==2.8.9 +iniconfig==2.3.1 +ipython==9.17.1 +ipython-pygments-lexers==1.1.1 +jedi==0.20.0 +jinja2==3.1.6 +joblib==1.6.0 +kiwisolver==1.5.1 +markupsafe==3.0.4 +matplotlib==3.11.1 +matplotlib-inline==0.2.2 +narwhals==2.27.1 +numpy==2.4.6 +packaging==26.3 +pandas==3.0.5 +parso==0.8.7 +patsy==1.0.3 +pexpect==4.9.0 +pillow==12.3.0 +pluggy==1.6.0 +prompt-toolkit==3.0.53 +psutil==7.2.2 +ptyprocess==0.7.0 +pure-eval==0.2.4 +pyarrow==25.0.1 +pycaleva==0.8.2 +pydantic==2.13.4 +pydantic-core==2.46.4 +pygments==2.21.0 +pyparsing==3.3.3 +pytest==9.1.1 +python-dateutil==2.9.0.post0 +pyyaml==6.0.3 +ruff==0.17.0 +scikit-learn==1.9.0 +scipy==1.17.1 +six==1.17.0 +stack-data==0.6.3 +statsmodels==0.14.6 +threadpoolctl==3.7.0 +tqdm==4.70.1 +traitlets==5.16.1 +typing-extensions==4.16.0 +typing-inspection==0.4.4 +wcwidth==0.9.2 diff --git a/benchmarks/alternatives/results/glue_inventory.json b/benchmarks/alternatives/results/glue_inventory.json new file mode 100644 index 0000000..b0dc698 --- /dev/null +++ b/benchmarks/alternatives/results/glue_inventory.json @@ -0,0 +1,12 @@ +{ + "physical_lines_including_comments": { + "reference_crossfit": 15, + "custom_bootstrap": 11, + "coverage_cases": 70, + "probably_adapter.R": 90, + "isitfair_adapter.py": 60 + }, + "interpretation": "Descriptive implementation size only. Includes validation/export code; not comparable developer time or a minimum required adapter.", + "native_predval_calls": "ladder; paired_brier_gain_interval; coverage_report/check_coverage/check_roster/check_common_selection", + "pycaleva_custom_scope": "External correction, grouped resampling, cohort/roster handling; point metrics native." +} diff --git a/benchmarks/alternatives/results/python/correction_metrics.csv b/benchmarks/alternatives/results/python/correction_metrics.csv new file mode 100644 index 0000000..c9a2da8 --- /dev/null +++ b/benchmarks/alternatives/results/python/correction_metrics.csv @@ -0,0 +1,13 @@ +fixture,tool,metric,value +wdbc_heldout,predval_rung2,auroc,0.98403077896068303 +wdbc_heldout,predval_rung2,brier,0.029256064983711545 +wdbc_heldout,statsmodels_custom_grouped,auroc,0.98403077896068303 +wdbc_heldout,statsmodels_custom_grouped,brier,0.029256064983711881 +wdbc_heldout,statsmodels_raw_probability,auroc,0.96874670601876245 +wdbc_heldout,statsmodels_raw_probability,brier,0.027717535594874616 +synthetic_clustered,predval_rung2,auroc,0.81855107327905252 +synthetic_clustered,predval_rung2,brier,0.17054132084865364 +synthetic_clustered,statsmodels_custom_grouped,auroc,0.81855107327905252 +synthetic_clustered,statsmodels_custom_grouped,brier,0.17054132084865359 +synthetic_clustered,statsmodels_raw_probability,auroc,0.81883558475203566 +synthetic_clustered,statsmodels_raw_probability,brier,0.17238338966503758 diff --git a/benchmarks/alternatives/results/python/coverage.csv b/benchmarks/alternatives/results/python/coverage.csv new file mode 100644 index 0000000..b542e9a --- /dev/null +++ b/benchmarks/alternatives/results/python/coverage.csv @@ -0,0 +1,3 @@ +model_id,n_scored,n_full,coverage_full,n_common,common_fraction,n_dropped_if_common,meets_minimum +full,1332,1332,1,919,0.68993993993993996,413,True +partial,919,1332,0.68993993993993996,919,0.68993993993993996,0,False diff --git a/benchmarks/alternatives/results/python/coverage_selection_metrics.csv b/benchmarks/alternatives/results/python/coverage_selection_metrics.csv new file mode 100644 index 0000000..ba6e87c --- /dev/null +++ b/benchmarks/alternatives/results/python/coverage_selection_metrics.csv @@ -0,0 +1,5 @@ +subset,metric,value,n +full_available,auroc,0.82189813101406362,1332 +full_available,brier,0.18140992475539261,1332 +common_intersection,auroc,0.78319621993512833,919 +common_intersection,brier,0.17792866586674272,919 diff --git a/benchmarks/alternatives/results/python/intervals.csv b/benchmarks/alternatives/results/python/intervals.csv new file mode 100644 index 0000000..2ff15cf --- /dev/null +++ b/benchmarks/alternatives/results/python/intervals.csv @@ -0,0 +1,11 @@ +fixture,tool,scheme,metric,low,high,width +wdbc_heldout,pycaleva_plus_custom_bootstrap,patient,brier,0.015593303087883301,0.048631062684011898,0.033037759596128598 +wdbc_heldout,pycaleva_plus_custom_bootstrap,patient,paired_brier_gain,-0.0053364764046153401,0.0094270783303540603,0.014763554734969399 +wdbc_heldout,pycaleva_plus_custom_bootstrap,row,brier,0.015723847480198692,0.048319757974591579,0.032595910494392887 +wdbc_heldout,pycaleva_plus_custom_bootstrap,row,paired_brier_gain,-0.0055358590458910581,0.0092397263447514654,0.014775585390642523 +wdbc_heldout,predval,patient,paired_brier_gain,-0.0053364764046153375,0.0094270783303540585,0.014763554734969396 +synthetic_clustered,pycaleva_plus_custom_bootstrap,patient,brier,0.1587361211804677,0.20688337470523394,0.048147253524766231 +synthetic_clustered,pycaleva_plus_custom_bootstrap,patient,paired_brier_gain,-0.0011632227475880863,0.022167365927628083,0.02333058867521617 +synthetic_clustered,pycaleva_plus_custom_bootstrap,row,brier,0.16852215094087003,0.19457984262889494,0.026057691688024909 +synthetic_clustered,pycaleva_plus_custom_bootstrap,row,paired_brier_gain,0.005338072340460855,0.016573381895931719,0.011235309555470864 +synthetic_clustered,predval,patient,paired_brier_gain,-0.0011632227475880895,0.022167365927628101,0.023330588675216191 diff --git a/benchmarks/alternatives/results/python/manifest.json b/benchmarks/alternatives/results/python/manifest.json new file mode 100644 index 0000000..c72cb8a --- /dev/null +++ b/benchmarks/alternatives/results/python/manifest.json @@ -0,0 +1,31 @@ +{ + "bootstrap_replicates": 1000, + "bootstrap_seed": 20261011, + "files": { + "correction_metrics.csv": "810d078fed3b8b96c69751ad2c04333e9730e13f64c2d7a9811faa4becacbd8a", + "coverage.csv": "3a6aa0df1d5da3aea520a81e7b1ff5287d12b1bad3ba5d36cbe90955061f8ca5", + "coverage_inputs/cohort.csv": "50127a122204e8fcbae05f4cf597e77f210791060015bbe120c837edc7f421c2", + "coverage_inputs/cohort.yaml": "fd3789802a5c4884f3d05a87ceb67bc67b5fbc97f9c736efe739f135780cf548", + "coverage_inputs/predictions.csv": "effe905a9fdc1117bf6d50f6bdfd9c20857552597d440259d37eb0accba5bc96", + "coverage_selection_metrics.csv": "2633e304ad3c482de0753cefcfce9c5c6835fc6882f0518310b2dc63d1f38712", + "intervals.csv": "65b5b3fc24cf9c603f7ca421d74b4a3353d8b75d92e2a7817d6428b125e44482", + "point_metrics.csv": "ba5cb17b04f400541e09b3c4b0108e41a211e75709707b55c9d4efee29f62480", + "summary.json": "03300c0306c33c4541e437b87426735a3bba16cabc2808a3144b282f54f41ac9", + "synthetic_clustered.csv": "50127a122204e8fcbae05f4cf597e77f210791060015bbe120c837edc7f421c2", + "synthetic_clustered_corrected.csv": "1bb4899c68ad53387f47804f6da41899c76ff5fc87165e5118cfd60a19e78568", + "wdbc_heldout.csv": "61a22060cc9072ef62ad74f7db619b982e8eeb3130c07d4cc275d602aa1ccd9c", + "wdbc_heldout_corrected.csv": "b47c07f8301ba3d7bac38a4ad9396593b0342036d71dabe1970d3630d5ce80c2" + }, + "protocol_sha256": "8630bb9af1f54eb4a7f8e7bacfa5a652dd4915dc60e1e319ac02105cbf920db1", + "python": "3.12.14", + "seed": 20261010, + "versions": { + "numpy": "2.4.6", + "pandas": "3.0.5", + "predval": "0.1.0", + "pycaleva": "0.8.2", + "scikit-learn": "1.9.0", + "scipy": "1.17.1", + "statsmodels": "0.14.6" + } +} diff --git a/benchmarks/alternatives/results/python/point_metrics.csv b/benchmarks/alternatives/results/python/point_metrics.csv new file mode 100644 index 0000000..285b7e5 --- /dev/null +++ b/benchmarks/alternatives/results/python/point_metrics.csv @@ -0,0 +1,21 @@ +fixture,tool,metric,value,abs_diff_reference +wdbc_heldout,predval,auroc,0.9880889638452619,0 +wdbc_heldout,predval,brier,0.031262751693950298,0 +wdbc_heldout,predval,calibration_intercept,1.8547783584918272,8.8817841970012523e-16 +wdbc_heldout,predval,calibration_slope,0.65895543819077251,1.4432899320127035e-15 +wdbc_heldout,pycaleva,auroc,0.9880889638452619,0 +wdbc_heldout,pycaleva,brier,0.031262751693950298,0 +wdbc_heldout,independent_reference,auroc,0.9880889638452619,0 +wdbc_heldout,independent_reference,brier,0.031262751693950298,0 +wdbc_heldout,independent_reference,calibration_intercept,1.854778358491828,0 +wdbc_heldout,independent_reference,calibration_slope,0.65895543819077107,0 +synthetic_clustered,predval,auroc,0.82189813101406362,0 +synthetic_clustered,predval,brier,0.18140992475539261,0 +synthetic_clustered,predval,calibration_intercept,0.71216526049354745,1.1102230246251565e-16 +synthetic_clustered,predval,calibration_slope,0.79441642455613248,2.2204460492503131e-16 +synthetic_clustered,pycaleva,auroc,0.82189813101406362,0 +synthetic_clustered,pycaleva,brier,0.18140992475539261,0 +synthetic_clustered,independent_reference,auroc,0.82189813101406362,0 +synthetic_clustered,independent_reference,brier,0.18140992475539261,0 +synthetic_clustered,independent_reference,calibration_intercept,0.71216526049354756,0 +synthetic_clustered,independent_reference,calibration_slope,0.79441642455613226,0 diff --git a/benchmarks/alternatives/results/python/resource_usage.json b/benchmarks/alternatives/results/python/resource_usage.json new file mode 100644 index 0000000..33735de --- /dev/null +++ b/benchmarks/alternatives/results/python/resource_usage.json @@ -0,0 +1,6 @@ +{ + "wall_seconds": 1.962911616999918, + "cpu_seconds": 1.9628360740000002, + "max_rss_kib": 291520, + "note": "Benchmark body, not a fair tool runtime comparison; excludes imports and installation." +} diff --git a/benchmarks/alternatives/results/python/summary.json b/benchmarks/alternatives/results/python/summary.json new file mode 100644 index 0000000..3bb2ec8 --- /dev/null +++ b/benchmarks/alternatives/results/python/summary.json @@ -0,0 +1,59 @@ +{ + "coverage": { + "errors": [ + { + "check": "minimum_coverage_90_percent", + "error": "1 model(s) fall below coverage.min_fraction; worst shown\n model: partial\n coverage: 0.689940\n required: 0.900000", + "status": "rejected" + }, + { + "check": "declared_absent_model", + "error": "declared in expected_models but absent from predictions: ['absent']", + "status": "rejected" + } + ], + "flags": [ + { + "code": "declared_but_absent", + "message": "declared in expected_models but absent from predictions: ['absent']", + "severity": "warning" + }, + { + "code": "informative_common_selection", + "message": "the common subset excludes 31.0% of cohort rows (> 20%); selection into the common subset may be informative", + "severity": "warning" + } + ], + "pycaleva_scope": "Accepts aligned y/p arrays; roster and cohort denominators need caller glue." + }, + "scope": "Executed: predval and PyCalEva; statsmodels/sklearn independent reference.", + "synthetic_clustered": { + "brier_gain": 0.010868603906738982, + "brier_gain_ci": [ + -0.0011632227475880895, + 0.0221673659276281 + ], + "events": 560, + "max_corrected_prediction_difference": 5.9119376061289586e-15, + "patients": 120, + "rows": 1332, + "suppressed_ladder_rungs": [] + }, + "wdbc_heldout": { + "brier_gain": 0.002006686710238758, + "brier_gain_ci": [ + -0.0053364764046153375, + 0.009427078330354059 + ], + "events": 106, + "max_corrected_prediction_difference": 3.785860513971784e-14, + "patients": 285, + "rows": 285, + "suppressed_ladder_rungs": [ + [ + "rung3", + "non-convergence" + ] + ] + } +} diff --git a/benchmarks/alternatives/results/r/r_agreement.csv b/benchmarks/alternatives/results/r/r_agreement.csv new file mode 100644 index 0000000..8f9cc40 --- /dev/null +++ b/benchmarks/alternatives/results/r/r_agreement.csv @@ -0,0 +1,3 @@ +fixture,max_abs_predval_vs_r_logit,max_abs_statsmodels_vs_probably_raw +wdbc_heldout,6.723954726339798e-12,1.0349785251051458e-8 +synthetic_clustered,5.773159728050814e-15,9.704870240767605e-11 diff --git a/benchmarks/alternatives/results/r/r_bootstrap_summary.csv b/benchmarks/alternatives/results/r/r_bootstrap_summary.csv new file mode 100644 index 0000000..b0a6040 --- /dev/null +++ b/benchmarks/alternatives/results/r/r_bootstrap_summary.csv @@ -0,0 +1,3 @@ +fixture,replicates,low,high,method +wdbc_heldout,1000,-0.0052822820622240177,0.0093309392234624533,rsample native group_bootstraps; fixed R logit-GLM predictions +synthetic_clustered,1000,-0.0010582283577589672,0.022889944278495382,rsample native group_bootstraps; fixed R logit-GLM predictions diff --git a/benchmarks/alternatives/results/r/r_metrics.csv b/benchmarks/alternatives/results/r/r_metrics.csv new file mode 100644 index 0000000..a9c0f5b --- /dev/null +++ b/benchmarks/alternatives/results/r/r_metrics.csv @@ -0,0 +1,9 @@ +fixture,tool,brier,auroc +wdbc_heldout,p,0.031262751693950305,0.9880889638452619 +wdbc_heldout,probably_raw_probability,0.027717535582638633,0.9687467060187626 +wdbc_heldout,r_glm_logit,0.02925606498367934,0.984030778960683 +wdbc_heldout,r_glm_raw_probability,0.027717535594874523,0.9687467060187626 +synthetic_clustered,p,0.18140992475539264,0.8218981310140636 +synthetic_clustered,probably_raw_probability,0.1723833896630457,0.8188355847520356 +synthetic_clustered,r_glm_logit,0.1705413208486536,0.8185510732790525 +synthetic_clustered,r_glm_raw_probability,0.1723833896650324,0.8188355847520356 diff --git a/benchmarks/alternatives/results/r/resource_usage.txt b/benchmarks/alternatives/results/r/resource_usage.txt new file mode 100644 index 0000000..d863511 --- /dev/null +++ b/benchmarks/alternatives/results/r/resource_usage.txt @@ -0,0 +1,2 @@ + user system elapsed +138.449 3.509 142.258 diff --git a/benchmarks/alternatives/results/r/sessionInfo.txt b/benchmarks/alternatives/results/r/sessionInfo.txt new file mode 100644 index 0000000..d51406a --- /dev/null +++ b/benchmarks/alternatives/results/r/sessionInfo.txt @@ -0,0 +1,45 @@ +R version 4.5.3 (2026-03-11) +Platform: x86_64-conda-linux-gnu +Running under: Debian GNU/Linux 13 (trixie) + +Matrix products: default +BLAS/LAPACK: /workspace/shared/predval-r-toolchain/env/lib/libopenblasp-r0.3.34.so; LAPACK version 3.12.0 + +locale: + [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8 + [4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8 + [7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C +[10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C + +time zone: America/New_York +tzcode source: system (glibc) + +attached base packages: +[1] stats graphics grDevices utils datasets methods base + +other attached packages: +[1] purrr_1.2.2 tidyr_1.3.2 readr_2.2.0 dplyr_1.2.1 +[5] yardstick_1.4.0 rsample_1.3.2 probably_1.2.0 + +loaded via a namespace (and not attached): + [1] gtable_0.3.6 ggplot2_4.0.3 recipes_1.4.0 + [4] lattice_0.23-1 tzdb_0.5.0 vctrs_0.7.3 + [7] tools_4.5.3 generics_0.1.4 parallel_4.5.3 +[10] tibble_3.3.1 pkgconfig_2.0.3 Matrix_1.7-6 +[13] data.table_1.18.6.1 RColorBrewer_1.1-3 S7_0.2.2 +[16] lifecycle_1.0.5 compiler_4.5.3 farver_2.1.2 +[19] codetools_0.2-20 DiceDesign_1.10 class_7.3-24 +[22] tune_2.1.0 prodlim_2026.03.11 pillar_1.11.1 +[25] furrr_0.4.0 crayon_1.5.3 MASS_7.3-66 +[28] gower_1.0.2 rpart_4.1.27 parallelly_1.48.0 +[31] lava_1.9.3 dials_1.4.4 tidyselect_1.2.1 +[34] digest_0.6.39 future_1.76.0 listenv_1.1.0 +[37] splines_4.5.3 parsnip_1.6.1 grid_4.5.3 +[40] cli_3.6.6 magrittr_2.0.5 survival_3.8-12 +[43] future.apply_1.20.2 withr_3.0.3 scales_1.4.0 +[46] bit64_4.8.6 lubridate_1.9.5 timechange_0.4.0 +[49] butcher_0.4.0 globals_0.19.1 bit_4.6.0 +[52] nnet_7.3-21 timeDate_4052.112 workflows_1.3.0 +[55] hms_1.1.4 hardhat_1.4.3 rlang_1.3.0 +[58] Rcpp_1.1.2 glue_1.8.1 ipred_0.9-15 +[61] vroom_1.7.1 R6_2.6.1 diff --git a/benchmarks/alternatives/results/r/verification.json b/benchmarks/alternatives/results/r/verification.json new file mode 100644 index 0000000..9520c7c --- /dev/null +++ b/benchmarks/alternatives/results/r/verification.json @@ -0,0 +1,17 @@ +{ + "python_manifest_sha256": "a2a541ec30fbffe83103e04d2c85a5e83114a671e9818488e896a2f34eaf510d", + "python_stable_files_verified": 13, + "r_files_sha256": { + "r_agreement.csv": "2c1e306addc003d941c80f84ad0f2e1ae476902e42eb5a8498152f147600f4bd", + "r_metrics.csv": "7a8d558a65c8d61d43b04406fbf1d46c5c18126e6fc5a2ebd1818fe9dfde0c59", + "sessionInfo.txt": "54be6cde818461ff3da62ff156022848d00cab4ae131e30af1c4c1ef720b3039", + "synthetic_clustered_r_corrected.csv": "5c13c21f257107c6893d001dc80388d1b19ab92448dadf951982e9358414cba6", + "synthetic_clustered_rsample_gains.csv": "bbedbfec1f6085cb31290b43f84feaf1dff7c6049cb1d276cccd2f26c24a3ccb", + "wdbc_heldout_r_corrected.csv": "8bc8df31c97730d9e2acd8671d2d0ae9a0816f9a1580b8e615129dacccacaf3f", + "wdbc_heldout_rsample_gains.csv": "3f1fc4df7f404beedc6255da517ee6e4bd634ff2588833f9299be0bff96237dd" + }, + "r_repeated_stable_files_identical": 7, + "r_timing_first": "user system elapsed\n138.449 3.509 142.258", + "r_timing_repeat": "user system elapsed\n181.453 4.849 187.484", + "scope": "Same final inputs and fixed dependencies. Runtime excluded from hash equality." +} diff --git a/benchmarks/alternatives/results/reproducibility.json b/benchmarks/alternatives/results/reproducibility.json new file mode 100644 index 0000000..56f53f4 --- /dev/null +++ b/benchmarks/alternatives/results/reproducibility.json @@ -0,0 +1,18 @@ +{ + "fresh_python_environments": 2, + "stable_manifests_identical": true, + "manifest_sha256": "a2a541ec30fbffe83103e04d2c85a5e83114a671e9818488e896a2f34eaf510d", + "primary_usage": { + "wall_seconds": 1.962911616999918, + "cpu_seconds": 1.9628360740000002, + "max_rss_kib": 291520, + "note": "Benchmark body, not a fair tool runtime comparison; excludes imports and installation." + }, + "clean_usage": { + "wall_seconds": 2.3836366670002462, + "cpu_seconds": 2.3833308649999996, + "max_rss_kib": 291860, + "note": "Benchmark body, not a fair tool runtime comparison; excludes imports and installation." + }, + "timing_scope": "Benchmark body after imports, excludes dependency setup. Not a tool speed comparison." +} diff --git a/benchmarks/alternatives/run_benchmark.py b/benchmarks/alternatives/run_benchmark.py new file mode 100644 index 0000000..200de3c --- /dev/null +++ b/benchmarks/alternatives/run_benchmark.py @@ -0,0 +1,339 @@ +#!/usr/bin/env python3 +"""Prespecified, bounded alternatives benchmark. See PROTOCOL.md before running.""" + +from __future__ import annotations + +import argparse +import hashlib +import importlib.metadata +import json +import platform +import resource +import time +from pathlib import Path + +import numpy as np +import pandas as pd +import statsmodels.api as sm +import yaml +from pycaleva import CalibrationEvaluator +from pycaleva.metrics import brier as pycaleva_brier +from scipy.special import expit, logit +from sklearn.datasets import load_breast_cancer +from sklearn.linear_model import LogisticRegression +from sklearn.metrics import brier_score_loss, roc_auc_score +from sklearn.model_selection import train_test_split +from sklearn.pipeline import make_pipeline +from sklearn.preprocessing import StandardScaler + +from predval import metrics as pm +from predval import recalibrate as pr +from predval import uncertainty as pu +from predval.errors import CohortSpecError, CoverageViolation +from predval.io import ( + check_common_selection, + check_coverage, + check_roster, + coverage_report, + load_cohort, + load_predictions, +) + +SEED = 20261010 +BOOT_SEED = 20261011 +N_BOOT = 1000 +HERE = Path(__file__).resolve().parent + + +def digest(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def frame(y, p, patient): + groups = np.asarray(patient).astype(str) + fold_lookup = {g: i % 5 for i, g in enumerate(np.unique(groups))} + return pd.DataFrame( + { + "subject_id": [f"row_{i:05}" for i in range(len(y))], + "patient_id": groups, + "y": np.asarray(y, dtype=int), + "p": p, + "fold": [fold_lookup[g] for g in groups], + } + ) + + +def fixtures(): + data = load_breast_cancer() + y = 1 - data.target # UCI malignant is the positive event. + train, test = train_test_split(np.arange(len(y)), test_size=0.5, stratify=y, random_state=SEED) + model = make_pipeline(StandardScaler(), LogisticRegression(C=1, max_iter=2000, solver="lbfgs")) + model.fit(data.data[train], y[train]) + p = model.predict_proba(data.data[test])[:, 1] + real = frame( + y[test], + expit(-0.4 + 1.4 * logit(np.clip(p, 1e-6, 1 - 1e-6))), + [f"uci_{i:04}" for i in test], + ) + rng = np.random.default_rng(SEED) + sizes = rng.integers(4, 21, 120) + ids = np.repeat(np.arange(120), sizes) + latent = rng.normal(0, 1.2, 120) + x = rng.normal(size=len(ids)) + risk = expit(-0.8 + 0.9 * x + latent[ids]) + # A shared uniform induces positive within-patient outcome dependence while + # preserving each row's conditional Bernoulli marginal probability. + shared = rng.uniform(size=120) + uniforms = np.where(rng.uniform(size=len(ids)) < 0.65, shared[ids], rng.uniform(size=len(ids))) + sy = (uniforms < risk).astype(int) + sp = expit(-0.4 + 1.4 * (-0.8 + 0.9 * x + latent[ids])) + synthetic = frame(sy, sp, [f"patient_{i:03}" for i in ids]) + return {"wdbc_heldout": real, "synthetic_clustered": synthetic} + + +def metrics(y, p): + return {"auroc": roc_auc_score(y, p), "brier": brier_score_loss(y, p)} + + +def reference_calibration(y, p): + z = logit(np.clip(p, 1e-6, 1 - 1e-6)) + offset = sm.GLM(y, np.ones((len(y), 1)), family=sm.families.Binomial(), offset=z) + joint = sm.GLM(y, sm.add_constant(z), family=sm.families.Binomial()) + a = offset.fit(tol=1e-11, maxiter=100) + b = joint.fit(tol=1e-11, maxiter=100) + assert a.converged and b.converged + return {"calibration_intercept": a.params[0], "calibration_slope": b.params[1]} + + +def reference_crossfit(df, *, logit_scale=True): + y, p = df.y.to_numpy(), df.p.to_numpy() + x = logit(np.clip(p, 1e-6, 1 - 1e-6)) if logit_scale else p + out = np.full(len(df), np.nan) + for f in sorted(df.fold.unique()): + train = df.fold.to_numpy() != f + fit = sm.GLM( + y[train], + sm.add_constant(x[train]), + family=sm.families.Binomial(), + ).fit(tol=1e-11, maxiter=100) + assert fit.converged + out[~train] = fit.predict(sm.add_constant(x[~train])) + assert np.isfinite(out).all() + return out + + +def custom_bootstrap(y, p, corrected, groups): + """Adapter code, NOT a native PyCalEva uncertainty feature.""" + clusters = [np.flatnonzero(groups == g) for g in np.unique(groups)] + rng = np.random.default_rng(BOOT_SEED) + base, gain = [], [] + for _ in range(N_BOOT): + idx = np.concatenate([clusters[k] for k in rng.integers(0, len(clusters), len(clusters))]) + b = pycaleva_brier(y[idx], p[idx]) + base.append(b) + gain.append(b - pycaleva_brier(y[idx], corrected[idx])) + return np.quantile(base, [0.025, 0.975]), np.quantile(gain, [0.025, 0.975]) + + +def coverage_cases(df, out): + case = out / "coverage_inputs" + case.mkdir() + df.to_csv(case / "cohort.csv", index=False, float_format="%.17g") + score = df.groupby("patient_id").p.mean() + excluded = set(score[score >= score.quantile(0.7)].index) + keep = ~df.patient_id.isin(excluded) + a = df[["subject_id", "p"]].rename(columns={"p": "predicted"}).assign(model_id="full") + b = df.loc[keep, ["subject_id", "p"]].rename(columns={"p": "predicted"}) + b = b.assign(model_id="partial") # Identical predictions; only availability differs. + pd.concat([a, b]).to_csv(case / "predictions.csv", index=False, float_format="%.17g") + spec = { + "cohort_id": "controlled_coverage", + "version": 0, + "subject_key": "subject_id", + "data": "cohort.csv", + "outcome": {"type": "binary", "field": "y", "positive_label": 1}, + "clustering": {"field": "patient_id", "name": "patient"}, + "coverage": {"min_fraction": 0.9, "compare_on": "both", "common_warn_frac": 0.2}, + "completeness": {"require_outcome": True, "on_violation": "fail"}, + "expected_models": ["full", "partial", "absent"], + "on_missing_model": "warn", + } + spec_path = case / "cohort.yaml" + spec_path.write_text(yaml.safe_dump(spec)) + cohort, preds = load_cohort(spec_path), load_predictions(case / "predictions.csv") + report = coverage_report(cohort, preds) + report.to_csv(out / "coverage.csv", index=False, float_format="%.17g") + flags = check_roster(cohort, preds) + check_common_selection(cohort, preds) + errors = [] + try: + check_coverage(cohort, preds) + except CoverageViolation as exc: + errors.append( + {"check": "minimum_coverage_90_percent", "status": "rejected", "error": str(exc)} + ) + else: + raise AssertionError("Missing coverage was not rejected") + spec["on_missing_model"] = "fail" + spec_path.write_text(yaml.safe_dump(spec)) + try: + check_roster(load_cohort(spec_path), preds) + except CohortSpecError as exc: + # Avoid run-specific absolute paths in deterministic outputs. + errors.append( + { + "check": "declared_absent_model", + "status": "rejected", + "error": exc.message if hasattr(exc, "message") else str(exc).split("\n")[0], + } + ) + else: + raise AssertionError("Missing declared model was not rejected") + rows = [] + for name, selection in [ + ("full_available", np.ones(len(df), dtype=bool)), + ("common_intersection", keep.to_numpy()), + ]: + for m, value in metrics(df.y.to_numpy()[selection], df.p.to_numpy()[selection]).items(): + rows.append({"subset": name, "metric": m, "value": value, "n": int(selection.sum())}) + pd.DataFrame(rows).to_csv( + out / "coverage_selection_metrics.csv", index=False, float_format="%.17g" + ) + return { + "flags": [f.as_dict() for f in flags], + "errors": errors, + "pycaleva_scope": ( + "Accepts aligned y/p arrays; roster and cohort denominators need caller glue." + ), + } + + +def run(out): + start, cpu = time.monotonic(), time.process_time() + out.mkdir(parents=True, exist_ok=False) + point_rows, correction_rows, interval_rows, summary = [], [], [], {} + for name, df in fixtures().items(): + df.to_csv(out / f"{name}.csv", index=False, float_format="%.17g") + y, p, groups = df.y.to_numpy(), df.p.to_numpy(), df.patient_id.to_numpy() + ce = CalibrationEvaluator(y, p, outsample=True, n_groups=10) + estimates = { + "predval": { + "auroc": pm.auroc(y, p), + "brier": pm.brier(y, p), + "calibration_intercept": pm.calibration_intercept(y, p), + "calibration_slope": pm.calibration_slope(y, p), + }, + "pycaleva": {"auroc": ce.auroc, "brier": ce.brier}, + "independent_reference": metrics(y, p) | reference_calibration(y, p), + } + for tool, values in estimates.items(): + for metric, value in values.items(): + ref = estimates["independent_reference"][metric] + tolerance = 1e-10 if metric in {"auroc", "brier"} else 1e-8 + difference = abs(value - ref) + assert difference < tolerance, (name, tool, metric, difference) + point_rows.append( + { + "fixture": name, + "tool": tool, + "metric": metric, + "value": value, + "abs_diff_reference": difference, + } + ) + _, ladder = pr.ladder(y, p, groups, max_folds=5) + predval_corrected = ladder.crossfit_predictions["rung2"] + reference_corrected = reference_crossfit(df) + max_diff = float(np.max(np.abs(predval_corrected - reference_corrected))) + assert max_diff < 1e-8, max_diff + df["predval_rung2"] = predval_corrected + df["statsmodels_rung2"] = reference_corrected + raw_reference = reference_crossfit(df, logit_scale=False) + df["statsmodels_raw_probability"] = raw_reference + df.to_csv(out / f"{name}_corrected.csv", index=False, float_format="%.17g") + for tool, corrected in [ + ("predval_rung2", predval_corrected), + ("statsmodels_custom_grouped", reference_corrected), + ("statsmodels_raw_probability", raw_reference), + ]: + for metric, value in metrics(y, corrected).items(): + correction_rows.append( + {"fixture": name, "tool": tool, "metric": metric, "value": value} + ) + point, ci = pu.paired_brier_gain_interval( + y, p, predval_corrected, groups, 0.95, n_boot=N_BOOT, seed=BOOT_SEED + ) + for scheme, g in [("patient", groups), ("row", np.arange(len(df)))]: + base, gain = custom_bootstrap(y, p, predval_corrected, g) + for metric, interval in [("brier", base), ("paired_brier_gain", gain)]: + interval_rows.append( + { + "fixture": name, + "tool": "pycaleva_plus_custom_bootstrap", + "scheme": scheme, + "metric": metric, + "low": interval[0], + "high": interval[1], + "width": interval[1] - interval[0], + } + ) + if scheme == "patient": + assert np.allclose(gain, [ci.low, ci.high], atol=1e-12, rtol=0) + interval_rows.append( + { + "fixture": name, + "tool": "predval", + "scheme": "patient", + "metric": "paired_brier_gain", + "low": ci.low, + "high": ci.high, + "width": ci.high - ci.low, + } + ) + summary[name] = { + "rows": len(df), + "patients": len(np.unique(groups)), + "events": int(y.sum()), + "max_corrected_prediction_difference": max_diff, + "brier_gain": point, + "brier_gain_ci": [ci.low, ci.high], + "suppressed_ladder_rungs": ladder.suppressed, + } + for name, rows in [ + ("point_metrics", point_rows), + ("correction_metrics", correction_rows), + ("intervals", interval_rows), + ]: + pd.DataFrame(rows).to_csv(out / f"{name}.csv", index=False, float_format="%.17g") + summary["coverage"] = coverage_cases(fixtures()["synthetic_clustered"], out) + summary["scope"] = "Executed: predval and PyCalEva; statsmodels/sklearn independent reference." + (out / "summary.json").write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n") + packages = ["predval", "pycaleva", "numpy", "pandas", "scipy", "scikit-learn", "statsmodels"] + manifest = { + "python": platform.python_version(), + "versions": {p: importlib.metadata.version(p) for p in packages}, + "seed": SEED, + "bootstrap_seed": BOOT_SEED, + "bootstrap_replicates": N_BOOT, + "protocol_sha256": digest(HERE / "PROTOCOL.md"), + "files": { + str(p.relative_to(out)): digest(p) for p in sorted(out.rglob("*")) if p.is_file() + }, + } + (out / "manifest.json").write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n") + usage = { + "wall_seconds": time.monotonic() - start, + "cpu_seconds": time.process_time() - cpu, + "max_rss_kib": resource.getrusage(resource.RUSAGE_SELF).ru_maxrss, + "note": ( + "Benchmark body, not a fair tool runtime comparison; excludes imports and installation." + ), + } + (out / "resource_usage.json").write_text(json.dumps(usage, indent=2) + "\n") + print(json.dumps(summary, indent=2)) + print(json.dumps(usage)) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--out", type=Path, required=True) + run(parser.parse_args().out) diff --git a/benchmarks/alternatives/tests/test_benchmark.py b/benchmarks/alternatives/tests/test_benchmark.py new file mode 100644 index 0000000..b95110e --- /dev/null +++ b/benchmarks/alternatives/tests/test_benchmark.py @@ -0,0 +1,69 @@ +"""Benchmark contract/failure tests; no optional competitor executes on import.""" + +import importlib.util +from pathlib import Path + +import numpy as np +import pytest +from pycaleva import CalibrationEvaluator + +SOURCE = Path(__file__).resolve().parents[1] / "run_benchmark.py" +spec = importlib.util.spec_from_file_location("benchmark", SOURCE) +bm = importlib.util.module_from_spec(spec) +spec.loader.exec_module(bm) + + +def test_fixtures_are_deterministic_and_patient_folds_disjoint(): + first, second = bm.fixtures(), bm.fixtures() + assert len(first["wdbc_heldout"]) == 285 + assert first["synthetic_clustered"].patient_id.nunique() == 120 + for name, df in first.items(): + assert df.equals(second[name]) + assert df.groupby("patient_id").fold.nunique().eq(1).all() + for fold in sorted(df.fold.unique()): + train = df[df.fold != fold] + test = df[df.fold == fold] + assert set(train.patient_id).isdisjoint(test.patient_id) + assert set(train.y) == {0, 1} + + +def test_crossfit_matches_independent_glm(): + for df in bm.fixtures().values(): + _, report = bm.pr.ladder(df.y.to_numpy(), df.p.to_numpy(), df.patient_id.to_numpy()) + assert np.isfinite(bm.reference_crossfit(df, logit_scale=False)).all() + assert np.allclose( + report.crossfit_predictions["rung2"], bm.reference_crossfit(df), atol=1e-8, rtol=0 + ) + + +def test_missing_coverage_and_absent_models_reject(tmp_path): + results = bm.coverage_cases(bm.fixtures()["synthetic_clustered"], tmp_path) + assert {f["code"] for f in results["flags"]} == { + "declared_but_absent", + "informative_common_selection", + } + assert all(e["status"] == "rejected" for e in results["errors"]) + assert len(results["errors"]) == 2 + + +def test_pycaleva_rejects_out_of_range_probabilities(): + with pytest.raises(ValueError): + CalibrationEvaluator(np.array([0, 1, 0, 1]), np.array([0.1, 1.2, 0.3, 0.9]), True) + + +def test_invalid_crossfit_is_unavailable_not_fabricated(): + # Every patient is one class, so holding either of the two patients out + # leaves a single-class training fold. PREDVAL must withhold fitted rungs. + y = np.array([0, 0, 1, 1]) + p = np.array([0.1, 0.2, 0.7, 0.8]) + _, report = bm.pr.ladder(y, p, np.array(["a", "a", "b", "b"]), max_folds=2) + assert report.crossfit_predictions == {} + assert len(report.suppressed) == 3 + + +def test_quasi_separation_must_not_return_finite_converged_slope(): + y = np.array([0, 0, 1, 1]) + p = np.array([0.2, 0.5, 0.5, 0.8]) + # This quasi-separated likelihood has no finite slope MLE. Exhausting the + # IRLS iteration budget must mark the slope unavailable. + assert not np.isfinite(bm.pm.calibration_slope(y, p)) diff --git a/benchmarks/alternatives/verification/benchmark-contract-tests.log b/benchmarks/alternatives/verification/benchmark-contract-tests.log new file mode 100644 index 0000000..e4b1d08 --- /dev/null +++ b/benchmarks/alternatives/verification/benchmark-contract-tests.log @@ -0,0 +1,10 @@ +.....x [100%] +==================================== PASSES ==================================== +=========================== short test summary info ============================ +PASSED benchmarks/alternatives/tests/test_benchmark.py::test_fixtures_are_deterministic_and_patient_folds_disjoint +PASSED benchmarks/alternatives/tests/test_benchmark.py::test_crossfit_matches_independent_glm +PASSED benchmarks/alternatives/tests/test_benchmark.py::test_missing_coverage_and_absent_models_reject +PASSED benchmarks/alternatives/tests/test_benchmark.py::test_pycaleva_rejects_out_of_range_probabilities +PASSED benchmarks/alternatives/tests/test_benchmark.py::test_invalid_crossfit_is_unavailable_not_fabricated +XFAIL benchmarks/alternatives/tests/test_benchmark.py::test_quasi_separation_must_not_return_finite_converged_slope - Known PREDVAL IRLS iteration-limit/quasi-separation defect +5 passed, 1 xfailed in 8.12s diff --git a/benchmarks/alternatives/verification/integrated/benchmark-contract-tests.log b/benchmarks/alternatives/verification/integrated/benchmark-contract-tests.log new file mode 100644 index 0000000..fd9f43c --- /dev/null +++ b/benchmarks/alternatives/verification/integrated/benchmark-contract-tests.log @@ -0,0 +1,2 @@ +...... [100%] +6 passed in 7.14s diff --git a/benchmarks/alternatives/verification/integrated/diff-check.log b/benchmarks/alternatives/verification/integrated/diff-check.log new file mode 100644 index 0000000..e69de29 diff --git a/benchmarks/alternatives/verification/integrated/format.log b/benchmarks/alternatives/verification/integrated/format.log new file mode 100644 index 0000000..4bf16a1 --- /dev/null +++ b/benchmarks/alternatives/verification/integrated/format.log @@ -0,0 +1 @@ +63 files already formatted diff --git a/benchmarks/alternatives/verification/integrated/pytest.log b/benchmarks/alternatives/verification/integrated/pytest.log new file mode 100644 index 0000000..32c46c4 --- /dev/null +++ b/benchmarks/alternatives/verification/integrated/pytest.log @@ -0,0 +1,24 @@ +........................................................................ [ 23%] +................sss..................................................... [ 47%] +......................sssssssssssss..............ssss................... [ 71%] +........................................................................ [ 95%] +.............. [100%] +=========================== short test summary info ============================ +SKIPPED [1] tests/test_golden_gusto.py:69: GUSTO fixture not built; run examples/gusto/fetch_data.py + make_predictions.py +SKIPPED [1] tests/test_golden_improve.py:67: IMPROVE fixture not built; run examples/improve/build_fixture.py +SKIPPED [1] tests/test_golden_pcam.py:69: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:40: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:46: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:52: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:58: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:63: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:68: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:73: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:79: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:85: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:92: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:97: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:108: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:144: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [4] tests/test_predrupdate_agreement.py:99: GUSTO fixture not built; run examples/gusto/fetch_data.py + make_predictions.py +282 passed, 20 skipped in 167.91s (0:02:47) diff --git a/benchmarks/alternatives/verification/integrated/python-a.log b/benchmarks/alternatives/verification/integrated/python-a.log new file mode 100644 index 0000000..c56c1e7 --- /dev/null +++ b/benchmarks/alternatives/verification/integrated/python-a.log @@ -0,0 +1,61 @@ +Matplotlib is building the font cache; this may take a moment. +{ + "wdbc_heldout": { + "rows": 285, + "patients": 285, + "events": 106, + "max_corrected_prediction_difference": 3.785860513971784e-14, + "brier_gain": 0.002006686710238758, + "brier_gain_ci": [ + -0.0053364764046153375, + 0.009427078330354059 + ], + "suppressed_ladder_rungs": [ + [ + "rung3", + "non-convergence" + ] + ] + }, + "synthetic_clustered": { + "rows": 1332, + "patients": 120, + "events": 560, + "max_corrected_prediction_difference": 5.9119376061289586e-15, + "brier_gain": 0.010868603906738982, + "brier_gain_ci": [ + -0.0011632227475880895, + 0.0221673659276281 + ], + "suppressed_ladder_rungs": [] + }, + "coverage": { + "flags": [ + { + "code": "declared_but_absent", + "severity": "warning", + "message": "declared in expected_models but absent from predictions: ['absent']" + }, + { + "code": "informative_common_selection", + "severity": "warning", + "message": "the common subset excludes 31.0% of cohort rows (> 20%); selection into the common subset may be informative" + } + ], + "errors": [ + { + "check": "minimum_coverage_90_percent", + "status": "rejected", + "error": "1 model(s) fall below coverage.min_fraction; worst shown\n model: partial\n coverage: 0.689940\n required: 0.900000" + }, + { + "check": "declared_absent_model", + "status": "rejected", + "error": "declared in expected_models but absent from predictions: ['absent']" + } + ], + "pycaleva_scope": "Accepts aligned y/p arrays; roster and cohort denominators need caller glue." + }, + "scope": "Executed: predval and PyCalEva; statsmodels/sklearn independent reference." +} +{"wall_seconds": 3.85623039300026, "cpu_seconds": 3.6602351980000005, "max_rss_kib": 291780, "note": "Benchmark body, not a fair tool runtime comparison; excludes imports and installation."} diff --git a/benchmarks/alternatives/verification/integrated/python-b.log b/benchmarks/alternatives/verification/integrated/python-b.log new file mode 100644 index 0000000..7e9a490 --- /dev/null +++ b/benchmarks/alternatives/verification/integrated/python-b.log @@ -0,0 +1,60 @@ +{ + "wdbc_heldout": { + "rows": 285, + "patients": 285, + "events": 106, + "max_corrected_prediction_difference": 3.785860513971784e-14, + "brier_gain": 0.002006686710238758, + "brier_gain_ci": [ + -0.0053364764046153375, + 0.009427078330354059 + ], + "suppressed_ladder_rungs": [ + [ + "rung3", + "non-convergence" + ] + ] + }, + "synthetic_clustered": { + "rows": 1332, + "patients": 120, + "events": 560, + "max_corrected_prediction_difference": 5.9119376061289586e-15, + "brier_gain": 0.010868603906738982, + "brier_gain_ci": [ + -0.0011632227475880895, + 0.0221673659276281 + ], + "suppressed_ladder_rungs": [] + }, + "coverage": { + "flags": [ + { + "code": "declared_but_absent", + "severity": "warning", + "message": "declared in expected_models but absent from predictions: ['absent']" + }, + { + "code": "informative_common_selection", + "severity": "warning", + "message": "the common subset excludes 31.0% of cohort rows (> 20%); selection into the common subset may be informative" + } + ], + "errors": [ + { + "check": "minimum_coverage_90_percent", + "status": "rejected", + "error": "1 model(s) fall below coverage.min_fraction; worst shown\n model: partial\n coverage: 0.689940\n required: 0.900000" + }, + { + "check": "declared_absent_model", + "status": "rejected", + "error": "declared in expected_models but absent from predictions: ['absent']" + } + ], + "pycaleva_scope": "Accepts aligned y/p arrays; roster and cohort denominators need caller glue." + }, + "scope": "Executed: predval and PyCalEva; statsmodels/sklearn independent reference." +} +{"wall_seconds": 5.465970102000028, "cpu_seconds": 5.272218276, "max_rss_kib": 293760, "note": "Benchmark body, not a fair tool runtime comparison; excludes imports and installation."} diff --git a/benchmarks/alternatives/verification/integrated/ruff.log b/benchmarks/alternatives/verification/integrated/ruff.log new file mode 100644 index 0000000..1f5f344 --- /dev/null +++ b/benchmarks/alternatives/verification/integrated/ruff.log @@ -0,0 +1 @@ +All checks passed! diff --git a/benchmarks/alternatives/verification/integrated/status.json b/benchmarks/alternatives/verification/integrated/status.json new file mode 100644 index 0000000..9aa2d6a --- /dev/null +++ b/benchmarks/alternatives/verification/integrated/status.json @@ -0,0 +1,125 @@ +{ + "benchmark_tests_separate": { + "passed": 6, + "seconds": 7.14, + "skipped": 0, + "xfailed": 0 + }, + "cherry_picked_guard_commit": "617ddac480849ac48de41c1c1b1d050ddc427492", + "combined_tests": { + "command": "python -m pytest tests benchmarks/alternatives/tests -ra", + "failed": 0, + "passed": 282, + "seconds": 167.91, + "skip_reason": "Unavailable PCam, GUSTO, and IMPROVE optional example/golden fixtures.", + "skipped": 20, + "xfailed": 0, + "xpassed": 0 + }, + "date": "2026-10-10", + "lint": { + "formatted_file_count": 63, + "git_diff_check": "passed", + "ruff_check": "passed", + "ruff_format_check": "passed" + }, + "numerical_outcome": "No change in either main benchmark fixture; all 13 stable input/result files match the original in each of two reruns.", + "original_comparator_commit": "868276ef960f0bc7f2bc75cc79926ff7b3c2e1f1", + "original_convergence_guard_commit": "62c8af368824d75e1dd915e78c131f46e333a340", + "original_protocol_result_verification_files": [ + "benchmarks/alternatives/PROTOCOL.md", + "benchmarks/alternatives/results/glue_inventory.json", + "benchmarks/alternatives/results/python/correction_metrics.csv", + "benchmarks/alternatives/results/python/coverage.csv", + "benchmarks/alternatives/results/python/coverage_selection_metrics.csv", + "benchmarks/alternatives/results/python/intervals.csv", + "benchmarks/alternatives/results/python/manifest.json", + "benchmarks/alternatives/results/python/point_metrics.csv", + "benchmarks/alternatives/results/python/resource_usage.json", + "benchmarks/alternatives/results/python/summary.json", + "benchmarks/alternatives/results/r/r_agreement.csv", + "benchmarks/alternatives/results/r/r_bootstrap_summary.csv", + "benchmarks/alternatives/results/r/r_metrics.csv", + "benchmarks/alternatives/results/r/resource_usage.txt", + "benchmarks/alternatives/results/r/sessionInfo.txt", + "benchmarks/alternatives/results/r/verification.json", + "benchmarks/alternatives/results/reproducibility.json", + "benchmarks/alternatives/verification/benchmark-contract-tests.log", + "benchmarks/alternatives/verification/predval-pytest.log", + "benchmarks/alternatives/verification/r-parse.log", + "benchmarks/alternatives/verification/r-repeat.log", + "benchmarks/alternatives/verification/r-run.log", + "benchmarks/alternatives/verification/repeat-check.log", + "benchmarks/alternatives/verification/ruff.log", + "benchmarks/alternatives/verification/status.json" + ], + "original_protocol_result_verification_files_preserved": 25, + "patient_partition_guard": "99ca3d72f1d4f8340df9a59de3616277e9ebdb7a", + "python_environment_note": "Two existing isolated Python 3.12.14 environments created from requirements-lock.txt for the original benchmark were reused. They were not freshly installed for integration. PYTHONPATH selected this integration worktree; both imports were verified.", + "python_reruns": [ + { + "all_stable_hashes_match_original": true, + "manifest_sha256": "a2a541ec30fbffe83103e04d2c85a5e83114a671e9818488e896a2f34eaf510d", + "resource_usage": { + "cpu_seconds": 3.6602351980000005, + "max_rss_kib": 291780, + "note": "Benchmark body, not a fair tool runtime comparison; excludes imports and installation.", + "wall_seconds": 3.85623039300026 + }, + "run": "python-a", + "stable_file_count": 13 + }, + { + "all_stable_hashes_match_original": true, + "manifest_sha256": "a2a541ec30fbffe83103e04d2c85a5e83114a671e9818488e896a2f34eaf510d", + "resource_usage": { + "cpu_seconds": 5.272218276, + "max_rss_kib": 293760, + "note": "Benchmark body, not a fair tool runtime comparison; excludes imports and installation.", + "wall_seconds": 5.465970102000028 + }, + "run": "python-b", + "stable_file_count": 13 + } + ], + "python_stable_file_sha256": { + "correction_metrics.csv": "810d078fed3b8b96c69751ad2c04333e9730e13f64c2d7a9811faa4becacbd8a", + "coverage.csv": "3a6aa0df1d5da3aea520a81e7b1ff5287d12b1bad3ba5d36cbe90955061f8ca5", + "coverage_inputs/cohort.csv": "50127a122204e8fcbae05f4cf597e77f210791060015bbe120c837edc7f421c2", + "coverage_inputs/cohort.yaml": "fd3789802a5c4884f3d05a87ceb67bc67b5fbc97f9c736efe739f135780cf548", + "coverage_inputs/predictions.csv": "effe905a9fdc1117bf6d50f6bdfd9c20857552597d440259d37eb0accba5bc96", + "coverage_selection_metrics.csv": "2633e304ad3c482de0753cefcfce9c5c6835fc6882f0518310b2dc63d1f38712", + "intervals.csv": "65b5b3fc24cf9c603f7ca421d74b4a3353d8b75d92e2a7817d6428b125e44482", + "point_metrics.csv": "ba5cb17b04f400541e09b3c4b0108e41a211e75709707b55c9d4efee29f62480", + "summary.json": "03300c0306c33c4541e437b87426735a3bba16cabc2808a3144b282f54f41ac9", + "synthetic_clustered.csv": "50127a122204e8fcbae05f4cf597e77f210791060015bbe120c837edc7f421c2", + "synthetic_clustered_corrected.csv": "1bb4899c68ad53387f47804f6da41899c76ff5fc87165e5118cfd60a19e78568", + "wdbc_heldout.csv": "61a22060cc9072ef62ad74f7db619b982e8eeb3130c07d4cc275d602aa1ccd9c", + "wdbc_heldout_corrected.csv": "b47c07f8301ba3d7bac38a4ad9396593b0342036d71dabe1970d3630d5ce80c2" + }, + "r_evidence": { + "execution_repeated_for_integration": false, + "original_stable_file_count_verified": 7, + "original_stable_file_sha256": { + "r_agreement.csv": "2c1e306addc003d941c80f84ad0f2e1ae476902e42eb5a8498152f147600f4bd", + "r_metrics.csv": "7a8d558a65c8d61d43b04406fbf1d46c5c18126e6fc5a2ebd1818fe9dfde0c59", + "sessionInfo.txt": "54be6cde818461ff3da62ff156022848d00cab4ae131e30af1c4c1ef720b3039", + "synthetic_clustered_r_corrected.csv": "5c13c21f257107c6893d001dc80388d1b19ab92448dadf951982e9358414cba6", + "synthetic_clustered_rsample_gains.csv": "bbedbfec1f6085cb31290b43f84feaf1dff7c6049cb1d276cccd2f26c24a3ccb", + "wdbc_heldout_r_corrected.csv": "8bc8df31c97730d9e2acd8671d2d0ae9a0816f9a1580b8e615129dacccacaf3f", + "wdbc_heldout_rsample_gains.csv": "3f1fc4df7f404beedc6255da517ee6e4bd634ff2588833f9299be0bff96237dd" + }, + "reason": "The unchanged R adapter consumes byte-identical Python fixture/corrected-prediction inputs. Historical R evidence is reused and explicitly labeled; no new R execution is claimed." + }, + "remote_actions": [], + "scope": "Local integration revalidation; methods/engineering evidence, not clinical or statistical validation.", + "source_sha256": { + "benchmarks/alternatives/PROTOCOL.md": "8630bb9af1f54eb4a7f8e7bacfa5a652dd4915dc60e1e319ac02105cbf920db1", + "benchmarks/alternatives/probably_adapter.R": "b81a00594a0e7407b4af5f023a01be594353f30bc16a090ac1f16d39aa4723e3", + "benchmarks/alternatives/run_benchmark.py": "a62a5f77c04e74ba79d9a043c98f4f2be37dd651112a49cc76b5d5fcbf360375", + "benchmarks/alternatives/tests/test_benchmark.py": "6b2270052c2df675b94638c7bff18b6d14ed64afb44412d1f4c89393449e2d8f", + "src/predval/metrics.py": "5a0f13aee0e1a4de17611f5aa13ff05421382af478b9fff788fd77132d9b81e6" + }, + "statistical_stress_integrated": false, + "tested_source_note": "The code at the cherry-picked guard commit, with the comparator strict xfail removed; documentation and evidence records do not change source code." +} diff --git a/benchmarks/alternatives/verification/predval-pytest.log b/benchmarks/alternatives/verification/predval-pytest.log new file mode 100644 index 0000000..9764441 --- /dev/null +++ b/benchmarks/alternatives/verification/predval-pytest.log @@ -0,0 +1,5 @@ +........................................................................ [ 24%] +................sss..................................................... [ 49%] +.................sssssssssssss..............ssss........................ [ 74%] +........................................................................ [ 99%] +. [100%] diff --git a/benchmarks/alternatives/verification/r-parse.log b/benchmarks/alternatives/verification/r-parse.log new file mode 100644 index 0000000..a752f4e --- /dev/null +++ b/benchmarks/alternatives/verification/r-parse.log @@ -0,0 +1 @@ +R adapter syntax and output-whitespace normalization: passed diff --git a/benchmarks/alternatives/verification/r-repeat.log b/benchmarks/alternatives/verification/r-repeat.log new file mode 100644 index 0000000..1e6e0f0 --- /dev/null +++ b/benchmarks/alternatives/verification/r-repeat.log @@ -0,0 +1,3 @@ +Registered S3 method overwritten by 'butcher': + method from + as.character.dev_topic generics diff --git a/benchmarks/alternatives/verification/r-run.log b/benchmarks/alternatives/verification/r-run.log new file mode 100644 index 0000000..1e6e0f0 --- /dev/null +++ b/benchmarks/alternatives/verification/r-run.log @@ -0,0 +1,3 @@ +Registered S3 method overwritten by 'butcher': + method from + as.character.dev_topic generics diff --git a/benchmarks/alternatives/verification/repeat-check.log b/benchmarks/alternatives/verification/repeat-check.log new file mode 100644 index 0000000..44e782a --- /dev/null +++ b/benchmarks/alternatives/verification/repeat-check.log @@ -0,0 +1,17 @@ +{ + "python_stable_files_verified": 13, + "python_manifest_sha256": "a2a541ec30fbffe83103e04d2c85a5e83114a671e9818488e896a2f34eaf510d", + "r_repeated_stable_files_identical": 7, + "r_files_sha256": { + "r_metrics.csv": "7a8d558a65c8d61d43b04406fbf1d46c5c18126e6fc5a2ebd1818fe9dfde0c59", + "r_agreement.csv": "2c1e306addc003d941c80f84ad0f2e1ae476902e42eb5a8498152f147600f4bd", + "sessionInfo.txt": "54be6cde818461ff3da62ff156022848d00cab4ae131e30af1c4c1ef720b3039", + "wdbc_heldout_r_corrected.csv": "8bc8df31c97730d9e2acd8671d2d0ae9a0816f9a1580b8e615129dacccacaf3f", + "wdbc_heldout_rsample_gains.csv": "3f1fc4df7f404beedc6255da517ee6e4bd634ff2588833f9299be0bff96237dd", + "synthetic_clustered_r_corrected.csv": "5c13c21f257107c6893d001dc80388d1b19ab92448dadf951982e9358414cba6", + "synthetic_clustered_rsample_gains.csv": "bbedbfec1f6085cb31290b43f84feaf1dff7c6049cb1d276cccd2f26c24a3ccb" + }, + "r_timing_first": "user system elapsed\n138.449 3.509 142.258", + "r_timing_repeat": "user system elapsed\n181.453 4.849 187.484", + "scope": "Same final inputs and fixed dependencies. Runtime excluded from hash equality." +} diff --git a/benchmarks/alternatives/verification/ruff.log b/benchmarks/alternatives/verification/ruff.log new file mode 100644 index 0000000..1f5f344 --- /dev/null +++ b/benchmarks/alternatives/verification/ruff.log @@ -0,0 +1 @@ +All checks passed! diff --git a/benchmarks/alternatives/verification/status.json b/benchmarks/alternatives/verification/status.json new file mode 100644 index 0000000..e87158f --- /dev/null +++ b/benchmarks/alternatives/verification/status.json @@ -0,0 +1,21 @@ +{ + "predval_base_commit": "99ca3d72f1d4f8340df9a59de3616277e9ebdb7a", + "core_code_changed": false, + "existing_suite": { + "passed": 269, + "skipped": 20, + "failure_count": 0, + "note": "Optional example/golden fixture checks skipped; not claimed as passed." + }, + "benchmark_contracts": { + "passed": 5, + "expected_failures": 1, + "note": "Strict xfail: preexisting IRLS quasi-separation/iteration-limit defect." + }, + "ruff": "passed", + "git_diff_check": "passed", + "python_reproduction": "two freshly created isolated environments; stable hashes identical", + "r_numerics": "native probably/grouped rsample and independent controls executed; see r_agreement.csv", + "isitfair": "source-reviewed; no competitor source executed", + "r_reproduction": "Same final inputs; seven stable outputs byte-identical across two runs" +} diff --git a/benchmarks/alternatives/verify_results.py b/benchmarks/alternatives/verify_results.py new file mode 100644 index 0000000..224da99 --- /dev/null +++ b/benchmarks/alternatives/verify_results.py @@ -0,0 +1,63 @@ +#!/usr/bin/env python3 +"""Verify clean repeats and emit compact, regenerable R summaries.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +from pathlib import Path + +import numpy as np +import pandas as pd + + +def sha(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def verify(py: Path, py_repeat: Path, r: Path, r_repeat: Path): + assert (py / "manifest.json").read_bytes() == (py_repeat / "manifest.json").read_bytes() + files = json.loads((py / "manifest.json").read_text())["files"] + for relative, expected in files.items(): + assert sha(py / relative) == sha(py_repeat / relative) == expected + shared_r = ["r_metrics.csv", "r_agreement.csv", "sessionInfo.txt"] + summary = [] + for name in ("wdbc_heldout", "synthetic_clustered"): + shared_r.extend([f"{name}_r_corrected.csv", f"{name}_rsample_gains.csv"]) + gains = pd.read_csv(r / f"{name}_rsample_gains.csv").gain.to_numpy() + assert len(gains) == 1000 and np.isfinite(gains).all() + lo, hi = np.quantile(gains, [0.025, 0.975]) + summary.append( + { + "fixture": name, + "replicates": len(gains), + "low": lo, + "high": hi, + "method": "rsample native group_bootstraps; fixed R logit-GLM predictions", + } + ) + for name in shared_r: + assert (r / name).read_bytes() == (r_repeat / name).read_bytes(), name + pd.DataFrame(summary).to_csv(r / "r_bootstrap_summary.csv", index=False, float_format="%.17g") + result = { + "python_stable_files_verified": len(files), + "python_manifest_sha256": sha(py / "manifest.json"), + "r_repeated_stable_files_identical": len(shared_r), + "r_files_sha256": {name: sha(r / name) for name in shared_r}, + "r_timing_first": (r / "resource_usage.txt").read_text().strip(), + "r_timing_repeat": (r_repeat / "resource_usage.txt").read_text().strip(), + "scope": "Same final inputs and fixed dependencies. Runtime excluded from hash equality.", + } + (r / "verification.json").write_text(json.dumps(result, indent=2, sort_keys=True) + "\n") + print(json.dumps(result, indent=2)) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("python", type=Path) + parser.add_argument("python_repeat", type=Path) + parser.add_argument("r", type=Path) + parser.add_argument("r_repeat", type=Path) + args = parser.parse_args() + verify(args.python, args.python_repeat, args.r, args.r_repeat) diff --git a/benchmarks/family_extension/PROTOCOL.md b/benchmarks/family_extension/PROTOCOL.md new file mode 100644 index 0000000..94e3c54 --- /dev/null +++ b/benchmarks/family_extension/PROTOCOL.md @@ -0,0 +1,80 @@ +# Prospective local family extension, 2026-10-10 + +This extension follows inspection of the original two fixtures and their results. It is +not a retroactive preregistration of those results. This protocol is committed locally +before executing the new extension. Local Git chronology is an audit trail, not independent +proof of preregistration or a trusted external timestamp. Baseline: integrated `150998f`. +Historical benchmarks/alternatives inputs, protocol and results remain untouched. + +## Fixed design + +Use 20 seeds, the consecutive integers 20261020 through 20261039 inclusive. For each seed: +120 independent patient clusters, uniform integer sizes 4–20 inclusive; patient effect +Normal(0,1.2), independent row covariate Normal(0,1), eta=-0.8+0.9*x+patient_effect. +Published p=expit(-0.4+1.4*eta). Share cluster sizes, covariates, p and uniform outcome +randomness between the two generating families: + +- logit_linear: q=expit(eta), so logit(q)=(logit(p)+0.4)/1.4; +- probability_linear: q=expit(-2.5+5*p), so logit(q) is linear in p. + +Outcome uniforms select a patient-shared uniform with probability 0.65, otherwise an +independent row uniform. y=1[uniform= 0) & (prediction <= 1)).all() + ): + raise ValueError("nonfinite or out-of-range prediction") + predictions[method] = prediction + statuses[method] = {"available": True, "reason": None, **extra} + except (ValueError, RuntimeError, np.linalg.LinAlgError) as exc: + statuses[method] = { + "available": False, + "reason": f"{type(exc).__name__}: {exc}", + **extra, + } + statuses[method]["warnings"] = [f"{type(w.message).__name__}: {w.message}" for w in caught] + if "predval_rung2" in predictions and "statsmodels_logit" in predictions: + delta = float( + np.max(np.abs(predictions["predval_rung2"] - predictions["statsmodels_logit"])) + ) + statuses["predval_rung2"]["max_abs_independent_difference"] = delta + assert delta < 1e-8, delta + for method, prediction in predictions.items(): + statuses[method]["prediction_sha256"] = hashlib.sha256( + np.asarray(prediction, dtype=" 0 else None, + ) + return result + + +def mean_precision(values): + a = np.asarray(values, dtype=float) + n = len(a) + if n == 0: + return {"n": 0, "mean": None, "mcse": None, "mc95_low": None, "mc95_high": None} + mean = float(a.mean()) + se = float(a.std(ddof=1) / np.sqrt(n)) if n > 1 else None + half = float(t.ppf(0.975, n - 1) * se) if se is not None else None + return { + "n": n, + "mean": mean, + "mcse": se, + "mc95_low": mean - half if half is not None else None, + "mc95_high": mean + half if half is not None else None, + } + + +def summarize(rows): + synthetic = [r for r in rows if r["family"] != "wdbc_undistorted"] + summaries, contrasts = [], [] + for family in ("logit_linear", "probability_linear"): + for method in METHODS: + subset = [r for r in synthetic if r["family"] == family and r["method"] == method] + for metric in OUTCOMES: + values = [r[metric] for r in subset if r["available"] and r.get(metric) is not None] + summaries.append( + { + "family": family, + "method": method, + "metric": metric, + "attempted": len(subset), + **mean_precision(values), + } + ) + for metric in ("brier", "auroc", "expected_brier", "risk_mse"): + values = [] + for seed in SEEDS: + pair = { + r["method"]: r for r in synthetic if r["family"] == family and r["seed"] == seed + } + a, b = pair["predval_rung2"], pair["statsmodels_probability"] + if ( + a["available"] + and b["available"] + and a.get(metric) is not None + and b.get(metric) is not None + ): + values.append(a[metric] - b[metric]) + contrasts.append( + { + "family": family, + "contrast": "predval_minus_probability_glm", + "metric": metric, + "attempted": len(SEEDS), + **mean_precision(values), + } + ) + return { + "simulation_summary": summaries, + "paired_method_contrasts": contrasts, + "limits": ( + "20 seeds; maximum binomial MCSE about 0.112. " + "Monte Carlo intervals are not patient CIs." + ), + } + + +def run(out): + started = time.monotonic() + out.mkdir(parents=True, exist_ok=False) + rows, cases = [], [] + fixtures = [(seed, name, d) for seed in SEEDS for name, d in paired_fixtures(seed).items()] + fixtures.append((20261010, "wdbc_undistorted", wdbc_control())) + for seed, family, d in fixtures: + predictions, statuses = fit_methods(d) + input_hash = hashlib.sha256( + d.to_csv(index=False, float_format="%.17g").encode() + ).hexdigest() + cases.append( + {"seed": seed, "family": family, "input_sha256": input_hash, "statuses": statuses} + ) + for method in METHODS: + row = { + "seed": seed, + "family": family, + "method": method, + "rows": len(d), + "clusters": int(d.patient.nunique()), + "events": int(d.y.sum()), + "available": statuses[method]["available"], + "reason": statuses[method]["reason"], + } + if method in predictions: + row.update(score(d, predictions[method])) + rows.append(row) + for name, value in ( + ("metrics.json", rows), + ("cases.json", cases), + ("summary.json", summarize(rows)), + ): + (out / name).write_text(json.dumps(value, indent=2, sort_keys=True, allow_nan=False) + "\n") + pd.DataFrame(rows).to_csv(out / "metrics.csv", index=False, float_format="%.17g") + manifest = { + "python": platform.python_version(), + "seeds": SEEDS, + "versions": { + p: importlib.metadata.version(p) + for p in ("numpy", "scipy", "pandas", "statsmodels", "scikit-learn", "predval") + }, + "protocol_sha256": sha(HERE / "PROTOCOL.md"), + "runner_sha256": sha(Path(__file__)), + "predval_source": str(Path(recalibrate.__file__).resolve()), + "recalibrate_sha256": sha(Path(recalibrate.__file__)), + "files": {p.name: sha(p) for p in sorted(out.iterdir())}, + } + (out / "manifest.json").write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n") + (out / "runtime.json").write_text( + json.dumps( + { + "seconds": time.monotonic() - started, + "scope": "combined workload, excludes imports/install", + }, + indent=2, + ) + + "\n" + ) + print( + json.dumps( + { + "cases": len(cases), + "method_cases": len(rows), + "unavailable": sum(not r["available"] for r in rows), + "out": str(out), + }, + indent=2, + ) + ) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--out", required=True, type=Path) + run(parser.parse_args().out) diff --git a/benchmarks/family_extension/tests/test_extension.py b/benchmarks/family_extension/tests/test_extension.py new file mode 100644 index 0000000..f1145b6 --- /dev/null +++ b/benchmarks/family_extension/tests/test_extension.py @@ -0,0 +1,95 @@ +"""Independent contract checks for the prospective family extension.""" + +import importlib.util +from pathlib import Path + +import numpy as np +import pytest +from scipy.special import expit, logit + +SPEC = importlib.util.spec_from_file_location( + "extension", Path(__file__).parents[1] / "run_extension.py" +) +ex = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(ex) + + +def test_fixed_seed_budget(): + assert ex.SEEDS == tuple(range(20261020, 20261040)) + + +def test_shared_design_and_families(): + pair = ex.paired_fixtures(ex.SEEDS[0]) + a, b = pair.values() + for column in ("p", "patient", "fold"): + np.testing.assert_array_equal(a[column], b[column]) + np.testing.assert_allclose(a.q, expit((logit(a.p) + 0.4) / 1.4), atol=1e-14) + np.testing.assert_allclose(logit(b.q), -2.5 + 5 * b.p, atol=1e-14) + # Same outcome uniforms imply monotonic coupling when a row's q changes. + assert np.all(a.y[a.q <= b.q] <= b.y[a.q <= b.q]) + assert np.all(a.y[a.q >= b.q] >= b.y[a.q >= b.q]) + assert a.patient.nunique() == 120 + assert a.groupby("patient").size().between(4, 20).all() + + +def test_grouped_partitions_match_native(): + d = ex.paired_fixtures(ex.SEEDS[1])["logit_linear"] + native = ex.recalibrate._grouped_folds(d.patient.to_numpy(), 5) + for f, rows in enumerate(native): + np.testing.assert_array_equal(rows, np.flatnonzero(d.fold.to_numpy() == f)) + assert not set(d.patient.iloc[rows]) & set(d.patient[d.fold != f]) + + +def test_expected_loss_identity_and_oracle(): + d = ex.paired_fixtures(ex.SEEDS[2])["probability_linear"] + p = np.full(len(d), 0.4) + actual = ex.score(d, p) + direct = np.mean(d.q * (1 - p) ** 2 + (1 - d.q) * p**2) + assert actual["expected_brier"] == pytest.approx(direct, abs=1e-14) + oracle = ex.score(d, d.q.to_numpy()) + assert oracle["risk_mse"] == 0 + assert oracle["expected_recovery"] == 1 + assert ex.score(d, d.p.to_numpy())["expected_recovery"] == 0 + + +@pytest.mark.parametrize("family", ["logit_linear", "probability_linear"]) +def test_independent_agreement_and_probabilities(family): + d = ex.paired_fixtures(ex.SEEDS[3])[family] + preds, statuses = ex.fit_methods(d) + assert all(status["available"] for status in statuses.values()) + assert len(preds) == 4 + np.testing.assert_allclose(preds["predval_rung2"], preds["statsmodels_logit"], atol=1e-8) + assert all(np.isfinite(p).all() and ((p >= 0) & (p <= 1)).all() for p in preds.values()) + + +def test_unavailability_is_explicit(): + d = ex.frame(np.zeros(20, dtype=int), np.linspace(0.1, 0.9, 20), np.arange(20)) + preds, statuses = ex.fit_methods(d) + assert set(preds) == {"raw"} + assert all(not statuses[m]["available"] and statuses[m]["reason"] for m in ex.METHODS[1:]) + + +def test_wdbc_is_not_oracle(): + d = ex.wdbc_control() + assert len(d) == 285 and d.y.sum() == 106 + assert "q" not in d + result = ex.score(d, d.p.to_numpy()) + assert "expected_brier" not in result and "risk_mse" not in result + assert result["brier"] == pytest.approx(0.02947826313148458, abs=1e-12) + + +def test_mc_precision_and_missing_case_accounting(): + result = ex.mean_precision([1, 2, 3]) + assert result["n"] == 3 and result["mean"] == 2 + assert result["mcse"] == pytest.approx(1 / np.sqrt(3)) + assert ex.mean_precision([])["n"] == 0 + assert ex.mean_precision([1])["mcse"] is None + rows = [ + {"family": f, "seed": s, "method": m, "available": False} + for f in ("logit_linear", "probability_linear") + for s in ex.SEEDS + for m in ex.METHODS + ] + summary = ex.summarize(rows) + assert all(r["attempted"] == 20 and r["n"] == 0 for r in summary["simulation_summary"]) + assert all(r["attempted"] == 20 and r["n"] == 0 for r in summary["paired_method_contrasts"]) diff --git a/benchmarks/family_extension/verification/diff-check.log b/benchmarks/family_extension/verification/diff-check.log new file mode 100644 index 0000000..e69de29 diff --git a/benchmarks/family_extension/verification/format.log b/benchmarks/family_extension/verification/format.log new file mode 100644 index 0000000..293af16 --- /dev/null +++ b/benchmarks/family_extension/verification/format.log @@ -0,0 +1 @@ +68 files already formatted diff --git a/benchmarks/family_extension/verification/pytest.log b/benchmarks/family_extension/verification/pytest.log new file mode 100644 index 0000000..8a9e3a4 --- /dev/null +++ b/benchmarks/family_extension/verification/pytest.log @@ -0,0 +1,24 @@ +........................................................................ [ 23%] +................sss..................................................... [ 46%] +......................sssssssssssss..............ssss................... [ 69%] +........................................................................ [ 92%] +....................... [100%] +=========================== short test summary info ============================ +SKIPPED [1] tests/test_golden_gusto.py:69: GUSTO fixture not built; run examples/gusto/fetch_data.py + make_predictions.py +SKIPPED [1] tests/test_golden_improve.py:67: IMPROVE fixture not built; run examples/improve/build_fixture.py +SKIPPED [1] tests/test_golden_pcam.py:69: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:40: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:46: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:52: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:58: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:63: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:68: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:73: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:79: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:85: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:92: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:97: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:108: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [1] tests/test_pcam_fixture.py:144: PCam fixture not built; run examples/pcam/build_fixture.py +SKIPPED [4] tests/test_predrupdate_agreement.py:99: GUSTO fixture not built; run examples/gusto/fetch_data.py + make_predictions.py +291 passed, 20 skipped in 110.19s (0:01:50) diff --git a/benchmarks/family_extension/verification/repeat.log b/benchmarks/family_extension/verification/repeat.log new file mode 100644 index 0000000..3a9bb55 --- /dev/null +++ b/benchmarks/family_extension/verification/repeat.log @@ -0,0 +1,6 @@ +{ + "cases": 41, + "method_cases": 164, + "unavailable": 0, + "out": "/tmp/predval-family-extension-b" +} diff --git a/benchmarks/family_extension/verification/reproducibility.json b/benchmarks/family_extension/verification/reproducibility.json new file mode 100644 index 0000000..758e34e --- /dev/null +++ b/benchmarks/family_extension/verification/reproducibility.json @@ -0,0 +1,7 @@ +{ + "stable_files_checked_per_run": 4, + "repeat_manifest_equal": true, + "cases": 41, + "method_cases": 164, + "all_fixed_seeds_accounted_for": true +} diff --git a/benchmarks/family_extension/verification/ruff.log b/benchmarks/family_extension/verification/ruff.log new file mode 100644 index 0000000..1f5f344 --- /dev/null +++ b/benchmarks/family_extension/verification/ruff.log @@ -0,0 +1 @@ +All checks passed! diff --git a/benchmarks/family_extension/verification/runtime-a.json b/benchmarks/family_extension/verification/runtime-a.json new file mode 100644 index 0000000..b5bad54 --- /dev/null +++ b/benchmarks/family_extension/verification/runtime-a.json @@ -0,0 +1,4 @@ +{ + "seconds": 11.137972092999917, + "scope": "combined workload, excludes imports/install" +} diff --git a/benchmarks/family_extension/verification/runtime-b.json b/benchmarks/family_extension/verification/runtime-b.json new file mode 100644 index 0000000..6dd388f --- /dev/null +++ b/benchmarks/family_extension/verification/runtime-b.json @@ -0,0 +1,4 @@ +{ + "seconds": 9.642061055999875, + "scope": "combined workload, excludes imports/install" +} diff --git a/benchmarks/family_extension/verification/status.json b/benchmarks/family_extension/verification/status.json new file mode 100644 index 0000000..ed23b45 --- /dev/null +++ b/benchmarks/family_extension/verification/status.json @@ -0,0 +1,52 @@ +{ + "baseline": "150998f76df1141c73b6c3621233e693c4eb0b49", + "calibrationcurves": "Primary documentation reviewed; not installed or executed.", + "execution_note": "First run had Matplotlib/fontconfig import cache warnings; repeat and suite set writable cache directories. No model-fitting warnings.", + "fitting_warnings": 0, + "git_diff_check": "passed", + "historical_and_production_unchanged": true, + "independent_verification": { + "blocking_findings": 0, + "independent_glm_families_refitted": 2, + "limitation": "Unexpected exceptions outside selected fitting exception types would abort the run. No such exceptions occurred in the fixed design.", + "max_abs_loss_discrepancy": 4.45e-15, + "protocol_unchanged": true, + "summary_and_MCSE_reproduced": true, + "synthetic_designs_reconstructed": 40, + "tracked_production_and_historical_unchanged": true + }, + "local_protocol_commit": "4a289159a937c49b1d06469e93db9079fef5f47f", + "manifests_equal": true, + "max_abs_predval_independent_logit_difference": 7.105427357601002e-15, + "method_cases": 164, + "native_R_extension_execution": false, + "provenance_limit": "Local commit chronology is not independently timestamped preregistration. Original results were inspected before this extension.", + "remote_actions": [], + "repeated_cases": 41, + "ruff": "passed", + "ruff_format": "passed", + "runtime_scope": "Combined workload after imports, not a package-speed comparison.", + "runtime_seconds": [ + 11.137972092999917, + 9.642061055999875 + ], + "source_sha256": { + "benchmarks/family_extension/PROTOCOL.md": "8a34e65b43b189e3a845a6c0fc7d92f11a992d57eafb62e199de62fef8954559", + "benchmarks/family_extension/run_extension.py": "cbffe569b0bd7ae556bbebad65cdebc28b5045c4573f29fc6e1a300bc2978bf2", + "benchmarks/family_extension/verify_results.py": "fab99b931dfa6cf3d1e7580e816aedc77ab590b71cc8487e05a018b5b97f9d65", + "src/predval/metrics.py": "5a0f13aee0e1a4de17611f5aa13ff05421382af478b9fff788fd77132d9b81e6", + "src/predval/recalibrate.py": "513d69580b00cdd2f4633060300210513c79f5524fc0027d038e25810d072a22" + }, + "stable_files_checked_per_run": 4, + "suppression": "Only WDBC rung3 non-convergence; rung2 available in all cases.", + "test_command": "PYTHONPATH=$PWD/src MPLCONFIGDIR=/tmp/predval-family-mpl XDG_CACHE_HOME=/tmp/predval-cache OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 /workspace/shared/predval-benchmark-20261010/.venv/bin/python -m pytest tests benchmarks/alternatives/tests benchmarks/family_extension/tests -ra", + "tests": { + "failed": 0, + "new_tests": 9, + "passed": 291, + "seconds": 110.19, + "skip_reason": "Optional PCam, GUSTO and IMPROVE example/golden fixtures unavailable.", + "skipped": 20 + }, + "unavailable_method_cases": 0 +} diff --git a/benchmarks/family_extension/verify_results.py b/benchmarks/family_extension/verify_results.py new file mode 100644 index 0000000..8a177f9 --- /dev/null +++ b/benchmarks/family_extension/verify_results.py @@ -0,0 +1,55 @@ +"""Verify repeat hashes and complete accounting for the fixed extension design.""" + +import argparse +import hashlib +import json +from pathlib import Path + + +def verify(a, b): + manifests = [] + for root in (a, b): + manifest = json.loads((root / "manifest.json").read_text()) + for name, expected in manifest["files"].items(): + assert hashlib.sha256((root / name).read_bytes()).hexdigest() == expected, name + rows = json.loads((root / "metrics.json").read_text()) + cases = json.loads((root / "cases.json").read_text()) + assert len(rows) == 164 and len(cases) == 41 + assert len({(r["seed"], r["family"], r["method"]) for r in rows}) == 164 + for family in ("logit_linear", "probability_linear"): + assert {r["seed"] for r in rows if r["family"] == family} == set( + range(20261020, 20261040) + ) + assert sum(r["family"] == "wdbc_undistorted" for r in rows) == 4 + assert all(r["available"] or r["reason"] for r in rows) + summaries = json.loads((root / "summary.json").read_text())["simulation_summary"] + for s in summaries: + selected = [ + r for r in rows if r["family"] == s["family"] and r["method"] == s["method"] + ] + assert s["attempted"] == len(selected) == 20 + assert s["n"] == sum( + r["available"] and r.get(s["metric"]) is not None for r in selected + ) + manifests.append(manifest) + assert manifests[0] == manifests[1] + print( + json.dumps( + { + "stable_files_checked_per_run": len(manifests[0]["files"]), + "repeat_manifest_equal": True, + "cases": 41, + "method_cases": 164, + "all_fixed_seeds_accounted_for": True, + }, + indent=2, + ) + ) + + +if __name__ == "__main__": + p = argparse.ArgumentParser(description=__doc__) + p.add_argument("run_a", type=Path) + p.add_argument("run_b", type=Path) + args = p.parse_args() + verify(args.run_a, args.run_b) diff --git a/docs/spec.md b/docs/spec.md index 2299263..c592d08 100644 --- a/docs/spec.md +++ b/docs/spec.md @@ -107,8 +107,12 @@ grouping unit within which subjects are correlated. This exists because the unit of analysis and the unit of independence are frequently not the same. In the PCam fixture, 19,999 image patches derive from just 22 whole slides; patches from one slide are heavily correlated. Confidence intervals computed as though there were 19,999 -independent observations are not slightly optimistic, they are wrong by roughly the square root -of the cluster size. S1 validates and plumbs this field. S2 consumes it for cluster-robust or +independent observations can seriously understate uncertainty. For an illustrative mean with +independent clusters of equal size `m`, equal marginal variances, and exchangeable within-cluster +correlation `ICC`, the standard-error inflation is `sqrt(1 + (m - 1) * ICC)`. It equals `sqrt(m)` only when +`ICC = 1`. Unequal cluster sizes and other statistics, including AUROC, need their own variance +analysis; the correlation relevant to a loss mean is that of the losses, not necessarily the +outcomes. S1 validates and plumbs this field. S2 consumes it for cluster-robust or cluster-bootstrap intervals over the grouping unit. If `clustering` is omitted, S2 treats subjects as independent and says so in the report. @@ -326,14 +330,24 @@ replicate *and* every fold — so that it carries the variance of the correction gain holds the cross-fit mapping fixed; the full refit-in-replicate interval is a design surface of its own and remains backlog rather than a silent omission. +Holding the predictions fixed during resampling does **not** by itself establish conditional +coverage: cross-fitted loss differences share fitted corrections and overlapping training data. +The current intervals do not account for the full fitting-and-selection procedure. Respecting +the cluster resampling unit alone does not guarantee nominal coverage, and the `few_clusters` +warning is not a validated boundary above which coverage is guaranteed. Any proposed replacement +interval needs evaluation against an explicitly stated target before adoption. + ### 4.6 Every metric at every rung, with two integrity checks (S3.1) The ladder recomputes **all** metrics on each corrected mapping — not only calibration. Threshold metrics move because recalibration shifts the operating point, and that shift is clinically real; -rung3 can reorder scores and so change AUROC and average precision. Emitting the full set is what -lets a report show, side by side, that recalibration does **not** buy discrimination under the -monotone rungs (rung1/rung2 AUROC equals rung0's exactly) while a decision threshold's sensitivity -genuinely changes. +rung3 can reorder scores and so change AUROC and average precision. A single strictly increasing +mapping preserves ranks: apparent rung1, and apparent rung2 with a positive slope, preserve +AUROC if clipping or numerical saturation does not introduce ties. This invariance does **not** +extend to pooled cross-fitted predictions: different fold-specific mappings can reorder scores +across folds even when each mapping is increasing. A nonpositive rung2 slope also precludes the +strictly increasing argument. All metrics are therefore recomputed in both fit modes; threshold +sensitivity can change even when ranks are preserved. Two identities hold **by construction** and must be annotated as such in any report, never presented as findings: rung1's apparent calibration intercept is ≈ 0, and rung2's apparent diff --git a/examples/improve/README.md b/examples/improve/README.md index b21f5b8..a77aa5f 100644 --- a/examples/improve/README.md +++ b/examples/improve/README.md @@ -27,10 +27,12 @@ through broad-scale validation of T cell recognition*, Frontiers in Immunology. patients; melanoma 5,921 / 26; a mixed "Basket" group 5,362 / 20). - 467 peptides are experimentally immunogenic — **2.666% prevalence**. - Predictions are the paper's own **out-of-fold** scores from its 5-fold cross-validation. That - CV was partitioned **by patient**: every patient sits in exactly one partition, verified in - `build_fixture.py`. No score here comes from a model that had seen another peptide from the - same patient, which is what makes the file admissible as a predictions-only validation input - rather than an apparent-fit artefact. + CV was partitioned **by patient**. `build_fixture.py` checks every released random-forest + table for non-missing patient IDs and integer partitions, and rejects any patient appearing + in more than one partition before writing either fixture. This verifies the tables' grouping + metadata; it does not independently audit upstream training or the NNAlign comparator's + split assignments. Interpreting the released scores as out-of-fold predictions still relies + on the source authors' training procedure. - Seven models: the IMPROVE random forest in its three released feature configurations (`Simple`; `TME_excluded`, adding cellular prevalence and a prioritisation score; `TME_included`, further adding cytolytic activity, HLA expression and MCP-counter estimates), diff --git a/examples/improve/build_fixture.py b/examples/improve/build_fixture.py index e3f2f1b..4effebb 100644 --- a/examples/improve/build_fixture.py +++ b/examples/improve/build_fixture.py @@ -39,11 +39,32 @@ def _sid(frame: pd.DataFrame) -> pd.Series: ) +def validate_patient_partitions(frame: pd.DataFrame, model_id: str) -> None: + """Fail closed on missing grouping metadata or patients split across CV folds. + + This checks the released RF tables, not the upstream training implementation or + the NNAlign comparator (whose Split field is not read by this builder). + """ + required = {"Patient", "Partition"} + if not required.issubset(frame.columns): + raise SystemExit(f"{model_id}: missing Patient or Partition column") + if frame["Patient"].isna().any() or frame["Patient"].astype(str).str.strip().eq("").any(): + raise SystemExit(f"{model_id}: missing patient identifiers") + partition = pd.to_numeric(frame["Partition"], errors="coerce") + if partition.isna().any() or (partition % 1 != 0).any(): + raise SystemExit(f"{model_id}: CV partitions must be non-missing integers") + # Use the same patient representation as the output cohort, without changing the data. + counts = partition.groupby(frame["Patient"].astype(str)).nunique() + if (counts > 1).any(): + raise SystemExit(f"{model_id}: a patient appears in multiple CV partitions") + + def load_rf(src: Path) -> dict[str, pd.DataFrame]: out = {} for model_id, (sub, name) in RF_MODELS.items(): path = src / "5_fold_CV" / sub / name frame = pd.read_csv(path, sep=r"\s+") + validate_patient_partitions(frame, model_id) frame["subject_id"] = _sid(frame) out[model_id] = frame return out diff --git a/src/predval/metrics.py b/src/predval/metrics.py index 5e027db..0f85e84 100644 --- a/src/predval/metrics.py +++ b/src/predval/metrics.py @@ -128,7 +128,8 @@ def _irls_logistic( if np.max(np.abs(new - beta)) < tol: return new beta = new - return beta + # A finite last iterate is not a converged fit, notably under quasi-separation. + return None def calibration_intercept(y: np.ndarray, p: np.ndarray) -> float: diff --git a/src/predval/recalibrate.py b/src/predval/recalibrate.py index 0634ba1..1223cda 100644 --- a/src/predval/recalibrate.py +++ b/src/predval/recalibrate.py @@ -35,8 +35,10 @@ # Every metric is recomputed on each recalibrated mapping -- not only calibration. Threshold # metrics move because recalibration shifts the operating point, and rung3 can reorder scores, # so AUROC/average precision are no longer invariant. Emitting all of them is what lets the -# report show that recalibration cannot buy discrimination (rung1/rung2 AUROC == rung0) while a -# threshold's sensitivity genuinely changes. The metric set is M.THRESHOLD_FREE + M.THRESHOLDED. +# report distinguish rank-preserving apparent fits from pooled cross-fit scores. A single +# strictly increasing mapping preserves AUROC unless clipping/saturation creates ties; rung2 +# requires a positive slope. Different fold-specific mappings can reorder pooled held-out scores +# even when each is increasing. The metric set is M.THRESHOLD_FREE + M.THRESHOLDED. APPARENT = "apparent" CROSSFIT = "crossfit" diff --git a/src/predval/report.py b/src/predval/report.py index b51d605..d9a4d01 100644 --- a/src/predval/report.py +++ b/src/predval/report.py @@ -46,7 +46,9 @@ def _template_source() -> str: "rung0, not competitors to it. The cross-fitted figures are the expected performance after " "local recalibration on a cohort like this one -- not evidence that the recalibrated model " "has been validated, which would require a cohort the correction was never fitted on. " - "Recalibration does not and cannot improve discrimination." + "Recalibration targets calibration. A single strictly increasing mapping preserves ranking " + "unless clipping or saturation creates ties. Other mappings, or pooling fold-specific " + "corrections, can change discrimination; inspect AUROC changes alongside Brier gain." ) #: Historical marker for a gain shown without an interval. The paired cross-fit gain now carries a diff --git a/src/predval/templates/report.html.j2 b/src/predval/templates/report.html.j2 index 54064df..ab6c1b4 100644 --- a/src/predval/templates/report.html.j2 +++ b/src/predval/templates/report.html.j2 @@ -89,8 +89,9 @@ cross-check in metrics.parquet agrees (none crossed the 0 loss boundary).

best admissible rung -- no free adjectives. BSS is anchored at 0 = no-skill (always predict prevalence) and 1 = perfect; "closes Y% of the gap" is BSS as a percentage. The gauge sentence states any detected miscalibration (slope or intercept interval excluding its -null), the lowest rung whose out-of-fold repair is demonstrated, and any rung whose recalibration -reliably harms. A member with no detected fault and no demonstrated repair carries no gauge +null), the lowest rung meeting the repair criterion, and any rung meeting the harm criterion. +These labels describe gain intervals under fixed-prediction resampling, with the small-cluster +and selection limitations explained in section 3; they do not establish benefit on a new cohort. A member with no detected fault and no demonstrated repair carries no gauge sentence: the test can fail to reject calibration but never establish it.