uncertainty: refit-in-replicate interval for the paired cross-fit gain (spec §4.5)

[aRealGem]

Requested by Precious · project thread

Before: the paired_gain_brier interval holds the out-of-fold correction fixed and resamples clusters of its per-row loss difference. It carries no variance from fitting the correction. In a clustered-logit simulation (10–50 clusters of 5 rows, true slope 0.7 or 0.5) its coverage at a nominal 95% was 74–93%, missing almost always on the high side.

After: cohort.yaml can declare uncertainty.gain_interval: refit. Every cluster-bootstrap replicate then re-runs the grouped K-fold cross-fit on the resampled rows and takes that replicate's gain (ci_method = paired_cluster_bootstrap_refit). The point estimate is unchanged. In the same simulation this interval covered at 90–97% in every cell, with lower power. The manifest records which interval was used.

Fold membership inside a replicate is keyed on the original cluster id, so a cluster drawn twice has both copies held out together. Relabelling copies as distinct clusters leaks a copy into the train fold and biases every replicate gain upward; a test pins the original-id behaviour.

How: a new paired_brier_gain_refit_interval in uncertainty.py takes a crossfit callable so it stays free of a recalibrate import; evaluate.py selects it from the spec; schema.py adds the gain_interval field with default fixed. The three golden manifests gain the new field only; no golden number changes. Flipping the default to refit is a separate change that needs the GUSTO, PCam and IMPROVE goldens regenerated. Spec §4.5 is rewritten to describe both intervals and the simulation evidence.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GTNTyyGWZbXyLAnT9qWWwV

---

Generated by Claude Code