Where: explainers/reject-inference.md - the "Detection Code" python block and its accompanying output table/prose.
The gap: the explainer's claimed output (Naive model: AUC 0.7812, age fairness gap 15.80 pts; IPW: AUC 0.8345, gap 2.30 pts; Soft Parceled: AUC 0.8115, gap 7.30 pts) does not match what the exact code block in the same file actually produces.
Repro (the code extracted verbatim from the file's own fenced python block, run directly):
$ python3 -c "
import re
text = open('explainers/reject-inference.md').read()
block = re.findall(r'\`\`\`python\n(.*?)\`\`\`', text, re.S)[0]
open('/tmp/reject_inference_extracted.py', 'w').write(block)
"
$ python3 /tmp/reject_inference_extracted.py
Population AUC Population Accuracy Older Approval Rate Younger Approval Rate Age Fairness Gap
Naive (Approved Only) 0.9460 0.8854 0.5420 0.5369 0.0051
IPW Reweighted 0.9456 0.8809 0.5511 0.5453 0.0059
Soft Parceled 0.9241 0.8855 0.5420 0.5372 0.0048
Reproduced identically across repeated runs (the script is seeded). The real gaps (0.51/0.59/0.48 points, not 15.80/2.30/7.30 points) and AUCs (0.946/0.946/0.924, not 0.7812/0.8345/0.8115) are nowhere close to the explainer's claimed table - and mathematically this is the expected outcome, since age is generated independently of the two model features by construction, so a near-zero population-level gap is correct, not a coincidence.
Why it matters: the entire "15.80-point gap corrected down to 2.30" narrative that anchors this explainer's main point is unsupported by the code sitting right next to it in the same file - a reader who runs the code themselves gets a completely different, much less dramatic story.
Suggested fix: either regenerate the output table from the current code (and adjust the surrounding prose to match whatever real gap that produces), or fix the code/toy-data generation so it actually produces a real age-correlated gap worth correcting - right now the two contradict each other.
Where:
explainers/reject-inference.md- the "Detection Code" python block and its accompanying output table/prose.The gap: the explainer's claimed output (Naive model: AUC 0.7812, age fairness gap 15.80 pts; IPW: AUC 0.8345, gap 2.30 pts; Soft Parceled: AUC 0.8115, gap 7.30 pts) does not match what the exact code block in the same file actually produces.
Repro (the code extracted verbatim from the file's own fenced python block, run directly):
Reproduced identically across repeated runs (the script is seeded). The real gaps (0.51/0.59/0.48 points, not 15.80/2.30/7.30 points) and AUCs (0.946/0.946/0.924, not 0.7812/0.8345/0.8115) are nowhere close to the explainer's claimed table - and mathematically this is the expected outcome, since
ageis generated independently of the two model features by construction, so a near-zero population-level gap is correct, not a coincidence.Why it matters: the entire "15.80-point gap corrected down to 2.30" narrative that anchors this explainer's main point is unsupported by the code sitting right next to it in the same file - a reader who runs the code themselves gets a completely different, much less dramatic story.
Suggested fix: either regenerate the output table from the current code (and adjust the surrounding prose to match whatever real gap that produces), or fix the code/toy-data generation so it actually produces a real age-correlated gap worth correcting - right now the two contradict each other.