Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion explainers/underdiagnosis-bias.html
Original file line number Diff line number Diff line change
Expand Up @@ -353,7 +353,7 @@ <h3 id="detection-code">Detection Code</h3>
print(&quot;Diagnostic label rate by biomarker band:&quot;)
print(diagnosis_rates)</code></pre>
<h4 id="output-interpretation">Output Interpretation</h4>
<p>1. <strong>The Evaluation Trap</strong>: On observed EHR labels (<code>obs_recall</code>), the model appears to perform decently across both groups (~0.85 vs ~0.76). But on true disease status (<code>true_recall</code>), the model catches <strong>80.5%</strong> of sick Group A patients versus only <strong>41.2%</strong> of sick Group B patients - a <strong>39.3-point true false-negative gap</strong> hidden from standard evaluation. 2. <strong>The Biomarker Audit</strong>: In the highest biomarker band (Q4), Group A patients have a <strong>0.887</strong> diagnosis rate while Group B has a <strong>0.536</strong> rate. Disparities in diagnostic coding among patients with matching objective clinical values flag underdiagnosis bias directly from EHR records.</p>
<p>1. <strong>The Evaluation Trap</strong>: Against observed EHR labels (<code>obs_recall</code>), the model already recalls Group B worse than Group A (<strong>~0.64</strong> vs <strong>~0.33</strong>), but standard evaluation reads that as the model faithfully matching a lower &quot;disease rate&quot; in Group B. Against true disease status (<code>true_recall</code>), the real picture is worse: the model catches <strong>62.5%</strong> of sick Group A patients versus only <strong>29.2%</strong> of sick Group B patients - a <strong>33.3-point true false-negative gap</strong>, most of which standard evaluation scores as correct True Negatives. 2. <strong>The Biomarker Audit</strong>: In the highest biomarker band (Q4), Group A patients have a <strong>0.785</strong> recorded-diagnosis rate while Group B has a <strong>0.497</strong> rate, despite sharing the same objective clinical values. Disparities in diagnostic coding within matched biomarker bands flag underdiagnosis bias directly from EHR records, without needing the unobserved true label.</p>
<h3 id="limitations">Limitations</h3>
<h4 id="1-unobserved-true-disease-state-y">1. Unobserved True Disease State (Y*)</h4>
<p>In observational healthcare data, true disease status Y* is rarely recorded. Identifying underdiagnosis requires objective proxy biomarkers (e.g., lab results, physiological waveforms), prospective screening studies, or external clinical audit samples.</p>
Expand Down
4 changes: 2 additions & 2 deletions explainers/underdiagnosis-bias.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,8 +215,8 @@ print(diagnosis_rates)

### Output Interpretation

1. **The Evaluation Trap**: On observed EHR labels (`obs_recall`), the model appears to perform decently across both groups (~0.85 vs ~0.76). But on true disease status (`true_recall`), the model catches **80.5%** of sick Group A patients versus only **41.2%** of sick Group B patients - a **39.3-point true false-negative gap** hidden from standard evaluation.
2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.887** diagnosis rate while Group B has a **0.536** rate. Disparities in diagnostic coding among patients with matching objective clinical values flag underdiagnosis bias directly from EHR records.
1. **The Evaluation Trap**: Against observed EHR labels (`obs_recall`), the model already recalls Group B worse than Group A (**~0.64** vs **~0.33**), but standard evaluation reads that as the model faithfully matching a lower "disease rate" in Group B. Against true disease status (`true_recall`), the real picture is worse: the model catches **62.5%** of sick Group A patients versus only **29.2%** of sick Group B patients - a **33.3-point true false-negative gap**, most of which standard evaluation scores as correct True Negatives.
2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.785** recorded-diagnosis rate while Group B has a **0.497** rate, despite sharing the same objective clinical values. Disparities in diagnostic coding within matched biomarker bands flag underdiagnosis bias directly from EHR records, without needing the unobserved true label.

## Limitations

Expand Down
4 changes: 2 additions & 2 deletions faircode/_explainers/underdiagnosis-bias.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,8 +215,8 @@ print(diagnosis_rates)

### Output Interpretation

1. **The Evaluation Trap**: On observed EHR labels (`obs_recall`), the model appears to perform decently across both groups (~0.85 vs ~0.76). But on true disease status (`true_recall`), the model catches **80.5%** of sick Group A patients versus only **41.2%** of sick Group B patients - a **39.3-point true false-negative gap** hidden from standard evaluation.
2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.887** diagnosis rate while Group B has a **0.536** rate. Disparities in diagnostic coding among patients with matching objective clinical values flag underdiagnosis bias directly from EHR records.
1. **The Evaluation Trap**: Against observed EHR labels (`obs_recall`), the model already recalls Group B worse than Group A (**~0.64** vs **~0.33**), but standard evaluation reads that as the model faithfully matching a lower "disease rate" in Group B. Against true disease status (`true_recall`), the real picture is worse: the model catches **62.5%** of sick Group A patients versus only **29.2%** of sick Group B patients - a **33.3-point true false-negative gap**, most of which standard evaluation scores as correct True Negatives.
2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.785** recorded-diagnosis rate while Group B has a **0.497** rate, despite sharing the same objective clinical values. Disparities in diagnostic coding within matched biomarker bands flag underdiagnosis bias directly from EHR records, without needing the unobserved true label.

## Limitations

Expand Down
4 changes: 2 additions & 2 deletions llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8877,8 +8877,8 @@ print(diagnosis_rates)

### Output Interpretation

1. **The Evaluation Trap**: On observed EHR labels (`obs_recall`), the model appears to perform decently across both groups (~0.85 vs ~0.76). But on true disease status (`true_recall`), the model catches **80.5%** of sick Group A patients versus only **41.2%** of sick Group B patients - a **39.3-point true false-negative gap** hidden from standard evaluation.
2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.887** diagnosis rate while Group B has a **0.536** rate. Disparities in diagnostic coding among patients with matching objective clinical values flag underdiagnosis bias directly from EHR records.
1. **The Evaluation Trap**: Against observed EHR labels (`obs_recall`), the model already recalls Group B worse than Group A (**~0.64** vs **~0.33**), but standard evaluation reads that as the model faithfully matching a lower "disease rate" in Group B. Against true disease status (`true_recall`), the real picture is worse: the model catches **62.5%** of sick Group A patients versus only **29.2%** of sick Group B patients - a **33.3-point true false-negative gap**, most of which standard evaluation scores as correct True Negatives.
2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.785** recorded-diagnosis rate while Group B has a **0.497** rate, despite sharing the same objective clinical values. Disparities in diagnostic coding within matched biomarker bands flag underdiagnosis bias directly from EHR records, without needing the unobserved true label.

## Limitations

Expand Down