diff --git a/assets/explainers-ui.js b/assets/explainers-ui.js index 1c9b5c1..3a8398d 100644 --- a/assets/explainers-ui.js +++ b/assets/explainers-ui.js @@ -109,8 +109,14 @@ return null; } - const headers = rows[0].split('|').slice(1, -1).map(cell => cell.trim()); - const bodyRows = rows.slice(2).map(row => row.split('|').slice(1, -1).map(cell => cell.trim())); + // Split on unescaped "|" only, then unescape "\|" -> "|" in each cell, so + // a literal pipe inside a cell (GFM's "\|") no longer starts a spurious + // column. Mirrors scripts/build_explainers.py's split_row(). + const splitRow = row => row.trim().split(/(? cell.trim().replace(/\\\|/g, '|')); + + const headers = splitRow(rows[0]); + const bodyRows = rows.slice(2).map(splitRow); const headerHtml = headers.map(cell => `${inlineMarkdown(cell)}`).join(''); const bodyHtml = bodyRows.map(row => `${row.map(cell => `${inlineMarkdown(cell)}`).join('')}`).join(''); @@ -214,7 +220,9 @@ flushParagraph(); flushList(); flushQuote(); - const level = headingMatch[1].length; + // +1 offset: the explainer page's hero already renders a real

. + // Mirrors scripts/build_explainers.py's render_markdown(). + const level = Math.min(headingMatch[1].length + 1, 6); const headingText = headingMatch[2]; const baseId = slugifyHeading(headingText); const nextCount = (headingCounts.get(baseId) || 0) + 1; diff --git a/explainers/base-rate-fallacy.html b/explainers/base-rate-fallacy.html index 210d9ca..9544417 100644 --- a/explainers/base-rate-fallacy.html +++ b/explainers/base-rate-fallacy.html @@ -205,7 +205,7 @@

The Mathematics of the Base Ra 
PPV = P(Y = 1 | Ŷ = 1) = (TPR * p) / (TPR * p + FPR * (1 - p))

Prevalence Impact on Reliability

To see how background prevalence dictates prediction reliability, consider a screening model with fixed TPR = 0.90 and FPR = 0.10 evaluated across varying base rates (p):

-
Base Rate (p)True Positives (TPR * p)False Positives (FPR * (1 - p))PPV (P(Y = 1 \Ŷ = 1))False Discovery Rate (1 - PPV)
1%0.00900.09908.33%91.67%
5%0.04500.095032.14%67.86%
10%0.09000.090050.00%50.00%
30%0.27000.070079.41%20.59%
50%0.45000.050090.00%10.00%
+
Base Rate (p)True Positives (TPR * p)False Positives (FPR * (1 - p))PPV (P(Y = 1 | Ŷ = 1))False Discovery Rate (1 - PPV)
1%0.00900.09908.33%91.67%
5%0.04500.095032.14%67.86%
10%0.09000.090050.00%50.00%
30%0.27000.070079.41%20.59%
50%0.45000.050090.00%10.00%

At a 1% base rate, over 91% of flagged individuals are false alarms, despite the model having 90% sensitivity and 90% specificity.

The Chouldechova Impossibility Identity

When evaluating models across demographic groups A and B, Chouldechova (2017) demonstrated that the false positive rate (FPR), false negative rate (FNR), positive predictive value (PPV), and base rate (p) are linked by a strict identity:

diff --git a/explainers/reject-inference.html b/explainers/reject-inference.html index 0cc8680..44fd8ac 100644 --- a/explainers/reject-inference.html +++ b/explainers/reject-inference.html @@ -226,7 +226,7 @@

The Missingness Me

If a bank historically required younger applicants to meet a higher credit bar than older applicants, then the younger applicants present in the approved dataset (S = 1) represent an artificially selected, ultra-qualified subset of all young applicants. A model trained on this sample will overestimate the credit standards required for young borrowers to succeed.

Core Reject Inference Techniques

Practitioners use four main statistical approaches to correct for reject inference:

-
MethodCore MechanismStrengthsKey Vulnerability
Hard Parceling (Pseudo-Labeling)Train initial model M1 on approved cases (S = 1); score rejected cases (S = 0); assign binary labels Y_hat via threshold; retrain M2 on all rows.Simple to implement in standard ML pipelines.Propagates initial model errors and thresholding artifacts into retraining.
Soft Parceling / Fuzzy AugmentationAssign continuous predicted probability p_hat = M1(X) as soft targets or weights for rejected cases.Avoids hard threshold cutoffs; preserves prediction uncertainty.Dilutes training signal if initial model probability estimates are miscalibrated.
Inverse Probability Weighting (IPW)Estimate selection propensity w(X) = P(S = 1X); weight approved cases by 1 / w(X) during training.Theoretically unbiased under Missing At Random (MAR) assumptions.Extreme weights when propensity P(S = 1X) ≈ 0 create high estimator variance.
Heckman Two-Stage ModelStage 1: Fit probit model for selection S. Stage 2: Add Inverse Mills Ratio λ(Zγ) to outcome model to absorb correlation ρ(u, ε).Explicitly models unobserved selection correlation ρ.Relies heavily on bivariate normality and valid exclusion restrictions (Z).
+
MethodCore MechanismStrengthsKey Vulnerability
Hard Parceling (Pseudo-Labeling)Train initial model M1 on approved cases (S = 1); score rejected cases (S = 0); assign binary labels Y_hat via threshold; retrain M2 on all rows.Simple to implement in standard ML pipelines.Propagates initial model errors and thresholding artifacts into retraining.
Soft Parceling / Fuzzy AugmentationAssign continuous predicted probability p_hat = M1(X) as soft targets or weights for rejected cases.Avoids hard threshold cutoffs; preserves prediction uncertainty.Dilutes training signal if initial model probability estimates are miscalibrated.
Inverse Probability Weighting (IPW)Estimate selection propensity w(X) = P(S = 1 | X); weight approved cases by 1 / w(X) during training.Theoretically unbiased under Missing At Random (MAR) assumptions.Extreme weights when propensity P(S = 1 | X) ≈ 0 create high estimator variance.
Heckman Two-Stage ModelStage 1: Fit probit model for selection S. Stage 2: Add Inverse Mills Ratio λ(Zγ) to outcome model to absorb correlation ρ(u, ε).Explicitly models unobserved selection correlation ρ.Relies heavily on bivariate normality and valid exclusion restrictions (Z).

Concrete Example: German Credit Lending - Audit 03

German Credit Lending/credit_customers.csv is the dataset behind Audit 03 in this repository. Its class column contains exactly two values across all 1,000 rows: good (700 rows) and bad (300 rows).

diff --git a/explainers/reject-inference.md b/explainers/reject-inference.md index 80acb72..eddf6ed 100644 --- a/explainers/reject-inference.md +++ b/explainers/reject-inference.md @@ -65,7 +65,7 @@ Practitioners use four main statistical approaches to correct for reject inferen |---|---|---|---| | **Hard Parceling (Pseudo-Labeling)** | Train initial model M1 on approved cases (S = 1); score rejected cases (S = 0); assign binary labels Y_hat via threshold; retrain M2 on all rows. | Simple to implement in standard ML pipelines. | Propagates initial model errors and thresholding artifacts into retraining. | | **Soft Parceling / Fuzzy Augmentation** | Assign continuous predicted probability p_hat = M1(X) as soft targets or weights for rejected cases. | Avoids hard threshold cutoffs; preserves prediction uncertainty. | Dilutes training signal if initial model probability estimates are miscalibrated. | -| **Inverse Probability Weighting (IPW)** | Estimate selection propensity w(X) = P(S = 1 | X); weight approved cases by 1 / w(X) during training. | Theoretically unbiased under Missing At Random (MAR) assumptions. | Extreme weights when propensity P(S = 1 | X) ≈ 0 create high estimator variance. | +| **Inverse Probability Weighting (IPW)** | Estimate selection propensity w(X) = P(S = 1 \| X); weight approved cases by 1 / w(X) during training. | Theoretically unbiased under Missing At Random (MAR) assumptions. | Extreme weights when propensity P(S = 1 \| X) ≈ 0 create high estimator variance. | | **Heckman Two-Stage Model** | Stage 1: Fit probit model for selection S. Stage 2: Add Inverse Mills Ratio λ(Zγ) to outcome model to absorb correlation ρ(u, ε). | Explicitly models unobserved selection correlation ρ. | Relies heavily on bivariate normality and valid exclusion restrictions (Z). | --- diff --git a/faircode/_explainers/reject-inference.md b/faircode/_explainers/reject-inference.md index 80acb72..eddf6ed 100644 --- a/faircode/_explainers/reject-inference.md +++ b/faircode/_explainers/reject-inference.md @@ -65,7 +65,7 @@ Practitioners use four main statistical approaches to correct for reject inferen |---|---|---|---| | **Hard Parceling (Pseudo-Labeling)** | Train initial model M1 on approved cases (S = 1); score rejected cases (S = 0); assign binary labels Y_hat via threshold; retrain M2 on all rows. | Simple to implement in standard ML pipelines. | Propagates initial model errors and thresholding artifacts into retraining. | | **Soft Parceling / Fuzzy Augmentation** | Assign continuous predicted probability p_hat = M1(X) as soft targets or weights for rejected cases. | Avoids hard threshold cutoffs; preserves prediction uncertainty. | Dilutes training signal if initial model probability estimates are miscalibrated. | -| **Inverse Probability Weighting (IPW)** | Estimate selection propensity w(X) = P(S = 1 | X); weight approved cases by 1 / w(X) during training. | Theoretically unbiased under Missing At Random (MAR) assumptions. | Extreme weights when propensity P(S = 1 | X) ≈ 0 create high estimator variance. | +| **Inverse Probability Weighting (IPW)** | Estimate selection propensity w(X) = P(S = 1 \| X); weight approved cases by 1 / w(X) during training. | Theoretically unbiased under Missing At Random (MAR) assumptions. | Extreme weights when propensity P(S = 1 \| X) ≈ 0 create high estimator variance. | | **Heckman Two-Stage Model** | Stage 1: Fit probit model for selection S. Stage 2: Add Inverse Mills Ratio λ(Zγ) to outcome model to absorb correlation ρ(u, ε). | Explicitly models unobserved selection correlation ρ. | Relies heavily on bivariate normality and valid exclusion restrictions (Z). | --- diff --git a/llms-full.txt b/llms-full.txt index a99002b..cefe54c 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -9235,7 +9235,7 @@ Practitioners use four main statistical approaches to correct for reject inferen |---|---|---|---| | **Hard Parceling (Pseudo-Labeling)** | Train initial model M1 on approved cases (S = 1); score rejected cases (S = 0); assign binary labels Y_hat via threshold; retrain M2 on all rows. | Simple to implement in standard ML pipelines. | Propagates initial model errors and thresholding artifacts into retraining. | | **Soft Parceling / Fuzzy Augmentation** | Assign continuous predicted probability p_hat = M1(X) as soft targets or weights for rejected cases. | Avoids hard threshold cutoffs; preserves prediction uncertainty. | Dilutes training signal if initial model probability estimates are miscalibrated. | -| **Inverse Probability Weighting (IPW)** | Estimate selection propensity w(X) = P(S = 1 | X); weight approved cases by 1 / w(X) during training. | Theoretically unbiased under Missing At Random (MAR) assumptions. | Extreme weights when propensity P(S = 1 | X) ≈ 0 create high estimator variance. | +| **Inverse Probability Weighting (IPW)** | Estimate selection propensity w(X) = P(S = 1 \| X); weight approved cases by 1 / w(X) during training. | Theoretically unbiased under Missing At Random (MAR) assumptions. | Extreme weights when propensity P(S = 1 \| X) ≈ 0 create high estimator variance. | | **Heckman Two-Stage Model** | Stage 1: Fit probit model for selection S. Stage 2: Add Inverse Mills Ratio λ(Zγ) to outcome model to absorb correlation ρ(u, ε). | Explicitly models unobserved selection correlation ρ. | Relies heavily on bivariate normality and valid exclusion restrictions (Z). | --- diff --git a/scripts/build_explainers.py b/scripts/build_explainers.py index 1bd5b5e..f8c2c3d 100644 --- a/scripts/build_explainers.py +++ b/scripts/build_explainers.py @@ -142,7 +142,12 @@ def parse_table(lines, start_index): return None def split_row(row): - return [cell.strip() for cell in row.split("|")[1:-1]] + # Split on unescaped "|" only, then unescape "\|" -> "|" in each cell, + # so a literal pipe inside a cell (GFM's "\|") no longer starts a + # spurious column. Rows carry a leading and trailing "|", so the + # first and last split fragments are empty and dropped. + cells = re.split(r"(? "|" + assert r"/(?, so the markdown + # body's headings are shifted down one level (h1 -> h2, capped at h6). + # assets/explainers-ui.js's renderMarkdown must match this (#553). + script = importlib.import_module("scripts.build_explainers") + + html = script.render_markdown("# Top\n\n## Sub\n\n###### Deep\n", set()) + + assert '

Top

' in html + assert '

Sub

' in html + assert '
Deep
' in html # h6 + 1 stays h6, not h7