Skip to content

docs: correct COMPAS reduction figure repo-wide (71% -> 82%) + Benefits Denial proxy count - #558

Merged
yakew7 merged 2 commits into
yakew7:mainfrom
propcgamer20-png:docs/results-glance-corrections
Sep 10, 2026
Merged

docs: correct COMPAS reduction figure repo-wide (71% -> 82%) + Benefits Denial proxy count#558
yakew7 merged 2 commits into
yakew7:mainfrom
propcgamer20-png:docs/results-glance-corrections

Conversation

@propcgamer20-png

@propcgamer20-png propcgamer20-png commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Two Results-at-a-Glance inaccuracies, one of which turned out to be repo-wide.

#545 - COMPAS "Reduction" uses the wrong formula (repo-wide)

Every audit's "Reduction" column is a relative reduction, (before - after) / before. COMPAS's alone was the raw point gap:

(86.77 - 15.69)          = 71.08  ->  "71%"    # what was printed everywhere
(86.77 - 15.69) / 86.77  = 81.9%  ->  "82%"    # the relative reduction the other 6 audits use

The other rows all verify against the relative formula (97.3%, 73.6%, ~60%, ~72%, 53%). The issue names the two summary tables, but grep shows "71% reduction" (and "71% of the gap") is COMPAS's number in 9 explainers + CHANGELOG.md too, so fixing only the tables would make the repo more inconsistent. This PR corrects every source reference and regenerates the explainer pages / faircode/_explainers/ mirror / llms-full.txt:

  • COMPAS/README.md, root README.md (summary row + "Result: N% reduction" line)
  • CHANGELOG.md, .github/PULL_REQUEST_TEMPLATE.md (example hint)
  • explainers: ai-objectivity-myth, confounding-variable (×3), data-leakage, equalized-odds, how-ai-detects-patterns, ml-bias, proxy-variables (×2), reinforcement-learning (×3)

The underlying gap numbers (86.77%, 15.69%) are untouched - only the derived reduction % changes. This was understating COMPAS's mitigation effectiveness.

#546 - Benefits Denial README omits fnlwgt from the proxy list

audit.yaml proxy_features has five entries (marital.status, occupation, relationship, hours.per.week, fnlwgt); fair.py removes all five with its own documented reasoning. Both READMEs listed four. Added fnlwgt to Benefits Denial/README.md, the root README summary row, the "four proxy variables" -> "five" counts (2 spots), and a # fnlwgt removed line in the fair.py code-block comment.

Checks

check_em_dash.py, check_broken_links.py, check_generated_files_current.py all pass.

Closes #545
Closes #546

@vercel

vercel Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

@propcgamer20-png is attempting to deploy a commit to the yashkewlani2020-gmailcom's projects Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

@yakew7 @Shreyash0712 - new PR touching a path you own, please review.

…ts Denial proxy count

yakew7#545: the COMPAS "Reduction" figure was the raw percentage-point gap
(86.77 - 15.69 = 71.08 -> "71%"), not the relative reduction every other
audit reports:

    (86.77 - 15.69) / 86.77 = 81.9%  ->  "82%"

The other six audits' rows all check out against the relative formula
(97.3%, 73.6%, ~60%, ~72%, 53%). "71%" was COMPAS's number not just in
the two summary tables but in nine explainers and CHANGELOG.md, so
fixing only the tables would have left the repo internally inconsistent.
Updated every "71% reduction" / "71% of the gap" COMPAS reference in
source (COMPAS/README.md, root README.md x3, CHANGELOG.md,
PULL_REQUEST_TEMPLATE.md example, and explainers ai-objectivity-myth,
confounding-variable, data-leakage, equalized-odds,
how-ai-detects-patterns, ml-bias, proxy-variables, reinforcement-learning)
and regenerated the explainer pages / package mirror / llms-full.txt.
The underlying gap numbers (86.77%, 15.69%) are unchanged.

yakew7#546: Benefits Denial's audit.yaml lists five proxy_features
(marital.status, occupation, relationship, hours.per.week, fnlwgt) and
fair.py removes all five with documented reasoning, but both READMEs
listed only four. Added fnlwgt to Benefits Denial/README.md and the root
README (summary row, the "four proxy variables" -> "five" counts, and the
fair.py code-block comment list).

Closes yakew7#545
Closes yakew7#546
@propcgamer20-png
propcgamer20-png force-pushed the docs/results-glance-corrections branch from 3c984a6 to 1f58fd2 Compare September 9, 2026 20:04
@propcgamer20-png propcgamer20-png changed the title docs: fix Results-at-a-Glance COMPAS reduction % and Benefits Denial proxy list docs: correct COMPAS reduction figure repo-wide (71% -> 82%) + Benefits Denial proxy count Sep 9, 2026
@yakew7
yakew7 merged commit c8c5a32 into yakew7:main Sep 10, 2026
13 of 14 checks passed
yakew7 added a commit that referenced this pull request Sep 10, 2026
… numbers

PR #557's fix for issue #545 (COMPAS's "71%" reduction figure should be
"82%") did a substring replace of "71" that also matched two completely
unrelated statistics still containing that digit sequence:

- "Black Defendants | 84.71%" (COMPAS's fair-model high-risk rate) became
  "84.82%" - breaking the table's own arithmetic (84.82 - 69.02 = 15.80,
  not the stated 15.69% New Fairness Gap; the original 84.71% is the only
  value that reproduces 15.69% exactly).
- "Male applicants | 25.71%" / "Female applicants | 7.71%" (Benefits
  Denial's ineligibility rates, from a different audit entirely) became
  "25.82%" / "7.82%".

Both reverted to their real values, confirmed against each table's own
stated gap (84.71 - 69.02 = 15.69; 25.71 - 7.71 = 18.00) and against git
history (both lines were untouched by PR #558's more careful, scoped fix
for the same underlying issue, and by every other reference to these two
audits' rates elsewhere in the repo).
yakew7 added a commit that referenced this pull request Sep 10, 2026
The combined #557/#558 bullet only credited the COMPAS-reduction and
Benefits Denial proxy fixes; #557 also updated ROADMAP.md's stale
traction table (29/33/60), which the bullet omitted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
yakew7 added a commit that referenced this pull request Sep 10, 2026
- Moved @propcgamer20-png into Core contributors (43 merged PRs, now
  code owner for /Makefile and /scripts/), with their bio extended to
  cover the #558-567 batch.
- Added @slsgzs-cloud (3 merged PRs, first merged 2026-09-08), missing
  from this file despite #503/#504 already being merged.
- Snapshot bumped to 2026-09-10, covering through PR #567.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants