fix: significance_report's confidence parameter now affects the significant flag - #560
Merged
yakew7 merged 1 commit intoSep 10, 2026
Conversation
…ficant flag significant was hardcoded p_value < 0.05 while confidence (default 0.95) only widened or narrowed the returned CI. A caller passing a stricter confidence (0.99) expecting a more conservative significance verdict got the same p < 0.05 threshold, silently. significant is now p_value < (1 - confidence): identical to the old p < 0.05 at the default confidence=0.95 (so every audit's unfair.py / fair.py output is unchanged), and correspondingly stricter for a caller that asks for it. Documented in the docstring. Closes yakew7#548
Contributor
|
@propcgamer20-png is attempting to deploy a commit to the yashkewlani2020-gmailcom's projects Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
faircode/significance.py'ssignificance_reporthardcoded"significant": p_value < 0.05, whileconfidence(default0.95) only affected the width of the returnedci_low/ci_high.A caller tightening
confidenceto0.99to get a more conservative verdict got the samep < 0.05threshold, with nothing indicatingconfidencehad no effect onsignificant.Fix
significantis nowp_value < (1 - confidence):confidence=0.95->p < 0.05, identical to the old behavior (every audit'sunfair.py/fair.pyuses the default, so their output is unchanged)confidence=0.99->p < 0.01, sop=0.0383is nowsignificant=FalseThe CI and the verdict move together. Documented in the docstring.
Test
test_confidence_tightens_the_significance_threshold_not_just_the_ci- a gap with0.01 < p < 0.05issignificantatconfidence=0.95but not atconfidence=0.99, same permutationp_valueboth times.pytest tests/test_significance.py-> 8 passed.ruffclean.Closes #548