fix(experiments): show configured statistics level on interval labels#16
Open
lordspline wants to merge 1 commit into
Open
fix(experiments): show configured statistics level on interval labels#16lordspline wants to merge 1 commit into
lordspline wants to merge 1 commit into
Conversation
…ics level Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Bayesian/frequentist experiments let you configure a statistics level (e.g.
Bayesian / 90%) on the experiment settings, and the results query already computes the credible/confidence interval at that configured level. But the results presentation hardcoded95%in the interval labels, so a 90%-configured experiment showed a correctly-computed 90% interval mislabeled asCredible interval (95%). This mismatch undermines trust in the numbers.Addresses PostHog#62236
Changes
The interval calculation was already correct — it reads
stats_config.bayesian.ci_level/stats_config.frequentist.alphaand feeds them into the Bayesian/frequentist engines. The defect was display-only. This PR makes the displayed percentage reflect the configured level instead of a hardcoded95%:getExperimentStatsLevel(experiment, statsMethod?)andformatStatsLevelPercent(level)helpers (plusgetExperimentStatsMethod) as a single source of truth inscenes/experiments/utils.ts, backed by newDEFAULT_BAYESIAN_CI_LEVEL/DEFAULT_FREQUENTIST_ALPHAconstants.ResultDetails.tsx) and the "How to read" tooltip (HowToReadTooltip.tsx) now derive the percentage from the configured level. Frequentist terminology ("Confidence interval") and Bayesian terminology ("Credible interval") are preserved; the label method follows the result's method.Info.tsx,SettingsTab.tsx,StatsMethodModal.tsx) onto the shared helper to prevent this drift from recurring.stats_configis already serialized by the experiment API.Legacy experiment behavior is intentionally untouched.
Screenshots
Verified the same metric fixture under
Bayesian / 90%andBayesian / 95%; the results column header tracks the configured level (CREDIBLE INTERVAL (90%)vsCREDIBLE INTERVAL (95%)), matching the settings panel. Screenshots are attached in the originating agent session.How did you test this code?
I'm an agent. Automated tests I actually ran:
products/experiments/backend/hogql_queries/test/test_stats_config.py— 43 passed, including new parameterized tests proving the Bayesianci_leveland frequentistalphachange the computed interval, and that missing/default config matches the explicit 95% interval.products/experiments/backend/test/test_presentation_api.py— 6 new parameterized API round-trip tests (Bayesian 90/95/99, default, frequentist 90/99) confirming the configured level survives create → serialize → read.frontend/src/scenes/experiments/utils.test.ts— new parameterized tests forgetExperimentStatsLevel(Bayesian levels, default fallback, frequentist1 - alpha, method override) andformatStatsLevelPercent.typescript:checkshows no new errors in changed files (pre-existing repo-wide kea*LogicType/implicit-any noise unrelated to this change).stats_configatci_level0.90 and 0.95 and confirmed the column header and settings panel agree (90% and 95% respectively).I did not exercise a live end-to-end experiment with real ingested data; verification was via the calculation/serializer tests above plus the Storybook render.
Automatic notifications
Docs update
No docs change needed.
🤖 Agent context
Autonomy: Human-driven (agent-assisted) — directed by the requester; please assign the DRI for review.
Authored with Captain Capy (Capy coding agent). Approach: traced the configured level from persistence (
Experiment.stats_config) through the new-engine query runner into the Bayesian/frequentist stats methods, and proved numerically that the interval already responds to the level (90% width ≈ 1.78 vs 95% ≈ 2.12 vs 99% ≈ 2.79; frequentist mirrors viaalpha). That isolated the bug to display only. Chose the smallest correct fix — derive the label from the already-serializedstats_config— over a larger backend change that returned the level in the response, since the calculation was never wrong. Consolidated existing duplicated level-derivation onto one helper to stop the labels and settings from drifting apart again. Agent-authored; requires human review (no self-merge).