Skip to content

fix(experiments): show configured statistics level on interval labels#16

Open
lordspline wants to merge 1 commit into
masterfrom
capy/fix-interval-labels-stats
Open

fix(experiments): show configured statistics level on interval labels#16
lordspline wants to merge 1 commit into
masterfrom
capy/fix-interval-labels-stats

Conversation

@lordspline

@lordspline lordspline commented Jun 17, 2026

Copy link
Copy Markdown

Problem

Bayesian/frequentist experiments let you configure a statistics level (e.g. Bayesian / 90%) on the experiment settings, and the results query already computes the credible/confidence interval at that configured level. But the results presentation hardcoded 95% in the interval labels, so a 90%-configured experiment showed a correctly-computed 90% interval mislabeled as Credible interval (95%). This mismatch undermines trust in the numbers.

Addresses PostHog#62236

Changes

The interval calculation was already correct — it reads stats_config.bayesian.ci_level / stats_config.frequentist.alpha and feeds them into the Bayesian/frequentist engines. The defect was display-only. This PR makes the displayed percentage reflect the configured level instead of a hardcoded 95%:

  • Added getExperimentStatsLevel(experiment, statsMethod?) and formatStatsLevelPercent(level) helpers (plus getExperimentStatsMethod) as a single source of truth in scenes/experiments/utils.ts, backed by new DEFAULT_BAYESIAN_CI_LEVEL / DEFAULT_FREQUENTIST_ALPHA constants.
  • New experiment results view: the results table column header (ResultDetails.tsx) and the "How to read" tooltip (HowToReadTooltip.tsx) now derive the percentage from the configured level. Frequentist terminology ("Confidence interval") and Bayesian terminology ("Credible interval") are preserved; the label method follows the result's method.
  • Consolidated the three already-correct sites that duplicated the same expression (Info.tsx, SettingsTab.tsx, StatsMethodModal.tsx) onto the shared helper to prevent this drift from recurring.
  • No hardcoded percentage is swapped for another, and no serializer/viewset change was needed — stats_config is already serialized by the experiment API.

Legacy experiment behavior is intentionally untouched.

Screenshots

Verified the same metric fixture under Bayesian / 90% and Bayesian / 95%; the results column header tracks the configured level (CREDIBLE INTERVAL (90%) vs CREDIBLE INTERVAL (95%)), matching the settings panel. Screenshots are attached in the originating agent session.

How did you test this code?

I'm an agent. Automated tests I actually ran:

  • Backend products/experiments/backend/hogql_queries/test/test_stats_config.py — 43 passed, including new parameterized tests proving the Bayesian ci_level and frequentist alpha change the computed interval, and that missing/default config matches the explicit 95% interval.
  • Backend products/experiments/backend/test/test_presentation_api.py — 6 new parameterized API round-trip tests (Bayesian 90/95/99, default, frequentist 90/99) confirming the configured level survives create → serialize → read.
  • Frontend frontend/src/scenes/experiments/utils.test.ts — new parameterized tests for getExperimentStatsLevel (Bayesian levels, default fallback, frequentist 1 - alpha, method override) and formatStatsLevelPercent.
  • Frontend formatter ran clean; typescript:check shows no new errors in changed files (pre-existing repo-wide kea *LogicType/implicit-any noise unrelated to this change).
  • Visual Rendered the experiment results Storybook story with stats_config at ci_level 0.90 and 0.95 and confirmed the column header and settings panel agree (90% and 95% respectively).

I did not exercise a live end-to-end experiment with real ingested data; verification was via the calculation/serializer tests above plus the Storybook render.

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Docs update

No docs change needed.

🤖 Agent context

Autonomy: Human-driven (agent-assisted) — directed by the requester; please assign the DRI for review.

Authored with Captain Capy (Capy coding agent). Approach: traced the configured level from persistence (Experiment.stats_config) through the new-engine query runner into the Bayesian/frequentist stats methods, and proved numerically that the interval already responds to the level (90% width ≈ 1.78 vs 95% ≈ 2.12 vs 99% ≈ 2.79; frequentist mirrors via alpha). That isolated the bug to display only. Chose the smallest correct fix — derive the label from the already-serialized stats_config — over a larger backend change that returned the level in the response, since the calculation was never wrong. Consolidated existing duplicated level-derivation onto one helper to stop the labels and settings from drifting apart again. Agent-authored; requires human review (no self-merge).

…ics level

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@lordspline lordspline added the capy Generated by capy.ai label Jun 17, 2026 — with Capy AI
@capy-ai capy-ai Bot changed the title fix(experiments): derive interval label from configured stats level fix(experiments): show configured statistics level on interval labels Jun 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

capy Generated by capy.ai

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant