feat(experiments): configurable analysis baseline variant

[lordspline]

Problem

Experiment analysis was hard-coded to compare every variant against a variant literally keyed control (see PostHog#30796). Teams that want to compare against a different arm — or whose baseline should change mid-flight — had no way to do it. This change lets the analysis baseline be any existing variant, selectable at creation and switchable later, while keeping every legacy experiment on its current control-first behavior.

The baseline is persisted as stats_config.baseline_variant_key. There is no migration and no schema change — it rides the existing stats_config JSON field, and any experiment without the key resolves to control exactly as before.

Changes

Backend already read baseline_variant_key through the query runner, validation, and result labels. This PR closes the remaining gaps:

• Graceful fallback when the referenced variant disappears. New resolve_baseline_variant_key() resolves the configured baseline against the variants actually present (configured → control → first variant) before splitting baseline/test. If the chosen baseline is later removed directly on the flag, analysis degrades instead of erroring. The experiment-update path still validates and rejects dangling baselines on the normal edit flow.
• Creation-time picker. The new-experiment wizard's analysis step gains a "Which variant is the baseline?" selector (defaults to control/first), so the baseline can be chosen up front rather than only after creation.
• Analysis settings hardening. The existing baseline selector now guards against double-submission while the save is in flight and surfaces a warning (with a safe fallback) when the stored baseline no longer matches a variant.
• Consistent FE/BE resolution. A frontend resolveBaselineVariantKey() mirrors the backend so the distribution table and settings agree with what the query runner computes.
• Activity logging of baseline changes is preserved (surfaced via the existing stats_config diff) and covered by a test.

Frontend changes: the baseline switch visibly re-pivots the results scorecard so the selected variant becomes the reference row and the others are compared against it. (Screenshots captured in the authoring session; the distribution-table "Baseline" tag is gated behind the EXPERIMENTS_EXCLUDED_VARIANTS flag.)

How did you test this code?

I'm an agent (Captain Capy). I did not hand-test beyond an automated browser session. Automated checks actually run:

• Backend (pytest): test_stats_config.py resolver/split/runner-baseline cases, test_experiment_service.py baseline validation + cross-state (draft/running/completed) update + duplication-preserves-baseline, and a test_presentation_api.py activity-log assertion — all green.
• Frontend (jest): resolveBaselineVariantKey unit cases — green.
• Migrations: makemigrations --check reports no changes (confirming the no-migration design).
• Browser: an agent-driven session logged in, opened a seeded 3-variant experiment, switched the baseline from control to test-1 in settings, and confirmed the metrics scorecard re-pivoted so test-1 became the baseline/reference row.

CI runs the full type-check, lint, and remaining suites.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Authored by Captain Capy. The approach was scoped deliberately to the backward-compatible "explicit baseline key" design: existing experiments default to control, customer variant keys are never renamed, and historical results are never silently reinterpreted. Relaxing the mandatory control variant at flag-creation/MCP level (so a baseline could have a fully arbitrary name with no control present) was intentionally deferred as a larger, cross-cutting follow-up — this PR keeps the control convention for flag creation and only makes the analysis baseline configurable. No serializer/MCP schema change was needed because the key flows through the existing untyped stats_config passthrough. Reviewers should sanity-check the fallback ordering in resolve_baseline_variant_key and the wizard placement of the creation-time picker.

[capy-ai]

Added 1 comment