Skip to content

feat(experiments): add configurable baseline variant#20

Open
lordspline wants to merge 3 commits into
masterfrom
capy/configurable-experiment-baseli-2
Open

feat(experiments): add configurable baseline variant#20
lordspline wants to merge 3 commits into
masterfrom
capy/configurable-experiment-baseli-2

Conversation

@lordspline

@lordspline lordspline commented Jun 19, 2026

Copy link
Copy Markdown

This PR introduces configurable baseline variants for experiments, allowing users to select any variant as the baseline instead of assuming control. The change is backward compatible and updates calculations, validation, and UI labels across the experiment stack.

Backend

  • products/experiments/backend/hogql_queries/experiment_trends_query_runner.py: Read baseline_variant_key from stats_config and use it to determine baseline instead of hardcoded CONTROL_VARIANT_KEY
  • products/experiments/backend/hogql_queries/experiment_funnels_query_runner.py: Same baseline configuration logic for funnel metrics
  • products/experiments/backend/hogql_queries/utils.py: Updated error messages to reference the configured baseline key
  • products/experiments/backend/hogql_queries/test/test_stats_config.py: Added tests for legacy query runners respecting baseline_variant_key

Frontend

  • frontend/src/scenes/experiments/MetricsView/shared/ErrorChecklist.tsx: Dynamic error messages using baseline_variant_key from stats_config
  • frontend/src/scenes/experiments/legacy/metricsView/LegacyErrorChecklist.tsx: Same baseline-aware error messaging
  • frontend/src/scenes/experiments/constants.ts: Updated conclusion descriptions to use "baseline" instead of "control"
  • frontend/src/scenes/experiments/stories/ExperimentFrequentistFiveVariants.stories.tsx: Renamed story to ExperimentFrequentistFiveVariantsCustomBaseline to demonstrate the feature

Fixtures

  • frontend/src/mocks/fixtures/api/experiments/experiment_frequentist_five_variants.json: Added baseline_variant_key: "test-2" to stats_config
  • frontend/src/mocks/fixtures/api/experiments/experiment_frequentist_five_variants_*_metric_result.json: Updated mock results to reflect test-2 as the baseline with other variants as test variants

Existing experiments remain compatible because baseline_variant_key defaults to control when not set.

Open POST-069 POST-069


View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.

…ontrol

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@lordspline lordspline added the capy Generated by capy.ai label Jun 19, 2026 — with Capy AI

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added 2 comments

Comment thread products/experiments/backend/hogql_queries/experiment_funnels_query_runner.py Outdated
lordspline and others added 2 commits June 19, 2026 04:10
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

capy Generated by capy.ai

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant