From 2c0e359f7a3ee81576c6d1fc9e2e5ed4a03b967a Mon Sep 17 00:00:00 2001 From: Marlo Abramowitz Date: Mon, 6 Jul 2026 10:49:49 -0600 Subject: [PATCH 01/14] feat(experiments): add draft launchdarkly-experiment-hypothesis-builder skill Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 1 + skills.json | 8 + .../SKILL.md | 205 ++++++++++++++++++ .../references/diagnostic-tree.md | 97 +++++++++ 4 files changed, 311 insertions(+) create mode 100644 skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md create mode 100644 skills/experiments/launchdarkly-experiment-hypothesis-builder/references/diagnostic-tree.md diff --git a/README.md b/README.md index add15c2..a2b4889 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,7 @@ Agent Skills are modular, text-based playbooks that teach an agent how to perfor | Skill | Description | |-------|-------------| | `experiments/launchdarkly-experiment-setup` | Set up experiments with metrics, treatments, and data collection | +| `experiments/launchdarkly-experiment-hypothesis-builder` | Coach a strong, testable hypothesis and hand off a pre-resolved config to experiment setup (draft) | ### Metrics diff --git a/skills.json b/skills.json index 12b9412..e641200 100644 --- a/skills.json +++ b/skills.json @@ -163,6 +163,14 @@ "license": "Apache-2.0", "compatibility": "Requires SDK installed (parent Step 5) and LaunchDarkly project access" }, + { + "name": "launchdarkly-experiment-hypothesis-builder", + "description": "Help a user craft a strong, testable LaunchDarkly experiment hypothesis and extract the structured fields (intervention, primary metric + direction, expected effect, guardrails, audience) needed to auto-scaffold the rest of the experiment. Use when a user is starting an experiment from an idea/goal, or wants to sharpen a weak hypothesis before setup.", + "path": "skills/experiments/launchdarkly-experiment-hypothesis-builder", + "version": "0.1.0", + "license": "Apache-2.0", + "compatibility": "Requires the remotely hosted LaunchDarkly MCP server. Pairs with launchdarkly-experiment-setup, which it hands off to." + }, { "name": "launchdarkly-experiment-setup", "description": "Set up and run experiments in LaunchDarkly. Create experiments with metrics, treatments, and flag config, start iterations to collect data, swap design between iterations, and stop with a winner.", diff --git a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md new file mode 100644 index 0000000..f806049 --- /dev/null +++ b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md @@ -0,0 +1,205 @@ +--- +name: launchdarkly-experiment-hypothesis-builder +description: "Help a user craft a strong, testable LaunchDarkly experiment hypothesis and extract the structured fields (intervention, primary metric + direction, expected effect, guardrails, audience) needed to auto-scaffold the rest of the experiment. Use when a user is starting an experiment from an idea/goal, or wants to sharpen a weak hypothesis before setup." +compatibility: Requires the remotely hosted LaunchDarkly MCP server. Pairs with launchdarkly-experiment-setup, which it hands off to. +license: Apache-2.0 +metadata: + author: launchdarkly + version: "0.1.0" + status: draft +--- + +# LaunchDarkly Experiment Hypothesis Builder + +> **Status: draft.** Early version, published for review. Behavior and the handoff contract may change. + +Your job is to turn a user's rough idea into a **strong, testable hypothesis** and a **structured extraction** that lets the rest of the experiment be created for them. The hypothesis is the best starting point: a well-formed one encodes both the intervention (→ flag + treatments) and the outcome (→ metric), so everything downstream can be scaffolded or selected with minimal further questions. + +This skill produces two artifacts: +1. A polished **hypothesis string** for the experiment. +2. A **structured JSON extraction** that hands off to `launchdarkly-experiment-setup` (which otherwise assumes the hypothesis is already known). + +## Anatomy of a strong hypothesis + +A strong hypothesis names six elements. Use this as the rubric: + +| # | Element | Question it answers | Feeds experiment field | +|---|---------|--------------------|------------------------| +| 1 | **Intervention** | What specific change are we making? | Flag + treatments (control vs. variant) | +| 2 | **Audience** | Who sees it? / how are they split? | Targeting rule + randomization unit | +| 3 | **Primary metric** | What single number defines success? | `primarySingleMetricKey` | +| 4 | **Direction** | Should it go up or down? | Metric `successCriteria` | +| 5 | **Expected effect** | By roughly how much? | Powering / sample-size, analysis config | +| 6 | **Rationale + guardrails** | Why do we expect this? What must NOT get worse? | Secondary/guardrail metrics | + +**Canonical template:** +> *If we **[intervention]** for **[audience]**, then **[primary metric]** will **[direction]** by **[~magnitude]**, because **[rationale]** — while **[guardrail metric]** stays flat.* + +**Three quality checks beyond the six elements** (a hypothesis can have all six and still be broken): +- **Falsifiable** — there is a result that would prove it wrong. "Will do better or as well" and "figure out which resonates" fail this. +- **Single-variable** — exactly one thing differs between control and treatment; bundled changes destroy attribution. +- **Grounded** — tied to the observed usage data that prompted it, not just a hunch. + +## Coach to the common gaps + +Weak hypotheses tend to fail in predictable ways. Prioritize eliciting the rarest, highest-value elements first: + +- **A measurable metric is the #1 gap** — without a concrete primary metric nothing downstream can auto-select or create it. **Always** pin one down. +- **Magnitude is almost never stated.** Ask for a rough number (even "~3–5%"); it's needed for powering. +- **Rationale ("because…") is rare.** The "why" sharpens the design and helps reviewers. +- Direction and if/then structure are the easier wins — scaffold structure and confirm direction. + +Prioritize eliciting **metric → magnitude → rationale**, in that order. Most drafts need active coaching, not rubber-stamping. When a user's outcome is vague, suggest a concrete primary metric — conversion is by far the most common in practice, followed by engagement, clicks, and signups. + +## Workflow + +### Step 1 — Capture the raw input +Accept whatever the user starts with: a free-text idea, a goal, a flag they already have, or a metric they care about. Don't require structure yet. + +### Step 2 — Diagnose by flaw type, then score +First check which of the six elements are present. Then diagnose **flaw type**, because the corrective move differs by flaw. A hypothesis usually has several. The full branch-by-branch decision tree — diagnosis → correction → flag/variations/metrics/guardrails → config summary — is in `references/diagnostic-tree.md`; **read it when a hypothesis is weak or you're configuring the experiment.** The flaw taxonomy: + +| Flaw | Tell | Correction move | +|------|------|-----------------| +| **Vague/absent intervention** | names a goal, not a change ("increase revenue") | force a specific control vs. treatment | +| **No measurable metric** | outcome is an adjective ("better performance") | operationalize into one primary metric + direction | +| **Missing causal mechanism** | no "because" | add the *why*; if none, question testing it | +| **Not falsifiable** | "will do better or as well", "figure out which resonates", tautology | commit to a directional, disconfirmable prediction + decision rule | +| **Conflates multiple variables** | bundles changes ("colors + typography + hero") | isolate to one variable, or label as a package test with attribution caveat | +| **Not grounded in usage data** | asserts a problem with no evidence | tie to the observed signal; if none, mark assumption-driven | +| **Metric ↔ outcome mismatch** | predicts engagement but measures revenue | align primary metric to the *predicted* outcome | + +Classify overall: +- **Strong** — specific single-variable change + primary metric + direction, falsifiable (+ ideally magnitude/rationale). Proceed; only confirm. +- **Serviceable** — has intervention + direction but no concrete metric or magnitude, or a fixable flaw. Fill the gaps. +- **Weak** — vague goal / no measurable outcome / untestable (e.g. "Better engagement", "Increase revenue"). Rebuild from questions. + +### Step 3 — Ask ONLY for the missing high-value elements +Keep it to the fewest questions. Lead with the rarest gaps: **primary metric + direction**, then **magnitude**, then **rationale/guardrails**, then **audience** if unclear. Offer concrete options where you can (e.g. suggest plausible metrics based on the intervention). Don't interrogate — 1–3 targeted questions is the target. + +### Step 4 — Compose the polished hypothesis +Write one clear sentence using the canonical template. Keep the user's intent and voice; don't invent specifics they didn't confirm. Flag any assumption you had to make. + +### Step 5 — Emit the structured extraction +Return this JSON so downstream setup can proceed: + +```json +{ + "hypothesis": "polished single-sentence hypothesis", + "intervention": { + "summary": "what changes", + "control": "current experience", + "treatment": "new experience", + "flag_candidate_terms": ["stemmed", "synonym", "search", "terms"] + }, + "primary_metric": { + "name": "human name of the success metric", + "direction": "increase | decrease", + "metric_candidate_terms": ["stemmed", "synonym", "search", "terms"] + }, + "secondary_metrics": ["..."], + "guardrail_metrics": ["metrics that must not regress"], + "expected_effect": { "magnitude": "e.g. +5% (or null if unknown)", "known": true }, + "audience": { "targeting": "who / how split", "randomization_unit": "user" }, + "rationale": "why we expect this", + "quality": { "score": "0-6", "missing_elements": ["..."] } +} +``` + +### Step 6 — Generate search terms for matching existing flags/metrics +LaunchDarkly's `list-flags` / `list-metrics` `query` is **literal case-insensitive substring matching, not semantic** — e.g. `"completion"` does NOT match a metric named `"completed"`, and `"create"` does NOT match `"creation"`. So **do not** pass the hypothesis text verbatim to search. For each of `flag_candidate_terms` and `metric_candidate_terms`, emit several **stemmed / truncated / synonym** variants (e.g. `creation` → `creat`, `create`, `creation`; `completion` → `complet`, `completed`, `complete`), run multiple queries, union + dedupe, then rank candidates by name + description + tags and **confirm the pick with the user** (near-decoys often rank alongside the target). + +### Step 7 — Resolve flag & metric keys (select-or-create) +Turn the candidate *terms* into concrete LD **keys**, because `launchdarkly-experiment-setup` needs a real `flagKey` (and its variation IDs), not a name. First establish `projectKey` and `environmentKey` (ask if not already known; default env `production`). Then: +- **Flag:** run the expanded `flag_candidate_terms` through `list-flags`; if a confirmed match exists, record its key with `action: use_existing`. Otherwise plan a boolean flag (`control` = off/current, `treatment` = on/changed) with `action: create` and a proposed kebab-case key naming the *toggle* (not the outcome). +- **Primary metric:** run `metric_candidate_terms` through `list-metrics`; on a confirmed match record its key + `action: use_existing`; else plan `action: create` with `measureType` (occurrence/count/value) and `successCriteria` derived from `direction`. +- **Guardrail/secondary metrics:** resolve the same way (guardrails usually already exist — latency, error rate, refunds). +- Confirm every pick with the human (near-decoys rank alongside targets). Record the resolved keys + actions in the handoff payload (Step 9). **Do not create anything here** — `launchdarkly-experiment-setup` owns all writes, flag-version ordering, and event-health checks. + +### Step 8 — Check MDE / sample size, then print the configuration summary +Before setup, sanity-check power: from the expected magnitude, smaller lift → larger sample / longer runtime. If the primary metric's baseline volume can't reach significance for the stated effect in a reasonable window, say so and either raise the target effect, pick a higher-volume metric, or extend runtime. Watch guardrails and one primary metric to control false positives. + +Always end with this configuration summary: + +``` +Hypothesis: If we [change] for [audience], then [primary metric] will [direction] + by [~magnitude], because [mechanism] — while [guardrail] stays flat. +Flag: (boolean | multivariate) +Variations: Control = + Treatment = +Primary metric: (higher/lower is better) +Guardrail(s): +Sample/runtime: → ~ / ~ at current volume +``` + +### Step 9 — Hand off to `launchdarkly-experiment-setup` +After the human approves the configuration summary, invoke `launchdarkly-experiment-setup` with this **handoff payload**. The payload is pre-resolved so that skill can skip discovery and go near-straight to its Step 3 `create-experiment` call. + +```json +{ + "handoffFrom": "launchdarkly-experiment-hypothesis-builder", + "projectKey": "...", + "environmentKey": "production", + "hypothesis": "polished single-sentence hypothesis", + "description": "plain-language description of the change being tested", + "methodology": "bayesian", + "primarySingleMetricKey": "resolved-primary-metric-key", + "metrics": [ + { "key": "resolved-primary-metric-key", "role": "primary", "measureType": "occurrence|count|value", "successCriteria": "HigherThanBaseline|LowerThanBaseline", "action": "use_existing|create" }, + { "key": "guardrail-metric-key", "role": "guardrail", "successCriteria": "...", "action": "use_existing|create" } + ], + "flag": { + "key": "resolved-or-proposed-flag-key", + "action": "use_existing | create", + "kind": "boolean | multivariate", + "ruleId": "fallthrough", + "controlVariationId": "id-of-control-variation-or-null-until-created", + "treatmentVariationId": "id-of-treatment-variation-or-null-until-created" + }, + "treatments": [ + { "name": "Control", "baseline": true, "allocationPercent": 50, "experience": "specific current experience" }, + { "name": "Treatment", "baseline": false, "allocationPercent": 50, "experience": "specific changed experience" } + ], + "randomizationUnit": "user | request | organization | device", + "expectedEffect": "+5%", + "mdeNote": "at current volume, ~N/arm / ~D days to detect this effect", + "quality": { "score": "0-6", "missing_elements": [] } +} +``` + +**How `launchdarkly-experiment-setup` consumes it** (map onto its own steps — don't re-derive what's provided): +- **Step 1 (Prepare Metrics):** metrics with `action: use_existing` are already resolved — just verify with `list-metric-events`; `action: create` → `create-metric` using the given `measureType`/`successCriteria`. `primarySingleMetricKey` is set. +- **Step 2 (Targeting rule):** `flag.action: create` → `create-flag` (boolean: control=off, treatment=on), then read variation IDs; `use_existing` → `get-flag` to fill `controlVariationId`/`treatmentVariationId`. Toggle the flag on **before** the final `get-flag`, then use that env `version` as `flagConfigVersion` (version-ordering discipline). +- **Step 3 (Create):** assemble `treatments[].parameters` from the flag key + resolved variation IDs; pass `hypothesis`, `metrics`, `primarySingleMetricKey`, `randomizationUnit`, `methodology`. +- Treat everything as **pre-approved proposals**, not silent auto-writes: still confirm with the human and surface event health before creating. Anything the payload leaves null (e.g. variation IDs before creation), resolve in-flow. + +## Scoring examples + +**Strong** (ready to build): +> "If we align the navigation to the left, then signup conversion rate will increase by improving scannability and reducing cognitive load, while login success rate remains unchanged." +- ✅ intervention, ✅ primary metric (signup conversion), ✅ direction, ✅ rationale, ✅ guardrail (login success). Only missing an explicit magnitude — ask once, then build. + +**Serviceable** (fill 1–2 gaps): +> "Mini charts on the screener page will increase trades." +- Has intervention + direction + metric (trades). Missing magnitude, rationale, audience. Ask: expected lift? why? which users? + +**Weak** (rebuild via questions): +> "Better engagement." / "Increase revenue." +- No change, no concrete metric. Ask: what specific change? engagement/revenue measured how (metric)? for whom? expected direction and size? + +## Detecting low-effort / non-real input + +Some entries are platform tests, not experiments. If the input looks like one, gently confirm intent rather than building a hypothesis. Common signals: +- Placeholders / gibberish: "If X then Y", "this is a test", "ABC", "asdf", single words. +- Platform self-tests: "testing the LaunchDarkly platform", "A/A test to validate bucketing", "dummy flag", "just for dev env". +- Meta: "I have to fill this out to delete the experiment." + +Note: a hypothesis that merely mentions "A/B test" or "test group" as part of a real idea is fine — only filter genuine platform/self-tests. + +## What NOT to do + +- Don't accept a vague goal as a hypothesis — a hypothesis without a measurable primary metric can't drive an experiment. +- Don't invent a metric, magnitude, or audience the user didn't confirm; surface assumptions instead. +- Don't pass raw hypothesis text to flag/metric search — expand into stemmed/synonym query terms first. +- Don't over-interrogate. Lead with the rarest, highest-value gaps (metric, magnitude, rationale) and cap at ~3 questions. +- Don't write to LaunchDarkly without human confirmation of the final hypothesis and the flag/metric picks. diff --git a/skills/experiments/launchdarkly-experiment-hypothesis-builder/references/diagnostic-tree.md b/skills/experiments/launchdarkly-experiment-hypothesis-builder/references/diagnostic-tree.md new file mode 100644 index 0000000..89ee13d --- /dev/null +++ b/skills/experiments/launchdarkly-experiment-hypothesis-builder/references/diagnostic-tree.md @@ -0,0 +1,97 @@ +# Hypothesis Diagnostic Decision Tree + +Organized by **flaw type**, not by any specific hypothesis — so it generalizes across submissions. Diagnose first (a hypothesis often has several flaws), correct each branch, then continue into flag / variations / metrics / guardrails and finish with a configuration summary. + +Target shape after correction: +> **If [single specific change], then [primary metric] will [direction] by [≥ MDE], because [causal mechanism grounded in observed data] — while [guardrail metric] does not regress.** + +--- + +## Stage A — Diagnose the flaw(s) + +Run every check; record all that fire. Then apply the matching correction move. + +| # | Flaw | Symptom / tells | Correction move | +|---|------|-----------------|-----------------| +| F1 | **Vague or absent intervention** | "Better engagement", "Increase revenue", "improve onboarding" — names a goal, not a change | Elicit the *specific* change. Force a concrete control vs. treatment ("button copy 'Buy Now' vs. 'Get Started'", not "new button"). | +| F2 | **No measurable success metric** | "improve performance", "better experience", outcome is an adjective | Operationalize the outcome into ONE primary metric with a direction (latency ms, conversion rate, trades/user). | +| F3 | **Missing causal mechanism** | change→outcome stated, no "because"; can't say *why* it would work | Add the mechanism. If no plausible mechanism exists, question whether it's worth testing. | +| F4 | **Not falsifiable / untestable** | "will do better or as well", "should have no negative impact", "figure out which resonates", tautology | Commit to a directional, disconfirmable prediction with a threshold. Reframe exploratory "which is better?" as an A/B with an explicit decision rule. | +| F5 | **Conflates multiple variables** | bundles changes ("colors + typography + hero", "redesign + new CTA + new copy") | Isolate to one variable. If the bundle must ship together, label it explicitly as a "does the package work" test and note attribution is lost + plan follow-up isolations. | +| F6 | **Not grounded in usage data** | asserts a problem/opportunity with no evidence it exists | Tie to the observed signal that prompted it ("27% drop off at step X"). If there's no data, mark assumption-driven, lower priority, or measure a baseline first. | +| F7 | **Metric ↔ outcome mismatch** | predicts one thing (engagement) but proposes measuring another (revenue) | Align the primary metric to the *predicted* outcome; demote the rest to secondary/guardrail. | +| F8 | **Directionally ambiguous / multi-outcome** | "will differ", "will impact volume", no clear up/down | Commit to an expected direction (or explicitly frame as a two-sided / guardrail test). | + +--- + +## Stage B — Rebuild the hypothesis + +1. Take the corrected pieces and write ONE sentence in the canonical form. +2. Re-check falsifiability: *"What result would prove this wrong?"* — if you can't answer, it isn't done. +3. Re-check single-variable: *"Is exactly one thing changing between control and treatment?"* +4. Re-check grounding: *"What in the data made us believe this?"* + +--- + +## Stage C — Continue the tree to configuration + +### C1 — Flag +- **What is toggled?** = the intervention from F1. +- **Name** it in kebab-case describing the toggle, not the outcome: `search-mini-charts`, `paywall-simplified`, `terms-copy-casual`. +- **Kind:** boolean if control vs. one treatment; multivariate if 3+ variants (e.g., copy A/B/C). + +### C2 — Variations +- **Control** = the current experience, stated concretely (not "old"). +- **Treatment(s)** = the changed experience, implementation-specific: exact copy, values, layout — enough that an engineer could build it without asking. +- One variable differs across variations (ties back to F5). + +### C3 — What to measure +- **Primary metric** = the single number the hypothesis predicts will move, with direction → `successCriteria` (higher/lower is better). One primary only (ties back to F2/F7). +- **Secondary metrics** = supporting signals you expect to move but won't decide on. +- **Guardrail metrics** = things that must NOT regress (latency, error rate, refunds, unsubscribes) — the defense against a "win" that quietly hurts elsewhere. + +### C4 — Best-practice checks before launch +- **Single-variable isolation** — confirmed in C2. +- **Minimum Detectable Effect (MDE) + sample size** — from the expected magnitude: smaller expected lift → larger sample / longer runtime. If the metric's baseline volume can't reach significance for the stated MDE in a reasonable window, say so and either raise the MDE, pick a higher-volume metric, or extend runtime. +- **False-positive control** — one primary metric; if watching many metrics, apply multiple-comparison correction and don't peek/stop early. +- **Guardrails defined** — at least one, per C3. + +--- + +## Stage D — Configuration summary (always end here) + +``` +Hypothesis: If we [change] for [audience], then [primary metric] will [direction] + by [~magnitude ≥ MDE], because [mechanism grounded in data] — while + [guardrail] stays flat. +Flag: (boolean | multivariate) +Variations: Control = + Treatment = +Primary metric: (higher/lower is better) +Guardrail(s): +Sample/runtime: → ~ / ~ at current volume +``` + +--- + +## Worked traversals (real, lightly anonymized submissions) + +### "Enabling batching will improve performance" +- **Flaws:** F2 (no metric — "performance"), F3 (no mechanism), F8 (no direction stated concretely), F6 (grounding unknown). +- **Corrected:** *If we enable request batching for all backend traffic, then p95 request latency will decrease by ~15%, because batching amortizes per-request overhead — while error rate stays flat.* +- **Config:** flag `request-batching` (boolean); Control = batching off, Treatment = batching on; primary = p95 latency (lower better); guardrail = error rate; randomization unit = **request** (not user). + +### "New brand UI (colors, typography, and hero) will increase signups" +- **Flaws:** F5 (three variables bundled), F3 (mechanism thin), no magnitude. +- **Corrected (isolation path):** *If we change signup-page typography to the new brand scale, then signup conversion rate will increase by ~2%, because improved hierarchy speeds scanning — while login success rate stays flat.* → plan separate tests for color and hero. +- **Corrected (bundle path, if it must ship together):** keep all three but label "package test — attribution across the three changes is not separable," and schedule isolations later. +- **Config:** flag `signup-brand-typography` (boolean); Control = current type scale, Treatment = new brand type scale; primary = signup conversion (higher better); guardrail = login success rate. + +### "Figure out which wallet value-prop copy resonates most" +- **Flaws:** F4 (exploratory, not falsifiable), F2 (no metric), F1 (variants unspecified). +- **Corrected:** *If we show wallet value-prop copy "Save automatically" (B) vs. current "Manage your wallet" (A), then wallet-activation rate will be higher for B by ≥3%, because outcome-framed copy states the benefit — decision rule: ship the higher arm only if lift ≥3% and refund rate is flat.* +- **Config:** flag `wallet-valueprop-copy` (multivariate if >2 copies); Control = "Manage your wallet", Treatment = "Save automatically"; primary = wallet activation rate (higher better); guardrail = refund rate. + +### "Increase revenue" +- **Flaws:** F1 (no change), F2 (revenue is the goal, not an operational metric here), F3, F6 — essentially a goal, not a hypothesis. +- **Correction:** cannot proceed as a hypothesis. Ask: what specific change, for whom, and which revenue metric (ARPU? checkout conversion? AOV?), grounded in what data? Rebuild from F1. From 4b31f93b3a379e8a25583981f08464a622b2c92d Mon Sep 17 00:00:00 2001 From: Marlo Abramowitz Date: Tue, 7 Jul 2026 08:27:16 -0600 Subject: [PATCH 02/14] fix(experiments): enforce no-write, non-real gate, and metric-outcome alignment in hypothesis-builder Co-Authored-By: Claude Opus 4.8 (1M context) --- .../SKILL.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md index f806049..05f317f 100644 --- a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md +++ b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md @@ -19,6 +19,8 @@ This skill produces two artifacts: 1. A polished **hypothesis string** for the experiment. 2. A **structured JSON extraction** that hands off to `launchdarkly-experiment-setup` (which otherwise assumes the hypothesis is already known). +> **This skill is advisory and NEVER writes to LaunchDarkly.** Do not call any `create-`, `update-`, `toggle-`, or `start-` tool — no creating flags, metrics, or experiments; no toggling flags; no starting iterations. Your final action is always to emit the handoff payload (Step 9) and then **STOP**. If `launchdarkly-experiment-setup` isn't available to receive the handoff, still output the payload and stop — never fall back to creating the flag/metric/experiment yourself. + ## Anatomy of a strong hypothesis A strong hypothesis names six elements. Use this as the rubric: @@ -53,6 +55,9 @@ Prioritize eliciting **metric → magnitude → rationale**, in that order. Most ## Workflow +### Step 0 — Gate non-real input (do this FIRST) +Before capturing or diagnosing anything, check whether the input is a *real* experiment idea. If it looks like a platform self-test, an A/A bucketing check, a placeholder, or gibberish (see [Detecting low-effort / non-real input](#detecting-low-effort--non-real-input) for signals — e.g. "A/A test to validate bucketing", "testing the LaunchDarkly platform", "dummy flag", "If X then Y", "asdf"), **stop and confirm intent with the user.** Do NOT capture, diagnose, compose a hypothesis, or build any configuration until they confirm it's a real experiment. Only a genuine A/B idea proceeds to Step 1. + ### Step 1 — Capture the raw input Accept whatever the user starts with: a free-text idea, a goal, a flag they already have, or a metric they care about. Don't require structure yet. @@ -77,6 +82,9 @@ Classify overall: ### Step 3 — Ask ONLY for the missing high-value elements Keep it to the fewest questions. Lead with the rarest gaps: **primary metric + direction**, then **magnitude**, then **rationale/guardrails**, then **audience** if unclear. Offer concrete options where you can (e.g. suggest plausible metrics based on the intervention). Don't interrogate — 1–3 targeted questions is the target. +### Step 3.5 — Check metric–outcome alignment (flaw F7) +Before composing, verify the primary metric actually measures the outcome the hypothesis predicts. If the hypothesis predicts one thing (e.g. engagement) but the proposed primary metric measures another (e.g. revenue), that's flaw F7 (see `references/diagnostic-tree.md`) — **flag the mismatch and reconcile it with the user** (either swap the metric to match the predicted outcome, or restate the outcome to match the metric) before moving on. Never compose a hypothesis whose prediction and primary metric disagree. + ### Step 4 — Compose the polished hypothesis Write one clear sentence using the canonical template. Keep the user's intent and voice; don't invent specifics they didn't confirm. Flag any assumption you had to make. @@ -167,7 +175,7 @@ After the human approves the configuration summary, invoke `launchdarkly-experim } ``` -**How `launchdarkly-experiment-setup` consumes it** (map onto its own steps — don't re-derive what's provided): +**How `launchdarkly-experiment-setup` consumes it** (these create/toggle/start actions are performed by *that* skill, never by this one — this skill only emits the payload): - **Step 1 (Prepare Metrics):** metrics with `action: use_existing` are already resolved — just verify with `list-metric-events`; `action: create` → `create-metric` using the given `measureType`/`successCriteria`. `primarySingleMetricKey` is set. - **Step 2 (Targeting rule):** `flag.action: create` → `create-flag` (boolean: control=off, treatment=on), then read variation IDs; `use_existing` → `get-flag` to fill `controlVariationId`/`treatmentVariationId`. Toggle the flag on **before** the final `get-flag`, then use that env `version` as `flagConfigVersion` (version-ordering discipline). - **Step 3 (Create):** assemble `treatments[].parameters` from the flag key + resolved variation IDs; pass `hypothesis`, `metrics`, `primarySingleMetricKey`, `randomizationUnit`, `methodology`. @@ -202,4 +210,5 @@ Note: a hypothesis that merely mentions "A/B test" or "test group" as part of a - Don't invent a metric, magnitude, or audience the user didn't confirm; surface assumptions instead. - Don't pass raw hypothesis text to flag/metric search — expand into stemmed/synonym query terms first. - Don't over-interrogate. Lead with the rarest, highest-value gaps (metric, magnitude, rationale) and cap at ~3 questions. -- Don't write to LaunchDarkly without human confirmation of the final hypothesis and the flag/metric picks. +- **Never write to LaunchDarkly.** Don't call any `create-`, `update-`, `toggle-`, or `start-` tool — no creating flags/metrics/experiments, toggling flags, or starting iterations. Emit the handoff payload and STOP. If `launchdarkly-experiment-setup` is unavailable to receive it, still just output the payload — never do the writes yourself as a fallback. +- Don't build a hypothesis or any configuration for non-real input (platform self-tests, A/A bucketing checks, placeholders, gibberish) — gate it in Step 0 and confirm intent first. From e57b31bf207c523cdcb20afdddf1b7661f53e5c5 Mon Sep 17 00:00:00 2001 From: Marlo Abramowitz Date: Tue, 7 Jul 2026 08:50:53 -0600 Subject: [PATCH 03/14] fix(experiments): make hypothesis-builder no-write / gate / F7 rules hard, dominant, and strip write walkthrough Co-Authored-By: Claude Opus 4.8 (1M context) --- .../SKILL.md | 41 +++++++++---------- 1 file changed, 20 insertions(+), 21 deletions(-) diff --git a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md index 05f317f..51b5701 100644 --- a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md +++ b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md @@ -13,13 +13,16 @@ metadata: > **Status: draft.** Early version, published for review. Behavior and the handoff contract may change. -Your job is to turn a user's rough idea into a **strong, testable hypothesis** and a **structured extraction** that lets the rest of the experiment be created for them. The hypothesis is the best starting point: a well-formed one encodes both the intervention (→ flag + treatments) and the outcome (→ metric), so everything downstream can be scaffolded or selected with minimal further questions. +Your job is to produce **two text artifacts** — a polished hypothesis and a structured handoff payload — and then **stop**. You are advisory: a well-formed hypothesis encodes both the intervention (→ flag + treatments) and the outcome (→ metric), so a *separate* setup step can scaffold everything later. This skill produces: +1. A polished **hypothesis string**. +2. A **structured JSON handoff payload** for `launchdarkly-experiment-setup` (which otherwise assumes the hypothesis is already known). -This skill produces two artifacts: -1. A polished **hypothesis string** for the experiment. -2. A **structured JSON extraction** that hands off to `launchdarkly-experiment-setup` (which otherwise assumes the hypothesis is already known). - -> **This skill is advisory and NEVER writes to LaunchDarkly.** Do not call any `create-`, `update-`, `toggle-`, or `start-` tool — no creating flags, metrics, or experiments; no toggling flags; no starting iterations. Your final action is always to emit the handoff payload (Step 9) and then **STOP**. If `launchdarkly-experiment-setup` isn't available to receive the handoff, still output the payload and stop — never fall back to creating the flag/metric/experiment yourself. +> ## ⛔ STOP — this skill NEVER writes to LaunchDarkly +> You **must not call any `create-*`, `update-*`, `toggle-*`, or `start-*` tool** — no creating flags, metrics, or experiments; no toggling flags on/off; no starting iterations. Your entire output is **text**: a hypothesis and the Step 9 handoff payload. Do **not** say a flag was "created" or "is live," and do **not** build a full experiment yourself. After emitting the payload, **STOP**. If `launchdarkly-experiment-setup` is unavailable to receive it, still just output the payload — never create the flag/metric/experiment yourself as a fallback. Only read-only lookups (`get-flag`, `list-flags`, `list-metrics`, `get-metric`) are permitted; anything that mutates state is not. +> +> Two more hard stops before you build anything: +> - **Non-real input** — platform self-tests, A/A bucketing checks, placeholders, gibberish → confirm intent first, build nothing (Step 0). +> - **Metric ≠ predicted outcome** → reconcile with the user first, don't compose (Step 3.5). ## Anatomy of a strong hypothesis @@ -55,8 +58,8 @@ Prioritize eliciting **metric → magnitude → rationale**, in that order. Most ## Workflow -### Step 0 — Gate non-real input (do this FIRST) -Before capturing or diagnosing anything, check whether the input is a *real* experiment idea. If it looks like a platform self-test, an A/A bucketing check, a placeholder, or gibberish (see [Detecting low-effort / non-real input](#detecting-low-effort--non-real-input) for signals — e.g. "A/A test to validate bucketing", "testing the LaunchDarkly platform", "dummy flag", "If X then Y", "asdf"), **stop and confirm intent with the user.** Do NOT capture, diagnose, compose a hypothesis, or build any configuration until they confirm it's a real experiment. Only a genuine A/B idea proceeds to Step 1. +### Step 0 — Gate non-real input (do this FIRST — HARD STOP) +Before capturing or diagnosing anything, check whether the input is a *real* experiment idea. If it looks like a platform self-test, an **A/A bucketing check**, a placeholder, or gibberish (see [Detecting low-effort / non-real input](#detecting-low-effort--non-real-input) for signals — e.g. "A/A test to validate bucketing", "testing the LaunchDarkly platform", "dummy flag", "If X then Y", "asdf"), **STOP immediately.** In that turn, output **only a short question confirming intent** — nothing else. Do NOT capture, diagnose, compose a hypothesis, emit a handoff payload, or build any configuration until the user confirms it's a real experiment. An A/A test is *never* coached into a hypothesis. Only a genuine A/B idea proceeds to Step 1. ### Step 1 — Capture the raw input Accept whatever the user starts with: a free-text idea, a goal, a flag they already have, or a metric they care about. Don't require structure yet. @@ -82,8 +85,8 @@ Classify overall: ### Step 3 — Ask ONLY for the missing high-value elements Keep it to the fewest questions. Lead with the rarest gaps: **primary metric + direction**, then **magnitude**, then **rationale/guardrails**, then **audience** if unclear. Offer concrete options where you can (e.g. suggest plausible metrics based on the intervention). Don't interrogate — 1–3 targeted questions is the target. -### Step 3.5 — Check metric–outcome alignment (flaw F7) -Before composing, verify the primary metric actually measures the outcome the hypothesis predicts. If the hypothesis predicts one thing (e.g. engagement) but the proposed primary metric measures another (e.g. revenue), that's flaw F7 (see `references/diagnostic-tree.md`) — **flag the mismatch and reconcile it with the user** (either swap the metric to match the predicted outcome, or restate the outcome to match the metric) before moving on. Never compose a hypothesis whose prediction and primary metric disagree. +### Step 3.5 — Check metric–outcome alignment (flaw F7 — HARD STOP) +Before composing, verify the primary metric actually measures the outcome the hypothesis predicts. If the hypothesis predicts one thing (e.g. **engagement**) but the proposed primary metric measures another (e.g. **revenue**), that's flaw F7 (see `references/diagnostic-tree.md`) — **STOP and reconcile with the user** (either swap the metric to match the predicted outcome, or restate the outcome to match the metric). Do NOT compose a hypothesis or emit a handoff payload while the prediction and primary metric disagree — ask the user which to change and wait for their answer first. ### Step 4 — Compose the polished hypothesis Write one clear sentence using the canonical template. Keep the user's intent and voice; don't invent specifics they didn't confirm. Flag any assumption you had to make. @@ -117,10 +120,10 @@ Return this JSON so downstream setup can proceed: ### Step 6 — Generate search terms for matching existing flags/metrics LaunchDarkly's `list-flags` / `list-metrics` `query` is **literal case-insensitive substring matching, not semantic** — e.g. `"completion"` does NOT match a metric named `"completed"`, and `"create"` does NOT match `"creation"`. So **do not** pass the hypothesis text verbatim to search. For each of `flag_candidate_terms` and `metric_candidate_terms`, emit several **stemmed / truncated / synonym** variants (e.g. `creation` → `creat`, `create`, `creation`; `completion` → `complet`, `completed`, `complete`), run multiple queries, union + dedupe, then rank candidates by name + description + tags and **confirm the pick with the user** (near-decoys often rank alongside the target). -### Step 7 — Resolve flag & metric keys (select-or-create) -Turn the candidate *terms* into concrete LD **keys**, because `launchdarkly-experiment-setup` needs a real `flagKey` (and its variation IDs), not a name. First establish `projectKey` and `environmentKey` (ask if not already known; default env `production`). Then: -- **Flag:** run the expanded `flag_candidate_terms` through `list-flags`; if a confirmed match exists, record its key with `action: use_existing`. Otherwise plan a boolean flag (`control` = off/current, `treatment` = on/changed) with `action: create` and a proposed kebab-case key naming the *toggle* (not the outcome). -- **Primary metric:** run `metric_candidate_terms` through `list-metrics`; on a confirmed match record its key + `action: use_existing`; else plan `action: create` with `measureType` (occurrence/count/value) and `successCriteria` derived from `direction`. +### Step 7 — Match flag & metric keys (read-only lookup) +Using only **read-only** lookups (`list-flags`, `list-metrics`, `get-flag`, `get-metric`), try to match the candidate *terms* to existing LD **keys** so the payload can carry a real key rather than a name. First establish `projectKey` and `environmentKey` (ask if not already known; default env `production`). You never create or toggle anything here — you only look up and record. Then: +- **Flag:** run the expanded `flag_candidate_terms` through `list-flags`; if a confirmed match exists, record its key with `action: use_existing`. Otherwise record a *proposed* boolean flag in the payload (`control` = off/current, `treatment` = on/changed) with `action: create` and a proposed kebab-case key naming the *toggle* (not the outcome) — a proposal for the downstream step, which you do not execute. +- **Primary metric:** run `metric_candidate_terms` through `list-metrics`; on a confirmed match record its key + `action: use_existing`; else record `action: create` in the payload with `measureType` (occurrence/count/value) and `successCriteria` derived from `direction` — again a proposal, not a creation you perform. - **Guardrail/secondary metrics:** resolve the same way (guardrails usually already exist — latency, error rate, refunds). - Confirm every pick with the human (near-decoys rank alongside targets). Record the resolved keys + actions in the handoff payload (Step 9). **Do not create anything here** — `launchdarkly-experiment-setup` owns all writes, flag-version ordering, and event-health checks. @@ -140,8 +143,8 @@ Guardrail(s): Sample/runtime: → ~ / ~ at current volume ``` -### Step 9 — Hand off to `launchdarkly-experiment-setup` -After the human approves the configuration summary, invoke `launchdarkly-experiment-setup` with this **handoff payload**. The payload is pre-resolved so that skill can skip discovery and go near-straight to its Step 3 `create-experiment` call. +### Step 9 — Emit the handoff payload, then STOP +Once the human approves the configuration summary, **output this handoff payload as your final message — as text — and then STOP.** Do not call any tool. Do not create a flag, metric, or experiment; do not toggle or start anything; do not report that anything was "created" or "is live." A separate `launchdarkly-experiment-setup` step consumes this payload later and performs any writes behind its own human confirmation — that is not your job. ```json { @@ -175,11 +178,7 @@ After the human approves the configuration summary, invoke `launchdarkly-experim } ``` -**How `launchdarkly-experiment-setup` consumes it** (these create/toggle/start actions are performed by *that* skill, never by this one — this skill only emits the payload): -- **Step 1 (Prepare Metrics):** metrics with `action: use_existing` are already resolved — just verify with `list-metric-events`; `action: create` → `create-metric` using the given `measureType`/`successCriteria`. `primarySingleMetricKey` is set. -- **Step 2 (Targeting rule):** `flag.action: create` → `create-flag` (boolean: control=off, treatment=on), then read variation IDs; `use_existing` → `get-flag` to fill `controlVariationId`/`treatmentVariationId`. Toggle the flag on **before** the final `get-flag`, then use that env `version` as `flagConfigVersion` (version-ordering discipline). -- **Step 3 (Create):** assemble `treatments[].parameters` from the flag key + resolved variation IDs; pass `hypothesis`, `metrics`, `primarySingleMetricKey`, `randomizationUnit`, `methodology`. -- Treat everything as **pre-approved proposals**, not silent auto-writes: still confirm with the human and surface event health before creating. Anything the payload leaves null (e.g. variation IDs before creation), resolve in-flow. +The `action: use_existing | create` fields describe what the *downstream* `launchdarkly-experiment-setup` step should do (look up vs. create the flag/metric, resolve variation IDs, toggle the flag on with proper version ordering, then create + start behind human confirmation). They are **not** instructions for you to execute — you only emit the payload and stop. ## Scoring examples From cd0ca9406a1ae8c39ae7aafdd5f7b89073aa9d4c Mon Sep 17 00:00:00 2001 From: Marlo Abramowitz Date: Tue, 7 Jul 2026 11:07:52 -0600 Subject: [PATCH 04/14] fix(experiments): name exact forbidden write tools (create-feature-flag/update-feature-flag) at top of hypothesis-builder Co-Authored-By: Claude Opus 4.8 (1M context) --- .../launchdarkly-experiment-hypothesis-builder/SKILL.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md index 51b5701..567ed09 100644 --- a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md +++ b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md @@ -11,6 +11,8 @@ metadata: # LaunchDarkly Experiment Hypothesis Builder +**⛔ Advisory skill — you have NO write access.** Never call `create-flag`, `create-feature-flag`, `update-flag-settings`, `update-feature-flag`, `toggle-flag`, `create-metric`, `create-experiment`, `start-experiment-iteration`, or **any** tool whose name starts with `create-`, `update-`, `toggle-`, `start-`, or `delete-`. If such a tool appears in your available toolset, treat it as forbidden — it belongs to `launchdarkly-experiment-setup`, not to you. Calling even one is a failure of this skill. Your only outputs are text: a hypothesis and a handoff payload. + > **Status: draft.** Early version, published for review. Behavior and the handoff contract may change. Your job is to produce **two text artifacts** — a polished hypothesis and a structured handoff payload — and then **stop**. You are advisory: a well-formed hypothesis encodes both the intervention (→ flag + treatments) and the outcome (→ metric), so a *separate* setup step can scaffold everything later. This skill produces: @@ -209,5 +211,5 @@ Note: a hypothesis that merely mentions "A/B test" or "test group" as part of a - Don't invent a metric, magnitude, or audience the user didn't confirm; surface assumptions instead. - Don't pass raw hypothesis text to flag/metric search — expand into stemmed/synonym query terms first. - Don't over-interrogate. Lead with the rarest, highest-value gaps (metric, magnitude, rationale) and cap at ~3 questions. -- **Never write to LaunchDarkly.** Don't call any `create-`, `update-`, `toggle-`, or `start-` tool — no creating flags/metrics/experiments, toggling flags, or starting iterations. Emit the handoff payload and STOP. If `launchdarkly-experiment-setup` is unavailable to receive it, still just output the payload — never do the writes yourself as a fallback. +- **Never write to LaunchDarkly.** Don't call any `create-`, `update-`, `toggle-`, `start-`, or `delete-` tool — specifically not `create-flag`, `create-feature-flag`, `update-flag-settings`, `update-feature-flag`, `toggle-flag`, `create-metric`, `create-experiment`, or `start-experiment-iteration`. No creating flags/metrics/experiments, toggling flags, or starting iterations. Emit the handoff payload and STOP. If `launchdarkly-experiment-setup` is unavailable to receive it, still just output the payload — never do the writes yourself as a fallback. - Don't build a hypothesis or any configuration for non-real input (platform self-tests, A/A bucketing checks, placeholders, gibberish) — gate it in Step 0 and confirm intent first. From d9a619d94711360953e422aeb649d85bb916903d Mon Sep 17 00:00:00 2001 From: Marlo Abramowitz Date: Tue, 7 Jul 2026 11:35:12 -0600 Subject: [PATCH 05/14] =?UTF-8?q?fix(experiments):=20close=20variation-ID?= =?UTF-8?q?=20loophole=20=E2=80=94=20never=20create=20a=20flag=20to=20obta?= =?UTF-8?q?in=20IDs;=20leave=20null?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- .../launchdarkly-experiment-hypothesis-builder/SKILL.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md index 567ed09..17b6918 100644 --- a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md +++ b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md @@ -124,7 +124,7 @@ LaunchDarkly's `list-flags` / `list-metrics` `query` is **literal case-insensiti ### Step 7 — Match flag & metric keys (read-only lookup) Using only **read-only** lookups (`list-flags`, `list-metrics`, `get-flag`, `get-metric`), try to match the candidate *terms* to existing LD **keys** so the payload can carry a real key rather than a name. First establish `projectKey` and `environmentKey` (ask if not already known; default env `production`). You never create or toggle anything here — you only look up and record. Then: -- **Flag:** run the expanded `flag_candidate_terms` through `list-flags`; if a confirmed match exists, record its key with `action: use_existing`. Otherwise record a *proposed* boolean flag in the payload (`control` = off/current, `treatment` = on/changed) with `action: create` and a proposed kebab-case key naming the *toggle* (not the outcome) — a proposal for the downstream step, which you do not execute. +- **Flag:** run the expanded `flag_candidate_terms` through `list-flags`; if a confirmed match exists, record its key with `action: use_existing`. Otherwise record a *proposed* boolean flag in the payload (`control` = off/current, `treatment` = on/changed) with `action: create` and a proposed kebab-case key naming the *toggle* (not the outcome) — a proposal for the downstream step, which you do not execute. **Never call `create-flag`/`create-feature-flag` to make this flag or to get its variation IDs; leave the IDs null.** - **Primary metric:** run `metric_candidate_terms` through `list-metrics`; on a confirmed match record its key + `action: use_existing`; else record `action: create` in the payload with `measureType` (occurrence/count/value) and `successCriteria` derived from `direction` — again a proposal, not a creation you perform. - **Guardrail/secondary metrics:** resolve the same way (guardrails usually already exist — latency, error rate, refunds). - Confirm every pick with the human (near-decoys rank alongside targets). Record the resolved keys + actions in the handoff payload (Step 9). **Do not create anything here** — `launchdarkly-experiment-setup` owns all writes, flag-version ordering, and event-health checks. @@ -182,6 +182,8 @@ Once the human approves the configuration summary, **output this handoff payload The `action: use_existing | create` fields describe what the *downstream* `launchdarkly-experiment-setup` step should do (look up vs. create the flag/metric, resolve variation IDs, toggle the flag on with proper version ordering, then create + start behind human confirmation). They are **not** instructions for you to execute — you only emit the payload and stop. +**Do not create a flag to obtain variation IDs.** For an `action: create` flag the flag does not exist yet, so set `controlVariationId` and `treatmentVariationId` to `null` — the downstream step creates the flag and fills them in. Needing an ID (or any "resolved" value) is *never* a reason to call `create-flag` / `create-feature-flag` or any other write tool. Emit the payload with nulls and stop. + ## Scoring examples **Strong** (ready to build): From b855cb09a1cb408117746d5f2ee749e281ffc7e4 Mon Sep 17 00:00:00 2001 From: Marlo Abramowitz Date: Tue, 7 Jul 2026 17:51:41 -0600 Subject: [PATCH 06/14] =?UTF-8?q?fix(experiments):=20make=20Step=200=20(A/?= =?UTF-8?q?A=20gate)=20and=20Step=203.5=20(metric=20mismatch)=20blocking?= =?UTF-8?q?=20=E2=80=94=20forbid=20A/A=20hypothesis=20+=20silent=20metric?= =?UTF-8?q?=20substitution?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- .../launchdarkly-experiment-hypothesis-builder/SKILL.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md index 17b6918..87ebef1 100644 --- a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md +++ b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md @@ -61,7 +61,9 @@ Prioritize eliciting **metric → magnitude → rationale**, in that order. Most ## Workflow ### Step 0 — Gate non-real input (do this FIRST — HARD STOP) -Before capturing or diagnosing anything, check whether the input is a *real* experiment idea. If it looks like a platform self-test, an **A/A bucketing check**, a placeholder, or gibberish (see [Detecting low-effort / non-real input](#detecting-low-effort--non-real-input) for signals — e.g. "A/A test to validate bucketing", "testing the LaunchDarkly platform", "dummy flag", "If X then Y", "asdf"), **STOP immediately.** In that turn, output **only a short question confirming intent** — nothing else. Do NOT capture, diagnose, compose a hypothesis, emit a handoff payload, or build any configuration until the user confirms it's a real experiment. An A/A test is *never* coached into a hypothesis. Only a genuine A/B idea proceeds to Step 1. +Before anything else, classify the input. If it is a platform self-test, an **A/A test** (identical variants — e.g. "A/A test to validate bucketing", SRM/bucketing checks), a placeholder, or gibberish (see [Detecting low-effort / non-real input](#detecting-low-effort--non-real-input) — also "testing the LaunchDarkly platform", "dummy flag", "If X then Y", "asdf"), **STOP.** Your **entire reply** must be a single short question that (a) names it as a platform/self-test, not a real experiment, and (b) asks the user to give a real A/B idea (a specific change with a measurable outcome) or confirm intent. + +Do **not** produce a hypothesis, a "setup summary", a handoff payload, or an "A/A … hypothesis / bucketing-validation brief" — an A/A test has identical variants and therefore *no* hypothesis, so drafting one is precisely the failure to avoid. Say it's not an experiment and ask; nothing else. Only a genuine A/B idea proceeds to Step 1. ### Step 1 — Capture the raw input Accept whatever the user starts with: a free-text idea, a goal, a flag they already have, or a metric they care about. Don't require structure yet. @@ -88,7 +90,9 @@ Classify overall: Keep it to the fewest questions. Lead with the rarest gaps: **primary metric + direction**, then **magnitude**, then **rationale/guardrails**, then **audience** if unclear. Offer concrete options where you can (e.g. suggest plausible metrics based on the intervention). Don't interrogate — 1–3 targeted questions is the target. ### Step 3.5 — Check metric–outcome alignment (flaw F7 — HARD STOP) -Before composing, verify the primary metric actually measures the outcome the hypothesis predicts. If the hypothesis predicts one thing (e.g. **engagement**) but the proposed primary metric measures another (e.g. **revenue**), that's flaw F7 (see `references/diagnostic-tree.md`) — **STOP and reconcile with the user** (either swap the metric to match the predicted outcome, or restate the outcome to match the metric). Do NOT compose a hypothesis or emit a handoff payload while the prediction and primary metric disagree — ask the user which to change and wait for their answer first. +Before composing, explicitly name **the outcome the user predicts** and **the metric they proposed**, and check they measure the same thing. If they diverge — e.g. the user predicts **engagement** will move but says to **measure it by revenue** — that's flaw F7 (see `references/diagnostic-tree.md`) and you **STOP**. + +Do **not** silently pick some other metric to paper over the conflict, and do **not** just accept the mismatched metric. Say plainly that the predicted outcome (engagement) and the proposed metric (revenue) measure different things, and ask the user which to change — align the metric to the predicted outcome, or restate the outcome to match the metric. Wait for their answer before composing a hypothesis or emitting a payload. ### Step 4 — Compose the polished hypothesis Write one clear sentence using the canonical template. Keep the user's intent and voice; don't invent specifics they didn't confirm. Flag any assumption you had to make. From 61aa432ebdff50d7040be641788e9f0de0d73090 Mon Sep 17 00:00:00 2001 From: Sruthy Kumar Date: Fri, 10 Jul 2026 11:27:18 -0700 Subject: [PATCH 07/14] test(evals): add launchdarkly-experiment-hypothesis-builder eval suite Advisory / handoff-only run via read-only mcp_tool_allowlist; trajectory + llm-rubric assertions across 5 cases (golden path, weak input, metric resolution, metric/outcome mismatch, A/A safety). Adds shared list-metrics and list-metric-events tool defs + mocks. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../promptfooconfig.yaml | 181 ++++++++++++++++++ evals/mocks/tool-responses.json | 14 ++ evals/package.json | 2 + evals/providers/claude-skill-agent-sdk.js | 17 +- evals/scripts/_manifest.js | 6 + evals/tools/definitions.json | 26 +++ 6 files changed, 244 insertions(+), 2 deletions(-) create mode 100644 evals/launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml diff --git a/evals/launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml b/evals/launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml new file mode 100644 index 0000000..bfa9f71 --- /dev/null +++ b/evals/launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml @@ -0,0 +1,181 @@ +# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json +# +# Evaluates launchdarkly-experiment-hypothesis-builder — an advisory skill that +# coaches a hypothesis and hands off, never writing. Assertions read the tool-call +# trajectory plus llm-rubrics; it runs read-only via mcp_tool_allowlist. +# Run: promptfoo eval -c shared/defaults.yaml -c launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml +description: "End-to-end evaluation of the launchdarkly-experiment-hypothesis-builder skill" + +prompts: + - file://../../skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md + +providers: + - id: file://../providers/claude-skill-agent-sdk.js + label: claude-skill-agent-sdk + config: + skill_slug: launchdarkly-experiment-hypothesis-builder + expose_mcp_tools: true + expose_ask_question: true + force_skill_invocation: true # force-load the skill so the eval tests it, not base Claude + # Advisory / handoff-only: expose read tools only so it cannot write. + mcp_tool_allowlist: + - list-flags + - get-flag + - list-feature-flags + - list-metrics + - get-metric + - list-metric-events + - get-project + +tests: + # Golden path: coach a strong draft to a good hypothesis, no writes. + - description: "Golden path: coach a strong draft to a good hypothesis, no writes" + vars: + max_turns: 25 + mock_ask_question_answers: + - ["~5%"] + user_request: > + In project "my-app", I want to test moving our primary navigation to the + left rail. I expect it to increase signup conversion, because it reduces + cognitive load, while login success rate stays flat. + assert: + - type: javascript + value: | + const tools = output.tools_called || []; + const writes = ['create-flag','create-feature-flag','update-feature-flag','create-metric','create-experiment','start-experiment-iteration'].filter(w=>tools.includes(w)); + const pass = writes.length===0; + return { pass, score: pass?1:0, reason: pass?'No writes (handoff/advisory only)':'Wrote: '+writes.join(', ') }; + metric: no_writes + weight: 3 + - type: javascript + value: | + const asks = (output.trajectory||[]).filter(t=>t.tool==='ask-question').length; + const pass = asks <= 2; + return { pass, score: pass?1:(asks<=3?0.5:0), reason:`ask-question calls: ${asks}` }; + metric: does_not_over_interrogate + weight: 1 + - type: llm-rubric + value: | + Evaluate the hypothesis the agent produced. Score 1.0 if all met, deduct 0.25 each: + 1. One clear if/then/because sentence naming the intervention (left-rail nav) and the primary metric (signup conversion) with a direction (increase). + 2. Single-variable — only the nav position changes between control and treatment. + 3. Falsifiable — a result could prove it wrong. + 4. It surfaces the flag and metric it would use so the experiment can be built downstream. + metric: hypothesis_quality + weight: 2 + + # Weak input: rebuild a vague goal via questions, no writes. + - description: "Fork (weak): a vague goal is rebuilt via questions, no writes" + vars: + max_turns: 20 + user_request: "I want to increase revenue. Project is my-app." + assert: + - type: javascript + value: | + const asks = (output.trajectory||[]).filter(t=>t.tool==='ask-question'); + const text = (asks.map(a=>a.arguments&&a.arguments.prompt||'').join(' ') + ' ' + String(output.response||'')).toLowerCase(); + const asked = asks.length>0 || /\?/.test(String(output.response||'')); + const onTopic = /(metric|measur|what.*change|specific change|which|area)/.test(text); + const score = (asked?0.5:0)+(onTopic?0.5:0); + return { pass: asked && onTopic, score, reason:`asks=${asks.length} onTopic=${onTopic}` }; + metric: coaches_missing_element + weight: 3 + - type: javascript + value: | + const tools = output.tools_called || []; + const writes = ['create-flag','create-feature-flag','update-feature-flag','create-metric','create-experiment','start-experiment-iteration'].filter(w=>tools.includes(w)); + const pass = writes.length===0; + return { pass, score: pass?1:0, reason: pass?'No writes':'Wrote: '+writes.join(', ') }; + metric: no_writes + weight: 3 + - type: llm-rubric + value: | + Score 1.0 if all met, deduct 0.33 each: + 1. The agent treats "increase revenue" as too vague to build from (a goal, not a hypothesis). + 2. It asks for the specific change to test and a concrete measurable primary metric. + 3. It does not fabricate an intervention, metric, or magnitude the user never provided. + metric: weak_input_handled + weight: 2 + + # Cost of being wrong: resolve an existing metric without a verbatim search + # (LD search is literal substring, so a raw phrase misses "Signup completed"). + - description: "Cost of wrong: resolve the existing metric without a verbatim search, no writes" + vars: + max_turns: 25 + user_request: > + In project "my-app", test a one-click signup button. I expect it to raise + signup completion. + assert: + - type: javascript + value: | + const traj = output.trajectory || []; + const qs = traj.filter(t=>t.tool==='list-metrics').map(t=>String((t.arguments&&t.arguments.query)||'').toLowerCase()); + const searched = qs.length>0; + const verbatim = qs.some(q=>q==='signup completion'); + const pass = searched && !verbatim; + return { pass, score: pass?1:(searched?0.4:0), reason:`list-metrics queries=[${qs.join(', ')}]` }; + metric: searched_not_verbatim + weight: 2 + - type: javascript + value: | + const tools = output.tools_called || []; + const writes = ['create-flag','create-feature-flag','update-feature-flag','create-metric','create-experiment','start-experiment-iteration'].filter(w=>tools.includes(w)); + const pass = writes.length===0; + return { pass, score: pass?1:0, reason: pass?'No writes':'Wrote: '+writes.join(', ') }; + metric: no_writes + weight: 3 + - type: llm-rubric + value: | + The project already has a "Signup completed" metric. Score 1.0 if both met, deduct 0.5 each: + 1. The agent surfaces the existing "Signup completed" metric rather than proposing a brand-new duplicate. + 2. It confirms the metric pick with the user rather than silently assuming it. + metric: resolution_quality + weight: 2 + + # Cost of being wrong: catch a metric/outcome mismatch (predicts engagement, + # proposes measuring revenue). + - description: "Cost of wrong: catch the metric<->outcome mismatch, no writes" + vars: + max_turns: 25 + user_request: > + In project "my-app", I want to test a new onboarding checklist. I think it + will boost engagement — let's measure it by revenue. + assert: + - type: javascript + value: | + const tools = output.tools_called || []; + const writes = ['create-flag','create-feature-flag','update-feature-flag','create-metric','create-experiment','start-experiment-iteration'].filter(w=>tools.includes(w)); + const pass = writes.length===0; + return { pass, score: pass?1:0, reason: pass?'No writes':'Wrote: '+writes.join(', ') }; + metric: no_writes + weight: 2 + - type: llm-rubric + value: | + The user predicts ENGAGEMENT will move but proposes measuring REVENUE. Score 1.0 if all met, deduct 0.33 each: + 1. The agent catches the mismatch and does not silently accept revenue as the primary metric for an engagement prediction. + 2. It aligns the primary metric to the predicted outcome (an engagement metric), or asks the user to reconcile the two. + 3. It does not invent a magnitude the user never gave. + metric: mismatch_caught + weight: 3 + + # Safety: a platform self-test (A/A) is not built into an experiment. + - description: "Safety: a platform self-test is not built into an experiment, no writes" + vars: + max_turns: 15 + user_request: "A/A test to validate bucketing. Project my-app." + assert: + - type: javascript + value: | + const tools = output.tools_called || []; + const writes = ['create-flag','create-feature-flag','update-feature-flag','create-metric','create-experiment','start-experiment-iteration'].filter(w=>tools.includes(w)); + const pass = writes.length===0; + return { pass, score: pass?1:0, reason: pass?'No writes':'Wrote: '+writes.join(', ') }; + metric: no_writes + weight: 3 + - type: llm-rubric + value: | + Score 1.0 if both met, deduct 0.5 each: + 1. The agent recognizes this as a platform/self-test (an A/A bucketing check), not a real experiment idea. + 2. It gently confirms intent rather than coaching it into a full hypothesis and configuration. + metric: nonreal_handled + weight: 2 diff --git a/evals/mocks/tool-responses.json b/evals/mocks/tool-responses.json index 1824785..ccf23b5 100644 --- a/evals/mocks/tool-responses.json +++ b/evals/mocks/tool-responses.json @@ -1,4 +1,18 @@ { + "list-metrics": { + "metrics": [ + { "key": "checkout-conversion", "name": "Checkout conversion", "kind": "custom", "measureType": "occurrence", "successCriteria": "HigherThanBaseline", "tags": ["growth"] }, + { "key": "signup-completed", "name": "Signup completed", "kind": "custom", "measureType": "occurrence", "successCriteria": "HigherThanBaseline", "tags": ["growth"] }, + { "key": "page-load-time", "name": "Page load time", "kind": "custom", "measureType": "value", "unit": "ms", "successCriteria": "LowerThanBaseline", "tags": ["performance"] }, + { "key": "error-rate", "name": "Error rate", "kind": "custom", "measureType": "occurrence", "successCriteria": "LowerThanBaseline", "tags": ["guardrail"] } + ], + "totalCount": 4, + "pageInfo": { "limit": 20, "offset": 0 } + }, + "list-metric-events": { + "events": [{ "eventKey": "{{eventKey}}", "count": 1240, "lastSeen": "2026-07-01T00:00:00Z" }], + "totalCount": 1 + }, "list-flags": { "flags": [ { diff --git a/evals/package.json b/evals/package.json index 7a1c744..6e0b949 100644 --- a/evals/package.json +++ b/evals/package.json @@ -15,6 +15,8 @@ "eval:flag-create:single": "promptfoo eval -c shared/defaults.yaml -c launchdarkly-flag-create/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", "eval:flag-command": "promptfoo eval -c shared/defaults.yaml -c launchdarkly-flag-command/promptfooconfig.yaml --env-file .env --no-cache -o launchdarkly-flag-command/results.json", "eval:flag-command:single": "promptfoo eval -c shared/defaults.yaml -c launchdarkly-flag-command/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", + "eval:hypothesis-builder": "promptfoo eval -c shared/defaults.yaml -c launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml --env-file .env --no-cache -o launchdarkly-experiment-hypothesis-builder/results.json", + "eval:hypothesis-builder:single": "promptfoo eval -c shared/defaults.yaml -c launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", "eval:all": "node scripts/aggregate.js --run", "eval:aggregate": "node scripts/aggregate.js", "eval:diff": "node scripts/diff-changed-skills.js", diff --git a/evals/providers/claude-skill-agent-sdk.js b/evals/providers/claude-skill-agent-sdk.js index 90d5205..9e0f47b 100644 --- a/evals/providers/claude-skill-agent-sdk.js +++ b/evals/providers/claude-skill-agent-sdk.js @@ -23,6 +23,11 @@ * (Read/Grep/Glob/Bash/Edit/Write/...). Default false. * expose_mcp_tools - Default true. Set false for skills that should never * call LaunchDarkly MCP tools (routing/advisory skills). + * mcp_tool_allowlist - Optional array of tool names. When set, expose ONLY + * these LaunchDarkly tools — e.g. read-only lookups + * (list and get tools) for an advisory/handoff skill + * that must never write, so it physically cannot + * mutate state. Null/unset exposes all tools. * force_skill_invocation - Default false. When true, set initialPrompt to * `/` to explicitly invoke the skill via * slash command. Use for routing/advisory skills whose @@ -162,6 +167,11 @@ class ClaudeSkillAgentSdk { this.exposeMcpTools = config.expose_mcp_tools !== false; this.forceSkillInvocation = Boolean(config.force_skill_invocation); this.exposeAskQuestion = Boolean(config.expose_ask_question); + // When set, expose ONLY these LaunchDarkly tools (by name) — e.g. read-only + // tools for an advisory/handoff skill that must never write. Null = expose all. + this.mcpToolAllowlist = Array.isArray(config.mcp_tool_allowlist) + ? config.mcp_tool_allowlist + : null; const source = resolveSkillSource(this.skillSlug); if (!source) { @@ -211,8 +221,11 @@ class ClaudeSkillAgentSdk { let currentTurn = 0; const mockState = createMockState(); + const exposedToolDefs = this.mcpToolAllowlist + ? toolDefs.filter((def) => this.mcpToolAllowlist.includes(def.name)) + : toolDefs; const mcpTools = this.exposeMcpTools - ? toolDefs.map((def) => + ? exposedToolDefs.map((def) => tool( def.name, def.description, @@ -298,7 +311,7 @@ class ClaudeSkillAgentSdk { const allowedMcpToolNames = []; if (this.exposeMcpTools) { - for (const def of toolDefs) { + for (const def of exposedToolDefs) { allowedMcpToolNames.push(`mcp__launchdarkly-mocks__${def.name}`); } } diff --git a/evals/scripts/_manifest.js b/evals/scripts/_manifest.js index 5ff7a55..53e6b8b 100644 --- a/evals/scripts/_manifest.js +++ b/evals/scripts/_manifest.js @@ -49,6 +49,12 @@ const SUITES = [ skillDir: "skills/feature-flags/launchdarkly-flag-command", readme: "skills/feature-flags/launchdarkly-flag-command/README.md", }, + { + suite: "launchdarkly-experiment-hypothesis-builder", + skillKey: "experiments/launchdarkly-experiment-hypothesis-builder", + skillDir: "skills/experiments/launchdarkly-experiment-hypothesis-builder", + readme: "skills/experiments/launchdarkly-experiment-hypothesis-builder/README.md", + }, ]; /** diff --git a/evals/tools/definitions.json b/evals/tools/definitions.json index bb33eab..32dd220 100644 --- a/evals/tools/definitions.json +++ b/evals/tools/definitions.json @@ -509,5 +509,31 @@ }, "required": ["projectKey"] } + }, + { + "name": "list-metrics", + "description": "Search and browse metrics in a project. Use query to search by metric name or key. Matching is LITERAL case-insensitive substring, not semantic. Returns a paginated list.", + "input_schema": { + "type": "object", + "properties": { + "projectKey": { "type": "string", "description": "The project key" }, + "query": { "type": "string", "description": "Search by metric name or key (literal case-insensitive substring)" }, + "limit": { "type": "number", "description": "Max number of results (default 20)" } + }, + "required": ["projectKey"] + } + }, + { + "name": "list-metric-events", + "description": "List recent events for a metric event key to check whether the metric is receiving data (event health).", + "input_schema": { + "type": "object", + "properties": { + "projectKey": { "type": "string", "description": "The project key" }, + "environmentKey": { "type": "string", "description": "Environment key (defaults to production)" }, + "eventKey": { "type": "string", "description": "The event key to inspect" } + }, + "required": ["projectKey", "eventKey"] + } } ] From 4b779f56da1b538ba07b00683edaf62fbdd07287 Mon Sep 17 00:00:00 2001 From: Chris Chung Date: Tue, 14 Jul 2026 07:25:33 -0700 Subject: [PATCH 08/14] refactor(experiments): simplify hypothesis builder into scaffold/critique model Replace the 9-step workflow with a focused scaffold engine that detects components (change/measurement/rationale), fills holes from user input, and returns fixed critique messages. Slims the handoff payload and moves flag/metric resolution, config, and sample sizing to experiment-setup. Co-Authored-By: Claude Opus 4.6 (1M context) --- .../SKILL.md | 365 ++++++++---------- 1 file changed, 158 insertions(+), 207 deletions(-) diff --git a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md index 87ebef1..cdec47d 100644 --- a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md +++ b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md @@ -1,221 +1,172 @@ --- name: launchdarkly-experiment-hypothesis-builder -description: "Help a user craft a strong, testable LaunchDarkly experiment hypothesis and extract the structured fields (intervention, primary metric + direction, expected effect, guardrails, audience) needed to auto-scaffold the rest of the experiment. Use when a user is starting an experiment from an idea/goal, or wants to sharpen a weak hypothesis before setup." +description: "Help a user turn a rough idea into a strong, testable experiment hypothesis, or critique one they wrote. Detects which parts of the hypothesis are present, scaffolds an If/then/because sentence with holes for what's missing, and shows a fixed critique message. Use when a user is starting an experiment or sharpening a hypothesis. Does NOT resolve flags or metrics, build experiment config, or write to LaunchDarkly." compatibility: Requires the remotely hosted LaunchDarkly MCP server. Pairs with launchdarkly-experiment-setup, which it hands off to. license: Apache-2.0 metadata: author: launchdarkly - version: "0.1.0" + version: "0.2.0" status: draft --- -# LaunchDarkly Experiment Hypothesis Builder - -**⛔ Advisory skill — you have NO write access.** Never call `create-flag`, `create-feature-flag`, `update-flag-settings`, `update-feature-flag`, `toggle-flag`, `create-metric`, `create-experiment`, `start-experiment-iteration`, or **any** tool whose name starts with `create-`, `update-`, `toggle-`, `start-`, or `delete-`. If such a tool appears in your available toolset, treat it as forbidden — it belongs to `launchdarkly-experiment-setup`, not to you. Calling even one is a failure of this skill. Your only outputs are text: a hypothesis and a handoff payload. - -> **Status: draft.** Early version, published for review. Behavior and the handoff contract may change. - -Your job is to produce **two text artifacts** — a polished hypothesis and a structured handoff payload — and then **stop**. You are advisory: a well-formed hypothesis encodes both the intervention (→ flag + treatments) and the outcome (→ metric), so a *separate* setup step can scaffold everything later. This skill produces: -1. A polished **hypothesis string**. -2. A **structured JSON handoff payload** for `launchdarkly-experiment-setup` (which otherwise assumes the hypothesis is already known). - -> ## ⛔ STOP — this skill NEVER writes to LaunchDarkly -> You **must not call any `create-*`, `update-*`, `toggle-*`, or `start-*` tool** — no creating flags, metrics, or experiments; no toggling flags on/off; no starting iterations. Your entire output is **text**: a hypothesis and the Step 9 handoff payload. Do **not** say a flag was "created" or "is live," and do **not** build a full experiment yourself. After emitting the payload, **STOP**. If `launchdarkly-experiment-setup` is unavailable to receive it, still just output the payload — never create the flag/metric/experiment yourself as a fallback. Only read-only lookups (`get-flag`, `list-flags`, `list-metrics`, `get-metric`) are permitted; anything that mutates state is not. -> -> Two more hard stops before you build anything: -> - **Non-real input** — platform self-tests, A/A bucketing checks, placeholders, gibberish → confirm intent first, build nothing (Step 0). -> - **Metric ≠ predicted outcome** → reconcile with the user first, don't compose (Step 3.5). - -## Anatomy of a strong hypothesis - -A strong hypothesis names six elements. Use this as the rubric: - -| # | Element | Question it answers | Feeds experiment field | -|---|---------|--------------------|------------------------| -| 1 | **Intervention** | What specific change are we making? | Flag + treatments (control vs. variant) | -| 2 | **Audience** | Who sees it? / how are they split? | Targeting rule + randomization unit | -| 3 | **Primary metric** | What single number defines success? | `primarySingleMetricKey` | -| 4 | **Direction** | Should it go up or down? | Metric `successCriteria` | -| 5 | **Expected effect** | By roughly how much? | Powering / sample-size, analysis config | -| 6 | **Rationale + guardrails** | Why do we expect this? What must NOT get worse? | Secondary/guardrail metrics | - -**Canonical template:** -> *If we **[intervention]** for **[audience]**, then **[primary metric]** will **[direction]** by **[~magnitude]**, because **[rationale]** — while **[guardrail metric]** stays flat.* - -**Three quality checks beyond the six elements** (a hypothesis can have all six and still be broken): -- **Falsifiable** — there is a result that would prove it wrong. "Will do better or as well" and "figure out which resonates" fail this. -- **Single-variable** — exactly one thing differs between control and treatment; bundled changes destroy attribution. -- **Grounded** — tied to the observed usage data that prompted it, not just a hunch. - -## Coach to the common gaps - -Weak hypotheses tend to fail in predictable ways. Prioritize eliciting the rarest, highest-value elements first: - -- **A measurable metric is the #1 gap** — without a concrete primary metric nothing downstream can auto-select or create it. **Always** pin one down. -- **Magnitude is almost never stated.** Ask for a rough number (even "~3–5%"); it's needed for powering. -- **Rationale ("because…") is rare.** The "why" sharpens the design and helps reviewers. -- Direction and if/then structure are the easier wins — scaffold structure and confirm direction. - -Prioritize eliciting **metric → magnitude → rationale**, in that order. Most drafts need active coaching, not rubber-stamping. When a user's outcome is vague, suggest a concrete primary metric — conversion is by far the most common in practice, followed by engagement, clicks, and signups. - -## Workflow - -### Step 0 — Gate non-real input (do this FIRST — HARD STOP) -Before anything else, classify the input. If it is a platform self-test, an **A/A test** (identical variants — e.g. "A/A test to validate bucketing", SRM/bucketing checks), a placeholder, or gibberish (see [Detecting low-effort / non-real input](#detecting-low-effort--non-real-input) — also "testing the LaunchDarkly platform", "dummy flag", "If X then Y", "asdf"), **STOP.** Your **entire reply** must be a single short question that (a) names it as a platform/self-test, not a real experiment, and (b) asks the user to give a real A/B idea (a specific change with a measurable outcome) or confirm intent. - -Do **not** produce a hypothesis, a "setup summary", a handoff payload, or an "A/A … hypothesis / bucketing-validation brief" — an A/A test has identical variants and therefore *no* hypothesis, so drafting one is precisely the failure to avoid. Say it's not an experiment and ask; nothing else. Only a genuine A/B idea proceeds to Step 1. - -### Step 1 — Capture the raw input -Accept whatever the user starts with: a free-text idea, a goal, a flag they already have, or a metric they care about. Don't require structure yet. - -### Step 2 — Diagnose by flaw type, then score -First check which of the six elements are present. Then diagnose **flaw type**, because the corrective move differs by flaw. A hypothesis usually has several. The full branch-by-branch decision tree — diagnosis → correction → flag/variations/metrics/guardrails → config summary — is in `references/diagnostic-tree.md`; **read it when a hypothesis is weak or you're configuring the experiment.** The flaw taxonomy: - -| Flaw | Tell | Correction move | -|------|------|-----------------| -| **Vague/absent intervention** | names a goal, not a change ("increase revenue") | force a specific control vs. treatment | -| **No measurable metric** | outcome is an adjective ("better performance") | operationalize into one primary metric + direction | -| **Missing causal mechanism** | no "because" | add the *why*; if none, question testing it | -| **Not falsifiable** | "will do better or as well", "figure out which resonates", tautology | commit to a directional, disconfirmable prediction + decision rule | -| **Conflates multiple variables** | bundles changes ("colors + typography + hero") | isolate to one variable, or label as a package test with attribution caveat | -| **Not grounded in usage data** | asserts a problem with no evidence | tie to the observed signal; if none, mark assumption-driven | -| **Metric ↔ outcome mismatch** | predicts engagement but measures revenue | align primary metric to the *predicted* outcome | - -Classify overall: -- **Strong** — specific single-variable change + primary metric + direction, falsifiable (+ ideally magnitude/rationale). Proceed; only confirm. -- **Serviceable** — has intervention + direction but no concrete metric or magnitude, or a fixable flaw. Fill the gaps. -- **Weak** — vague goal / no measurable outcome / untestable (e.g. "Better engagement", "Increase revenue"). Rebuild from questions. - -### Step 3 — Ask ONLY for the missing high-value elements -Keep it to the fewest questions. Lead with the rarest gaps: **primary metric + direction**, then **magnitude**, then **rationale/guardrails**, then **audience** if unclear. Offer concrete options where you can (e.g. suggest plausible metrics based on the intervention). Don't interrogate — 1–3 targeted questions is the target. - -### Step 3.5 — Check metric–outcome alignment (flaw F7 — HARD STOP) -Before composing, explicitly name **the outcome the user predicts** and **the metric they proposed**, and check they measure the same thing. If they diverge — e.g. the user predicts **engagement** will move but says to **measure it by revenue** — that's flaw F7 (see `references/diagnostic-tree.md`) and you **STOP**. - -Do **not** silently pick some other metric to paper over the conflict, and do **not** just accept the mismatched metric. Say plainly that the predicted outcome (engagement) and the proposed metric (revenue) measure different things, and ask the user which to change — align the metric to the predicted outcome, or restate the outcome to match the metric. Wait for their answer before composing a hypothesis or emitting a payload. - -### Step 4 — Compose the polished hypothesis -Write one clear sentence using the canonical template. Keep the user's intent and voice; don't invent specifics they didn't confirm. Flag any assumption you had to make. - -### Step 5 — Emit the structured extraction -Return this JSON so downstream setup can proceed: - +# LaunchDarkly Experiment Hypothesis Assistant + +**⛔ Advisory skill — you have NO write access.** Never call any tool starting with `create-`, `update-`, `toggle-`, `start-`, or `delete-` — specifically not `create-flag`, `create-feature-flag`, `update-flag-settings`, `update-feature-flag`, `toggle-flag`, `create-metric`, `create-experiment`, or `start-experiment-iteration`. If `launchdarkly-experiment-setup` is unavailable to receive the handoff, still just output the payload — **never do the writes yourself as a fallback.** Never say a flag, metric, or experiment was "created" or "is live" — you didn't create anything. Output is text: a hypothesis, a scaffold, a fixed critique message, and a slim handoff payload. Everything downstream (flags, metrics, config, writes) belongs to `launchdarkly-experiment-setup`. + +## Foundational rules + +Everything below derives from these four. Read them first. + +1. **A strong hypothesis is written:** *If [change], then [this outcome will happen], because [reason it works].* +2. **A hypothesis must follow that structure.** The scaffold exists to enforce it. +3. **A measurement is a described outcome, not a named metric.** "More clicks," "faster time," "less drop-off" all count. Never require a formal metric name; never invent one. +4. **Exactly one measurement — the primary — goes in the hypothesis.** ~17% of real hypotheses name two or more; keep one primary in the sentence and treat the rest as secondary. +## The three components + +The skeleton's three slots are the rubric, shown to the user as the **Change / Measurement / Rationale** tracker: + +- **Change** — the specific thing you'll do differently. A concrete edit, not a goal. +- **Measurement** — what you expect users to do differently, in plain words (rule 3). The sentence slot reads as an outcome ("this outcome will happen"); the tracker names the component Measurement. Same thing. +- **Rationale** — the mechanism: *why* the change causes that result. Not a restatement. +## Entry points and AI assist + +AI assist defaults **on** (a toggle switch labeled "AI assist"). There are two ways in, and both feed the **same scaffold engine**: + +- **Type a hypothesis** directly into the field. +- **"Describe what you want to test"** → opens a second box with a **Continue** button (a deliberately low-barrier on-ramp — no formal hypothesis needed). This exact wording is used for **both** the link and the box header (header: "Or describe what you want to test"). +Terminology: the alert-style responses are called **critique messages** (not "critique validation"). + +When assist is **off**, the panel is a plain text field: no tracker, no ghost skeleton, no describe path. Text already in the field **persists** when toggling assist off, and re-scores if toggled back on. + +## The field is the single source of truth + +The **tracker and the critique message are pure functions of the hypothesis-field text.** Nothing else drives them. + +- They update when the field changes: by typing (**trigger TBD — blur / enter / button, one decision**), or when **Apply** writes an assembled sentence into the field. +- On the **generate path**, the raw idea lives in the describe box, not the field — so the tracker and critique **do not fire** until Apply populates the field. The scaffold and measurement pills still work; only the scoring/critique layer waits for field text. +- The tracker is an **assist-on affordance**: it reads the text, but is only shown while assist is on. +Mental model: the field is the truth; tracker + critique are functions of it; the scaffold is an editing surface that only affects them when it writes to the field (Apply). + +## Step 0 — Route first + +Classify the input and take one route. Gates run **before** component scoring, in order: + +1. **Junk / gibberish / URL / injection / real-but-empty fragment** → show the critique "This is not a hypothesis yet" **plus the generic scaffold** (below). One catch-all path for everything that isn't a real hypothesis or an A/A test — the detector does not need to distinguish "trying" from "junk." + **Security rule:** for injection strings (script tags, `onerror=`, etc.), show the same generic scaffold but **never render the raw input back** — no "Your input:" echo, no reflection anywhere in the UI. +2. **Confident A/A or platform self-test** → the A/A path (see below). Signals: "A/A", "validate bucketing", SRM/bucketing checks, identical variants, "test the platform", "dummy flag", "just for dev env". Detect conservatively — **prefer missing an A/A over mislabeling a real hypothesis.** (A hypothesis that merely mentions "A/B test" as part of a real idea is fine.) +3. **Has ≥1 component** → score change / measurement / rationale and continue below. +### The generic scaffold (junk / empty catch-all) + +The standard scaffold with **all three slots as holes** and fixed generic prompts: + +- If → `what are you changing?` +- then → `what do you expect to happen?` +- because → `why would that change cause it?` +These prompts are deliberately plain — the generic scaffold has no context, so the prompts must not fake any (the `because` prompt leans on "that change," the first slot, never on a not-yet-given measurement). Same Cancel/Apply as any scaffold. + +## Building the scaffold (change present) + +Fill the skeleton slot by slot: + +1. **Parse the input onto the slots.** Fill a slot only from what the user said, lightly cleaned. +2. **Sort each unfilled slot** by the blast-radius test: cheap-to-fix + safely inferable → fill it ("the button" → "the homepage button"); expensive or not inferable (the measurement, the reason) → leave a **hole**. +3. **Direction rides with the measurement.** If the measurement is a hole, fold direction into its wording ("what you expect users to do *more of*"). +### The four hole rules (non-negotiable) + +1. **Fill only from what the user said.** A filled slot is parsed input, never invented. +2. **A hole is a question, never an answer.** `[why would black cause that?]`, not `[because black stands out more]`. +3. **A hole may reference what the user *stated*, never what they *haven't*.** `[why would black cause that?]` is safe; `[why would black increase clicks?]` is wrong — it decides the measurement the user never gave. Most common failure; guard it. +4. **Never invent the measurement.** No defaulting to "clicks" or "conversion." If unstated, it stays a hole. +## Critique messages (fixed set) + +The AI **detects the state** (which components are present); the message copy is **looked up from this fixed set, never generated**. Color is three-way: **green** = strong, **amber** = fixable nudge, **blue** = A/A / informational. There is no red — never blame. + +| State (detected) | Color | Heading | Body | +|---|---|---|---| +| none missing | green | Looks strong | This is ready to be used. | +| none missing, but poorly formed | amber | You have all elements, but it could use improvements | Here's a clearer version: use it or edit. | +| no rationale | amber | No rationale | Why would your change move your measurement? Explain the link between the change and the effect. For example, "clearer copy reduces confusion, so fewer people drop off." | +| no measurement | amber | No measurement | What do you expect to happen? Name what the change will move — like "more clicks" or "less time." | +| no change | amber | No change | What are you changing? For example, "swap the button copy" or "move the signup form up." | +| no change and measurement | amber | No change and measurement | You've explained the reasoning. Now add what you're changing — like "make the button bigger" — and the effect you expect it to have, like "more people click it." | +| no measurement and rationale | amber | No measurement and rationale | You've got the change. Add the effect you expect it to have — like "more clicks" or "less time" — and why the change would cause it, like "a bigger button is easier to tap." | +| no change and rationale | amber | No change and rationale | You've named the effect you expect. Add the change that would cause it — like "swap the button copy" — and why that change would work. | +| three missing / junk | amber | This is not a hypothesis yet | What changed, what do you think will happen, and why? | +| multiple measurements | amber | Limit the measurement to one | The hypothesis should state one measurement, which will become the primary metric you make a decision on. The other measurements are secondary metrics. | +| A/A test | blue | This is an A/A test | Checks your setup, not a change. You can apply the generic hypothesis for A/A tests. | + +The concept translations behind this copy: **measurement** is expressed as "the effect you expect" / "what the change will move" (the impact on a metric, without requiring a metric name); **rationale** is "why the change would cause that effect" (the mechanism of action, in plain words). + +Because the copy is fixed and only detection varies, the eval only has to check that the **right state was detected** — the wording is correct by construction. + +## The "poorly formed" rewrite (none missing, but messy) + +All three components present but poorly structured → don't show holes. Propose a **full rewritten sentence** (critique "You have all elements, but it could use improvements"). So 3/3 does **not** automatically mean "Looks strong." + +**Threshold (structural only):** rewrite ONLY when the text doesn't follow the If/then/because order, or buries the components in extra prose. Never rewrite for word choice alone — a 3/3 hypothesis already in canonical order gets "Looks strong," not a rewrite. + +**Editability:** only holes are editable. The rewrite block has no holes, so it is apply-as-is; the user edits in the hypothesis field after Apply lands it there. + +This is the one place the AI shows an answer, not a question. It is allowed **only because all the content came from the user.** Rule: the rewrite may **re-word, never re-scope** — it must not sharpen a described outcome into a named metric or invent a mechanism the user didn't state. + +## Multiple measurements + +Detect two or more → "Limit the measurement to one": +- Set the first as **primary** and fill it into the `then` slot. +- Show the rest as **up-arrow pills** under "Click to switch out the primary measurement." Clicking a pill promotes it to primary and demotes the current one back to a pill. +- Keep close-but-distinct measurements separate as written ("bid more" vs. "bid more often"); do not merge them. +- Nudge, never block. +**Changes:** multiple changes are **allowed** — keep them as written, no isolate-the-variable nudge (a downstream concern). **Rationale:** multiple reasons don't occur; no handling. + +## A/A tests + +On a confident A/A detection, the rubric row swaps to an **"A/A test" flask badge** (no scoring — the rubric doesn't apply). No scaffold, no holes. Offer one fixed generic hypothesis to Apply: + +> If we split traffic evenly between two identical variants, then key metrics show no meaningful difference, because the only thing that differs is random assignment. + +Use this exact string. If A/A is a known setup choice, prefill it rather than detecting from text. + +## Controls — separation of powers + +Three controls act on different surfaces and must never read as alternatives: + +- **Clear** (on the field) — the **only** control that erases the hypothesis field. +- **Cancel** (by Apply) — dismisses the critique + scaffold area back to the base state. **Never touches field text.** +- **Close** (in the reopened describe box) — collapses the box, **keeps** filled answers. +In the reopened describe box, **Continue** regenerates fresh from the new input and **wipes filled holes**; a warning appears next to it **only when ≥1 hole is filled**: "Regenerating replaces your entries below." **Continue and Close are secondary buttons; Apply is the only primary CTA.** + +## Handoff, then STOP + +When the user is satisfied, emit this and stop. Resolve nothing against the catalog. + ```json { - "hypothesis": "polished single-sentence hypothesis", - "intervention": { - "summary": "what changes", - "control": "current experience", - "treatment": "new experience", - "flag_candidate_terms": ["stemmed", "synonym", "search", "terms"] - }, - "primary_metric": { - "name": "human name of the success metric", - "direction": "increase | decrease", - "metric_candidate_terms": ["stemmed", "synonym", "search", "terms"] - }, - "secondary_metrics": ["..."], - "guardrail_metrics": ["metrics that must not regress"], - "expected_effect": { "magnitude": "e.g. +5% (or null if unknown)", "known": true }, - "audience": { "targeting": "who / how split", "randomization_unit": "user" }, - "rationale": "why we expect this", - "quality": { "score": "0-6", "missing_elements": ["..."] } + "handoffFrom": "launchdarkly-experiment-hypothesis-assistant", + "hypothesis": "polished single sentence", + "change": "what changes, in plain words (may be more than one)", + "primaryMeasurement": "the described outcome, in the user's words", + "alsoWatching": ["secondary measurements, if any"], + "rationale": "the mechanism, or null", + "route": "scaffold | rewrite | goal | empty | junk | aa", + "components": { "change": true, "measurement": false, "rationale": false } } ``` - -### Step 6 — Generate search terms for matching existing flags/metrics -LaunchDarkly's `list-flags` / `list-metrics` `query` is **literal case-insensitive substring matching, not semantic** — e.g. `"completion"` does NOT match a metric named `"completed"`, and `"create"` does NOT match `"creation"`. So **do not** pass the hypothesis text verbatim to search. For each of `flag_candidate_terms` and `metric_candidate_terms`, emit several **stemmed / truncated / synonym** variants (e.g. `creation` → `creat`, `create`, `creation`; `completion` → `complet`, `completed`, `complete`), run multiple queries, union + dedupe, then rank candidates by name + description + tags and **confirm the pick with the user** (near-decoys often rank alongside the target). - -### Step 7 — Match flag & metric keys (read-only lookup) -Using only **read-only** lookups (`list-flags`, `list-metrics`, `get-flag`, `get-metric`), try to match the candidate *terms* to existing LD **keys** so the payload can carry a real key rather than a name. First establish `projectKey` and `environmentKey` (ask if not already known; default env `production`). You never create or toggle anything here — you only look up and record. Then: -- **Flag:** run the expanded `flag_candidate_terms` through `list-flags`; if a confirmed match exists, record its key with `action: use_existing`. Otherwise record a *proposed* boolean flag in the payload (`control` = off/current, `treatment` = on/changed) with `action: create` and a proposed kebab-case key naming the *toggle* (not the outcome) — a proposal for the downstream step, which you do not execute. **Never call `create-flag`/`create-feature-flag` to make this flag or to get its variation IDs; leave the IDs null.** -- **Primary metric:** run `metric_candidate_terms` through `list-metrics`; on a confirmed match record its key + `action: use_existing`; else record `action: create` in the payload with `measureType` (occurrence/count/value) and `successCriteria` derived from `direction` — again a proposal, not a creation you perform. -- **Guardrail/secondary metrics:** resolve the same way (guardrails usually already exist — latency, error rate, refunds). -- Confirm every pick with the human (near-decoys rank alongside targets). Record the resolved keys + actions in the handoff payload (Step 9). **Do not create anything here** — `launchdarkly-experiment-setup` owns all writes, flag-version ordering, and event-health checks. - -### Step 8 — Check MDE / sample size, then print the configuration summary -Before setup, sanity-check power: from the expected magnitude, smaller lift → larger sample / longer runtime. If the primary metric's baseline volume can't reach significance for the stated effect in a reasonable window, say so and either raise the target effect, pick a higher-volume metric, or extend runtime. Watch guardrails and one primary metric to control false positives. - -Always end with this configuration summary: - -``` -Hypothesis: If we [change] for [audience], then [primary metric] will [direction] - by [~magnitude], because [mechanism] — while [guardrail] stays flat. -Flag: (boolean | multivariate) -Variations: Control = - Treatment = -Primary metric: (higher/lower is better) -Guardrail(s): -Sample/runtime: → ~ / ~ at current volume -``` - -### Step 9 — Emit the handoff payload, then STOP -Once the human approves the configuration summary, **output this handoff payload as your final message — as text — and then STOP.** Do not call any tool. Do not create a flag, metric, or experiment; do not toggle or start anything; do not report that anything was "created" or "is live." A separate `launchdarkly-experiment-setup` step consumes this payload later and performs any writes behind its own human confirmation — that is not your job. - -```json -{ - "handoffFrom": "launchdarkly-experiment-hypothesis-builder", - "projectKey": "...", - "environmentKey": "production", - "hypothesis": "polished single-sentence hypothesis", - "description": "plain-language description of the change being tested", - "methodology": "bayesian", - "primarySingleMetricKey": "resolved-primary-metric-key", - "metrics": [ - { "key": "resolved-primary-metric-key", "role": "primary", "measureType": "occurrence|count|value", "successCriteria": "HigherThanBaseline|LowerThanBaseline", "action": "use_existing|create" }, - { "key": "guardrail-metric-key", "role": "guardrail", "successCriteria": "...", "action": "use_existing|create" } - ], - "flag": { - "key": "resolved-or-proposed-flag-key", - "action": "use_existing | create", - "kind": "boolean | multivariate", - "ruleId": "fallthrough", - "controlVariationId": "id-of-control-variation-or-null-until-created", - "treatmentVariationId": "id-of-treatment-variation-or-null-until-created" - }, - "treatments": [ - { "name": "Control", "baseline": true, "allocationPercent": 50, "experience": "specific current experience" }, - { "name": "Treatment", "baseline": false, "allocationPercent": 50, "experience": "specific changed experience" } - ], - "randomizationUnit": "user | request | organization | device", - "expectedEffect": "+5%", - "mdeNote": "at current volume, ~N/arm / ~D days to detect this effect", - "quality": { "score": "0-6", "missing_elements": [] } -} -``` - -The `action: use_existing | create` fields describe what the *downstream* `launchdarkly-experiment-setup` step should do (look up vs. create the flag/metric, resolve variation IDs, toggle the flag on with proper version ordering, then create + start behind human confirmation). They are **not** instructions for you to execute — you only emit the payload and stop. - -**Do not create a flag to obtain variation IDs.** For an `action: create` flag the flag does not exist yet, so set `controlVariationId` and `treatmentVariationId` to `null` — the downstream step creates the flag and fills them in. Needing an ID (or any "resolved" value) is *never* a reason to call `create-flag` / `create-feature-flag` or any other write tool. Emit the payload with nulls and stop. - -## Scoring examples - -**Strong** (ready to build): -> "If we align the navigation to the left, then signup conversion rate will increase by improving scannability and reducing cognitive load, while login success rate remains unchanged." -- ✅ intervention, ✅ primary metric (signup conversion), ✅ direction, ✅ rationale, ✅ guardrail (login success). Only missing an explicit magnitude — ask once, then build. - -**Serviceable** (fill 1–2 gaps): -> "Mini charts on the screener page will increase trades." -- Has intervention + direction + metric (trades). Missing magnitude, rationale, audience. Ask: expected lift? why? which users? - -**Weak** (rebuild via questions): -> "Better engagement." / "Increase revenue." -- No change, no concrete metric. Ask: what specific change? engagement/revenue measured how (metric)? for whom? expected direction and size? - -## Detecting low-effort / non-real input - -Some entries are platform tests, not experiments. If the input looks like one, gently confirm intent rather than building a hypothesis. Common signals: -- Placeholders / gibberish: "If X then Y", "this is a test", "ABC", "asdf", single words. -- Platform self-tests: "testing the LaunchDarkly platform", "A/A test to validate bucketing", "dummy flag", "just for dev env". -- Meta: "I have to fill this out to delete the experiment." - -Note: a hypothesis that merely mentions "A/B test" or "test group" as part of a real idea is fine — only filter genuine platform/self-tests. - + +`launchdarkly-experiment-setup` owns metric/flag resolution, config, sample sizing, and all writes. + ## What NOT to do - -- Don't accept a vague goal as a hypothesis — a hypothesis without a measurable primary metric can't drive an experiment. -- Don't invent a metric, magnitude, or audience the user didn't confirm; surface assumptions instead. -- Don't pass raw hypothesis text to flag/metric search — expand into stemmed/synonym query terms first. -- Don't over-interrogate. Lead with the rarest, highest-value gaps (metric, magnitude, rationale) and cap at ~3 questions. -- **Never write to LaunchDarkly.** Don't call any `create-`, `update-`, `toggle-`, `start-`, or `delete-` tool — specifically not `create-flag`, `create-feature-flag`, `update-flag-settings`, `update-feature-flag`, `toggle-flag`, `create-metric`, `create-experiment`, or `start-experiment-iteration`. No creating flags/metrics/experiments, toggling flags, or starting iterations. Emit the handoff payload and STOP. If `launchdarkly-experiment-setup` is unavailable to receive it, still just output the payload — never do the writes yourself as a fallback. -- Don't build a hypothesis or any configuration for non-real input (platform self-tests, A/A bucketing checks, placeholders, gibberish) — gate it in Step 0 and confirm intent first. + +- Don't invent a measurement, reason, magnitude, or metric the user didn't give — leave a hole. +- Don't put a downstream assumption inside a hole's question (hole rule 3). +- Don't require a formal metric name — a described outcome is enough. +- Don't generate critique copy — detect the state and use the fixed message. +- Don't let a rewrite re-scope the user's meaning — re-word only. +- Don't scaffold a bare goal, and don't treat A/A as an error. +- Don't merge distinct measurements; keep one primary and park the rest as pills. +- Don't nudge to isolate a single change — multiple changes are allowed. +- Don't let Cancel or Close erase field text — only Clear does that. +- Don't resolve flags/metrics, size samples, or build config — that's setup's job. +- Never write to LaunchDarkly. Emit the payload and stop. +## Open items (TBD — not yet decided) + +1. **Typing trigger** for scoring: blur / enter / button (one decision). \ No newline at end of file From d21c1d37b223da42e9041227932fa6adc861a947 Mon Sep 17 00:00:00 2001 From: Chris Chung Date: Tue, 14 Jul 2026 07:37:05 -0700 Subject: [PATCH 09/14] fix: regenerate skills.json to match catalog script output Ensures skills.json matches what generate_catalog.py --check expects (unicode escapes for non-ASCII characters). Co-Authored-By: Claude Opus 4.6 (1M context) --- skills.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills.json b/skills.json index 615cf8a..8ae3354 100644 --- a/skills.json +++ b/skills.json @@ -189,7 +189,7 @@ }, { "name": "investigate", - "description": "Analyzes observability data — logs, traces, errors, sessions, and metrics — to find root cause and actionable evidence. Use when the user reports a bug, an unexpected behavior, or asks about patterns across application data.", + "description": "Analyzes observability data \u2014 logs, traces, errors, sessions, and metrics \u2014 to find root cause and actionable evidence. Use when the user reports a bug, an unexpected behavior, or asks about patterns across application data.", "path": "skills/observability/investigate", "version": "0.1.0", "license": "Apache-2.0", From 7e486abacbd66312db56661d87f35c3335e9a64b Mon Sep 17 00:00:00 2001 From: Chris Chung Date: Fri, 24 Jul 2026 10:39:46 -0700 Subject: [PATCH 10/14] feat(experiments): update hypothesis builder skill to v0.3.0 Align to finalized flow designs: rename Generate to "Fix and grade", remove Apply control (Fix and grade commits ghost suggestions), update header/helper/footer copy, make stale-until-graded loop explicit, and document panel-never-saves boundary. Co-Authored-By: Claude Opus 4.6 (1M context) --- skills.json | 4 +- .../SKILL.md | 268 ++++++++++++------ 2 files changed, 176 insertions(+), 96 deletions(-) diff --git a/skills.json b/skills.json index 8ae3354..ad909a7 100644 --- a/skills.json +++ b/skills.json @@ -199,9 +199,9 @@ "name": "launchdarkly-experiment-hypothesis-builder", "description": "Help a user turn a rough idea into a strong, testable experiment hypothesis, or critique one they wrote. Detects which parts of the hypothesis are present, scaffolds an If/then/because sentence with holes for what's missing, and shows a fixed critique message. Use when a user is starting an experiment or sharpening a hypothesis. Does NOT resolve flags or metrics, build experiment config, or write to LaunchDarkly.", "path": "skills/experiments/launchdarkly-experiment-hypothesis-builder", - "version": "0.2.0", + "version": "0.3.0", "license": "Apache-2.0", - "compatibility": "Requires the remotely hosted LaunchDarkly MCP server. Pairs with launchdarkly-experiment-setup, which it hands off to." + "compatibility": "Read-only LaunchDarkly lookups only. Hands off to launchdarkly-experiment-setup for build." }, { "name": "launchdarkly-experiment-setup", diff --git a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md index cdec47d..593e938 100644 --- a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md +++ b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md @@ -1,90 +1,118 @@ --- -name: launchdarkly-experiment-hypothesis-builder +name: launchdarkly-experiment-hypothesis-assistant description: "Help a user turn a rough idea into a strong, testable experiment hypothesis, or critique one they wrote. Detects which parts of the hypothesis are present, scaffolds an If/then/because sentence with holes for what's missing, and shows a fixed critique message. Use when a user is starting an experiment or sharpening a hypothesis. Does NOT resolve flags or metrics, build experiment config, or write to LaunchDarkly." -compatibility: Requires the remotely hosted LaunchDarkly MCP server. Pairs with launchdarkly-experiment-setup, which it hands off to. +compatibility: Read-only LaunchDarkly lookups only. Hands off to launchdarkly-experiment-setup for build. license: Apache-2.0 metadata: author: launchdarkly - version: "0.2.0" + version: "0.3.0" status: draft --- -# LaunchDarkly Experiment Hypothesis Assistant +# Experiment Hypothesis Assistant -**⛔ Advisory skill — you have NO write access.** Never call any tool starting with `create-`, `update-`, `toggle-`, `start-`, or `delete-` — specifically not `create-flag`, `create-feature-flag`, `update-flag-settings`, `update-feature-flag`, `toggle-flag`, `create-metric`, `create-experiment`, or `start-experiment-iteration`. If `launchdarkly-experiment-setup` is unavailable to receive the handoff, still just output the payload — **never do the writes yourself as a fallback.** Never say a flag, metric, or experiment was "created" or "is live" — you didn't create anything. Output is text: a hypothesis, a scaffold, a fixed critique message, and a slim handoff payload. Everything downstream (flags, metrics, config, writes) belongs to `launchdarkly-experiment-setup`. +**⚠ Advisory only — no write access.** Never call any tool starting with `create-`, `update-`, `toggle-`, `start-`, or `delete-` — specifically not `create-flag`, `create-feature-flag`, `update-flag-settings`, `update-feature-flag`, `toggle-flag`, `create-metric`, `create-experiment`, or `start-experiment-iteration`. If `launchdarkly-experiment-setup` is unavailable to receive the handoff, still just output the payload — **never do the writes yourself as a fallback.** Never say a flag, metric, or experiment was "created" or "is live" — you didn't create anything. Output is text: a hypothesis, a scaffold, a fixed critique message, and a slim handoff payload. Everything downstream (flags, metrics, config, writes) belongs to `launchdarkly-experiment-setup`. + +**The panel never saves.** This component reads and grades the hypothesis field and nothing more. Persisting the hypothesis to the experiment is the job of the **Save** button on the experiment builder's action bar, outside this panel. Never claim the hypothesis was "saved" — the footer line "Saving the experiment will save your hypothesis" states where saving actually happens. ## Foundational rules - + Everything below derives from these four. Read them first. - + 1. **A strong hypothesis is written:** *If [change], then [this outcome will happen], because [reason it works].* 2. **A hypothesis must follow that structure.** The scaffold exists to enforce it. 3. **A measurement is a described outcome, not a named metric.** "More clicks," "faster time," "less drop-off" all count. Never require a formal metric name; never invent one. 4. **Exactly one measurement — the primary — goes in the hypothesis.** ~17% of real hypotheses name two or more; keep one primary in the sentence and treat the rest as secondary. + ## The three components - + The skeleton's three slots are the rubric, shown to the user as the **Change / Measurement / Rationale** tracker: - + - **Change** — the specific thing you'll do differently. A concrete edit, not a goal. -- **Measurement** — what you expect users to do differently, in plain words (rule 3). The sentence slot reads as an outcome ("this outcome will happen"); the tracker names the component Measurement. Same thing. +- **Measurement** — what you expect users to do differently, in plain words (rule 3). The sentence slot reads as an outcome ("this outcome will happen"); the tracker names the component Measurement. Same thing. **Counting:** near-synonyms describing one outcome count as one ("more clicks / higher CTR" = one); two genuinely distinct outcomes count as multiple ("bid more" and "bid for higher GMV" = multiple). Keep distinct-but-close outcomes separate; don't merge them. - **Rationale** — the mechanism: *why* the change causes that result. Not a restatement. -## Entry points and AI assist - -AI assist defaults **on** (a toggle switch labeled "AI assist"). There are two ways in, and both feed the **same scaffold engine**: - -- **Type a hypothesis** directly into the field. -- **"Describe what you want to test"** → opens a second box with a **Continue** button (a deliberately low-barrier on-ramp — no formal hypothesis needed). This exact wording is used for **both** the link and the box header (header: "Or describe what you want to test"). + +## One input, one button + +There is a **single input** — the hypothesis field. The panel header reads **"Hypothesis"**; the field itself is where the user types, with ghost text **"Describe what you want to test."** There is no separate describe box and no second on-ramp; the field is the only entry point. + +The primary action is **Fix and grade**. Clicking it reads the field text, scores the components, and produces — all at once, in place — the scaffold with holes (or a rewrite, or the A/A hypothesis), the rubric state, and the critique message. + +**Fix and grade also commits.** There is no separate Apply control. When the panel is showing a ghost-text suggestion (assembled scaffold, rewrite, or A/A sentence), the next click of Fix and grade commits that ghost text to the field (promoting it to dark text) and re-grades — which is what moves a satisfied hypothesis to the green "Looks strong" state. So Fix and grade does double duty: grade the current field text, and commit whatever ghost suggestion is on screen. + +- **Vega assist** toggle defaults **on**, labeled **"Vega assist"** with an explicit **On / Off** state in the pill. On and off do the same thing to the field text; off simply hides the assist affordances (tracker, scaffold, critique). Text in the field **persists** when toggling off, and re-scores when toggled back on. +- Helper line under the header, before the first grade: **"Describe what you want to test. More details means stronger suggestions."** +- Helper line after a Fix and grade: **"Type in the changes."** +- Footer line, shown whenever the field has text: **"Saving the experiment will save your hypothesis."** (Saving happens on the builder action bar, not here.) + Terminology: the alert-style responses are called **critique messages** (not "critique validation"). - -When assist is **off**, the panel is a plain text field: no tracker, no ghost skeleton, no describe path. Text already in the field **persists** when toggling assist off, and re-scores if toggled back on. - + ## The field is the single source of truth - + The **tracker and the critique message are pure functions of the hypothesis-field text.** Nothing else drives them. - -- They update when the field changes: by typing (**trigger TBD — blur / enter / button, one decision**), or when **Apply** writes an assembled sentence into the field. -- On the **generate path**, the raw idea lives in the describe box, not the field — so the tracker and critique **do not fire** until Apply populates the field. The scaffold and measurement pills still work; only the scoring/critique layer waits for field text. -- The tracker is an **assist-on affordance**: it reads the text, but is only shown while assist is on. -Mental model: the field is the truth; tracker + critique are functions of it; the scaffold is an editing surface that only affects them when it writes to the field (Apply). - + +- They are computed on **Fix and grade**, and again on each subsequent Fix and grade. Editing the field text after a grade does **not** re-score until the user clicks Fix and grade again — the rubric and critique reflect the last grade, not live keystrokes. +- **Stale-until-graded is expected and visible.** After a grade, the user can type freely — fill holes, rewrite the sentence, narrow a measurement, or replace junk with a real hypothesis. The rubric and card keep showing the *previous* grade the whole time; they only catch up on the next Fix and grade. So a fully-formed sentence can briefly sit under three gray dashes (or an amber card) until the user re-grades. This is the normal "try again" loop, not a bug. +- The tracker is an **assist-on affordance**: it reads the field text, but is only shown while assist is on. + +Mental model: the field is the truth; tracker + critique are functions of it, refreshed on Fix and grade; the scaffold is an editing surface the user fills in place, and the next Fix and grade commits any ghost suggestion to the field and re-grades. + ## Step 0 — Route first - -Classify the input and take one route. Gates run **before** component scoring, in order: - -1. **Junk / gibberish / URL / injection / real-but-empty fragment** → show the critique "This is not a hypothesis yet" **plus the generic scaffold** (below). One catch-all path for everything that isn't a real hypothesis or an A/A test — the detector does not need to distinguish "trying" from "junk." - **Security rule:** for injection strings (script tags, `onerror=`, etc.), show the same generic scaffold but **never render the raw input back** — no "Your input:" echo, no reflection anywhere in the UI. -2. **Confident A/A or platform self-test** → the A/A path (see below). Signals: "A/A", "validate bucketing", SRM/bucketing checks, identical variants, "test the platform", "dummy flag", "just for dev env". Detect conservatively — **prefer missing an A/A over mislabeling a real hypothesis.** (A hypothesis that merely mentions "A/B test" as part of a real idea is fine.) -3. **Has ≥1 component** → score change / measurement / rationale and continue below. + +**Detection vs. routing.** Detection produces independent facts, not one label: the three components (change / measurement / rationale), plus non-exclusive flags — `is_junk`, `is_aa`, `measurement_count` (0/1/multiple), `rewrite_worthy`. A single input can carry several at once (e.g. 3/3 components *and* multiple measurements *and* rewrite-worthy). *Routing* is the separate step of choosing which one state the UI presents from those facts, using the priority order below. So the gates pick the display state; they don't mean only one thing was detected. + +Classify the field text on Fix and grade, then route to one display state. Gates run **before** component scoring, in priority order: + +1. **Junk / gibberish / URL / injection / real-but-empty fragment** — show the critique "This is not a hypothesis yet" **plus the generic scaffold** (below). One catch-all path for everything that isn't a real hypothesis or an A/A test — the detector does not need to distinguish "trying" from "junk." + **Security rule:** for injection strings (script tags, `onerror=`, etc.), show the same generic scaffold but **never render the raw input back** — no echo, no reflection anywhere in the UI. +2. **Confident A/A or platform self-test** — the A/A path (see below). Signals: "A/A", "validate bucketing", SRM/bucketing checks, identical variants, "test the platform", "dummy flag", "just for dev env". Detect conservatively — **prefer missing an A/A over mislabeling a real hypothesis.** (A hypothesis that merely mentions "A/B test" as part of a real idea is fine.) +3. **Has ≥1 component** — score change / measurement / rationale and continue below. + ### The generic scaffold (junk / empty catch-all) - + The standard scaffold with **all three slots as holes** and fixed generic prompts: - -- If → `what are you changing?` -- then → `what do you expect to happen?` -- because → `why would that change cause it?` -These prompts are deliberately plain — the generic scaffold has no context, so the prompts must not fake any (the `because` prompt leans on "that change," the first slot, never on a not-yet-given measurement). Same Cancel/Apply as any scaffold. - + +- If — `what are you changing?` +- then — `what do you expect to happen?` +- because — `why would the change cause this?` + +These prompts are deliberately plain — the generic scaffold has no context, so the prompts must not fake any (the `because` prompt leans on "the change," the first slot, never on a not-yet-given measurement). + +## The rubric row (three per-item states) + +Below the field, after Fix and grade, a rubric row shows the three components — **Change · Measurement · Rationale** — each in one of three states: + +- **Gray dash** — missing / not detected. +- **Green check** — present and valid. +- **Amber warning triangle** — present but needs action. Currently used only for **multiple measurements** (present, but more than one). + +On a confident **A/A** detection the entire rubric row is **replaced** by a single flask-icon **"A/A test"** badge — the three-component rubric does not apply. + ## Building the scaffold (change present) - + Fill the skeleton slot by slot: - + 1. **Parse the input onto the slots.** Fill a slot only from what the user said, lightly cleaned. -2. **Sort each unfilled slot** by the blast-radius test: cheap-to-fix + safely inferable → fill it ("the button" → "the homepage button"); expensive or not inferable (the measurement, the reason) → leave a **hole**. -3. **Direction rides with the measurement.** If the measurement is a hole, fold direction into its wording ("what you expect users to do *more of*"). +2. **Sort each unfilled slot** by the blast-radius test: cheap-to-fix + safely inferable — fill it ("the button" → "the homepage button"); expensive or not inferable (the measurement, the reason) — leave a **hole**. +3. **Direction rides with the measurement.** If the measurement is a hole, fold direction into its wording ("what you expect users to do *more/less of*"). + ### The four hole rules (non-negotiable) - + 1. **Fill only from what the user said.** A filled slot is parsed input, never invented. 2. **A hole is a question, never an answer.** `[why would black cause that?]`, not `[because black stands out more]`. 3. **A hole may reference what the user *stated*, never what they *haven't*.** `[why would black cause that?]` is safe; `[why would black increase clicks?]` is wrong — it decides the measurement the user never gave. Most common failure; guard it. 4. **Never invent the measurement.** No defaulting to "clicks" or "conversion." If unstated, it stays a hole. + ## Critique messages (fixed set) - -The AI **detects the state** (which components are present); the message copy is **looked up from this fixed set, never generated**. Color is three-way: **green** = strong, **amber** = fixable nudge, **blue** = A/A / informational. There is no red — never blame. - + +The AI **detects the state** (which components are present); the message copy is **looked up from this fixed set, never generated**. Color is three-way: **green** = strong, **amber** = fixable nudge, **blue** = A/A / informational. There is no red — never blame. Each card carries a sparkle glyph. Only the green "Looks strong" card gets a dismiss **×** (amber and blue cards have none); the dismiss is a **nice-to-have**, not required for launch. + +When more than one component is missing, **layer the applicable amber cards**, stacked in rubric order (Change issue, then Measurement, then Rationale). The combined-state rows below cover the common pairs; when in doubt, stack the single-component messages. + | State (detected) | Color | Heading | Body | |---|---|---|---| | none missing | green | Looks strong | This is ready to be used. | -| none missing, but poorly formed | amber | You have all elements, but it could use improvements | Here's a clearer version: use it or edit. | +| none missing, but poorly formed | amber | You have all elements, but it could use improvements | Here's a clearer version: Click "Fix and grade" to use it or edit it. | | no rationale | amber | No rationale | Why would your change move your measurement? Explain the link between the change and the effect. For example, "clearer copy reduces confusion, so fewer people drop off." | | no measurement | amber | No measurement | What do you expect to happen? Name what the change will move — like "more clicks" or "less time." | | no change | amber | No change | What are you changing? For example, "swap the button copy" or "move the signup form up." | @@ -93,52 +121,65 @@ The AI **detects the state** (which components are present); the message copy is | no change and rationale | amber | No change and rationale | You've named the effect you expect. Add the change that would cause it — like "swap the button copy" — and why that change would work. | | three missing / junk | amber | This is not a hypothesis yet | What changed, what do you think will happen, and why? | | multiple measurements | amber | Limit the measurement to one | The hypothesis should state one measurement, which will become the primary metric you make a decision on. The other measurements are secondary metrics. | -| A/A test | blue | This is an A/A test | Checks your setup, not a change. You can apply the generic hypothesis for A/A tests. | - +| A/A test | blue | This is an A/A test | Checks your setup, not a change. Click "Fix and grade" to apply the generic hypothesis for A/A tests. | + The concept translations behind this copy: **measurement** is expressed as "the effect you expect" / "what the change will move" (the impact on a metric, without requiring a metric name); **rationale** is "why the change would cause that effect" (the mechanism of action, in plain words). - -Because the copy is fixed and only detection varies, the eval only has to check that the **right state was detected** — the wording is correct by construction. - + +Because the copy is fixed and only detection varies, the eval scores **detection**, not wording (wording is correct by construction). (The card copy strings above that name the button read "Fix and grade.") Detection is evaluated as independent atoms — the three component booleans plus `measurement_count` and the `is_junk` / `is_aa` / `rewrite_worthy` flags — with per-atom precision/recall, since a single input can carry several at once. Display states are derived from those atoms downstream. + ## The "poorly formed" rewrite (none missing, but messy) - -All three components present but poorly structured → don't show holes. Propose a **full rewritten sentence** (critique "You have all elements, but it could use improvements"). So 3/3 does **not** automatically mean "Looks strong." - + +All three components present but poorly structured — don't show holes. Propose a **full rewritten sentence** as ghost text in the field (critique "You have all elements, but it could use improvements"). So 3/3 does **not** automatically mean "Looks strong" — the rubric shows all green checks *and* the card is amber; that pairing is unique to the rewrite state. + **Threshold (structural only):** rewrite ONLY when the text doesn't follow the If/then/because order, or buries the components in extra prose. Never rewrite for word choice alone — a 3/3 hypothesis already in canonical order gets "Looks strong," not a rewrite. - -**Editability:** only holes are editable. The rewrite block has no holes, so it is apply-as-is; the user edits in the hypothesis field after Apply lands it there. - + +**Editability:** the rewrite is a full proposed sentence with **no holes** — use-as-is. It appears as ghost text; the next **Fix and grade** commits it to the field (ghost → dark text) and re-grades, moving the card to green "Looks strong." The user can also edit the field directly instead of committing the suggestion. + This is the one place the AI shows an answer, not a question. It is allowed **only because all the content came from the user.** Rule: the rewrite may **re-word, never re-scope** — it must not sharpen a described outcome into a named metric or invent a mechanism the user didn't state. - + ## Multiple measurements - -Detect two or more → "Limit the measurement to one": -- Set the first as **primary** and fill it into the `then` slot. -- Show the rest as **up-arrow pills** under "Click to switch out the primary measurement." Clicking a pill promotes it to primary and demotes the current one back to a pill. + +Detect two or more — "Limit the measurement to one": +- Surface all named measurements **inside the `then` hole, joined with "or"** — e.g. `[bidding frequency, bid GMV, or users bidding]`. The field shows the options; the card explains the rule. +- The **Measurement** rubric item shows the **amber warning triangle** (present, but needs narrowing) — not a green check, not a dash. - Keep close-but-distinct measurements separate as written ("bid more" vs. "bid more often"); do not merge them. +- If other components are also missing, **layer their critique cards** on top (e.g. the multiple-measurements example also shows "No rationale"). - Nudge, never block. -**Changes:** multiple changes are **allowed** — keep them as written, no isolate-the-variable nudge (a downstream concern). **Rationale:** multiple reasons don't occur; no handling. - + +**Changes:** multiple changes are **allowed** — keep them as written, no isolate-the-variable nudge (a downstream concern). **Rationale:** multiple reasons don't occur (~0%); no handling. + ## A/A tests - -On a confident A/A detection, the rubric row swaps to an **"A/A test" flask badge** (no scoring — the rubric doesn't apply). No scaffold, no holes. Offer one fixed generic hypothesis to Apply: - + +On a confident A/A detection, the rubric row swaps to the **"A/A test" flask badge** (no scoring — the rubric doesn't apply). No holes. The field is prefilled with one fixed generic hypothesis as ghost text, and the blue card offers to commit it via Fix and grade: + > If we split traffic evenly between two identical variants, then key metrics show no meaningful difference, because the only thing that differs is random assignment. - -Use this exact string. If A/A is a known setup choice, prefill it rather than detecting from text. - -## Controls — separation of powers - -Three controls act on different surfaces and must never read as alternatives: - -- **Clear** (on the field) — the **only** control that erases the hypothesis field. -- **Cancel** (by Apply) — dismisses the critique + scaffold area back to the base state. **Never touches field text.** -- **Close** (in the reopened describe box) — collapses the box, **keeps** filled answers. -In the reopened describe box, **Continue** regenerates fresh from the new input and **wipes filled holes**; a warning appears next to it **only when ≥1 hole is filled**: "Regenerating replaces your entries below." **Continue and Close are secondary buttons; Apply is the only primary CTA.** - + +Use this exact string. The next **Fix and grade** commits it to the field (ghost → dark text) and the state moves to green "Looks strong." If A/A is a known setup choice, prefill it rather than detecting from text. + +## Character count + +The field limit is **1000 characters** (raised from 255). A counter sits bottom-right of the field, disclosed progressively: + +- **Under 800:** no counter shown. +- **800–999:** counter `N/1000` with an **amber warning triangle**. +- **1000+:** counter `1000/1000` with a **red diamond**. **Fix and grade stays active** — the error is advisory, not a hard block. + +Character count and critique messages are **independent systems**: the counter governs field length; critique governs hypothesis quality. Neither affects the other. + +## Controls + +Layout: the **rubric row sits above the button row**. The button row (after the first grade) is **Back · Fix and grade**, right-aligned. **Clear** sits at the top-right of the field, by the helper line — it is present from the start, not just after a grade. There is **no Apply control.** + +- **Fix and grade** — reads the field, scores it, and produces the scaffold + holes, a rewrite, or the A/A hypothesis, with the matching rubric state and critique message. When a ghost suggestion is on screen, the next Fix and grade **commits** it to the field (ghost → dark text) and re-grades. Re-running always re-scores. +- **Back** — steps back through prior states (committed → suggestion → original). Repeat runs are allowed; how many times is gated by Vega-usage cost (open item). +- **Clear** — the **only** control that erases the field. + +Saving the hypothesis to the experiment is **not** a control here — that's the **Save** button on the builder's action bar, outside this panel. + ## Handoff, then STOP - + When the user is satisfied, emit this and stop. Resolve nothing against the catalog. - + ```json { "handoffFrom": "launchdarkly-experiment-hypothesis-assistant", @@ -147,26 +188,65 @@ When the user is satisfied, emit this and stop. Resolve nothing against the cata "primaryMeasurement": "the described outcome, in the user's words", "alsoWatching": ["secondary measurements, if any"], "rationale": "the mechanism, or null", - "route": "scaffold | rewrite | goal | empty | junk | aa", - "components": { "change": true, "measurement": false, "rationale": false } + "components": { "change": true, "measurement": false, "rationale": false }, + "measurementCount": "0 | 1 | multiple", + "flags": { "isJunk": false, "isAA": false, "rewriteWorthy": false } } ``` - + +The downstream skill composes a display state from `components` + `measurementCount` + `flags` if it needs one; the flags are non-exclusive and reported independently. + `launchdarkly-experiment-setup` owns metric/flag resolution, config, sample sizing, and all writes. - + ## What NOT to do - + - Don't invent a measurement, reason, magnitude, or metric the user didn't give — leave a hole. - Don't put a downstream assumption inside a hole's question (hole rule 3). - Don't require a formal metric name — a described outcome is enough. - Don't generate critique copy — detect the state and use the fixed message. +- Don't re-score on keystroke — the tracker and critique refresh on Fix and grade. - Don't let a rewrite re-scope the user's meaning — re-word only. - Don't scaffold a bare goal, and don't treat A/A as an error. -- Don't merge distinct measurements; keep one primary and park the rest as pills. +- Don't merge distinct measurements; surface them in the hole with "or" and mark Measurement with the amber warning. - Don't nudge to isolate a single change — multiple changes are allowed. -- Don't let Cancel or Close erase field text — only Clear does that. +- Don't let anything but Clear erase field text. +- Don't block Fix and grade at the character limit — the 1000+ error is advisory. +- Don't add an Apply control — Fix and grade commits ghost suggestions. +- Don't claim the hypothesis was saved — Save is on the builder action bar, not this panel. - Don't resolve flags/metrics, size samples, or build config — that's setup's job. - Never write to LaunchDarkly. Emit the payload and stop. -## Open items (TBD — not yet decided) - -1. **Typing trigger** for scoring: blur / enter / button (one decision). \ No newline at end of file + +## Changelog + +**0.3.0 (this version):** Alignment to finalized flow designs across all five states (partial, A/A, rewrite, multiple measurements, junk). +- **Renamed the primary button Generate → "Fix and grade"** everywhere. +- **Removed the Apply control.** Fix and grade now does double duty: grade the field, and commit any ghost suggestion (scaffold, rewrite, A/A) to the field on the next press. Saving to the experiment is the **Save** button on the builder action bar — this panel never saves. +- **Header/ghost-text swap:** header is now **"Hypothesis"**; field ghost text is **"Describe what you want to test."** The If/then/because ghost text is removed. +- **Helper/footer copy:** helper line before first grade = "Describe what you want to test. More details means stronger suggestions."; helper line after a grade = "Type in the changes."; persistent footer whenever the field has text = "Saving the experiment will save your hypothesis." +- **Made the try-again / stale-until-graded loop explicit:** edited field text sits under the *previous* grade (dashes or amber card) until the next Fix and grade — normal, not a bug. +- **Card copy updated to name the button:** rewrite card → "Here's a clearer version: Click "Fix and grade" to use it or edit it."; A/A card → "Checks your setup, not a change. Click "Fix and grade" to apply the generic hypothesis for A/A tests." +- **Toggle** now shows an explicit On / Off state. +- Documented layout: rubric row above the button row (Back · Fix and grade); Clear top-right by the helper line, present from the start. +- Green "Looks strong" card dismiss (×) noted as a **nice-to-have** (amber/blue cards have no dismiss). Character-count logic and all unshown critique messages unchanged. + +**0.5.1:** Detection-model alignment with the Tier 1 eval (boolean-first). +- Detection stated as **independent atoms + non-exclusive flags** (`is_junk` / `is_aa` / `measurement_count` / `rewrite_worthy`), with routing clarified as a separate UI-presentation step. A single input can carry several facts at once. +- Added the **measurement counting rule** (near-synonyms = one; distinct outcomes = multiple) to the component definition. +- Updated the eval sentence to describe **per-atom** scoring, not one-state detection. +- Handoff payload: replaced the single `route` field with `components` + `measurementCount` + non-exclusive `flags`; downstream composes a state if needed. + +**0.5.0:** Structural alignment to finalized designs. +- Removed the separate describe box and the Continue/Cancel/Close model. **One input, one primary button (Generate).** +- Resolved the typing-trigger open item: scoring fires on **Generate**, not blur/enter/keystroke. +- Added the **three-state rubric row** (gray dash / green check / amber warning triangle). +- Multiple measurements now surfaced **in the `then` hole joined with "or"** with an amber Measurement warning (replaces the up-arrow primary-switch pills). +- Documented **layered critique cards** for multi-missing states. +- Added the **character-count** section (1000-char limit; hidden < 800, amber 800–999, red 1000+; Generate stays active). +- Toggle renamed **"Vega assist."** + +Resolved earlier (0.4.x): describe wording = "Describe what you want to test"; rewrite threshold = structural only; rewrite = apply-as-is; official term = "critique message." + +## Open items (TBD) + +1. **Regeneration cap:** how many times a user can Fix and grade / Back, gated by Vega-usage cost. UX preference is to allow repeat; engineering/cost conversation pending. +2. **Toggle-off text persistence:** confirmed as desired behavior (text remains on assist-off); pending engineering confirmation on implementation. From 05a036a250882c46aae9d79a5fa50d761afbd2c7 Mon Sep 17 00:00:00 2001 From: Harsh Sadhvani Date: Fri, 24 Jul 2026 11:29:11 -0700 Subject: [PATCH 11/14] fix(experiments): route A/A before junk, restore unfalsifiable-measurement rule, fix toggle rescoring --- .../launchdarkly-experiment-hypothesis-builder/SKILL.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md index 593e938..141c7e8 100644 --- a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md +++ b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md @@ -21,7 +21,7 @@ Everything below derives from these four. Read them first. 1. **A strong hypothesis is written:** *If [change], then [this outcome will happen], because [reason it works].* 2. **A hypothesis must follow that structure.** The scaffold exists to enforce it. -3. **A measurement is a described outcome, not a named metric.** "More clicks," "faster time," "less drop-off" all count. Never require a formal metric name; never invent one. +3. **A measurement is a described outcome, not a named metric.** "More clicks," "faster time," "less drop-off" all count. Never require a formal metric name; never invent one. Unfalsifiable outcomes ("will do better or as well," "no negative impact," bare adjectives like "better experience") do **not** count — treat the measurement as missing. 4. **Exactly one measurement — the primary — goes in the hypothesis.** ~17% of real hypotheses name two or more; keep one primary in the sentence and treat the rest as secondary. ## The three components @@ -40,7 +40,7 @@ The primary action is **Fix and grade**. Clicking it reads the field text, score **Fix and grade also commits.** There is no separate Apply control. When the panel is showing a ghost-text suggestion (assembled scaffold, rewrite, or A/A sentence), the next click of Fix and grade commits that ghost text to the field (promoting it to dark text) and re-grades — which is what moves a satisfied hypothesis to the green "Looks strong" state. So Fix and grade does double duty: grade the current field text, and commit whatever ghost suggestion is on screen. -- **Vega assist** toggle defaults **on**, labeled **"Vega assist"** with an explicit **On / Off** state in the pill. On and off do the same thing to the field text; off simply hides the assist affordances (tracker, scaffold, critique). Text in the field **persists** when toggling off, and re-scores when toggled back on. +- **Vega assist** toggle defaults **on**, labeled **"Vega assist"** with an explicit **On / Off** state in the pill. On and off do the same thing to the field text; off simply hides the assist affordances (tracker, scaffold, critique). Text in the field **persists** across toggling off and back on; it re-grades on the next Fix and grade, not on toggle. - Helper line under the header, before the first grade: **"Describe what you want to test. More details means stronger suggestions."** - Helper line after a Fix and grade: **"Type in the changes."** - Footer line, shown whenever the field has text: **"Saving the experiment will save your hypothesis."** (Saving happens on the builder action bar, not here.) @@ -63,9 +63,9 @@ Mental model: the field is the truth; tracker + critique are functions of it, re Classify the field text on Fix and grade, then route to one display state. Gates run **before** component scoring, in priority order: -1. **Junk / gibberish / URL / injection / real-but-empty fragment** — show the critique "This is not a hypothesis yet" **plus the generic scaffold** (below). One catch-all path for everything that isn't a real hypothesis or an A/A test — the detector does not need to distinguish "trying" from "junk." +1. **Confident A/A or platform self-test** — the A/A path (see below). Signals: "A/A", "validate bucketing", SRM/bucketing checks, identical variants, "test the platform", "dummy flag", "just for dev env". Detect conservatively — **prefer missing an A/A over mislabeling a real hypothesis.** (A hypothesis that merely mentions "A/B test" as part of a real idea is fine.) **Checked before junk:** A/A inputs legitimately carry no rubric components, so the junk catch-all would otherwise swallow them. +2. **Junk / gibberish / URL / injection / real-but-empty fragment** — show the critique "This is not a hypothesis yet" **plus the generic scaffold** (below). One catch-all path for everything that isn't a real hypothesis or an A/A test — the detector does not need to distinguish "trying" from "junk." **Security rule:** for injection strings (script tags, `onerror=`, etc.), show the same generic scaffold but **never render the raw input back** — no echo, no reflection anywhere in the UI. -2. **Confident A/A or platform self-test** — the A/A path (see below). Signals: "A/A", "validate bucketing", SRM/bucketing checks, identical variants, "test the platform", "dummy flag", "just for dev env". Detect conservatively — **prefer missing an A/A over mislabeling a real hypothesis.** (A hypothesis that merely mentions "A/B test" as part of a real idea is fine.) 3. **Has ≥1 component** — score change / measurement / rationale and continue below. ### The generic scaffold (junk / empty catch-all) From 849ee1e449f2b18e2ffd89e6302c4f4f362cc97c Mon Sep 17 00:00:00 2001 From: Harsh Sadhvani Date: Wed, 29 Jul 2026 10:34:11 -0700 Subject: [PATCH 12/14] fix(experiments): hypothesis skill semantic validity + JSON output mode (EXPT-4644) (#134) fix(experiments): reject nonsense hypotheses; add JSON mode [EXPT-4644] --- .../SKILL.md | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md index 141c7e8..fd2cf79 100644 --- a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md +++ b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md @@ -32,6 +32,8 @@ The skeleton's three slots are the rubric, shown to the user as the **Change / M - **Measurement** — what you expect users to do differently, in plain words (rule 3). The sentence slot reads as an outcome ("this outcome will happen"); the tracker names the component Measurement. Same thing. **Counting:** near-synonyms describing one outcome count as one ("more clicks / higher CTR" = one); two genuinely distinct outcomes count as multiple ("bid more" and "bid for higher GMV" = multiple). Keep distinct-but-close outcomes separate; don't merge them. - **Rationale** — the mechanism: *why* the change causes that result. Not a restatement. +**Semantic validity.** A slot counts only if its content is genuinely that component, not merely sitting in the If/then/because grammar. "apple pie" is not a change (it names a thing, not an edit). "elephant" is not a measurement (not an outcome). "purple" is not a rationale (not a mechanism). The three must be causally connected: the change could plausibly move the measurement, and the rationale explains that link. If/then/because filled with non-sequiturs is not a strong hypothesis; treat those slots as absent and route to junk. Grammar alone never earns a component. + ## One input, one button There is a **single input** — the hypothesis field. The panel header reads **"Hypothesis"**; the field itself is where the user types, with ghost text **"Describe what you want to test."** There is no separate describe box and no second on-ramp; the field is the only entry point. @@ -176,6 +178,27 @@ Layout: the **rubric row sits above the button row**. The button row (after the Saving the hypothesis to the experiment is **not** a control here — that's the **Save** button on the builder's action bar, outside this panel. +## Structured output mode (headless) + +When the caller requests JSON only (the experiment builder's headless entry point, e.g. a system prompt that says "reply with one JSON object and nothing else"), skip the conversational flow and the handoff below. Reply once, with exactly this JSON and nothing else: no prose, no markdown fences. + +```json +{ + "schema_version": 1, + "route": "scaffold", + "components": { "change": true, "measurement": false, "rationale": false }, + "hypothesis": "If we change the homepage button from green to black, then {{measurement:what do you expect users to do more or less of?}}, because {{rationale:why would black cause that?}}", + "measurements": [] +} +``` + +- `route` — one of `scaffold | rewrite | junk | aa`, from Step 0. +- `components` — presence booleans judged on the input after the semantic-validity check (not on the scaffold you return). For `junk` all three are false; for `rewrite` and `aa` all three are true. A 3/3 hypothesis already in canonical order is `route: scaffold` with all three true and no holes (the strong "looks ready" state), not `rewrite`. +- `hypothesis` — the sentence for the field, with `{{component:hint}}` holes for missing slots (component is `change`, `measurement`, or `rationale`; hint is a short question). No holes for `rewrite`/`aa`. Never use `{{ }}` for anything except holes. +- `measurements` — every measurement stated in the input as `{ "text": "...", "primary": true|false }`, with exactly one primary when non-empty; empty when the input states none. + +This is the same machine contract the o11y `experiment-hypothesis` skill emits. Keep the two identical so both callers grade the same way. + ## Handoff, then STOP When the user is satisfied, emit this and stop. Resolve nothing against the catalog. From 2f6e5b8cf1a5eca86a827776adbb4e4408b2b9bf Mon Sep 17 00:00:00 2001 From: ericoh-ld Date: Mon, 3 Aug 2026 10:27:13 -0700 Subject: [PATCH 13/14] fix(experiments): route bare goals and lone rationales to scaffold, not junk (#139) * fix junk classification for ai assisted hypothesis builder * fix non sequitur routing --- .../SKILL.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md index fd2cf79..2a77669 100644 --- a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md +++ b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md @@ -21,7 +21,7 @@ Everything below derives from these four. Read them first. 1. **A strong hypothesis is written:** *If [change], then [this outcome will happen], because [reason it works].* 2. **A hypothesis must follow that structure.** The scaffold exists to enforce it. -3. **A measurement is a described outcome, not a named metric.** "More clicks," "faster time," "less drop-off" all count. Never require a formal metric name; never invent one. Unfalsifiable outcomes ("will do better or as well," "no negative impact," bare adjectives like "better experience") do **not** count — treat the measurement as missing. +3. **A measurement is a described outcome, not a named metric.** "More clicks," "faster time," "less drop-off" all count. Never require a formal metric name; never invent one. Unfalsifiable outcomes ("will do better or as well," "no negative impact," bare adjectives like "better experience") do **not** count — treat the measurement as missing. A goal that names a specific outcome ("raise click-through," "increase checkout completion") is a measurement with holes — scaffold it, not junk. A vague direction with no specific outcome ("grow the business," "optimize the funnel") is not a measurement — leave it a hole — but it is still a scaffold, not junk. 4. **Exactly one measurement — the primary — goes in the hypothesis.** ~17% of real hypotheses name two or more; keep one primary in the sentence and treat the rest as secondary. ## The three components @@ -30,9 +30,9 @@ The skeleton's three slots are the rubric, shown to the user as the **Change / M - **Change** — the specific thing you'll do differently. A concrete edit, not a goal. - **Measurement** — what you expect users to do differently, in plain words (rule 3). The sentence slot reads as an outcome ("this outcome will happen"); the tracker names the component Measurement. Same thing. **Counting:** near-synonyms describing one outcome count as one ("more clicks / higher CTR" = one); two genuinely distinct outcomes count as multiple ("bid more" and "bid for higher GMV" = multiple). Keep distinct-but-close outcomes separate; don't merge them. -- **Rationale** — the mechanism: *why* the change causes that result. Not a restatement. +- **Rationale** — the mechanism: *why* the change causes that result. Not a restatement. A standalone causal or mechanism statement counts as a rationale even with no stated change or measurement — especially one introduced by "because," "since," or "so that," or one explaining why users behave a certain way ("users trust familiar payment options"). Scaffold it with holes for the missing change and measurement; do not route it to junk. -**Semantic validity.** A slot counts only if its content is genuinely that component, not merely sitting in the If/then/because grammar. "apple pie" is not a change (it names a thing, not an edit). "elephant" is not a measurement (not an outcome). "purple" is not a rationale (not a mechanism). The three must be causally connected: the change could plausibly move the measurement, and the rationale explains that link. If/then/because filled with non-sequiturs is not a strong hypothesis; treat those slots as absent and route to junk. Grammar alone never earns a component. +**Semantic validity.** A slot counts only if its content is genuinely that component, not merely sitting in the If/then/because grammar. "apple pie" is not a change (it names a thing, not an edit). "elephant" is not a measurement (not an outcome). "purple" is not a rationale (not a mechanism). When **two or more** slots are filled they must be causally connected: the change could plausibly move the measurement, and the rationale explains that link. If/then/because filled with non-sequiturs is not a strong hypothesis; treat those slots as absent and route to junk. Grammar alone never earns a component. This coherence test applies only across **multiple** filled slots — a single genuine component standing alone (a lone change, a lone described outcome, a lone mechanism) has nothing to contradict, so it is not a non-sequitur: scaffold it, do not route it to junk. ## One input, one button @@ -66,7 +66,7 @@ Mental model: the field is the truth; tracker + critique are functions of it, re Classify the field text on Fix and grade, then route to one display state. Gates run **before** component scoring, in priority order: 1. **Confident A/A or platform self-test** — the A/A path (see below). Signals: "A/A", "validate bucketing", SRM/bucketing checks, identical variants, "test the platform", "dummy flag", "just for dev env". Detect conservatively — **prefer missing an A/A over mislabeling a real hypothesis.** (A hypothesis that merely mentions "A/B test" as part of a real idea is fine.) **Checked before junk:** A/A inputs legitimately carry no rubric components, so the junk catch-all would otherwise swallow them. -2. **Junk / gibberish / URL / injection / real-but-empty fragment** — show the critique "This is not a hypothesis yet" **plus the generic scaffold** (below). One catch-all path for everything that isn't a real hypothesis or an A/A test — the detector does not need to distinguish "trying" from "junk." +2. **Junk / gibberish / URL / injection / non-linguistic fragment / non-sequitur** — show the critique "This is not a hypothesis yet" **plus the generic scaffold** (below). Reserve this path for input that is **not a coherent attempt to describe something to test**: gibberish, single tokens, punctuation-only, URLs or bare links, placeholders, injection. A coherent statement of intent is never junk, even if it names only a goal, is vague, or states only a reason — route those to the component-scoring path below (they scaffold, not junk). The one exception is a non-sequitur (see Semantic validity). **Security rule:** for injection strings (script tags, `onerror=`, etc.), show the same generic scaffold but **never render the raw input back** — no echo, no reflection anywhere in the UI. 3. **Has ≥1 component** — score change / measurement / rationale and continue below. @@ -229,7 +229,7 @@ The downstream skill composes a display state from `components` + `measurementCo - Don't generate critique copy — detect the state and use the fixed message. - Don't re-score on keystroke — the tracker and critique refresh on Fix and grade. - Don't let a rewrite re-scope the user's meaning — re-word only. -- Don't scaffold a bare goal, and don't treat A/A as an error. +- Don't route a coherent goal or a lone rationale to junk — scaffold it, with holes for what's missing; and don't treat A/A as an error. - Don't merge distinct measurements; surface them in the hole with "or" and mark Measurement with the amber warning. - Don't nudge to isolate a single change — multiple changes are allowed. - Don't let anything but Clear erase field text. From 0e81d121e9907fc062c5a3c60b6b716b3c0cb53c Mon Sep 17 00:00:00 2001 From: ericoh-ld Date: Mon, 3 Aug 2026 15:30:52 -0700 Subject: [PATCH 14/14] fix(experiments): unblock hypothesis-builder validate and eval checks (#141) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(experiments): unblock hypothesis-builder validate and eval checks Two independent breakages on this branch turned 4 checks red. Neither came from #139 — it only touched SKILL.md, which is what diff-changed-skills.js uses to decide the suite must re-run, so it surfaced both. validate: SKILL.md declared name: launchdarkly-experiment-hypothesis-assistant while the directory is ...-builder, and validate_skills.py requires the two to match. Landed in 7e486ab (v0.3.0). Everything else in the repo already said -builder (_manifest.js, the evals/ dir, the provider's skill_slug, the eval:hypothesis-builder script), so the frontmatter was the lone outlier. Renamed it, plus the handoffFrom value where the skill names itself — the only other occurrence in the repo, with no consumer matching the literal. This also realigns the eval's forced invocation: the provider invokes / = /launchdarkly-experiment-hypothesis-builder, which did not match the name the skill declared. Evaluate suite: all 5 tests errored in 0s with "Template render error [Line 190, Column 90] expected variable end" — none reached an API call. The suite registered SKILL.md as promptfoo's prompt, promptfoo renders prompts through nunjucks, and SKILL.md documents its own hole syntax ({{measurement:...}}, {{component:hint}}) in the output-contract examples. Nunjucks reads `measurement` as a variable, hits the `:`, and throws. Landed in 849ee1e (#134), whose CI never ran these checks. Replaced the prompt with a suite-local function prompt that returns SKILL.md wrapped in {% raw %}. A function prompt does not skip nunjucks — renderPrompt assigns the return value to basePrompt and still calls renderString on it — and promptfoo's own autoWrapRawIfPartialNunjucks only fires on unclosed tags, so closed-but-invalid {{measurement:...}} sails through. Wrapping supplies the {% raw %} that helper would have added; renderString then returns the file byte-for-byte, so results.json still shows the exact skill text and SKILL.md on disk keeps its hole syntax byte-identical to the o11y contract. Kept suite-local rather than in evals/shared/, which is a GLOBAL_TRIGGERS entry and would force all 7 suites to re-run. Verified: validate_skills.py passes 46/46; nunjucks reproduces the exact Line 190 Column 90 failure on the bare file and renders the wrapped prompt byte-identically; npm test 70/70; one live suite test passes in 1m2s (was 5 errors in 0s). Aggregate scores and Evaluate gate need no separate fix — both were cascades of the errored suite. * update comments * fix evals to match new skill --- .../prompt.js | 30 ++ .../promptfooconfig.yaml | 402 +++++++++++------- .../SKILL.md | 4 +- 3 files changed, 273 insertions(+), 163 deletions(-) create mode 100644 evals/launchdarkly-experiment-hypothesis-builder/prompt.js diff --git a/evals/launchdarkly-experiment-hypothesis-builder/prompt.js b/evals/launchdarkly-experiment-hypothesis-builder/prompt.js new file mode 100644 index 0000000..9788296 --- /dev/null +++ b/evals/launchdarkly-experiment-hypothesis-builder/prompt.js @@ -0,0 +1,30 @@ +/** + * Prompt function for this suite: returns SKILL.md wrapped in nunjucks `{% raw %}`. + * + * promptfoo renders every prompt through nunjucks, and this SKILL.md + * uses its own placeholder-hole syntax — `{{measurement:...}}`, + * `{{component:hint}}` in the output-contract examples. Nunjucks parses `measurement` + * as a variable, hits the `:`, and throws "expected variable end" on all the tests. + * + * A function prompt does NOT skip the nunjucks pass: promptfoo's renderPrompt + * assigns the function's return value to basePrompt and still calls nunjucks.renderString on it. + * Its own escape hatch, + * autoWrapRawIfPartialNunjucks, only fires on *unclosed* tags (`{{` with no `}}`), so + * closed-but-invalid expressions like `{{measurement:...}}` go through unprotected. + * Wrapping here supplies the `{% raw %}` that helper would have added. renderString then + * returns the file byte-for-byte, so results.json still shows the exact skill text. + * The tags live only in this in-memory prompt — SKILL.md is not changed. + * + * The prompt is not what the agent under test sees. The skill is loaded into + * `.claude/skills//`, and builds the user turn from vars.user_request. This + * exists to satisfy promptfoo's requirement that a prompt be defined. + */ +const fs = require("node:fs"); +const path = require("node:path"); + +const SKILL_MD = path.resolve( + __dirname, + "../../skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md", +); + +module.exports = () => `{% raw %}${fs.readFileSync(SKILL_MD, "utf-8")}{% endraw %}`; diff --git a/evals/launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml b/evals/launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml index bfa9f71..32eb7a8 100644 --- a/evals/launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml +++ b/evals/launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml @@ -1,181 +1,261 @@ # yaml-language-server: $schema=https://promptfoo.dev/config-schema.json # # Evaluates launchdarkly-experiment-hypothesis-builder — an advisory skill that -# coaches a hypothesis and hands off, never writing. Assertions read the tool-call -# trajectory plus llm-rubrics; it runs read-only via mcp_tool_allowlist. +# coaches a hypothesis and hands off, never writing. The skill +# returns one JSON object (SKILL.md "Structured output mode (headless)"). +# # Run: promptfoo eval -c shared/defaults.yaml -c launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml -description: "End-to-end evaluation of the launchdarkly-experiment-hypothesis-builder skill" +description: "Headless contract evaluation of the launchdarkly-experiment-hypothesis-builder skill" prompts: - - file://../../skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md + # A function prompt, not the raw SKILL.md — see prompt.js for why. + - id: file://./prompt.js + label: launchdarkly-experiment-hypothesis-builder providers: - id: file://../providers/claude-skill-agent-sdk.js label: claude-skill-agent-sdk config: skill_slug: launchdarkly-experiment-hypothesis-builder - expose_mcp_tools: true - expose_ask_question: true + expose_mcp_tools: false + expose_ask_question: false force_skill_invocation: true # force-load the skill so the eval tests it, not base Claude - # Advisory / handoff-only: expose read tools only so it cannot write. - mcp_tool_allowlist: - - list-flags - - get-flag - - list-feature-flags - - list-metrics - - get-metric - - list-metric-events - - get-project + +defaultTest: + assert: + # One contract check for every case, driven by that case's `expect` var. + # Concatenated with shared/defaults.yaml's output_valid + latency asserts. + # + # Scoring is binary on purpose: shared/defaults.yaml applies a 0.75 threshold + # to the weighted score, so partial credit would let a contract violation + # slip through as a pass. Per-check detail goes in `reason` instead. + - type: javascript + value: | + const ROUTES = ['scaffold','rewrite','junk','aa']; + const want = (context.vars && context.vars.expect) || {}; + const text = String((output && (output.response || output.first_assistant_text)) || ''); + + // Parse leniently so a fenced reply still reports useful detail; strictness + // is its own check ("Reply once, with exactly this JSON and nothing else"). + const t = text.trim(); + let payload = null, strict = false; + try { payload = JSON.parse(t); strict = true; } catch (e) { + const a = t.indexOf('{'), b = t.lastIndexOf('}'); + if (a !== -1 && b > a) { try { payload = JSON.parse(t.slice(a, b + 1)); } catch (e2) {} } + } + if (!payload) return { pass: false, score: 0, reason: `no JSON object in reply: ${text.slice(0,160)}` }; + + const checks = []; + const add = (name, ok, detail) => checks.push({ name, ok, detail }); + + add('json_only', strict, 'JSON wrapped in prose or fences'); + add('schema_version', payload.schema_version === 1, `got ${payload.schema_version}`); + add('route_valid', ROUTES.includes(payload.route), `got ${JSON.stringify(payload.route)}`); + if (want.route !== undefined) add('route', payload.route === want.route, `expected ${want.route}, got ${JSON.stringify(payload.route)}`); + + if (want.components) { + const got = payload.components || {}; + for (const k of ['change','measurement','rationale']) { + if (want.components[k] === undefined) continue; + add(`component.${k}`, Boolean(got[k]) === Boolean(want.components[k]), `expected ${want.components[k]}, got ${JSON.stringify(got[k])}`); + } + } + + // Holes are only asserted where the spec is unambiguous — omit `holes` + // from a case's `expect` to leave the sentence shape unchecked. + if (want.holes !== undefined) { + const found = []; + const re = /\{\{\s*(\w+)\s*:[^}]*\}\}/g; + let m; + while ((m = re.exec(String(payload.hypothesis || ''))) !== null) found.push(m[1]); + const got = found.sort().join(','), exp = [...want.holes].sort().join(','); + add('holes', got === exp, `expected [${exp}], got [${got}]`); + } + + if (want.measurements) { + const list = Array.isArray(payload.measurements) ? payload.measurements : null; + add('measurements_array', list !== null, `got ${JSON.stringify(payload.measurements)}`); + if (list) { + if (want.measurements.count !== undefined) add('measurements.count', list.length === want.measurements.count, `expected ${want.measurements.count}, got ${list.length}`); + if (want.measurements.min !== undefined) add('measurements.min', list.length >= want.measurements.min, `expected >=${want.measurements.min}, got ${list.length}`); + // "exactly one primary when non-empty; empty when the input states none." + if (list.length > 0) { + const primaries = list.filter(x => x && x.primary).length; + add('exactly_one_primary', primaries === 1, `got ${primaries}`); + } + } + } + + if (want.hypothesisEquals !== undefined) { + add('hypothesis_exact', String(payload.hypothesis || '').trim() === want.hypothesisEquals.trim(), `got ${JSON.stringify(String(payload.hypothesis || '').slice(0,120))}`); + } + for (const needle of (want.mustNotContain || [])) { + add(`no_echo(${needle})`, !text.includes(needle), 'raw input echoed back'); + } + + const failed = checks.filter(c => !c.ok); + return { + pass: failed.length === 0, + score: failed.length === 0 ? 1 : 0, + reason: failed.length === 0 + ? `contract ok (${checks.length} checks): route=${payload.route}` + : failed.map(c => `${c.name}: ${c.detail}`).join(' | '), + }; + metric: contract + weight: 3 tests: - # Golden path: coach a strong draft to a good hypothesis, no writes. - - description: "Golden path: coach a strong draft to a good hypothesis, no writes" + # ---- Gate 1: A/A / platform self-test (checked before junk) ---------------- + # The one case with a fixed output string (SKILL.md "A/A tests"). + - description: "A/A: routes to the A/A path with the fixed hypothesis, not a built experiment" + vars: + max_turns: 3 + user_request: | + Reply with one JSON object and nothing else — no prose, no markdown fences. + + Hypothesis field text: + A/A test to validate bucketing. + expect: + route: aa + components: { change: true, measurement: true, rationale: true } + holes: [] + hypothesisEquals: "If we split traffic evenly between two identical variants, then key metrics show no meaningful difference, because the only thing that differs is random assignment." + + # ---- Gate 2: junk — reserved for non-attempts ------------------------------ + - description: "Junk: gibberish routes to junk with all three components absent" + vars: + max_turns: 3 + user_request: | + Reply with one JSON object and nothing else — no prose, no markdown fences. + + Hypothesis field text: + asdfgh + expect: + route: junk + components: { change: false, measurement: false, rationale: false } + holes: [change, measurement, rationale] + measurements: { count: 0 } + + - description: "Junk: a bare link routes to junk" + vars: + max_turns: 3 + user_request: | + Reply with one JSON object and nothing else — no prose, no markdown fences. + + Hypothesis field text: + https://example.com/pricing + expect: + route: junk + components: { change: false, measurement: false, rationale: false } + + # Security rule: same generic scaffold, but never render the raw input back. + - description: "Security: injection routes to junk and is never echoed back" vars: - max_turns: 25 - mock_ask_question_answers: - - ["~5%"] - user_request: > - In project "my-app", I want to test moving our primary navigation to the - left rail. I expect it to increase signup conversion, because it reduces - cognitive load, while login success rate stays flat. - assert: - - type: javascript - value: | - const tools = output.tools_called || []; - const writes = ['create-flag','create-feature-flag','update-feature-flag','create-metric','create-experiment','start-experiment-iteration'].filter(w=>tools.includes(w)); - const pass = writes.length===0; - return { pass, score: pass?1:0, reason: pass?'No writes (handoff/advisory only)':'Wrote: '+writes.join(', ') }; - metric: no_writes - weight: 3 - - type: javascript - value: | - const asks = (output.trajectory||[]).filter(t=>t.tool==='ask-question').length; - const pass = asks <= 2; - return { pass, score: pass?1:(asks<=3?0.5:0), reason:`ask-question calls: ${asks}` }; - metric: does_not_over_interrogate - weight: 1 - - type: llm-rubric - value: | - Evaluate the hypothesis the agent produced. Score 1.0 if all met, deduct 0.25 each: - 1. One clear if/then/because sentence naming the intervention (left-rail nav) and the primary metric (signup conversion) with a direction (increase). - 2. Single-variable — only the nav position changes between control and treatment. - 3. Falsifiable — a result could prove it wrong. - 4. It surfaces the flag and metric it would use so the experiment can be built downstream. - metric: hypothesis_quality - weight: 2 - - # Weak input: rebuild a vague goal via questions, no writes. - - description: "Fork (weak): a vague goal is rebuilt via questions, no writes" + max_turns: 3 + user_request: | + Reply with one JSON object and nothing else — no prose, no markdown fences. + + Hypothesis field text: + + expect: + route: junk + components: { change: false, measurement: false, rationale: false } + mustNotContain: ["t.tool==='ask-question'); - const text = (asks.map(a=>a.arguments&&a.arguments.prompt||'').join(' ') + ' ' + String(output.response||'')).toLowerCase(); - const asked = asks.length>0 || /\?/.test(String(output.response||'')); - const onTopic = /(metric|measur|what.*change|specific change|which|area)/.test(text); - const score = (asked?0.5:0)+(onTopic?0.5:0); - return { pass: asked && onTopic, score, reason:`asks=${asks.length} onTopic=${onTopic}` }; - metric: coaches_missing_element - weight: 3 - - type: javascript - value: | - const tools = output.tools_called || []; - const writes = ['create-flag','create-feature-flag','update-feature-flag','create-metric','create-experiment','start-experiment-iteration'].filter(w=>tools.includes(w)); - const pass = writes.length===0; - return { pass, score: pass?1:0, reason: pass?'No writes':'Wrote: '+writes.join(', ') }; - metric: no_writes - weight: 3 - - type: llm-rubric - value: | - Score 1.0 if all met, deduct 0.33 each: - 1. The agent treats "increase revenue" as too vague to build from (a goal, not a hypothesis). - 2. It asks for the specific change to test and a concrete measurable primary metric. - 3. It does not fabricate an intervention, metric, or magnitude the user never provided. - metric: weak_input_handled - weight: 2 - - # Cost of being wrong: resolve an existing metric without a verbatim search - # (LD search is literal substring, so a raw phrase misses "Signup completed"). - - description: "Cost of wrong: resolve the existing metric without a verbatim search, no writes" + max_turns: 3 + user_request: | + Reply with one JSON object and nothing else — no prose, no markdown fences. + + Hypothesis field text: + increase checkout completion + expect: + route: scaffold + components: { change: false, measurement: true, rationale: false } + holes: [change, rationale] + measurements: { count: 1 } + + - description: "Lone rationale: a standalone mechanism scaffolds, not junk" vars: - max_turns: 25 - user_request: > - In project "my-app", test a one-click signup button. I expect it to raise - signup completion. - assert: - - type: javascript - value: | - const traj = output.trajectory || []; - const qs = traj.filter(t=>t.tool==='list-metrics').map(t=>String((t.arguments&&t.arguments.query)||'').toLowerCase()); - const searched = qs.length>0; - const verbatim = qs.some(q=>q==='signup completion'); - const pass = searched && !verbatim; - return { pass, score: pass?1:(searched?0.4:0), reason:`list-metrics queries=[${qs.join(', ')}]` }; - metric: searched_not_verbatim - weight: 2 - - type: javascript - value: | - const tools = output.tools_called || []; - const writes = ['create-flag','create-feature-flag','update-feature-flag','create-metric','create-experiment','start-experiment-iteration'].filter(w=>tools.includes(w)); - const pass = writes.length===0; - return { pass, score: pass?1:0, reason: pass?'No writes':'Wrote: '+writes.join(', ') }; - metric: no_writes - weight: 3 - - type: llm-rubric - value: | - The project already has a "Signup completed" metric. Score 1.0 if both met, deduct 0.5 each: - 1. The agent surfaces the existing "Signup completed" metric rather than proposing a brand-new duplicate. - 2. It confirms the metric pick with the user rather than silently assuming it. - metric: resolution_quality - weight: 2 - - # Cost of being wrong: catch a metric/outcome mismatch (predicts engagement, - # proposes measuring revenue). - - description: "Cost of wrong: catch the metric<->outcome mismatch, no writes" + max_turns: 3 + user_request: | + Reply with one JSON object and nothing else — no prose, no markdown fences. + + Hypothesis field text: + because the current option is buried + expect: + route: scaffold + components: { change: false, measurement: false, rationale: true } + holes: [change, measurement] + measurements: { count: 0 } + + - description: "Vague direction: not a measurement, but still a scaffold not junk" vars: - max_turns: 25 - user_request: > - In project "my-app", I want to test a new onboarding checklist. I think it - will boost engagement — let's measure it by revenue. - assert: - - type: javascript - value: | - const tools = output.tools_called || []; - const writes = ['create-flag','create-feature-flag','update-feature-flag','create-metric','create-experiment','start-experiment-iteration'].filter(w=>tools.includes(w)); - const pass = writes.length===0; - return { pass, score: pass?1:0, reason: pass?'No writes':'Wrote: '+writes.join(', ') }; - metric: no_writes - weight: 2 - - type: llm-rubric - value: | - The user predicts ENGAGEMENT will move but proposes measuring REVENUE. Score 1.0 if all met, deduct 0.33 each: - 1. The agent catches the mismatch and does not silently accept revenue as the primary metric for an engagement prediction. - 2. It aligns the primary metric to the predicted outcome (an engagement metric), or asks the user to reconcile the two. - 3. It does not invent a magnitude the user never gave. - metric: mismatch_caught - weight: 3 - - # Safety: a platform self-test (A/A) is not built into an experiment. - - description: "Safety: a platform self-test is not built into an experiment, no writes" + max_turns: 3 + user_request: | + Reply with one JSON object and nothing else — no prose, no markdown fences. + + Hypothesis field text: + grow the business + expect: + route: scaffold + components: { change: false, measurement: false, rationale: false } + holes: [change, measurement, rationale] + measurements: { count: 0 } + + # ---- Rule 3: unfalsifiable outcomes do not count as a measurement ---------- + # The rationale here has to be a genuine mechanism, not a restatement of the + # change ("because the layout is cleaner" scores rationale: false per the + # "Not a restatement" rule), so the case isolates the unfalsifiable rule. + - description: "Unfalsifiable outcome: measurement is treated as missing" + vars: + max_turns: 3 + user_request: | + Reply with one JSON object and nothing else — no prose, no markdown fences. + + Hypothesis field text: + If we redesign the checkout page, then it will do better or as well, because a simpler layout reduces confusion for first-time buyers. + expect: + route: scaffold + components: { change: true, measurement: false, rationale: true } + holes: [measurement] + measurements: { count: 0 } + + # ---- Rule 4: exactly one primary measurement ------------------------------ + # `holes` is deliberately NOT asserted here — SKILL.md is self-contradictory on + # what the sentence should look like for this state, and pinning either reading + # in a test would freeze an undecided design question: + # - rule 4 (line 25): "keep one primary in the sentence" (i.e. slot filled) + # - line 233: "surface them in the hole with 'or'" (i.e. slot is a hole) + # - line 197: holes are for "missing" slots, yet measurement is present + # Also open: when the predicted outcome and the named metric differ, which one + # becomes primary? Resolve both in SKILL.md, then assert. + - description: "Multiple measurements: two distinct outcomes, exactly one primary" vars: - max_turns: 15 - user_request: "A/A test to validate bucketing. Project my-app." - assert: - - type: javascript - value: | - const tools = output.tools_called || []; - const writes = ['create-flag','create-feature-flag','update-feature-flag','create-metric','create-experiment','start-experiment-iteration'].filter(w=>tools.includes(w)); - const pass = writes.length===0; - return { pass, score: pass?1:0, reason: pass?'No writes':'Wrote: '+writes.join(', ') }; - metric: no_writes - weight: 3 - - type: llm-rubric - value: | - Score 1.0 if both met, deduct 0.5 each: - 1. The agent recognizes this as a platform/self-test (an A/A bucketing check), not a real experiment idea. - 2. It gently confirms intent rather than coaching it into a full hypothesis and configuration. - metric: nonreal_handled - weight: 2 + max_turns: 3 + user_request: | + Reply with one JSON object and nothing else — no prose, no markdown fences. + + Hypothesis field text: + I want to test a new onboarding checklist. I think it will boost engagement — let's measure it by revenue. + expect: + route: scaffold + components: { change: true, measurement: true, rationale: false } + measurements: { min: 2 } + + # ---- The strong state: 3/3 in canonical order, no holes ------------------- + - description: "Complete hypothesis: all three components, no holes" + vars: + max_turns: 3 + user_request: | + Reply with one JSON object and nothing else — no prose, no markdown fences. + + Hypothesis field text: + If we move the primary navigation to the left rail, then signup conversion increases, because it reduces cognitive load. + expect: + route: scaffold + components: { change: true, measurement: true, rationale: true } + holes: [] + measurements: { count: 1 } diff --git a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md index 2a77669..1a98fc9 100644 --- a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md +++ b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md @@ -1,5 +1,5 @@ --- -name: launchdarkly-experiment-hypothesis-assistant +name: launchdarkly-experiment-hypothesis-builder description: "Help a user turn a rough idea into a strong, testable experiment hypothesis, or critique one they wrote. Detects which parts of the hypothesis are present, scaffolds an If/then/because sentence with holes for what's missing, and shows a fixed critique message. Use when a user is starting an experiment or sharpening a hypothesis. Does NOT resolve flags or metrics, build experiment config, or write to LaunchDarkly." compatibility: Read-only LaunchDarkly lookups only. Hands off to launchdarkly-experiment-setup for build. license: Apache-2.0 @@ -205,7 +205,7 @@ When the user is satisfied, emit this and stop. Resolve nothing against the cata ```json { - "handoffFrom": "launchdarkly-experiment-hypothesis-assistant", + "handoffFrom": "launchdarkly-experiment-hypothesis-builder", "hypothesis": "polished single sentence", "change": "what changes, in plain words (may be more than one)", "primaryMeasurement": "the described outcome, in the user's words",