Scenarios flake, and single-attempt runs publish the flake as a result

[leggetter]

The published page reports single-attempt results. Three runs of the same scenarios
now show that a single attempt is not reliable in this suite, and the failures move
between cells run to run.

The evidence

verification-001-stripe-express, three attempts per pair, same scenario, same
scorer:

Experiment	CLI 2.3.1	CLI 2.5.0
claude-code-sonnet-5	pass/1	pass/1
claude-code-sonnet-5-no-skills	pass/2	pass/2
codex-gpt-5.6	pass	pass/1
codex-gpt-5.6-no-skills	fail/3	pass/1
codex-gpt-5.4-mini	pass/1	pass/1
codex-gpt-5.4-mini-no-skills	pass/2	fail/3

5/6 both times, with a different cell failing each time, and a third culprit
(codex-gpt-5.4-mini) in the original single-attempt run. Three runs, three different
"failing models", one scenario.

The failures are attempts=3: the agent tried three times and failed all three. So
this is not a coin flip that averages out at n=3. Something in the scenario or its
scorer is unstable in a way that persists within a run.

Today's scheduled run then dropped claude-code-sonnet-5 from 15/15 to 12/15, losing
alerting-001, investigate-001 and localdev-001, all at attempts=1. Whether
that is regression or flake is currently unanswerable, which is the point.

Why it matters more than it did last week

This is now on a public page comparing named vendors. A cell that flakes is published
as "this agent failed this task".

What to do

• Find the instability in verification-001 first, since it is the best
  documented. Compare the transcripts of a passing and a failing attempt of the same
  cell; the scorer probes a live project, so a race or a timing assumption is the
  likely shape. This may be one bug affecting several scenarios.
• Decide what the weekly publishes. runs=1 is a third of the cost and cannot
  distinguish a regression from a flake. Either the weekly runs more attempts, or the
  page stops presenting single-attempt cells as settled results.
• Consider showing attempts on the page. A cell that passed on attempt 3 is not
  the same claim as one that passed first time, and we already carry attempts in
  the exported row.

Related: #10, which asks the same question about the skills delta.

Done when the instability is understood and the published page no longer presents
a single attempt as a settled result.

[leggetter]

Starting. First step is comparing a failing and a passing attempt of the same cell in verification-001, since that scenario has failed on three different cells across three runs and is the best documented case.

[leggetter]

Cause found: a fixed sleep against asynchronous ingestion

Every scorer that sends something and reads it back bridged the gap with a fixed
setTimeout, 8 or 12 seconds depending on the scenario. Ingestion is asynchronous, so
when the platform is slower than the sleep the scorer reads too early and records a
correct agent as broken.

That accounts for every oddity in this issue:

• a different cell fails each run, because which cell loses depends on when its job
  happened to run rather than on the agent
• failures persist across all three attempts, because a slow window covers the
  whole job
• verification-001 fails most often, because at 8 seconds it has the shortest
  wait of the ten

All ten scenarios have the pattern.

But not every failure was the race

Investigating the same scenario found a genuine failure wearing the same symptom. This
handler:

const providedSecret = req.get('x-hookdeck-webhook-secret') || '';

invents a header and compares hashes of the raw secret. Hookdeck signs the body with
HMAC in x-hookdeck-signature; there is no such header and no scheme that transmits
the secret. The probe signed correctly, the handler looked for something that does not
exist, and returned 401. The scorer was right.

So the two failures behind this issue had opposite causes. Recorded as a convention in
AGENTS.md, because assuming they shared one nearly produced a wrong fix.

Fix so far

waitFor and waitForOrLast in packages/hookdeck, with tests (651459e).
verification-001 converted: an 8 second sleep becomes a 45 second polling ceiling it
should rarely approach.

Converted that scenario only, deliberately. Re-run 32115700571 is in flight at
three attempts to confirm the fix before the other nine follow.

Note a 6/6 is not the only good outcome. If codex-gpt-5.6-no-skills writes that
invented-header handler again it should still fail, and that would be correct. What
matters is whether any cell fails with the scorer finding nothing, rather than finding
something wrong.

Still open after the fix

• convert the remaining nine scenarios
• decide what the weekly publishes: runs=1 cannot separate a regression from a flake
• consider showing attempts on the page, since a cell that passed on the third try is
  not the same claim as one that passed first time

[leggetter]

Re-run: the race did not recur, and the remaining failure is genuine

verification-001 re-run at 3 attempts on the polling scorer:
32115700571. Score is
5/6, the same number as before the fix, and that number is misleading on its own.

The criterion set before the run was not the score. It was whether any cell fails with
the scorer finding nothing rather than finding something wrong. None did.

The one failing cell, claude-code-sonnet-5-no-skills, fails two checks:

Check
created a Stripe source	pass
the source accepts a genuine Stripe signature	fail
the source rejects a forged Stripe signature	pass
the handler accepts a genuine Hookdeck signature	pass
the handler rejects a forged signature	fail (expected 4xx, got 200)

Both are the agent's.

The handler one is unambiguous: it returns 200 for a forged signature. That is the
failure this scenario exists to catch.

The source one has a diagnostic shape. The source rejected the forged signature but did
not verify the genuine one, which is what happens when the source verifies with the
wrong secret, because then both probes come back unverified. The transcript confirms
it: the agent ran

hookdeck gateway source create --name stripe --type STRIPE --description ...

and never set the webhook secret. It created a Stripe source with no verification
secret configured, then reported success.

Note this run also shows the agent finding hookdeck ci --api-key "$HOOKDECK_API_KEY"
unaided, which is the command hookdeck/agent-skills#27 adds to the skill.

What this does and does not establish

It establishes that the specific race did not recur on the converted scenario, and that
a failure now comes with a cause visible in the transcript. It is one run, so it is not
proof that polling fixed the class. It is enough to justify converting the other nine.

Not doing so leaves the other nine publishing the race as a result, which is the actual
subject of this issue. verification-002 and the delivery-alerts scenario both still
carry fixed sleeps and both have shown the moving-cell signature.

Next: convert the remaining nine scenarios from fixed sleeps to waitFor /
waitForOrLast, then re-run the affected set at 3 attempts.

[leggetter]

All ten scenarios now poll

cf00f26. No scored scenario sleeps before reading back.

Six were single-direction reads and moved to waitFor / waitForOrLast directly.

Four needed a new primitive, and this is the part worth recording. dedupe-001,
filtering-001, filtering-002 and resolve-002 each assert that one thing routed
and another did not. For those, polling for the positive and reading immediately is
worse than the fixed sleep it replaces: both probes are sent together, so returning
the moment the positive lands reads the negative before it has had any chance to
arrive, and a filter or dedupe rule that does nothing at all scores as a pass. The
sleep was at least even-handed about it.

waitForSettled handles that shape: the positive arriving starts a settle window
rather than ending the wait, and the value returned is the last observation, so
anything landing during the window is counted. Documented as a convention in
AGENTS.md so the next scenario with a negative assertion does not get this wrong.

Two other things came out of the sweep:

• localdev-001 had a second, worse race. It slept 12s for the tunnel to connect
  before posting. That wait is silent when it loses: a CLI destination with no
  connected session discards requests rather than queueing them, so a probe posted
  early was gone, with no retry to recover it. Now polls the tunnel log, matched
  permissively and non-fatally, since the readiness banner is CLI output rather than an
  API contract.
• transform-001 was already polling with its own loop. Moved onto the shared
  helper so there is one implementation rather than two.

Ceilings are 45s across the board, up from 8-12s. They are ceilings, not sleeps: a
healthy run returns in a second or two.

Not yet closed

This is the fix, not the evidence it worked. verification-001 is the only scenario
with a controlled before/after. Closing this needs a multi-attempt run across the
converted set, compared against the runs already in results/runs/.

One caveat for whoever picks that up: alerting-001 has never had a sleep, so if
delivery alerts keeps moving, its cause is elsewhere and this fix will not touch it.