This benchmark surfaced a class of CLI behaviour that is now fixed upstream, and the
fix needs measuring from our side.
Upstream: hookdeck/hookdeck-cli#340, "Epic: make the CLI safe and predictable for
agents and automation". Nine issues closed by hookdeck/hookdeck-cli#342 in 2.5.0,
described there as sharing one failure shape: the CLI silently does something other
than what the caller asked, and the first symptom is missing traffic rather than an
error. The most relevant is #334, "listen ignores HOOKDECK_API_KEY and silently
creates a guest account".
What we measured. At the pinned CLI version (2.3.1), listen did not read
HOOKDECK_API_KEY at all. An agent that never ran hookdeck ci was operating on a
temporary guest project with nothing telling it. In verification-001-stripe-express
the weak model hit this, reported
"I also tried Hookdeck CLI account access, but the provided API key failed
authentication in this sandbox, so I left the local hookdeck listen setup
documented rather than creating cloud resources here."
and created nothing. Earlier, a different model reported success while working
entirely inside a guest project, which is the same defect with a worse outcome
because it is invisible.
Our side. The sandbox is bumped to 2.5.0. The three CLI-dependent scenarios,
localdev-001, verification-001 and verification-002, are re-running at three
attempts each on the new CLI with the skill unchanged, so the delta is attributable
to the CLI alone.
Done when the re-run shows what the CLI fix bought, and that goes into LOOPS.md
(#9) as a completed loop and into the release notes.
This benchmark surfaced a class of CLI behaviour that is now fixed upstream, and the
fix needs measuring from our side.
Upstream: hookdeck/hookdeck-cli#340, "Epic: make the CLI safe and predictable for
agents and automation". Nine issues closed by hookdeck/hookdeck-cli#342 in 2.5.0,
described there as sharing one failure shape: the CLI silently does something other
than what the caller asked, and the first symptom is missing traffic rather than an
error. The most relevant is #334, "listen ignores HOOKDECK_API_KEY and silently
creates a guest account".
What we measured. At the pinned CLI version (2.3.1),
listendid not readHOOKDECK_API_KEYat all. An agent that never ranhookdeck ciwas operating on atemporary guest project with nothing telling it. In
verification-001-stripe-expressthe weak model hit this, reported
and created nothing. Earlier, a different model reported success while working
entirely inside a guest project, which is the same defect with a worse outcome
because it is invisible.
Our side. The sandbox is bumped to 2.5.0. The three CLI-dependent scenarios,
localdev-001,verification-001andverification-002, are re-running at threeattempts each on the new CLI with the skill unchanged, so the delta is attributable
to the CLI alone.
Done when the re-run shows what the CLI fix bought, and that goes into
LOOPS.md(#9) as a completed loop and into the release notes.