Skip to content

Test the nodes against a real Hookdeck project - #8

Merged
garethx merged 1 commit into
mainfrom
test/live-suites
Aug 13, 2026
Merged

Test the nodes against a real Hookdeck project#8
garethx merged 1 commit into
mainfrom
test/live-suites

Conversation

@garethx

@garethx garethx commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Adds live test suites that exercise the built nodes against a real Hookdeck
project, so the README's claims are backed by measurements rather than by the
unit suite's stubs.

The unit suite proves what the node sends. These prove what Hookdeck does
with it — which is where every finding below came from.

test/integration.test.mjs moves in here too, as
test/live/provisioning.test.mjs. The repo had two live-test conventions
covering different ground, each with its own API client, node contexts and
cleanup; that split is now legible rather than accidental. CI runs the whole
test/live/ directory in place of test:integration.

Suites

File Covers Needs
api.test.mjs every provisioning option, all three generic verification schemes, and all 29 action-node operations API key
verification.test.mjs Stripe and GitHub signature schemes, signed and forged, plus the node's own 401/400 paths API key
delivery.test.mjs real events through hookdeck listen, output shape, retry metadata, dedup, pause/resume API key + hookdeck CLI
provisioning.test.mjs source adoption vs rewrite, CLI-destination fallback, Source Get or Create API key
stripe.test.mjs a genuine Stripe webhook, endpoint created and torn down via the Stripe CLI + stripe login

All skip with a stated reason when prerequisites are missing, so forks and the
default npm test are unaffected. The delivery suite now checks for the Hookdeck
CLI on PATH rather than spawning it blind — CI installs neither CLI, so it would
otherwise fail with ENOENT and read like a broken node.

Moving the integration suite also brought it under the delete guard. Its
resources were previously removed by an exact-name match, which would have
deleted a same-named production source; they now carry the run id like
everything else.

What running them turned up

  • A platform-typed source reveals nothing about whether it verifies. A
    STRIPE source with a secret and one without return byte-identical config.
    Generic WEBHOOK sources do expose config.auth_type. So the README's
    "looks identical to an unconfigured one" is right for platform sources and
    wrong for generic ones.
  • Verification failure is not uniformly a 200. STRIPE answers 200 with
    verified: false and rejection_cause: VERIFICATION_FAILED; GITHUB refuses
    outright. Never infer verification from the status code, in either direction.
  • attemptTrigger is INITIAL/AUTOMATIC/MANUAL/BULK_RETRY/UNPAUSE
    never RETRY. Anything branching on 'RETRY' never fires.
  • Real Stripe signs with t,v1,v0 — three schemes, where the documented
    example shows two.
  • Delivery groups are a plan entitlement; without it the entire connection
    upsert is rejected. That case skips with the reason rather than passing quietly.
  • Source adoption is load-bearing for secrets. Provisioning adopts an
    existing source untouched, so applying a secret to one that already exists
    needs updateExistingSource. Without it the secret is accepted, silently
    discarded, and deliveries arrive unverified — caught by the Stripe suite
    against a real endpoint.
  • Rate limiting lives at destination.config.rate_limit; deleted resources
    answer 410, not 404; there is no /events/count.

Safety

These run against a project that also holds production sources, so destroy()
refuses to delete anything not carrying the run's own id, across all three naming
forms — including the one the node derives from the workflow id, which was
missed at first and left resources behind. Cleanup failures are reported loudly
rather than swallowed.

hookdeck listen is spawned detached and killed by process group: the hookdeck
on PATH is an npm wrapper around the real binary, and signalling only the
wrapper leaves that binary holding its connection open and the parent process
alive. Fixing that took one suite from a 600s timeout to 64s.

Verification

Rebased on current main. npm run test:live45 pass, 1 skip (delivery
groups, not enabled on the test org), exit 0, no leaked resources, no orphaned
processes. npm test 106 pass. Skip paths checked directly: no API key skips
every suite, and a missing CLI skips delivery and Stripe.

Unrelated and left alone: both nodes now trip
icon-prefer-themed-variants — the icon is a single file:hookdeck.svg where it
was previously the { light, dark } form, and only hookdeck.svg remains on
disk. That is a verification-programme rule, so worth a separate fix.

The unit suite drives the nodes against a stub, so it can prove what a node
sends but never what Hookdeck does with it. These four suites close that gap:
they provision real resources, put real events through the public ingest URL,
and feed the resulting requests into the node's own webhook handler.

  api           every provisioning option and all 29 action operations
  verification  Stripe and GitHub signatures, signed and forged
  delivery      real events over `hookdeck listen`, retries, dedup, pause
  stripe        a genuine Stripe webhook, endpoint created and torn down

These sit alongside test/integration.test.mjs rather than replacing it: that
suite covers source adoption and the CLI destination, which these do not.

Each skips with a stated reason when its prerequisites are absent, so the
default test run needs no credentials. They run serially, because count
assertions are project-wide and cannot be isolated by naming.

Running them corrected several assumptions. A platform source reveals nothing
about whether it verifies, while a generic one exposes config.auth_type. A
forged payload is answered 200 by Stripe and refused outright by GitHub, so the
status code is not the verdict in either direction. An automatic retry reports
AUTOMATIC, never RETRY. Real Stripe signs with three schemes, not the documented
two. Delivery groups are a plan entitlement, so that case skips rather than
passing quietly.

The live suites run against a project that also carries production sources, so
cleanup deletes only names carrying the run's own id — including the one the
node derives from the workflow id — and reports anything it could not remove
instead of swallowing the failure. `hookdeck listen` is spawned detached and
killed by process group: the `hookdeck` on PATH is an npm wrapper around the
real binary, and signalling only the wrapper leaves that binary holding the
connection open.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@garethx
garethx merged commit 6349138 into main Aug 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant