Acceptance tests leak sources and destinations: ~36k of each in the CI Gateway project

[leggetter]

What

The Gateway project used by acceptance slices has accumulated roughly 36,000 sources and 36,500 destinations. It grew by about 250 of each in a single hour of CI runs while this was being investigated, so it is still growing at the rate we run the suite.

Counts, taken 2026-08-20:

gateway source count      → 36144
gateway destination count → 36489

Why it happens

The tests do clean up — just not all of it. Counting helper calls across test/acceptance/*_test.go:

Helper	Call sites
deleteConnection	157
deleteSource	30
deleteDestination	24
cleanupConnections	0 (defined, never used)

Deleting a connection does not delete the source and destination it was built from. Most tests create a connection with --source-name / --destination-name, which creates all three, then register t.Cleanup(deleteConnection) and stop there. The connection goes; the other two are orphaned. That ratio — 157 against 30 and 24 — is the leak.

Two smaller contributors:

• hookdeck listen auto-creates a source when the named one does not exist, and a CLI destination alongside it. Nothing in the listen tests tracks either, so every run leaves a test-* source and a cli-test-* destination behind. This is the harder half: the test never learns the ids.
• cleanupConnections exists in helpers.go:1216 and is called from nowhere.

Why it matters

• It is unbounded. Nothing prunes it, and every run makes it worse.
• It makes lookups slower and noisier for anything that lists rather than filters by name. hookdeck listen '*' calls ListSources with no filter, so a developer pointing listen '*' at this project pulls the whole set.
• It hides real problems. A test failing because it hit a resource limit would look like an unrelated failure, and with this much noise nobody would suspect the project.
• It made a genuine CI investigation harder: 36k sources was a plausible cause of a listen failure and took time to rule out.

What to do

• One-off cleanup of the existing ~36k sources and ~36.5k destinations in the CI Gateway project. Only touch test-*, cli-test-* and probe-* names, and confirm nothing else in that project depends on them.
• Delete sources and destinations, not just connections. Where a test creates a connection with --source-name/--destination-name, capture the ids from the response (conn.Source.ID, conn.Destination.ID) and register cleanup for all three. deleteSource and deleteDestination already exist.
• Handle the listen auto-created resources. The test knows the source name it passed, so it can look the id up afterwards and delete it, along with the cli-<name> destination the tunnel creates.
• Remove or use cleanupConnections.
• Consider a guard — a periodic job, or a check in the suite that fails loudly above some threshold, so this cannot silently return.

Worth deciding

Whether the CI Gateway project should be recreated rather than pruned. Deleting 72,000 resources through the API is itself a long job, and a fresh project with the same secret would be faster — at the cost of losing whatever history is in there. The Outpost CI project does not have this problem (its tests clean up tenants, and deleting a tenant removes its destinations), so this is Gateway-specific.

Related

• v3.0.0: Outpost support, Gateway MCP write mode, and the gateway_ tool rename #348 — the v3.0.0 PR, where this surfaced while investigating a listen CI failure
• Run acceptance tests only when CLI or related code changes #240 — run acceptance tests only when CLI code changes, which would slow the growth but not stop it

[leggetter]

Regrowth measured: cleanup alone achieves nothing

Two of the three Gateway test projects were cleared yesterday. Counts today, after roughly a day of test runs:

Project	After cleanup	Today
tm_023eAwT83r5o (slice 1)	1 source / 0 destinations	124 / 123
tm_eOos3Jg5sqLV (slice 2)	0 / 0	60 / 60
tm_htnSCvufI7RK (slice 0)	partial (36,376 → 27,043 sources)	27,500 / 37,203

Slice 0's destinations have grown from 36,735 to 37,203 — the acceptance suite ran several times today during pre-release testing, and each run left more behind.

So the leak rate is roughly 60–125 resources per project per day of active testing, and slice 0 is still accumulating faster than it was cleaned.

This settles the ordering: fix the tests first, clear down second. A cleanup done before the leak is closed is undone within days, and clearing slice 0 is an ~11 hour job (see #366) that nobody wants to repeat.

It also confirms the diagnosis in the issue body. The regrowth is not from tests that create nothing — it is connections being deleted while the sources and destinations they were built from are left behind, plus listen auto-creating a source and a cli-* destination that no test tracks.