Skip to content

registry: detach the token singleflight from the leader's context - #59

Merged
Coldwings merged 1 commit into
mainfrom
fix/issue-51-token-flight-context
Aug 26, 2026
Merged

registry: detach the token singleflight from the leader's context#59
Coldwings merged 1 commit into
mainfrom
fix/issue-51-token-flight-context

Conversation

@Coldwings

@Coldwings Coldwings commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Fixes #51

What

AuthTransport.fetchTokenShared ran the shared token exchange on the elected leader's request context. Cancelling that one caller failed the entire concurrent cold-authentication cohort with context.Canceled, even for followers whose contexts were still live — one short-lived first request could fail every concurrent pull of the same repository.

The exchange now runs on a bounded background flight context (1 minute), exactly like fetch.Coalescing:

  • each caller's own context bounds only how long that caller waits — the leader waits on the flight like any follower;
  • the flight completes for the remaining waiters and still warms the token cache even if no waiter remains;
  • a stalled token endpoint (accepted, then silent) cannot pin the cache key past the bound — the flight's context expires, the key is unpublished, and the next caller leads a fresh exchange.

Triage evidence

Verified authentic against the exact commit the issue names (84aac4e, current main):

// internal/registry/auth.go:379 @ 84aac4e
c.value, c.expiresAt, c.err = a.fetchToken(ctx, ch, cred)

ctx is the leader's request context; followers wait on c.done and return the shared c.err, so a leader cancellation is published to every live follower. Reproduced deterministically before the fix:

--- FAIL: TestLeaderCancellationDoesNotPoisonFollowers (0.00s)
    issue51_test.go:118: round 0: follower 0 poisoned by the leader's cancellation: context canceled

Classification: implementation defect — the deviation is from a contract the repo itself already established. fetch.Coalescing (internal/fetch/fetch.go:457-463) runs shared flights on a bounded background context precisely so "one caller's cancellation does not abort it"; the registry singleflight contradicted that stance, and docs/registry.md already cites the Coalescing semantics as the reason credentials cannot ride request contexts.

Fixes

  • fetchTokenShared: the first caller now starts the flight (in a dedicated goroutine) and then waits on c.done like every follower; each caller's own ctx bounds only its own wait.
  • New runTokenExchange: performs the exchange on context.WithTimeout(context.Background(), maxTokenExchange) (1 minute), publishes to the cache before removing the in-flight marker (the existing one-critical-section ordering is preserved), and closes done only after the result fields are written and the lock is released — waiters get a happens-before edge on the result, nothing runs under a.mu.
  • docs/registry.md: new contract bullet for the singleflight flight semantics + test-table rows for the new regression tests.

No signature changes (authorize is the sole caller of fetchTokenShared); no new dependencies; main module go.sum still empty.

Regression tests

internal/registry/issue51_test.go, against a stub token transport that stalls mid-exchange and honors request-context cancellation like a real transport (that fidelity is what makes the tests fail on the old code):

  • TestLeaderCancellationDoesNotPoisonFollowers — the issue's exact scenario: the elected leader is cancelled while the token endpoint is blocked; 4 followers with live contexts joined the same flight must still complete with the flight's token, the cohort must cost exactly one exchange, and the result must be cached. Looped 25 rounds across scheduling windows (same pattern as TestConcurrentColdRequestsShareOneExchange). Fails on the old code in round 0 (follower 0 poisoned ...: context canceled); passes after the fix.
  • TestCancelledFollowerDoesNotAbortFlight — symmetric contract pin: a departing follower must not cancel the shared exchange either; the leader still completes and exactly one exchange occurred.

Verification

All run in the fix worktree with the repo-pinned toolchain (go1.22.12):

gofmt -l internal/ cmd/          # clean
go vet ./...                     # OK
go test ./... -race -count=1     # all 13 packages ok (incl. internal/registry 1.396s)
go test ./... -cover -count=1    # all ok; internal/registry coverage 89.0%

And the named regression demonstrated failing pre-fix, passing post-fix under -race.

The shared token exchange ran on the elected leader's request context, so
cancelling that one caller failed the whole concurrent cold-auth cohort with
context.Canceled — even followers whose contexts were still live. A
short-lived first request could thus fail every concurrent pull of the same
repository.

Run the exchange on a bounded background flight context (1 minute) exactly
like fetch.Coalescing: each caller's own context now bounds only how long
that caller waits, the flight completes for the remaining waiters (and warms
the cache even if none remain), and a stalled token endpoint cannot pin the
cache key past the bound.

Fixes #51
@Coldwings

Copy link
Copy Markdown
Collaborator Author

@copilot please review this PR. Focus points:

  1. The new flight detachment in internal/registry/auth.go: fetchTokenShared now starts runTokenExchange in a goroutine and every caller (leader included) waits on c.done. Check the happens-before edge (result fields written before close(c.done), cache stored before the in-flight marker is removed) and that no waiter can observe a partially published result.
  2. The maxTokenExchange (1 minute) bound: is it coherent with the rest of the registry/auth path, and is a permanently stalled token endpoint handled acceptably?
  3. The regression tests in internal/registry/issue51_test.go: does the stalled-transport orchestration faithfully reproduce the leader-cancellation poisoning, and is the looped scheduler-yield rendezvous sound?

Please do NOT push commits to this branch; post findings as review comments only.

Copilot AI commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

@copilot please review this PR. Focus points:

  1. The new flight detachment in internal/registry/auth.go: fetchTokenShared now starts runTokenExchange in a goroutine and every caller (leader inc...

Reviewed c6b086e. Findings: none.

  • The result write happens before close(c.done), and the close synchronizes with all waiters before they read c.value/c.expiresAt/c.err; cache store and in-flight deletion are kept in one a.mu critical section before done is closed.
  • The 1-minute detached flight bound matches the existing fetch.Coalescing pattern and prevents a permanently stalled token endpoint from pinning the in-flight key; after timeout the key is removed and the next caller can lead a fresh exchange.
  • The regression transport faithfully models a token endpoint that blocks while respecting request context cancellation. The repeated Gosched rendezvous is best-effort, but combined with the exchange-count/cache assertions it exercises the old leader-cancellation poisoning path. Local go test ./internal/registry -race -count=1 and CI vet + test (Go 1.22) passed.

No commits were pushed.

@Coldwings
Coldwings merged commit bd6418c into main Aug 26, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] registry: token singleflight leader cancellation poisons live followers

2 participants