Skip to content

fix: preserve legacy merge progress when a worker cannot connect - #317

Merged
beinan merged 3 commits into
mainfrom
codex/legacy-worker-connect-recovery
Oct 5, 2026
Merged

beinan merged 3 commits into
mainfrom
codex/legacy-worker-connect-recovery

Conversation

@beinan

@beinan beinan commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

When an ordinary worker drains during deployment, its connection failure currently stops legacy WAL merge fan-out before later healthy workers and drops reclaimed-generation metrics for already completed workers.

Continue the serial fan-out only after a connection-establishment failure, retain completed progress, and leave the canonical task failed/retryable for unavailable shards. Stop on HTTP errors, lost responses and post-send timeouts because an unfenced legacy writer may still be running. The internal worker client disables redirects so a failed redirected POST cannot be mistaken for a request that was never admitted; only connection establishment has a two-second deadline.

Validation:

  • Four real TCP regressions pass: healthy progress on both sides of a refused connection, HTTP failure stopping before the next writer, redirect to an unreachable endpoint even with a valid merge payload, and timeout after server admission.
  • Full master unit suite: 70 passed, 84 existing isolated-etcd tests ignored, zero failed.
  • Separately ran all eight merge-execution recovery tests against a fresh isolated local etcd: all passed, including orphan fencing, surviving-writer protection, partial success/retry exhaustion and no legacy downgrade.
  • Formatting and diff checks pass. Remaining CI is running; merge only after every workflow passes.

@beinan
beinan marked this pull request as ready for review October 5, 2026 22:40
@beinan
beinan merged commit 9803aa7 into main Oct 5, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant