Repository navigation
Wait out a shadow restart in oracle resolve - #235
Open
iskakaushik wants to merge 1 commit into
Open
iskakaushik wants to merge 1 commit into
iskakaushik wants to merge 1 commit into
Conversation
serprex
force-pushed
the
oracle-wait-shadow-restart
branch
from
October 23, 2026 02:59
20d8b0e to
93efac7
Compare
Shadow restart and crash recovery can outlast oracle retry budget, causing resolver to mark pipeline fatal while shadow is recovering. Retry transport failures without an attempt or elapsed-time limit, using existing capped backoff. Stop retries and interrupt backoff when session shuts down or failed pipeline leg drains, so shadow outage cannot hold up teardown. Continue draining healthy batches. Keep worker-reported errors fatal and leave retry_max_attempts bounding ClickHouse inserts only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
serprex
approved these changes
Oct 23, 2026
serprex
force-pushed
the
oracle-wait-shadow-restart
branch
from
October 23, 2026 03:33
93efac7 to
f803107
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continue work from #234.
When shadow PostgreSQL crashes, restarting it and recovering can take longer than oracle connection retries allow. In deployment described in #233, recovery took 2.5 minutes. Walshadow gave up while shadow was still recovering and exited.
Keep retrying oracle connection failures while session remains active, with increasing delays capped by existing backoff settings. Replace proposed 15-minute limit with retries that stop when session shuts down or a failed pipeline leg drains. Interrupt retry delays during shutdown so an unavailable shadow cannot hold up teardown. Healthy batches can still finish.
Errors returned by a running worker still fail immediately.
retry_max_attemptscontinues to limit ClickHouse inserts but no longer limits oracle retries.Regression tests cover retries continuing beyond configured attempt count and several simulated hours, shutdown interrupting a long retry delay, and worker errors failing without retries.