Skip to content

chore(smoke): tolerate indexer catching up + relax threshold to 500 blocks - #296

Merged
TuCopFi merged 1 commit into
Developmentfrom
chore/smoke-tolerate-indexer-catchup
Jul 28, 2026
Merged

chore(smoke): tolerate indexer catching up + relax threshold to 500 blocks#296
TuCopFi merged 1 commit into
Developmentfrom
chore/smoke-tolerate-indexer-catchup

Conversation

@TuCopFi

@TuCopFi TuCopFi commented Jul 28, 2026

Copy link
Copy Markdown
Member

Summary

  • Relaxes the pre-release backend smoke's indexer-health check from a hard 100-block ceiling to 500 blocks, and, if that's exceeded, gives the indexer 10 seconds to prove it's actively catching up (delta >= 50 blocks) before failing.
  • Post-mortem from the 1.118.8 pre-release: indexer briefly hit 3876 blocks lag after an RPC hiccup, then caught up seconds later. The old threshold flagged this as a hard failure and blocked the release for no real backend issue.

Behavior

  1. First sample: if lagBlocks <= 500, pass immediately.
  2. Otherwise wait 10s and re-sample.
  3. Pass if the second sample is either under 500, or dropped by at least 50 blocks (proof the indexer is draining backlog).
  4. Fail only when the second sample is still above 500 AND is not catching up.

Test plan

  • bash -n scripts/pre-release-backend-smoke.sh (syntax check)
  • Full run against https://tucop-backend-production.up.railway.app (all 6 checks pass, indexer healthy at time of test)
  • Next release cycle validates the retry branch during a real catch-up window

…locks

The previous 100-block threshold flakes when the indexer briefly falls behind
after an RPC hiccup, even though it's actively catching up. Post-mortem from
the 1.118.8 pre-release run: indexer was at 3876 blocks lag, caught up
seconds later without any real backend issue.

Relaxes the check in two ways:

1. Baseline threshold raised 100 -> 500 blocks (still tight enough to catch
   a stuck indexer).
2. If lag is above 500, re-sample after 10s. Pass if either (a) it dropped
   under 500, or (b) it moved at least 50 blocks in the retry window
   (proof that the indexer is actively draining backlog).

Only fails if the second sample is still above 500 AND is not catching up.
@TuCopFi
TuCopFi merged commit d817882 into Development Jul 28, 2026
10 checks passed
@TuCopFi
TuCopFi deleted the chore/smoke-tolerate-indexer-catchup branch July 28, 2026 19:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant