core/txpool, eth, internal/ethapi: improve transaction propagation reliability by 0xrukimedo · Pull Request #2510 · 0xPolygon/bor

0xrukimedo · 2026-10-20T15:52:39Z

Summary

Recover transaction propagation after full gossip queues or initial sync, and recover block fetches after silent peers without timeout-based jailing.

Select bounded rebroadcast batches fairly by oldest attempt/admission time. Track attempts separately from queue acceptance, respect retry intervals and maximum age, and clean up tracking when transactions leave the pool.
Announce pending transactions to connected peers after initial sync, retrying queue-rejected hashes with bounded batches and per-peer cursors. Recheck pool membership and private/conditional filters before each retry.
Reserve privacy before local admission and commit it only on success. Private submissions bypass the public retry journal. Relay-enabled nodes still forward temporary local-pool rejections to producers; permanent rejection and producer-only rejection remain errors.
Bound header/body fetches to 16 per peer and 256 overall, retain alternative announcers, and release only matching request ownership. Discarding already-known or too-old queued blocks now clears every fetch stage and retained announcement credit.
Keep fresh block gather deadlines independent of deferred retries. New announcements can bring the timer forward without resetting existing backoff or scanning every retained hash.
Apply a bounded, non-escalating 10-second cooldown after block-fetch timeouts. Downloader timeouts retain their 10-second backoff without adding jail strikes. Non-timeout failure policy remains unchanged.

Latest review fixes

Actual peer-queue offers drive rebroadcast attempt accounting. Suppressed batches retain their retry opportunity; rejected queue offers still count for fairness.
Private transactions return already known for exact pool-resident duplicates without entering the public journal. Public re-admission while an absent transaction remains private returns a conflict.
Private markers expire after 30 minutes, with a one-minute sweep interval, allowing stalled pool-resident transactions to become publicly eligible. Pool retention is unchanged. Active admissions and refreshed markers remain protected. See private transaction lifetime.
Bodies waiting on cooling suppliers recover through healthy retained alternatives. Fresh body deadlines remain independent of deferred retry backoff.

Latest fixes: bdd681bab. Full short race suites for eth and eth/relay passed, final focused race regressions passed, and scoped golangci-lint v2.11.4 reports zero new issues. Diffguard against 27a2047b9 passed with all 18/18 mutations killed (tier-1 13/13). Fresh CI must validate this revision.

Validation

For the latest follow-up:

Private API/backend race regressions pass, covering temporary rejection, relay success/failure, producer-only behavior, permanent errors, and marker cleanup.
Full fetcher short race suite and focused scheduling/cleanup race tests pass.
Scoped lint reports zero issues; formatting and diff checks pass.
Diffguard with mutations enabled against c15ccd87b: tier-1 logic 11/11 (100%), total 25/27 (92.6%), exit 0. Two statement/branch-deletion survivors remain; the installed tool cannot apply statement deletions inside select clauses. No thresholds were changed.

The preceding full-branch Diffguard run against origin/develop also passed: tier-1 159/169 (94.1%), total 364/426 (85.4%). Earlier revisions passed race regressions for downloader, pool, relay, sync completion, and queue recovery. Existing churn warnings remain in large functions.

Current-head CI must validate repository-wide tests, integration/e2e, and security checks. Local regressions do not establish live-network performance or guarantee transaction inclusion.

Rollout

No consensus rule, hardfork activation, wire format, database migration, or operator configuration changes. Existing rebroadcast settings continue to apply. Timeout-only peers become eligible again after the 10-second cooldown. Private submissions now have a 30-minute window before public fallback; callers must not assume indefinite confidentiality. Exact pool-resident duplicates preserve the already known response used by preconfirmation relays.

…liability

codecov · 2026-10-20T16:22:42Z

Codecov Report

❌ Patch coverage is 97.69585% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.95%. Comparing base (1cdf43c) to head (eaeae9c).
⚠️ Report is 4 commits behind head on develop.

Files with missing lines	Patch %	Lines
eth/fetcher/block_fetcher_schedule.go	94.40%	6 Missing and 3 partials ⚠️
eth/filters/IBackend.go	0.00%	8 Missing ⚠️
internal/ethapi/api_private.go	83.33%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2510      +/-   ##
===========================================
+ Coverage    57.76%   57.95%   +0.19%     
===========================================
  Files          966      982      +16     
  Lines       176609   177156     +547     
===========================================
+ Hits        102025   102678     +653     
+ Misses       68921    68836      -85     
+ Partials      5663     5642      -21

Files with missing lines	Coverage Δ
core/txpool/legacypool/legacypool.go	`80.04% <100.00%> (+1.55%)`	⬆️
core/txpool/legacypool/legacypool_stranded.go	`100.00% <100.00%> (ø)`
core/txpool/legacypool/rebroadcast.go	`100.00% <100.00%> (ø)`
core/txpool/legacypool/rebroadcast_attempt.go	`100.00% <100.00%> (ø)`
core/txpool/legacypool/rebroadcast_selection.go	`100.00% <100.00%> (ø)`
core/txpool/rebroadcast_attempt.go	`100.00% <100.00%> (ø)`
eth/api_backend.go	`35.81% <100.00%> (+2.42%)`	⬆️
eth/api_backend_private.go	`100.00% <100.00%> (ø)`
eth/downloader/bor_fetchers_concurrent.go	`78.93% <100.00%> (ø)`
eth/downloader/peer.go	`92.56% <100.00%> (ø)`
... and 23 more

... and 20 files with indirect coverage changes

Files with missing lines	Coverage Δ
core/txpool/legacypool/legacypool.go	`80.04% <100.00%> (+1.55%)`	⬆️
core/txpool/legacypool/legacypool_stranded.go	`100.00% <100.00%> (ø)`
core/txpool/legacypool/rebroadcast.go	`100.00% <100.00%> (ø)`
core/txpool/legacypool/rebroadcast_attempt.go	`100.00% <100.00%> (ø)`
core/txpool/legacypool/rebroadcast_selection.go	`100.00% <100.00%> (ø)`
core/txpool/rebroadcast_attempt.go	`100.00% <100.00%> (ø)`
eth/api_backend.go	`35.81% <100.00%> (+2.42%)`	⬆️
eth/api_backend_private.go	`100.00% <100.00%> (ø)`
eth/downloader/bor_fetchers_concurrent.go	`78.93% <100.00%> (ø)`
eth/downloader/peer.go	`92.56% <100.00%> (ø)`
... and 23 more

... and 20 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

0xrukimedo · 2026-10-21T05:02:12Z

@claude review

Copilot

🔵 Needs a closer look

Cross-component concurrency and privacy changes need human review, with unresolved propagation issues and the required mutation gate still failing.

2 open issues

What changed in this PR

Improves Bor’s transaction propagation and block-fetch recovery under congestion, initial sync, and peer timeouts.

Changes:

Adds fair rebroadcast scheduling and pending announcements after initial sync.
Preserves private transaction state through admission and pool residence.
Bounds block-fetch requests and separates timeout backoff from failure escalation.

File	Description
internal/ethapi/transaction_args_test.go	Updates backend mock.
internal/ethapi/backend.go	Adds privacy reservation interface.
internal/ethapi/api.go	Reserves privacy before admission.
internal/ethapi/api_test.go	Updates private-submission tests.
internal/ethapi/api_private_test.go	Tests concurrent private submissions.
eth/sync.go	Shares pending-hash filtering.
eth/sync_completion_test.go	Tests post-sync announcements.
eth/relay/relay.go	Exposes privacy reservations.
eth/relay/private_tx_store.go	Retains privacy during pool residence.
eth/relay/private_tx_store_test.go	Tests retention and sweeping.
eth/relay/private_tx_store_submission.go	Implements shared admission reservations.
eth/relay/private_tx_store_submission_test.go	Tests reservation lifecycle.
eth/rebroadcast_delivery_test.go	Adjusts transaction-age fixture.
eth/handler.go	Announces pending transactions after sync.
eth/filters/IBackend.go	Updates generated backend mock.
eth/fetcher/block_fetcher.go	Integrates bounded fetch scheduling.
eth/fetcher/block_fetcher_timeout_recovery_test.go	Tests timeout recovery.
eth/fetcher/block_fetcher_schedule.go	Implements deferred request scheduling.
eth/fetcher/block_fetcher_schedule_test.go	Tests scheduling capacity and priority.
eth/fetcher/block_fetcher_schedule_edges_test.go	Tests scheduling edge cases.
eth/fetcher/block_fetcher_request.go	Handles request execution and completion.
eth/fetcher/block_fetcher_release.go	Releases matching request ownership.
eth/fetcher/block_fetcher_release_test.go	Tests ownership-safe cleanup.
eth/fetcher/block_fetcher_pending.go	Bounds pending announcement origins.
eth/fetcher/block_fetcher_pending_test.go	Tests announcement accounting.
eth/fetcher/block_fetcher_bounds_test.go	Tests blocked-dispatch limits.
eth/fetcher/block_fetcher_body_request_test.go	Tests body-request cleanup.
eth/fetcher/block_fetcher_admission.go	Adds request capacity limits.
eth/fetcher/block_fetcher_admission_test.go	Tests admission limits and retries.
eth/downloader/skeleton.go	Avoids timeout strike escalation.
eth/downloader/peer.go	Makes backoff reason-aware.
eth/downloader/peer_timeout_test.go	Tests repeated timeout handling.
eth/downloader/peer_response.go	Passes failure reason to backoff.
eth/downloader/peer_response_test.go	Preserves non-timeout escalation tests.
eth/downloader/bor_fetchers_concurrent.go	Classifies stale requests as timeouts.
eth/downloader/bor_fetchers_concurrent_test.go	Tests departed-peer timeout handling.
eth/downloader/bor_fetch_helpers_test.go	Updates stale-request expectations.
eth/api_backend.go	Excludes private transactions from retry journaling.
eth/api_backend_private_test.go	Tests private admission and journaling.
eth/api_backend_private_lifecycle_test.go	Tests privacy lifecycle and duplicates.
core/txpool/legacypool/rebroadcast.go	Tracks attempts separately from acceptance.
core/txpool/legacypool/rebroadcast_test.go	Updates attempt-expiry expectations.
core/txpool/legacypool/rebroadcast_selection.go	Selects bounded, oldest-activity batches.
core/txpool/legacypool/rebroadcast_scheduling_test.go	Tests rotation and queue recovery.
core/txpool/legacypool/rebroadcast_fairness_test.go	Tests fairness amid new arrivals.
core/txpool/legacypool/rebroadcast_cleanup_test.go	Tests attempt-history cleanup.
core/txpool/legacypool/rebroadcast_boundaries_test.go	Tests eligibility and ordering boundaries.
core/txpool/legacypool/legacypool.go	Integrates attempt tracking and cleanup.
core/txpool/legacypool/legacypool_stranded.go	Clears history for stranded transactions.

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Cross-component concurrency, request ownership, and transaction privacy changes warrant final human review and confirmation of green full-suite CI.

No open issues 🎉

2 resolved since last review

_{Review effort: Balanced}

0xrukimedo · 2026-10-22T09:49:42Z

@claude review

claude

Findings marked 🟡 are optional suggestions and need no follow-up push.

Still open from earlier reviews (1):

🔴 eth/relay/private_tx_store.go:197 — Private transactions that block producers reject can now stay hidden in the relay's pool indefinitely. They are never g…

If you have decided not to act on one of these findings, resolve its thread (a reply alone leaves it open) and the next review stops counting it. To review this commit again now, use Re-run on its "Claude Code Review" check.

Additional findings (outside the current diff — GitHub can't attach inline comments there):

🔴 eth/fetcher/block_fetcher.go — Nodes that catch up through the downloader can permanently keep other peers' block announcements and their quota credits. Eventually announcements from new peers are refused. When a fetcher-queued block is popped as already known or too old, line 577 calls only forgetBlock. The stale f.completing[hash] keeps fetchInProgress true, so the hash's alternatives in f.announced are deferred forever. Fix: clear every stage for a discarded queued block (call forgetHash along with forgetBlock, as the done case does), so that alternatives and their f.announces credits are released.

Why this was flagged

The fetcher fetches block B (within maxQueueDist) and its body. B is enqueued with f.completing[B] set and waits because number > height+1 (block_fetcher.go:562). While B sits in completing/queued, other peers announce B. hasAnnouncer (block_fetcher_alternates.go:51) only rejects the same origin, so each new announcer is appended to f.announced[B] and f.announces[origin] is incremented (block_fetcher.go:634-635). The downloader then imports B. The import loop sees getBlock(B) != nil or number+maxUncleDist < height and calls only forgetBlock (block_fetcher.go:575-578). From then on, prepareHeaderFetch returns (nil, true) at fetchInProgress (block_fetcher_schedule.go:64-66) before readyToFetch can drop B, so the alternatives and their credits are never released. On the base branch the stale completing entry made the notify handler ignore further announcements of B, so no announcement or credit was retained.

Verification: When the queued B is popped as already known or too old, block_fetcher.go:575-577 calls only f.forgetBlock(hash). That removes queued/queues and leaves f.completing[B] behind permanently. Each other peer's announce for B is therefore appended, and f.announces[origin] = count is charged (block_fetcher.go:634-635).

claude

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

Copilot

🟡 Changes recommended

Producer-rejected private transactions still lack a bounded automatic recovery policy that preserves the endpoint’s privacy contract.

2 open issues

_{Review effort: Balanced}

0xrukimedo · 2026-10-22T10:49:24Z

Fixed the out-of-diff queued-block cleanup finding in 2b88eb6. When a queued block is already known or too old, the import loop now calls both forgetHash and forgetBlock. This clears stale completion state, retained alternatives, and their announcement credits as well as queue ownership. TestBlockFetchDiscardReleasesAlternatives drives the real fetcher loop for both cases and verifies stage/quota cleanup.

The private-retention recovery finding remains open. The two reviews request conflicting policies: automatic public fallback versus preserving the private payload. I have asked the PR author to choose the policy; this commit does not silently publish private transactions.

Copilot

🟡 Changes recommended

The acknowledged private-transaction recovery policy after producer retry exhaustion still needs an explicit decision, implementation, and tests before merging.

1 open issue

Pending transactions lack bounded recovery after relay exhaustion

1 resolved since last review

Shared backoff delays newly announced block fetches

_{Review effort: Balanced}

0xrukimedo · 2026-10-22T11:20:46Z

@claude review

claude

Findings marked 🟡 are optional suggestions and need no follow-up push.

Still open from earlier reviews (1):

🔴 eth/relay/private_tx_store.go:197 — Private transactions that block producers reject can now stay hidden in the relay's pool indefinitely. They are never g…

If you have decided not to act on one of these findings, resolve its thread (a reply alone leaves it open) and the next review stops counting it. To review this commit again now, use Re-run on its "Claude Code Review" check.

Copilot

🟡 Changes recommended

The acknowledged private-transaction recovery policy after producer retries exhaust still needs agreement, implementation, and regression coverage before merging.

No open issues 🎉

1 resolved since last review

Pending transactions lack bounded recovery after relay exhaustion

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

The acknowledged bounded recovery policy for private transactions after producer retry exhaustion still needs implementation and regression coverage.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Bounded private-transaction recovery after producer retry exhaustion remains an acknowledged merge blocker requiring an agreed policy, implementation, and regression tests.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Bounded private-transaction recovery after producer retries exhaust remains unresolved, with implementation and regression tests still outstanding.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The concurrent recovery and privacy changes require human review and current-head CI validation, with partial-body-response recovery still unresolved.

1 open issue

Requeue omitted block bodies after partial responses · New

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Cross-component retry scheduling and confidentiality changes require human sign-off and current-head CI validation.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Clarify privacy marker expiry for pool-resident transactions

eth/relay/CLAUDE.md:84

This guidance says privacy markers remain while the pool retains a transaction, but privateTxExpired now expires them after 30 minutes even for pool-resident transactions. The original payload then becomes eligible for public propagation without a replacement or public resubmission. Document the bounded privacy window and the in-flight admission exception here, consistent with docs/private-transactions.md, so the recovery guidance does not imply indefinite confidentiality.

1 resolved since last review

Requeue omitted block bodies after partial responses

_{Review effort: Balanced}

0xrukimedo · 2026-10-23T13:29:25Z

@claude review

claude

Findings marked 🟡 are optional suggestions and need no follow-up push.

Copilot

🔵 Needs a closer look

Privacy-sensitive public fallback and concurrent fetch scheduling require human sign-off and current-head integration validation.

No open issues 🎉

_{Review effort: Balanced}

core/txpool, eth, internal/ethapi: improve transaction propagation re…

457d7d4

…liability

0xrukimedo requested a balanced review from Copilot October 21, 2026 05:01

Copilot started reviewing on behalf of 0xrukimedo October 21, 2026 05:02

View session

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread eth/fetcher/block_fetcher_schedule.go Outdated

Comment thread eth/handler.go Outdated

claude Bot reviewed Oct 21, 2026

View reviewed changes

Comment thread eth/fetcher/block_fetcher_request.go

Comment thread eth/relay/private_tx_store.go

Comment thread eth/fetcher/block_fetcher_schedule.go Outdated

0xrukimedo added 2 commits October 21, 2026 14:20

eth, eth/fetcher: preserve propagation retries under peer backpressure

fefec2f

eth, eth/relay: verify private transaction replacement recovery

928d3b0

0xrukimedo requested a balanced review from Copilot October 22, 2026 09:34

Copilot started reviewing on behalf of 0xrukimedo October 22, 2026 09:35

View session

Copilot AI reviewed Oct 22, 2026

View reviewed changes

eth/fetcher: back off peers after block request timeouts

c15ccd8

claude Bot reviewed Oct 22, 2026

View reviewed changes

Comment thread eth/api_backend.go Outdated

Comment thread eth/fetcher/block_fetcher.go Outdated

0xrukimedo marked this pull request as ready for review October 22, 2026 10:21

Copilot AI balanced review requested due to automatic review settings October 22, 2026 10:21

claude Bot reviewed Oct 22, 2026

View reviewed changes

0xrukimedo requested review from lucca30 and marcello33 October 22, 2026 10:21

Copilot started reviewing on behalf of 0xrukimedo October 22, 2026 10:21

View session

Copilot AI reviewed Oct 22, 2026

View reviewed changes

Comment thread eth/relay/private_tx_store.go Outdated

Comment thread eth/fetcher/block_fetcher_alternates.go

eth, ethapi: recover private relay submissions and fetch scheduling

2b88eb6

Copilot AI balanced review requested due to automatic review settings October 22, 2026 10:49

Copilot started reviewing on behalf of 0xrukimedo October 22, 2026 10:49

View session

Copilot AI reviewed Oct 22, 2026

View reviewed changes

claude Bot reviewed Oct 22, 2026

View reviewed changes

Comment thread eth/fetcher/block_fetcher_admission.go

Comment thread core/txpool/legacypool/rebroadcast.go Outdated

lucca30 reviewed Oct 22, 2026

View reviewed changes

Comment thread core/txpool/legacypool/rebroadcast.go Outdated

Copilot AI balanced review requested due to automatic review settings October 22, 2026 12:51

Copilot started reviewing on behalf of 0xrukimedo October 22, 2026 12:52

View session

Copilot AI reviewed Oct 22, 2026

View reviewed changes

eth: model queue offers in sync recovery regression

52b0935

Copilot AI balanced review requested due to automatic review settings October 22, 2026 14:58

Copilot started reviewing on behalf of 0xrukimedo October 22, 2026 14:59

View session

Copilot AI reviewed Oct 22, 2026

View reviewed changes

ci: use published Bor release for pipeline baseline

dbb69db

Copilot AI balanced review requested due to automatic review settings October 22, 2026 15:31

Copilot started reviewing on behalf of 0xrukimedo October 22, 2026 15:31

View session

Copilot AI reviewed Oct 22, 2026

View reviewed changes

lucca30 reviewed Oct 22, 2026

View reviewed changes

Comment thread eth/api_backend.go Outdated

ci: remove YAML dependency from pipeline image selection

27a2047

Copilot AI balanced review requested due to automatic review settings October 23, 2026 07:58

Copilot started reviewing on behalf of 0xrukimedo October 23, 2026 07:59

View session

Copilot AI reviewed Oct 23, 2026

View reviewed changes

eth, relay: bound private lifetime and preserve duplicate responses

bdd681b

Copilot AI balanced review requested due to automatic review settings October 23, 2026 10:14

Copilot started reviewing on behalf of 0xrukimedo October 23, 2026 10:15

View session

Copilot AI reviewed Oct 23, 2026

View reviewed changes

Comment thread eth/fetcher/block_fetcher_release.go

eth/fetcher: retry omitted bodies after partial responses

499d674

Copilot AI balanced review requested due to automatic review settings October 23, 2026 12:40

Copilot started reviewing on behalf of 0xrukimedo October 23, 2026 12:40

View session

Copilot AI reviewed Oct 23, 2026

View reviewed changes

claude Bot reviewed Oct 23, 2026

View reviewed changes

Comment thread core/txpool/legacypool/rebroadcast_attempt.go

Comment thread eth/relay/CLAUDE.md Outdated

core/txpool, eth/relay: retain retries after refused queue offers

eaeae9c

Copilot AI balanced review requested due to automatic review settings October 23, 2026 14:13

Copilot started reviewing on behalf of 0xrukimedo October 23, 2026 14:13

View session

Copilot AI reviewed Oct 23, 2026

View reviewed changes

Conversation

0xrukimedo commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Latest review fixes

Validation

Rollout

Uh oh!

codecov Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

0xrukimedo commented Oct 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

0xrukimedo commented Oct 22, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

0xrukimedo commented Oct 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

0xrukimedo commented Oct 22, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

0xrukimedo commented Oct 20, 2026 •

edited

Loading

codecov Bot commented Oct 20, 2026 •

edited

Loading