Skip to content

Retry transient block upload failures in place - #2

Merged
ClaudiuSchuster merged 1 commit into
fix/drain-upload-block-workersfrom
fix/retry-transient-block-uploads
Aug 28, 2026
Merged

Retry transient block upload failures in place#2
ClaudiuSchuster merged 1 commit into
fix/drain-upload-block-workersfrom
fix/retry-transient-block-uploads

Conversation

@ClaudiuSchuster

Copy link
Copy Markdown
Member

Summary

  • retry only transiently failed encrypted blocks from the current upload batch
  • request fresh signed links and create a fresh reader for every retry
  • retain successful blocks instead of returning control to a whole-file retry
  • keep the retry budget bounded and context-aware
  • preserve the existing worker-drain and semaphore guarantees

Motivation

A sanitized live PDrive reproduction showed one 30.4 GiB VFS upload sending
more than 65 GB of aggregate payload while its current attempt remained near
49%. Six separated Proton Storage 502 cycles occurred in the same rclone
process. The service-level recovery guard correctly avoided a restart whenever
payload traffic resumed, but each bridge error still caused rclone to retry the
complete file stream.

The bridge already holds at most eight encrypted 4 MiB blocks for the current
batch. Retrying only the failed block indexes preserves completed work and
avoids turning a transient storage response into a multi-hour retransmission.

Tracked in oss-singularity/proton-drive-linux#42.

Safety boundaries

  • five total attempts with 1, 2, 4 and 8 second context-aware backoffs
  • retries only API 5xx responses and the bridge's concrete network error
  • client-side 4xx responses and context cancellation remain terminal
  • fresh signed upload links are requested only for failed block indexes
  • every started worker is drained before retry or return
  • retry logs contain counts and timing only, never signed URLs, tokens, link
    IDs or file paths
  • no draft, revision, cache or remote-file deletion behavior changes

Stacked review

This branch is intentionally based on fix/drain-upload-block-workers, which
is proposed upstream in rclone#8. The retry decision relies on
that change's guarantee that all concurrent worker results are drained before
the batch returns. This pull request contains one additional focused commit and
can be rebased onto upstream master after the dependency merges.

Verification

  • go test -count=1 ./...
  • go test -race -count=1 ./...
  • golangci-lint run --timeout=180s with v2.9.0
  • rclone backend/protondrive tests compiled and passed with a temporary
    module replacement pointing to this checkout
  • live PDrive state confirmed that measurable payload progress keeps the
    existing service PID and suppresses guarded restarts

Retry only failed encrypted blocks with fresh upload links and bounded context-aware backoff. Preserve successful blocks and return terminal or exhausted errors without replaying the complete file stream.

Refs oss-singularity/proton-drive-linux#42
@ClaudiuSchuster ClaudiuSchuster added the bug Something isn't working label Aug 28, 2026
@ClaudiuSchuster
ClaudiuSchuster merged commit c059d75 into fix/drain-upload-block-workers Aug 28, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant