Skip to content

Make the S3 extension production-ready at million-object scale - #105

Merged
forhappy merged 25 commits into
mainfrom
codex/s3-production-readiness
Aug 17, 2026
Merged

Make the S3 extension production-ready at million-object scale#105
forhappy merged 25 commits into
mainfrom
codex/s3-production-readiness

Conversation

@forhappy

Copy link
Copy Markdown
Contributor

What changed

  • keep commit-session authority stable across lease renewal while fencing real takeover
  • add bounded streaming bulk ingestion, ordered publication, resumable checkpoints, and branch-aware traversal
  • add tiny-object packs, exact live-byte inventory, sparse-pack candidates, pack-aware GC, and snapshot-safe repacking
  • accelerate deep history, detached-commit listing, sparse diff/merge, persistent caching, fsck, and million-object traversal
  • add cross-process maintenance barriers, writer fencing, durable GC checkpoints, and provider policy attestation
  • enforce the large-object ownership boundary: Prolly stores one logical object as one final provider object and owns no upload IDs, parts, chunk manifests, or chunk GC
  • expose a whole-object handoff so an external provider transfer manager can upload/resume, after which Prolly verifies and publishes the completed object

Why

The original benchmark work exposed correctness failures under authority renewal and severe request/byte amplification at bucket scale. Follow-up large-object work also crossed an architectural boundary by putting transfer-part lifecycle into Prolly. This PR keeps versioned repository metadata in Prolly, tiny-object packing as an explicit repository optimization, and transport multipart state entirely with the provider uploader.

Validation

  • full prolly-s3-core all-target/all-feature test suite
  • full prolly-s3-client all-target/all-feature test suite
  • live RustFS 65 MiB single-PutObject stream gate
  • live RustFS 33 MiB external-upload handoff across commit-session restart
  • 10K/20K correctness and performance gates plus 100K/500K/1M metadata qualification recorded in the performance envelope

The ignored AWS qualification and scale gates still require an operator-owned real bucket and explicit SLO configuration.

Remove Prolly-facing multipart upload IDs, part geometry, ListParts reconciliation, and chunk lifecycle state. Hand externally uploaded final objects back by whole-object identity and verify them before publication. Also add exact sparse-pack candidate accounting and snapshot-safe repack publication.
Combine durable object age with scheduler-observed temperature when repacking tiny objects. Keep recent or hot payloads in smaller packs, promote only old cold payloads to larger targets, and make maintenance sessions ephemeral.
Check the repository-wide GC coordinator before preparing new publications, preserve idempotent replay, and reconcile only genuinely unknown publication outcomes. Verify an independent RustFS writer remains retryably fenced across GC owner restart and becomes usable after completion.
@forhappy
forhappy marked this pull request as ready for review August 17, 2026 01:09
@forhappy
forhappy merged commit 891a64f into main Aug 17, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant