Split off from #10360, which fixed CI but only bought us time.
What happened
The CI S3 tests run against a MinIO server downloaded at job setup. MinIO has withdrawn its community binaries:
https://dl.min.io/server/minio/release/linux-{amd64,arm64}/minio and the matching mc URLs now answer HTTP 410 Gone. curl -f exits 22, so the step failed before borg was even installed.
- Their GitHub releases from
RELEASE.2025-10-15T17-29-55Z on carry 0 assets. The last release that still publishes binaries is RELEASE.2025-09-07T16-13-09Z.
That broke every PR and would have broken master on its next run: 3 native_tests jobs, both oldglibc_binary jobs, the codecov upload (no tests ran, so no coverage), plus fail-fast cancellations.
#10360 pins those last two releases (server + mc) from GitHub and verifies each download's sha256. CI is green again, but we are now pinned to a frozen snapshot of a project that has stopped shipping community binaries. It will not get security fixes, and nothing stops the remaining assets from being withdrawn too.
Where S3 is used in CI
.github/workflows/ci.yml, step Install and configure MinIO S3 server (test only) — starts a server and sets BORG_TEST_S3_REPO, which enables test_s3_repo_basics in src/borg/testsuite/archiver/remote_repo_test.py. That test is one round trip: repo-create, create, repo-list, list, extract, delete -a, repo-delete.
.github/workflows/ci.yml, step Smoke-test the binary against an S3 server in oldglibc_binary — runs the same shape of check against the frozen PyInstaller binary, to exercise the botocore models the spec bundles. It creates the bucket with boto3 rather than mc.
Only the first step needs the mc client at all, and only to create a bucket — step 2 already shows that boto3 does the job, so a replacement could drop that download entirely.
What a replacement has to support
borgstore's S3 backend (borgstore/backends/s3.py) is what has to work, not the AWS surface at large:
put_object, get_object, head_object, delete_object, copy_object
list_objects_v2 with Prefix, Delimiter, MaxKeys and continuation tokens
- ranged GETs (
get_object(..., Range=...)) — these are load-bearing, not an optimization: borg reads individual objects out of packs by range, and borg compact rewrites packs through the generic defrag in borgstore/backends/_base.py, which is ranged load() calls plus one put_object.
- No multipart upload is used.
- Note the existing compatibility workaround: the backend sets
request_checksum_calculation="when_required" / response_checksum_validation="when_required" and strips x-amz-checksum-crc32 and x-amz-sdk-checksum-algorithm from requests. A candidate server should not depend on those headers.
Candidates worth evaluating
|
License / runtime |
Distribution |
Notes |
| Garage |
AGPL-3.0, Rust |
static binaries, published per release |
actively maintained, publishes an S3 compatibility matrix; needs a small config file |
| SeaweedFS |
Apache-2.0, Go |
GitHub release binaries |
weed s3 subcommand; large project, S3 gateway is one component |
| s3proxy (gaul/s3proxy) |
Apache-2.0, Java |
jar |
maps S3 onto a filesystem; needs a JRE on the runner |
| adobe/s3mock |
Apache-2.0, Java |
jar / container |
a test double rather than a server; fine for a round-trip test, unclear on ranged GETs at our usage |
moto (moto[s3,server]) |
Apache-2.0, Python |
pip |
no external download at all — the runners already have Python and we already install requirements; it is a mock, so fidelity needs checking |
| MinIO pinned |
AGPL-3.0, Go |
last GitHub release with assets |
status quo after #10360 |
moto is attractive for CI specifically because it removes the external binary download that caused this outage in the first place; whether its ranged GET and list_objects_v2 behaviour is faithful enough for borgstore is the thing to test. Garage looks like the strongest choice if we want a real server.
Suggested next step
Stand one or two candidates up in a branch, point BORG_TEST_S3_REPO at them and run test_s3_repo_basics plus a borg compact on an S3 repo (compact is the part that leans hardest on ranged reads, and it is not covered by the current S3 test — worth adding either way).
Not urgent, since #10360 unblocked CI, but worth doing before the pinned assets disappear too.
Split off from #10360, which fixed CI but only bought us time.
What happened
The CI S3 tests run against a MinIO server downloaded at job setup. MinIO has withdrawn its community binaries:
https://dl.min.io/server/minio/release/linux-{amd64,arm64}/minioand the matchingmcURLs now answer HTTP 410 Gone.curl -fexits 22, so the step failed before borg was even installed.RELEASE.2025-10-15T17-29-55Zon carry 0 assets. The last release that still publishes binaries isRELEASE.2025-09-07T16-13-09Z.That broke every PR and would have broken master on its next run: 3
native_testsjobs, botholdglibc_binaryjobs, the codecov upload (no tests ran, so no coverage), plus fail-fast cancellations.#10360 pins those last two releases (server +
mc) from GitHub and verifies each download's sha256. CI is green again, but we are now pinned to a frozen snapshot of a project that has stopped shipping community binaries. It will not get security fixes, and nothing stops the remaining assets from being withdrawn too.Where S3 is used in CI
.github/workflows/ci.yml, step Install and configure MinIO S3 server (test only) — starts a server and setsBORG_TEST_S3_REPO, which enablestest_s3_repo_basicsinsrc/borg/testsuite/archiver/remote_repo_test.py. That test is one round trip:repo-create,create,repo-list,list,extract,delete -a,repo-delete..github/workflows/ci.yml, step Smoke-test the binary against an S3 server inoldglibc_binary— runs the same shape of check against the frozen PyInstaller binary, to exercise the botocore models the spec bundles. It creates the bucket with boto3 rather thanmc.Only the first step needs the
mcclient at all, and only to create a bucket — step 2 already shows that boto3 does the job, so a replacement could drop that download entirely.What a replacement has to support
borgstore's S3 backend (
borgstore/backends/s3.py) is what has to work, not the AWS surface at large:put_object,get_object,head_object,delete_object,copy_objectlist_objects_v2withPrefix,Delimiter,MaxKeysand continuation tokensget_object(..., Range=...)) — these are load-bearing, not an optimization: borg reads individual objects out of packs by range, andborg compactrewrites packs through the genericdefraginborgstore/backends/_base.py, which is rangedload()calls plus oneput_object.request_checksum_calculation="when_required"/response_checksum_validation="when_required"and stripsx-amz-checksum-crc32andx-amz-sdk-checksum-algorithmfrom requests. A candidate server should not depend on those headers.Candidates worth evaluating
weed s3subcommand; large project, S3 gateway is one componentmoto[s3,server])motois attractive for CI specifically because it removes the external binary download that caused this outage in the first place; whether its ranged GET andlist_objects_v2behaviour is faithful enough for borgstore is the thing to test. Garage looks like the strongest choice if we want a real server.Suggested next step
Stand one or two candidates up in a branch, point
BORG_TEST_S3_REPOat them and runtest_s3_repo_basicsplus aborg compacton an S3 repo (compact is the part that leans hardest on ranged reads, and it is not covered by the current S3 test — worth adding either way).Not urgent, since #10360 unblocked CI, but worth doing before the pinned assets disappear too.