CI: run the S3 tests against moto instead of MinIO - #224
Merged
Merged
Conversation
dl.min.io answers "410 Gone" for the minio and mc release binaries now, so the "Install and configure MinIO S3 server" step failed in every test job (and fail-fast cancelled the rest of the matrix). Use moto in server mode (moto_server) instead, like borg does since borgbackup/borg#10386: it comes from PyPI, so there is no server binary to download. The versions are pinned in requirements.d/s3-test-server.txt, which dependabot watches. moto's "server" extra pulls in the dependencies of all the AWS services it can mock; for S3, flask and flask-cors on top of the "s3" extra are enough. The step moved behind the Python setup, because it needs pip. moto keeps the objects in memory and accepts any credentials; the bucket is created with boto3. curl is not needed anymore. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
dl.min.ioanswers HTTP 410 Gone for theminioandmcrelease binaries now (checked withcurl -sIon 2026-09-19), so the step "Install and configure MinIO S3 server (test only)" fails in every Linux test job (curl: (22) The requested URL returned error: 410, first seen in the py314 job of #223), andfail-fastcancels the rest of the matrix. CI is broken for every PR.Change
Run the S3 tests against moto in server mode (
moto_server), the same way borg does since borgbackup/borg#10386 (borgbackup/borg#10361):requirements.d/s3-test-server.txt:moto[s3],flask,flask-cors, pinned - dependabot already watches/requirements.d. moto's ownserverextra pulls in the dependencies of all the AWS services it can mock (47 packages, some of them compiled); for S3 the server only needs flask and flask-cors on top of thes3extra.pip install, startmoto_server -H 127.0.0.1 -p 9000, wait for the port, create the buckettestwith boto3 (a moto dependency), exportBORGSTORE_TEST_S3_URL=s3:test:test@http://127.0.0.1:9000/test/path(same format as before; moto accepts any credentials and keeps the objects in memory).curlis not installed anymore (it was only added for the MinIO download), and the moto log is shown by the "Display diagnostics on failure" step.Testing
Locally (macOS, Python 3.11, moto 5.2.3): full test suite with
BORGSTORE_TEST_S3_URLpointing atmoto_server: 290 passed, 19 skipped - none of the skips is an S3 one (SFTP / REST / blake3 are not set up on that machine). The S3 tests alone: 26 passed; moto's access log shows 18206 Partial Contentresponses for them, so the ranged GETs ofS3.load(name, size=..., offset=...)are really served as ranges.🤖 Generated with Claude Code