s3cache

A transparent, fully S3-compatible caching proxy. Point any S3 client at it instead of the real endpoint; it forwards to an upstream S3 (e.g. Cloudflare R2) while cutting request costs for chatty clients.

It is a service that speaks S3 — not a client library. Set the client's S3 endpoint to this proxy; no client code changes.

What it does

• LIST and HEAD from an in-memory key index. Because every request funnels through the proxy, it sees every write and maintains an authoritative key index — so ListObjectsV2 is answered locally with no upstream LIST call. LISTs are the expensive S3 tier (R2 Class A), and clients that poll/list constantly (e.g. a log-structured store allocating slots) dominate the bill; this removes them. The same index answers HeadObject for a synced bucket: a faithful full record is local, and in single-node mode an absent key is an immediate 404. With replication, an index miss is forwarded because an out-of-band origin write has no feed event; a replicated cache therefore never invents a false 404. An entry only answers a HEAD once it is faithful: a bootstrap LIST row or a peer's gossiped write proves the key exists and carries what LIST reports, but not the Content-Type or x-amz-meta-* a HEAD does, so the first HEAD of such a key is forwarded once and its answer completes the entry in place (index_backfills). Nothing local is ever served that would differ from the origin — see Testing coherence and parity.
• Layered GET/HEAD object cache — hot / warm / cold. Cacheable reads (no range / part / conditional headers) of objects up to S3CACHE_MAX_OBJECT_BYTES are served from a hot node-local in-memory LRU (S3CACHE_CACHE_BYTES) in front of an optional warm node-local disk cache (S3CACHE_DISK_CACHE), falling through to cold — the S3 origin — on a miss. Concurrent misses for the same cacheable object share one origin fill; different keys remain fully concurrent, and known oversized objects stream straight through. Always layered, no mode to pick. Integer and suffix byte ranges slice the cached whole object. A first integer range promotes a small object when the index knows its size. An uncached suffix stays a small origin range request. HEAD is served from the same cache. See Cache tiers.
• Write-through + invalidation, and fill-on-write. PutObject / DeleteObject / multipart / CopyObject forward to the upstream (which stays the authority for conditional/OCC writes — identical semantics). A PutObject invalidates the object cache before forwarding, then updates the index after success, so even an origin write whose response is lost cannot leave an old hot or disk body behind. Ambiguous failures fence the key and rebuild its bucket index from the origin. A successful PUT goes one further: its body is already in hand, so the writing node keeps it (write_fill) instead of leaving the object's first read to be a guaranteed origin GET. Peers are still invalidated over the write feed — only the writer, which knows the new bytes, is refilled. The fill is taken only when the write knows exactly what a read of it will report: a Content-Type (without one the origin invents one), an ETag on the write response, a body within S3CACHE_MAX_OBJECT_BYTES, and none of the request forms whose stored object or response headers a kept copy could not reproduce (SSE-C, an append, a named storage class, object lock, tagging, a website redirect, Expires). Everything else streams through; refused writes retain the origin's response semantics.
• Full passthrough for everything else — all 98 S3 operations are implemented (generated from the s3s S3 trait), so arbitrary S3 clients work, not just one.

Correctness rests on one property: the proxy is the only writer to the bucket, so its index can't go stale. A restart just re-syncs the index (one LIST) and refills on demand — correctness never depends on the cache surviving.

Cache tiers

The body cache is always layered — there is no mode to select:

hot (in-memory, small)  ->  warm (node-local disk, large, optional)  ->  cold (S3 origin)

• hot — an in-memory LRU (S3CACHE_CACHE_BYTES), always on.
• warm — an optional node-local disk cache under S3CACHE_DISK_CACHE (S3CACHE_DISK_CACHE_BYTES). It's inclusive (every object written to hot is also written to disk) and survives restarts — a fresh pod re-indexes the on-disk files and comes up warm instead of stampeding the origin. Entries are retained by LRU. Bodies are served as memmap2-backed Bytes; only the small metadata header is decoded onto the heap, so mapped pages remain kernel-evictable and warm-hit memory is bounded by I/O/page-cache pressure rather than the disk-cache size. Cache files written by earlier releases (bincode, either layout) remain readable and age out normally; new entries use a postcard header. Size warm larger than hot to help. All disk ops are best-effort: an I/O error is a miss, never a data-plane failure.
• cold — the S3 origin.

Both tiers are node-local; cross-node coherence is handled separately (below): a peer's write updates the index and awaits removal of the local hot copy. The superseded warm copy is then deleted off the acknowledgement path, so dead bodies never crowd live ones out of the disk budget; a copy read in the meantime decodes suspect and must prove itself against that newer index before serving.

Cross-node coherence (gossip write feed)

By default each replica keeps its own LIST index, so running more than one would let their indexes drift. Setting S3CACHE_GOSSIP_BIND (with S3CACHE_GOSSIP_SEEDS as comma-separated id=host:port pairs) turns on the gossip write feed — groupnet's consistency layer, no broker, no consensus service, no extra infrastructure:

• every durable write publishes one compact event (put/del + bucket/key/size/ts) into this node's feed, pushed to live peers at network latency;
• every replica runs an apply loop that folds peers' events into its own LIST index (per-key last-writer-wins; deletes win timestamp ties via tombstones), then awaits hot invalidation before advancing its frontier/ack. Warm-disk deletion is not on this latency path: a retained warm body is suspect and the updated index either proves it or evicts/refetches only that key.

The fast path stays in local memory (LIST is served from RAM; nothing remote is on the read path). Loss is detected, never silent: a peer that falls behind the feed's ring, or a peer restart, surfaces as a gap — the node distrusts every local body and resyncs its index from the origin, which is the authority the index caches, so a copy is served again only once that index has proved it current. Only seeds need static addressing; every other peer resolves through gossiped advertisements. The ring is bounded by both event count and encoded bytes: sustained long-key writes shorten its replay window and surface the same explicit gap instead of growing one state entry past the transport envelope and silently arresting peer application. After a strong/strong-acks write has a real all-readers acknowledgement (or every straggler has lost serving authority), its safe feed prefix is retired while the newest head remains advertised as an anchor. Healthy feeds therefore carry an anchor plus only in-flight writes instead of remaining pinned near the encoded-frame ceiling. bounded writes and stalled waits never retire history because neither is an all-readers watermark.

With the feed on, multiple replicas are safe — this lifts the historical single-replica constraint with zero extra services.

Readiness and rollouts

The pod's readiness probe is GET /ready on the metrics port (CachingProxy::probe_ready). A pod is ready when it holds a complete index of its own, or when no live peer may hold one. Each node adds the s3cache:indexed gossip capability once its initial index and coherence warm-up complete (the GET /index-ready latch), and a peer may hold an index when this node sees it Alive or Suspect and it declares that capability or has declared nothing yet.

• Cold start. No peer holds an index, so every pod is ready and forwards to the origin while one builds. Once one declares its index, a pod still without one reads not ready until it installs that image; the kubelet marks it unready only after failureThreshold failed probes, so a quick install keeps it in the Service throughout.
• Rolling update. A replacement beside an index-holding peer is not ready until it installs or builds its own index, so the StatefulSet controller does not stop the next pod, the last index holder, first.
• A peer without an index (alive, still building) does not hold a pod back.
• A momentarily lapsed peer. A Suspect peer still counts. The kubelet marks a pod ready on one successful probe, so reading a slow peer as gone could let a rollout stop the last index holder; a peer that really died turns Dead within the membership detection window (a few seconds at the default S3CACHE_LEASE_MS). A lapsed serve-lease or a gap does not clear the latch: the peer still holds that index and rebuilds it.
• Crash, single pod left. A Dead or reaped peer never counts, whatever it last declared, so the remaining pod is ready and forwards to the origin while it builds. A replacement whose donor died is ready from then on as well, so nothing waits on a dead peer.
• Origin outage. A replacement that has to scan the origin stays not ready and the rollout stalls, while the index holder keeps serving: the last index is not stopped.
• Unknown peers. A starting pod's roster is empty until its seed resolver registers each gossip seed, and a seed then has no declaration until the first gossip with it. A seed counts as possibly indexed until it appears in the roster (or its address fails to resolve within the resolver's first round, about 30 s), and a peer until its declaration arrives. Every node declares a non-empty set when it joins, and a restarted node's new declaration replaces its previous life's.

The probe is served only once the S3 listener is bound. The chart's startup probe (TCP on the S3 port, availability.startupProbe) holds liveness and readiness off while the warm tier opens. Disabling the metrics listener selects a TCP readiness check, and rollouts then no longer wait for indexes. The headless gossip Service publishes not-ready addresses so peers can connect during startup. The chart renders a PodDisruptionBudget with maxUnavailable: 1; a not-ready replacement counts as unavailable, so an eviction cannot take the index holder either. Each ordinal owns its own warm PVC, so rollout availability does not require sharing cache files.

Index startup

The index scan can run concurrent LIST chains over disjoint key ranges. The default concurrency follows the process's available CPU parallelism, capped at 64. Set S3CACHE_INDEX_SCAN_CONCURRENCY or Helm indexScan.concurrency to override it. A value of 1 selects a serial scan.

Boundary discovery samples directory prefixes without assuming an application path layout. Each final range covers every key between its boundaries. Keys outside the sampled prefixes remain covered. The full index becomes available only after every range finishes in the same rebuild generation. An incomplete or failed discovery can reduce parallelism without omitting keys.

Discovery uses up to 16 additional LIST requests per rebuild by default. S3CACHE_INDEX_SCAN_DISCOVERY_REQUESTS or Helm indexScan.discoveryRequests sets that budget. Zero disables discovery and uses a serial scan. Each additional range can also read one page past its boundary. These are startup or recovery costs. They do not add a periodic scan. S3CACHE_RECOVERY_REARM=true (Helm recovery.rearm) optionally retries a full origin scan after a recovery episode exhausts its finite budget. The interval starts at 5 seconds and caps at 60 seconds; local reads remain origin-routed until a new generation is affirmed. The default is off and continues to use explicit restart after an exhausted episode.

Consistency

Clients should never have to know s3cache exists. In the default mode they don't:

strong (default) — lease-backed. Indistinguishable from talking to one S3 node, for every client, no headers, no cooperation. The mechanism is a coherence lease (Gray–Cheriton freshness leases, groupnet's T3 tier), and it replaces the view-stability heuristic earlier releases used:

• A node may serve locally only while it holds a lease. Every node continuously renews a serve-lease (one small gossip entry every D/3), and every other node grants the renewals it has adopted. A node answers a LIST, a faithful positive HEAD, or a GET from local state only while a lease confirmed by every peer covers this instant. Replicated index misses always go to the origin; no lease can prove that an object was not created there out of band. The lease is one mechanism for every local positive, not an absence oracle, and false covers everything: booting, warming up, lapsed, awaiting a resync, a peer gone silent, or a partition. Every one of them sends the read to the origin: slower, never stale (unhealthy_bypasses).
• Writes end at an ack or at a lapse. A write returns only after the origin acked it AND every lease-holder either applied the invalidation (the fast path — one in-cluster ack round, ~2 gossip hops, behind the origin round-trip already paid) or had its serve-lease expire in the writer's own engine (the slow path — bounded by D, counted as write_lease_lapses). So PUT via A; GET via B is deterministically fresh, and a peer that stops answering costs one write up to D once, not an ack timeout on every write forever.
• That second path is the whole point. The old mode ended an unacked write in a degradation: the writer proceeded and correctness then depended on the stale peer learning it should stand down — exactly what an asymmetrically-partitioned peer cannot do. A lapse is a guarantee instead: the straggler's own clock has closed its serve window, and a lapsed node serves nothing cached until it re-synchronizes from the origin and affirms it.
• A gap stands the lease down first. A write-feed gap (ring overflow, a peer restart) is proof this node missed invalidations, so it drops its right to serve before the remediation runs — and only the resync that actually ran may hand it back. Groupnet now owns one cold origin scan per node in gossip mode. Until that scan completes and the node affirms recovery, positive GETs forward to origin; requests do not wait for LIST, and local cache hits become available after recovery. Persisted bodies remain suspect until validated. LIST and index-backed absence still require a complete bucket index. A feed gap during boot supersedes the scan and requires a fresh one.
• A lapse with no gap behind it gets a recovery, and usually keeps the cache. Not every way of losing the licence arrives as an event: a peer that departs (a planned stop or scale-in) is dropped from the roster with a lapse, and a peer restarted while the write feed was quiet freezes this node's confirmation with no gap to notice. The lapse latches, so a watcher on the lease recovers from it — and a lapse is not proof that anything changed, only that this node stopped being allowed to serve, so the watcher proves the cache rather than throwing it away. It stands the licence down, waits for every granter separately to adopt a renewal published after the lapse (so every wait held against the old one is resolved — read per granter rather than off the confirmed watermark, which is a min over the roster and advances when the member pinning it merely departs), settles for the fabric's entry-propagation bound (2 × anti_entropy_interval), checks that no peer which was live at the lapse has vanished from membership since, and then barriers on every peer's advertised feed head — re-running the vanished check once more afterwards, because the barrier waits and a reap can land inside that wait. Past the barrier the apply loop has updated every changed key and removed its stale hot copy; retained warm bodies must validate against that index, while unrelated hot bodies keep their proof. The node then re-affirms warm. Groupnet owns the bounded generation and retry schedule. Any stage that cannot get its proof — a live peer reaped before the recovery could affirm, so its feed frame went with it; a head that never arrives; a granter that never re-grants — falls back to the gap's guarded origin rebuild instead. Without any of this a lapse would be permanent origin-serving for a node whose surviving peers are perfectly healthy.

What it rests on, stated as failure modes (groupnet's consistency::lease honesty box is the long form):

• Bounded clock rate skew, not bounded connectivity. A reader computes its window on its own monotonic clock and its granters expire on theirs. A reader whose clock runs slow relative to a granter's by more than the rate margin (max(D/100, 5ms)) over one lease duration can believe it holds a lease the granter already expired. It is an assumption about rates — a few hundred ppm on any healthy host — never about wall-clock steps, which cannot affect it.
• A write is never acknowledged without the guarantee. A reader that keeps renewing while it stops applying — a stuck apply loop, a partition that carries gossip but not writes — is granted no new renewal by the writer until it applies the write, so its lease lapses within D and the write ends on that lapse. What is left — a peer whose current life has not yet proven it counts the writer, a writer still in its warm-up window, a mixed-version fleet mid-rollout, an un-leased peer that does not ack — runs to the deadline (D + 1s) and is answered 503 ServiceUnavailable rather than success, counted as ack_timeouts and logged naming the node. The write may have reached the origin and is already on the write feed, so every node converges on it; a retry (or a read) settles which outcome the client got.
• A pod that dies without departing keeps the survivors on the origin until it returns. Confirmation is a min over the roster, and the roster only grows: a granter stays in it through suspect, dead and reap, and leaves only once its grant map says it departed. So after a crash (or a partition) every other reader serves from the origin — correct, slower — until the pod comes back under the same id (a StatefulSet pod grants again on boot) or the survivor restarts. A planned stop does not latch it: on SIGTERM (a rollout, a scale-in) the node departs the lease set before it drains, and readers drop it as soon as they see the departure — even while membership still lists it — re-synchronize once (cheaply, without an origin scan), and serve locally again. Each read sent to the origin for this reason counts read_absent_granter_bypasses, and the survivor logs serving reads from the origin: lease granters <ids> are counted but not alive whenever that set changes (and once more when they are all back), so the pod it is waiting on is named. Before this rule the reap released the readers after about 3 s, which is how a partition outliving the reap horizon let a reader serve with no granter at all.
• First write after a pod dies costs up to D. That is the lapse being waited out, once. It shows up as write_lease_lapses, not ack_timeouts, because the guarantee held. A planned stop does not pay it: the departing node retracts its serve-lease before it drains, so there is no lapse to wait out. While an asymmetric partition lasts, every write pays it, because each ends on a lapse.
• A planned restart keeps the peers' index; a crash costs one origin scan. On SIGTERM a stopping node answers new writes 503 and, while its HTTP drain runs (10 s at most), seals its write feed once its in-flight writes finish, waiting up to 5 s for peers to acknowledge the seal. A slow read still draining does not delay the seal. A peer that saw it crosses the restart without a gap (feed_renewals), and the returning node copies that peer's index without an origin LIST. A crash, a write still running after 5 s, or a seal no peer saw is an ordinary gap: the pair rebuilds with exactly one origin scan, which the other node follows. Keep the pod's termination grace period (Helm availability.terminationGracePeriodSeconds, default 30) above the 10 s the stop can take.
• A write whose outcome the origin did not settle is fenced everywhere before its error is answered. A 5xx, a timeout, a missing response, a 412 against a precondition the local index vouched for, a multipart completion whose upload is gone, or a version-scoped delete leaves the key's state unknown. The writer fences the key, advertises it as a put carrying neither ETag nor size, and holds the response until every lease-holder has fenced it too (or lapsed); each node then answers the key from the origin until one origin HEAD of its own resolves it. A node from before this rule indexes that event skeletally, which also sends the key's reads to the origin. Every mutation runs on a task of its own, so a client hanging up mid-write cannot strand an applied DELETE, copy or completion outside the index.
• A feed gap discards the gapped node's index. The node missed writes it can no longer order its rows against, so it keeps nothing for a peer-image install to carry forward; the install or the origin rebuild repopulates it.
• An origin answer only completes the version it describes. A forwarded HEAD lends an index entry its metadata only if the entry carries the same ETag and size and no write of the key was applied while the HEAD was in flight.
• The absolute arbiter for conflicting writers remains the origin: conditional If-Match/If-None-Match writes pass through untouched (OCC — no lost updates, regardless of node), and the index heals from the origin (gap resync, startup bootstrap). Cross-writer index races resolve by timestamp, deletes winning ties.

strong-acks (S3CACHE_CONSISTENCY=strong-acks) — deprecated, one release only. The pre-lease coherence mechanism, kept so a deployment can roll the lease tier through the fleet in stages: writes wait on every alive peer that participates in the ack tier, an ack timeout ends without a coherence guarantee, and the read-side licence is a fully-alive membership view. Replicated index misses still go to the origin in every mode.

It is a rollback lever for that mechanism, not a time machine. Three things changed underneath every mode, and this one gets them too:

• Membership is tuned differently, in every mode. dead_timeout is pulled from groupnet's 10s default down to max(D, 2s) uniformly — a mixed fleet must have one membership timing, not two — which puts the reap horizon at ~4s instead of ~20s. A partition outliving that lands on the write-feed gap path (distrust + origin re-LIST) rather than reconciling through a digest catch-up: loud, correct, and not what this mode did before.
• There is no replicated 404 trust window. A healthy view or valid lease proves feed participation, not the absence of writes made directly at the origin. A synced replicated index miss always forwards; faithful positive entries remain local.
• The ack wait skips peers that declare themselves bounded. A peer advertising s3cache:bounded is no longer waited out per write. A peer that advertises nothing still is — absence is not non-participation — so a pre-upgrade fleet is unaffected.

It is removed in the next release. Nothing new should choose it, and a fleet running it is running the brittleness the lease tier exists to remove. Mixed fleets are safe in both directions — a leased writer waits for strong-acks and pre-upgrade peers through the ordinary ack round, since they publish no lease for it to wait on or expire.

bounded (S3CACHE_CONSISTENCY=bounded, set uniformly). For clusters too large to pay a per-write ack round: writes return on the origin ack, reads are fresh within ~one push hop (the freshness barrier), and no ack-ledger traffic flows. A barrier that cannot reach an advertised head by its deadline fails closed to the origin. Session tokens then offer per-client strictness: every write response carries x-s3cache-write-token: <writer>:<epoch>:<seq>; echoing it on a read as x-s3cache-read-token barriers on that specific write, and an unverifiable token routes the read to the origin — never silently downgraded. (Tokens work in strong mode too; they're just redundant there.) Requests the cache cannot reproduce faithfully — a specific versionId, ChecksumMode, or SSE-C — bypass the cache and are served by the origin.

Client cookbook: tokens and cross-node OCC

Nothing in this section is required. In the default strong mode there is no client cooperation to opt into — every client is strict automatically, tokens are redundant. Tokens matter only if you switch to bounded, and even there an unaware client is fresh within ~one push hop and never silently stale; the token upgrades one client to read-its-own-writes strictness without paying strong's per-write ack round.

Getting / using the session token (opt-in; standard SDKs work fine without it):

# boto3: capture each write's token, echo it on reads.
TOKEN = {"v": None}
def inject(request, **_kw):
    if TOKEN["v"]:
        request.headers["x-s3cache-read-token"] = TOKEN["v"]
cli.meta.events.register("before-send.s3", inject)

resp = cli.put_object(Bucket=b, Key=k, Body=data)
TOKEN["v"] = resp["ResponseMetadata"]["HTTPHeaders"]["x-s3cache-write-token"]
# every following read via ANY node now reflects that write, or is served
# by the origin — never stale, never silently downgraded

Zero-client-change alternative: session affinity at the load balancer — a client pinned to one node reads its own writes locally by construction.

Cross-node OCC (read-modify-write) needs no token — conditional operations bypass the cache and the origin arbitrates, so no update is ever lost:

1. GET key (any node) → body + ETag.
2. PUT If-Match: <etag> with the new value. 200 ⇒ you won (keep the write token for your session's reads).
3. 412 ⇒ someone else won: re-read fresh and retry. GET If-None-Match: <your-stale-etag> is guaranteed fresh in one shot (conditional GETs pass through to the origin); a plain re-read is fresh within ~1 RTT.
4. Create-only: PUT If-None-Match: * — exactly one creator across all nodes.

Testing coherence and parity

• Unit / protocol tests (included in cargo test --locked --workspace --all-features): the unit cases are fully in-process — real groupnet nodes over an in-memory transport, no external services. They cover peer-write index folding + hot-only invalidation (including retained, suspect warm bodies), out-of-order LWW convergence (tombstones, delete-wins-ties, no-resurrection), and the freshness barrier. Groupnet's sans-IO core and simulator cover recovery generations, lease-lapse barriers, vanished peers, retry budgets, and stale callbacks. S3cache's MinIO startup tests cover origin fallback, body distrust, guarded retry pages, and explicit restart after a prolonged outage.
• Integration tests (cargo test --locked --workspace --all-features, needs a Docker daemon): tests/it/e2e.rs and tests/it/coherence.rs run the real CachingProxy against a real MinIO origin (testcontainers) reached through a transparent request counter, so every claim is asserted twice — what the client saw, and what the origin was asked for. LIST/HEAD from the index cost the origin nothing; a GET misses once; over-cap objects bypass; ranges slice the cached body; conditional writes (If-None-Match: *, If-Match) keep the origin's 412 semantics and leave the index and cache consistent with the outcome. tests/it/coherence.rs does the same with two nodes gossiping over loopback UDP: a write on A is in B's index by the time it returns, an overwrite on A makes B validate/refetch only the changed body, replicated misses and deletes ask the origin rather than inventing a 404, an origin-routed whole GET cannot re-probe stale cache state, a contested create-if-absent is arbitrated by the origin, and 640 long-key writes prove acknowledged feed retirement keeps apply advancing past the old transport-envelope arrest point.
• Differential tests (tests/it/differential.rs, same Docker origin): every row asks one question twice — once through the proxy, once straight at MinIO — and asserts a client could not tell which answered, over the status, the body and the headers it branches on (ETag, Content-Length, Content-Range, Last-Modified, Content-Type, Accept-Ranges, x-amz-meta-*) plus the whole ListObjectsV2 envelope. It is the referee for anything the proxy chooses to answer locally: the conditional-request matrix, the range matrix in every serving state, the LIST matrix walked to exhaustion, response-* overrides, max-keys=0, and a partially-refused batch delete.
• End-to-end (scripts/coherence-e2e.sh): spins up MinIO and runs boto3 harnesses against real s3cache nodes gossiping over loopback UDP. Needs podman/docker and python3 + boto3.
  • parity_e2e.py — differential parity: every operation through the cache returns the same as talking to S3 directly (GET bytes/headers/metadata, HEAD, ranged GET, LIST with prefix/delimiter/pagination/StartAfter, conditional PUT & GET / OCC, DELETE, multipart, copy), plus cross-node reads.
  • coherence_e2e.py — a write on one node is seen by another (LIST, no-stale GET, DELETE, both directions).
  • resilience_e2e.py — a peer outage is not a data-plane outage: with its peer down, a node's PUT/GET/LIST stay correct and fast (the barrier never waits on a dead peer), and coherence resumes both ways after the peer restarts — including the fresh-epoch gap path (distrust + origin resync), exercised end to end.

Development and release checks

Dependencies are pinned by the committed Cargo.lock. Run the locked core gate after Rust or dependency changes and before a release:

cargo metadata --locked --no-deps --format-version 1
cargo fmt --all --check
cargo build --locked --workspace --all-targets --all-features
cargo clippy --locked --workspace --all-targets --all-features
cargo test --locked --workspace --all-features
RUSTDOCFLAGS='-D warnings' cargo doc --locked --workspace --all-features --no-deps

The integration tests start MinIO through testcontainers and require a reachable Docker daemon. Build the pinned test origin before the Rust test gate. The build fetches MinIO source at commit 7aac2a2c5b7c882e68c1ce017d8256be2feea27f and compiles it in the official Go 1.24.8 image. It tags the local image as s3cache-minio-test:7aac2a2c:

bash scripts/build-minio-test-image.sh

Set RUNTIME=podman to build the image with Podman. The shell end-to-end harness builds the image before it starts MinIO. CI runs unit tests and the Docker-free metrics_endpoint, tier_cache, and startup_readiness integration modules. Run the real MinIO tests locally. After Helm chart changes, lint and render with the required upstream value:

helm lint deploy/helm/s3cache --set upstream.endpoint=https://example.com
helm template deploy/helm/s3cache --set upstream.endpoint=https://example.com

After dependency, Dockerfile/.dockerignore, or release changes, build the same container locally:

docker build --tag s3cache:check .

The .dockerignore allowlist limits application build inputs to Cargo.toml, Cargo.lock, and src/. Add any new required input to that allowlist deliberately. Unrelated documentation-only edits do not require a container build.

Config (env)

Var	Default	Meaning
S3CACHE_LISTEN	0.0.0.0:8014	S3 API listen address
S3CACHE_UPSTREAM_ENDPOINT	(required)	Upstream S3 endpoint URL (e.g. R2)
S3CACHE_BUCKETS	(empty)	Comma-separated buckets to index eagerly at startup
S3CACHE_RECOVERY_REARM	false	Opt in to Groupnet-owned capped full-recovery retries after a finite episode exhausts (5s initially, 60s cap); no S3 control writes
S3CACHE_CACHE_BYTES	268435456 (256 MB)	Hot (in-memory) tier capacity (bytes)
S3CACHE_MAX_OBJECT_BYTES	8388608 (8 MB)	Per-object cache cap; bigger objects stream through
S3CACHE_DISK_CACHE	(empty)	Directory for the warm disk tier; unset = no disk tier
S3CACHE_DISK_CACHE_BYTES	10737418240 (10 GB)	Warm disk tier capacity (bytes)
S3CACHE_GOSSIP_BIND	(empty)	UDP bind for the gossip write feed (e.g. 0.0.0.0:7946); unset = single-node, no coherence layer
S3CACHE_GOSSIP_SEEDS	(empty)	Comma-separated seed peers as id=host:port; only seeds need static addressing
S3CACHE_GOSSIP_ADVERTISE	bind addr	Address peers should dial back (set under NAT/container networking)
S3CACHE_CONSISTENCY	strong	strong: lease-backed — a node serves locally only while it holds a serve-lease, and a write ends at an ack or at a lapse. strong-acks: the pre-lease ack-only mechanism, deprecated, removed next release. bounded: ~one-hop freshness, no per-write ack round (large clusters; set uniformly)
S3CACHE_LEASE_MS	2000	Coherence-lease duration D (strong only): the writer's worst-case stall on a silent peer, each reader's window between confirmations, and the fleet's membership dead_timeout. Renewal traffic is one small entry per node per D/3
S3CACHE_STATS_SECS	60	Stats log interval (seconds)
S3CACHE_METRICS_LISTEN	(empty)	Listen address for the Prometheus text endpoint (e.g. 0.0.0.0:9090); unset = no endpoint
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION	—	Upstream creds (R2: region auto)

Clients authenticate to the proxy with the same key; the proxy re-signs to the upstream.

Metrics

Every counter is logged as one s3cache stats: line each S3CACHE_STATS_SECS, and — when S3CACHE_METRICS_LISTEN is set — served as Prometheus text at GET /metrics on that address, s3cache_-prefixed (metrics.enabled in the chart). Both are generated from one declaration, so a counter cannot exist in one and not the other. The same listener serves the readiness probe at GET /ready (see Readiness and rollouts): 200 once this node's index is complete or while no live peer may hold one, 503 otherwise. It is bound only after the S3 listener. GET /index-ready returns 503 until the configured indexes and initial coherence warm-up complete. It then stays at 200. The S3 read barrier still routes reads to the origin whenever local state cannot safely answer them.

What they attribute: LIST (list_from_index vs list_passthrough), GET (get_hit / get_miss / get_bypass, range_*), HEAD (head_hit from a cached body, head_index and single-node-only head_404 from the key index, head_miss forwarded upstream), the writes folded into the index by operation (writes_indexed_put / _copy / _multipart, each a separately billed upstream class-A call, plus _observed for keys learned on the read path), write_fill (writes whose body was kept rather than dropped — each one an origin GET the object's first read no longer costs), index_backfills (index entries completed from a forwarded answer — see below), the warm tier (warm_hit / warm_miss / warm_error, with warm_rejects for objects the per-object cap declined, kept apart so warm_error stays alertable), its live Tierstore view (warm_entries, warm_mapped_entries, warm_disk_bytes, warm_disk_budget_bytes, warm_evictions, and warm_evicted_bytes), and the gossip write feed (feed_*, ack_timeouts, write_lease_lapses, recovery_origin_scans, recovery_fallbacks, recovery_ready_recaptures, unhealthy_bypasses).

Origin requests are counted where they leave the process — an interceptor on the upstream client, below every forwarding, fill, probe, index-scan and recovery path — once per HTTP attempt that got an answer (retries included, since the origin bills each), whatever the status. By R2/S3 billing class: origin_class_a_requests (PutObject, CopyObject, multipart create/upload/complete, every List*, bucket Put*/Create*), origin_class_b_requests (GetObject, HeadObject, every other Get*/Head*) and origin_free_requests (Delete*, AbortMultipartUpload); their sum is everything sent upstream. Paid answers that moved no data are broken out within those totals: origin_get_not_found and origin_head_not_found (absent keys a replicated index miss forwards — misses stay non-authoritative there, so these are the cost of that rule) and, for object writes, origin_write_precondition_failed (412: a conditional PUT/copy such as If-None-Match: * that lost) and origin_write_refused (any other 4xx/5xx).

Those are the coherence tier's, and the split between lease lapses and timeouts is the one worth wiring an alert around: write_lease_lapses is the guarantee working — a peer stopped acknowledging, its serve-lease expired, and the write completed knowing that peer can serve nothing cached until it re-synchronizes. Sustained movement means a pod is unresponsive and each such write cost up to one lease duration, but no read was ever stale. ack_timeouts is the absence of the guarantee: a peer still live and still behind when the wait's deadline passed, and that write was answered 503. Alert on that one. unhealthy_bypasses counts reads sent to the origin because this node held no licence or could not reach every advertised feed head before the freshness-barrier deadline. read_licence_bypasses and read_freshness_bypasses split those causes, and read_absent_granter_bypasses is the part of read_licence_bypasses spent waiting on a lease granter that crashed without departing (the log names it). They do not include a read whose session token could not be verified in time. Some unhealthy bypasses are expected at startup and after a feed gap. body_revalidation_evictions counts suspect cached bodies rejected by the synced index. Its reason counters are body_revalidation_index_absent, body_revalidation_missing_identity, body_revalidation_etag_mismatch, and body_revalidation_timestamp_mismatch. These reasons keep the same trust decision. A body in an unsynced bucket is invalidated without incrementing body_revalidation_evictions. recovery_origin_scans counts one attempt each time this node starts the full origin-index arm, including cold boot, gaps, and any lapse whose feed/lease proof could not retain its index. The origin LIST counter prices the actual request cost. recovery_ready_recaptures counts each Ready recapture a node starts: a guarded snapshot of its whole index for peers at C, then a measure and encode of that snapshot off the index lock. C holds the recovery fence and the index write lock for O(buckets) work, microseconds at production size, so a recapture costs CPU on the blocking pool but never pauses the node. Expect one after a node's own scan, and one after each lease lapse or membership change. Groupnet paces failed attempts, and a capture retired before it has been Ready for a claim window counts as one, so sustained movement means the donor image keeps failing, and pods joining meanwhile scan the origin.