feat: Add optional external storage plugins with a pass-through example

[brightsparc]

Adds a plugin seam so capabilities can be added to walgit's storage layer without forking it or changing the ObjectStore trait. An operator-installed shared library decorates the configured S3, GCS or memory store.

With no plugin configured there is no overhead — not low, zero. Same stock binaries, same backend behaviour, same stored formats.

What this unlocks

• New capabilities without a fork. A decorator is an ordinary ObjectStore, so anything expressible against that trait can be added downstream — encryption at rest, audit interception, compression, transparent double-writing during a bucket migration, extra caching — with no upstream change per capability.
• Policy walgit should never carry. Encryption keys, tenant identity and key management stay in the plugin. The decorator sees logical keys and bytes; nothing about who owns them enters this repo.
• The stock binary ships as-is. Operators install capabilities as a .so alongside it — no branch to rebase, no private build to keep current.
• Per-deployment behaviour from one build. Different deployments can layer different capabilities against the same released binary.

This PR adds the seam, not any particular capability. The only functional example in-tree is a no-op pass-through, and nothing here adds encryption, tenant identity, key-management policy, or Azure Blob support.

How it works

[store.plugin]
library = "/opt/walgit/libmy_store.so"   # absolute path, operator-installed
options = { ... }                        # opaque JSON, handed to the factory

One CLI store constructor (open_store) applies the decorator across serving, maintenance and administrative commands. The factory receives the configured backend with its global prefix already applied once, so the decorator only ever sees logical keys. A downstream crate returns any ObjectStore via walgit_store_plugin::export_plugin!(factory).

Built on abi_stable 0.11.3 — checked root modules, sabi_trait endpoints, owned RBox/RVec/RString, following Rotel's Rust processor SDK and its async bridge.

Boundary details and constraints

• Object bytes stream in frames of at most 1 MiB; operation metadata is bounded JSON.
• Versions and conditional operations cross unchanged; response streams retain their endpoints.
• Dropping a stream stops further polling without draining, but cannot interrupt a blocking call already in flight.
• Load/init failures stop startup. The SDK catches unwinding factory/request/poll panics as errors.
• Each library's root-module layout is checked before its storage factory is invoked. Mappings stay loaded until process exit.
• Plugins must target the same OS/arch and a compatible SDK/abi_stable version, and declare crate-type = ["cdylib"] plus a direct abi_stable dependency.

Native plugins are trusted code. ABI checks are not a sandbox — aborts or panicking destructors can still take down the process.

Rejected combinations. Bucket mounts are refused alongside plugins, because a mount bypasses the decorator entirely. A transforming plugin must also disable or correctly implement signed URLs, acceleration and native composition; the SDK cannot infer a plugin's transformation policy. Known MIME hints are preserved, unknown advisory ones omitted.

Cost

Unconfigured: zero, structurally. open_store returns the backend store itself when [store.plugin] is unset — no decorator is constructed, so none of the code below is on the request path. DynStore is Arc<dyn ObjectStore> with or without this change, so an undecorated store gains no indirection either, and there is no per-operation branch to skip. Nothing in the tables below is paid by a deployment that doesn't load a plugin; the only residual cost of the seam existing is compile time and binary size for abi_stable.

Loaded: measured below. The hand-written ABI prototype (raw pointer handles, vtables, release callbacks) was replaced by the checked one. It also got faster, so the two probes are recorded for the record rather than as an argument.

Interface latency — macOS arm64, test-profile MemoryStore full GETs, three alternating runs, median of per-run means:

Object bytes	Manual ABI prototype	abi_stable
1 KiB	66.12 µs	66.92 µs
1 MiB	127.93 µs	97.84 µs
8 MiB	840.27 µs	602.14 µs

This does not isolate the ABI library's cost — the new implementation also removes a receiver-side frame copy.

Encryption overhead — this one does isolate the boundary, running the same AES-256-GCM workload in-process and again behind the plugin. Linux x86-64, release profile, MemoryStore, 16 MiB objects read in 1 MiB BLOCK_SIZE-aligned ranges (a pack's real read shape), best-of-4:

Configuration	Manual ABI prototype	abi_stable
Pass-through plugin — does no work at all	1.21–1.58 GB/s	1.70–1.99 GB/s
AES-256-GCM in-process, no plugin	1.66–1.76 GB/s	1.67–1.76 GB/s
AES-256-GCM through the plugin	0.55–0.74 GB/s	0.72–0.80 GB/s

Two things the latency table cannot show:

1. The boundary is not free at throughput. Under the manual ABI, a plugin doing nothing was slower than doing AES-256-GCM in-process.
2. The checked implementation is 25–30% faster, because an owned RVec lets Bytes adopt the plugin's allocation instead of copying out of it.

The cost is per byte, not per request — a whole-object read matches sixteen separate 1 MiB reads, and raising MAX_FRAME from 1 MiB to 8 MiB measured worse. It is not tunable by framing.

⚠️ Still MemoryStore with no KMS. A decorator that calls out to a key service pays round trips none of this measures. No network, Git or KMS is in any figure above; these are not production throughput claims.

Validation

The full validation is not all green. Details below.

Checked at 2e1f168 on macOS arm64, Rust 1.97.1, Git 2.55.0:

Check	Result
just web-build	✅ Passed, incl. oxlint + TypeScript. Vite reports its pre-existing bundle-size advisory.
just warnings	✅ Passed across all workspace targets.
just test	✅ Passed, incl. all three native plugin contract/load-failure tests.
cargo test -p walgit-server --test sim	✅ All 20 simulations passed.
S3 contract (just test-s3)	✅ Passed against a local RustFS service, using local dev credentials and bucket rather than the recipe's hard-coded standalone defaults.
just clippy	❌ Two clippy::cast_lossless errors at walgit-wal/src/registry.rs:494–495 on macOS. This file is identical to the PR base.
just e2e	⚠️ 42 passed, 1 failed, 1 ignored. fetch_from_front_that_serves_the_base_remotely failed after a commit-graph lock conflict left has_commit_graph=false; passed when run alone.
just ci	❌ Failed at Clippy; constituent recipes then run separately, as above.

The e2e failure is pre-existing and already documented in the base. Its test and harness are unchanged and use the in-memory store with no plugin. The isolated pass is diagnostic, not a substitute for a green full run. No assertions were relaxed and no failing checks suppressed.

Live GCS contracts and the ignored benchmark/soak tiers were not run — the GCS cases in the S3 test binary skipped because no GCS bucket was supplied.

Since the latest commits (docs + lint fix): cargo clippy --workspace --all-targets -- -D warnings now exits 0 on Linux x86-64. The base failed it on a missing-backticks doc lint in walgit-cli. The macOS cast_lossless errors above are base-identical and were not re-checked on macOS.

Test coverage

The native-library contract covers streaming and file uploads, ranges, conditional reads/writes/deletes, CAS races, listing, single prefix application, interrupted uploads, and response lifetime after the store is dropped. A deliberately incompatible module verifies layout rejection before the storage factory is invoked; missing libraries and failed factories also fail. just test builds and runs these native fixtures; the timing probes stay opt-in (passthrough_overhead, --ignored --nocapture --test-threads=1 with WALGIT_TEST_PLUGIN set).

Docs

• docs/STORAGE_PLUGINS.md — the interface and the adapter's obligations.
• docs/ROUNDTRIPS.md — the round-trip budget row for a decorated store.

[brightsparc]

@tobi The simplification in #52–#58: one inventory, immutable packs, no second catalog is nice.

Rebased onto main after #52–#58 by merging main into this branch (1706cae). The conflicts were all in the bundle removal's wake: bundle_cmd.rs is dropped (this PR's only change there was routing it through the plugin-aware open_store, and the file no longer exists), the D42 plugin decision now sits beside your new D47 in AGENTS.md, the [store.plugin] paragraph in CONTRACT.md precedes the post-bundle command line, and just test-plugin is appended to the reworked test recipe. Cargo.lock was regenerated by cargo. Workspace clippy and the native plugin boundary test are green on the merged head.