[hannesrudolph]

The portable store publishes a new gzip archive every hour. Gzip output cannot delta against the previous generation, so each snapshot adds about 68 MB to openclaw/gitcrawl-store and to every subscriber. The subscriber checkout here has reached 47 GiB. This PR adds a split packaging that Git can delta, plus reader support. No history is trimmed and no published ref changes.

Export. portable export --compression split commits the finalized, uncompressed SQLite bytes as ordered parts: NAME.part-000, NAME.part-001 and so on, 32 MiB each, or --max-archive-bytes when that is smaller. At most 64 parts are allowed. The manifest records compression: "split" and archiveParts: [{path, bytes, sha256}]. Because the parts are the database, archiveBytes and archiveSha256 equal outputBytes and sha256. JSON output lists archive_parts. Gzip and raw exports are unchanged.

Readers. Refresh, init, status, runtime staging and malformed-source recovery all accept split stores. They check part names, sizes, digests, the per-part budget and the digest of the ordered concatenation before reassembling and validating the database as usual. Older readers fail closed with unsupported compression and keep their last good mirror.

Subscriber storage. When the checkout's generation is split, strict refresh runs git repack -d --geometric=2 within the same deadline and growth budget. Each generation is then kept as a delta instead of eight new full objects. This also runs on a ref no-op, because implicit reader fast-forwards fetch without repacking. The result is reported as repack: "geometric", or "current" when nothing is loose. If more than 512 loose objects are present, for example years of loose gzip generations, the repack is skipped and reported as "skipped-loose-backlog". A failure is reported but does not undo the validated generation. Raw and gzip stores keep the existing no-repack contract.

Measured on real data. I exported two consecutive production generations (13:26Z and 14:26Z today) with this build:

• Each export produced eight parts totalling 256 MB.
• The second generation transferred as a 531 KB thin pack.
• Packed together, the two took 71.1 MiB, against 70.6 MiB for the first alone.
• A gzip generation is 69.5 MB.
• An 8-generation simulation pushed 505-637 KiB per generation after the first.

Validation. On Go 1.27.3:

• tidy, gofmt on tracked files, vet and deadcode: clean.
• govulncheck against a curl-fetched DB: no vulnerabilities.
• smoke, test-release and docs: passed.
• go test -race on internal/portable and internal/cli: passed.
• New tests cover the refresh transition from gzip to split, tamper refusal, invalid metadata, status without a runtime, the repack backlog guard and export part edge cases.

The gitcrawl-store publisher change that switches the hourly workflow to --compression split is a separate PR in that repository and pins this commit.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[hannesrudolph]

Real-data subscriber proof. I built d570dc4 (pre-fix; the fix adds no-op consolidation only) and ran it on a local publisher that holds the real 7036f37 gzip generation, followed by two split generations exported from the real 13:26Z and 14:26Z store snapshots. init succeeded, then portable refresh --json:

{"stage":"complete","result":"updated","artifact_bytes":256167936,"mirror_result":"promoted","repack":"geometric","elapsed_ms":52463}
{"stage":"complete","result":"updated","artifact_bytes":256270336,"mirror_result":"promoted","repack":"geometric","elapsed_ms":34303}

Both runs promoted the mirror with the manifest sha256. Each refresh finished well inside the 5-minute deadline, and the repack stage took 15s. In the simulation, geometric repacking repeatedly folded subscriber object storage back to roughly one generation plus deltas (sawtooth 83-330 MB over 8 generations), where loose objects grew without bound by about 80 MB per generation.

Owner decisions (Hannes delegated the storage decisions, "decide on the rest"):

• Split is a supported persisted portable format.
• Strict refresh gets a split-only, bounded consolidation exception.
• Raw and gzip stores keep the no-repack contract.
• History is never trimmed.

[hannesrudolph]

Proof at b45bc59 (same real-data publisher):

• An ordinary reader command (threads openclaw/openclaw --numbers 1 --json) implicitly fast-forwarded the subscriber to a new split commit and left 2 packs plus 1 loose object.
• Strict portable refresh --json: {"result":"no-op","repack":"geometric","elapsed_ms":19659}, leaving 1 pack and 0 loose objects. That one pack (142.33 MiB) holds the 69.5 MB gzip generation and all three split generations.
• A repeat strict refresh: {"result":"no-op","repack":"current","elapsed_ms":6781}, with nothing changed.

New tests cover packed fetches with fetch.unpackLimit=1 across three generations, the reader fast-forward case and the single-pack current result. The upgrade from the deployed release (v0.9.5), keeping the existing config and the writable, preserved local mirror, will be shown on the production subscriber during rollout.

[hannesrudolph]

Failure-path proof at 6ea6428, using the shipped CLI on the same real-data subscriber with .git/objects/pack made read-only:

{"result":"updated","mirror_result":"promoted","repack":"failed","repack_reason":"Git command failed; verify Git version, repository state and remote access: exit status 128; the validated generation is unchanged; check that the store's .git/objects directory is writable, then rerun refresh","sha256":"94cbb766..."}
stderr: gitcrawl: portable refresh: repack failed: <same reason>

After the permission was restored, the retry returned {"result":"no-op","mirror_result":"unchanged","repack":"geometric","sha256":"94cbb766..."}, and the mirror's SHA-256 still equals the manifest sha256 (94cbb766a658...). Local ClawSweeper review at 6ea6428: patch is correct, with no findings.

[clawsweeper]

ClawSweeper status: review started.

I am starting a fresh review of this pull request: feat(portable): publish split archives that Git can delta This is item 1/1 in the current shard. Shard 0/1.

This temporary status tracks the active review worker. The completed review will appear in the durable ClawSweeper review comment.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

[hannesrudolph]

Production subscriber upgrade. The deployed subscriber ran the official v0.9.5 release. It now runs a build of merged main (04d7441), with its existing refresh.toml, its checkout and its writable local mirror all preserved:

{"stage":"complete","result":"no-op","mirror_result":"preserved-local","after_commit":"3f106eea...","elapsed_ms":23733}

As before the upgrade, the writable local mirror stays preserved-local. The store is still gzip, so repack is correctly absent.

Operational note: under launchd, a locally built binary must be signed with the fleet's local code-signing identity, and the unchanged agent plist re-registered (bootout/bootstrap). Without that, macOS killed the job with OS_REASON_CODESIGNING. Each attempt was rolled back to v0.9.5 within a minute, and refreshes kept succeeding.