feat(portable): publish split archives that Git can delta

CHANGELOG.md

@@ -2,6 +2,10 @@
 
 ## Unreleased
 
+- Request `synchronous=FULL` on every SQLite connection and, on macOS, `fullfsync` for commits and checkpoints. Go test binaries skip the macOS full flush.
+- Let the native code producer publish from a captured upstream commit when core progress is fresh, even while provider coverage is incomplete. Retries and coverage are not changed.
+- Add `portable export --compression split`, which publishes the uncompressed portable database as verified 32 MiB parts so Git stores each new generation as a small delta instead of a new 68 MB gzip blob. Refresh, init, status and recovery read split stores; older readers refuse them and keep their last good mirror. After advancing to a split generation, strict refresh geometrically repacks recent objects so subscribers also keep each generation as a delta, skipping a loose backlog of more than 512 objects. Raw and gzip stores are still never repacked.
+
 ## 0.16.0 - 2026-10-20
 
 **Highlights:** Time-filtered neighbor discovery, Go security fixes, and reliable large-result lookups.

docs/portable-stores.md

@@ -55,6 +55,24 @@ size and SHA-256, runs SQLite `quick_check`, and only then atomically replaces
 the runtime mirror. Legacy raw SQLite stores continue to use the same manifest
 without compression fields.
 
+Stores may instead commit a split artifact: `compression: "split"` and an
+ordered `archiveParts` list of `{path, bytes, sha256}` basenames beside the
+manifest (`NAME.part-000`, `NAME.part-001`, ...). The parts are the
+uncompressed SQLite bytes, so `archiveBytes`/`archiveSha256` equal
+`outputBytes`/`sha256`. Gitcrawl checks every part's size and digest, the digest
+of their ordered concatenation and the per-part `maxArchiveBytes`, then
+reassembles and validates the database like any other artifact. Readers older
+than split support refuse the manifest as an unsupported compression and keep
+their last good runtime mirror.
+
+Split exists for Git-hosted stores. Each gzip generation is a new compressed
+blob that cannot delta against its predecessor, so every hourly snapshot added
+about 68 MB to the store. Consecutive uncompressed parts share most bytes; a
+measured pair of consecutive production generations transferred as a 0.5 MB
+thin pack and added about 0.5 MB to a fully packed repository. Subscribers
+receive each new part as a full object; strict refresh's post-advance geometric
+repack then folds it into deltas (see below).
+
 Initialization validates portable arguments before invoking Git and validates
 the artifact before saving configuration. Repeated initialization and a
 publisher's raw-to-gzip transition do not require a raw `.db` in the checkout.
@@ -106,8 +124,10 @@ checkout files**, hidden index entries, unrelated origins, divergent history,
 active Git locks or operation state, orphan/temporary packs, hooks, filters,
 attributes, submodules, linked worktrees, alternate object stores, config
 includes/redirections and unsupported extensions. Resolve these deliberately
-outside the subscriber command. It never resets, cleans, prunes, repacks,
-reclones, deletes backups or sidecars, or invokes reader/doctor auto-repair.
+outside the subscriber command. It never resets, cleans, prunes, reclones,
+deletes backups or sidecars, or invokes reader/doctor auto-repair. The only
+repack is the bounded post-advance geometric repack of a split store described
+below; raw and gzip stores are never repacked.
 
 Admission examines every exposed Git configuration scope. Refusals identify
 the scope and unsafe category without printing keys or values. Single-valued
@@ -185,14 +205,28 @@ operator inspection before another strict refresh.
 
 JSON reports `stage`, `result` (`updated`, `no-op`, `refused`, `partial`),
 `before_commit`, `after_commit`, `target_commit`, `artifact_id`, `sha256`,
-`artifact_bytes`, `mirror_destination`, `mirror_result`, `capacity` and
-`elapsed_ms`. A refusal includes a bounded `reason` and exits nonzero. Stderr
+`artifact_bytes`, `mirror_destination`, `mirror_result`, `capacity`,
+`elapsed_ms` and, for split stores, `repack`. A refusal includes a bounded `reason` and exits nonzero. Stderr
 contains stage/elapsed/result diagnostics; stdout remains data. A no-op means
 the refs already match; a missing runtime may still be materialized. Git failures
 use fixed, credential-safe repository, authentication, connection or disk-space
 diagnostics when recognized, with general troubleshooting guidance otherwise;
 raw remote/helper stderr is never included.
 
+When the checkout's generation is split, strict refresh runs
+`git repack -d --geometric=2` inside the same deadline and growth budget, so
+new generations are stored as deltas against their predecessors
+(`repack: "geometric"`). This also covers objects fetched by an implicit reader
+fast-forward, which never repacks, and generations kept as fetched packs
+(for example with `fetch.unpackLimit`), so a no-op refresh can still
+consolidate. With nothing loose and a single pack it reports
+`repack: "current"` and does nothing. A larger
+history pack is left untouched. With more than 512 loose objects, such as years of loose gzip
+generations, refresh skips this (`skipped-loose-backlog`); pack that backlog
+once with an owner-run `git repack -d` first. A consolidation failure is reported
+as `repack: "failed"` with a bounded `repack_reason` and next step, without
+undoing the validated generation.
+
 All portable Git entry points explicitly set `maintenance.auto=false` and
 `gc.auto=0`, including init, implicit reader fetch/merge and retained recovery
 clone/reset helpers. Fetch also uses `--no-auto-maintenance`. Hooks, fsmonitor
@@ -350,7 +384,10 @@ host path. `--body-chars` defaults to `256`. `--max-bytes` is an optional,
 inclusive limit for raw database generations. `--compression gzip` instead
 commits only the gzip archive and manifest in the generation. Pair it with
 `--max-archive-bytes`; a Git host requiring an artifact smaller than
-100,000,000 bytes should pass `99999999`.
+100,000,000 bytes should pass `99999999`. `--compression split` commits the
+manifest and ordered uncompressed parts of 32 MiB (or `--max-archive-bytes`,
+when smaller); `--max-archive-bytes` then caps each part, and more than 64 parts
+is refused.
 The optional `--repository owner/repo` is semantic export behavior: Gitcrawl
 removes all other repositories and their dependent rows from the disposable
 snapshot, verifies the exact remaining identity, and records it in the manifest.
@@ -372,7 +409,9 @@ file SHA while retaining the same meaningful portable state.
 For gzip generations, the manifest also records `compression`, `archivePath`,
 `archiveBytes`, `archiveSha256`, and `maxArchiveBytes`. Gitcrawl validates the
 archive against both compressed and expanded identities before atomically
-committing the generation.
+committing the generation. Split generations record `archiveParts` instead of
+`archivePath`; JSON command output lists them as `archive_parts` with output
+paths.
 Destructive `portable prune` continues to record its operation time as
 `portable_metadata.exported_at`.
 

internal/cli/help.go

@@ -345,7 +345,7 @@ const portableUsageText = `gitcrawl portable manages local portable-store snapsh
 Usage:
   gitcrawl portable refresh --expected-remote URL [--store-dir PATH] [--portable-db PATH] [--branch main] [--git PATH] [--timeout 2m] [--min-free-bytes N] [--max-growth-bytes N] [--json]
   gitcrawl portable prune [--body-chars N] [--no-vacuum] [--include-sync-failures] [--no-publish] [--json]
-  gitcrawl portable export --profile current-state-v1 --output-dir PATH [--repository owner/repo] [--database-name NAME] [--public-path PATH] [--body-chars N] [--max-bytes N] [--compression gzip] [--max-archive-bytes N] [--consume-source] [--json]
+  gitcrawl portable export --profile current-state-v1 --output-dir PATH [--repository owner/repo] [--database-name NAME] [--public-path PATH] [--body-chars N] [--max-bytes N] [--compression gzip|split] [--max-archive-bytes N] [--consume-source] [--json]
 
 Subcommands:
   refresh             validate and fast-forward a clean configured subscriber
@@ -361,6 +361,10 @@ Export never changes the active database or publishes the artifact. It creates
 a complete database and manifest generation at a previously nonexistent path.
 With --compression gzip, it commits the gzip archive and manifest without the
 uncompressed database; --max-archive-bytes applies to that published archive.
+With --compression split, it commits the uncompressed database as ordered
+32 MiB parts (NAME.part-000, ...) and the manifest. Git can store a later
+generation as a small delta against these parts; --max-archive-bytes caps each
+part. Readers reassemble and verify every part and the whole database.
 --repository semantically restricts the disposable snapshot to one owner/repo.
 Refresh never regenerates config or invokes repair. It requires a matching
 origin, clean checkout and manifest-backed artifact. --git requires an absolute

internal/cli/portable_commands.go

@@ -44,7 +44,7 @@ func (a *App) runPortableExport(ctx context.Context, args []string) error {
 	publicPath := fs.String("public-path", "", "logical portable database path")
 	repositoryRaw := fs.String("repository", "", "restrict the artifact to one owner/repo")
 	maxBytesRaw := fs.String("max-bytes", "", "maximum finalized database bytes")
-	compression := fs.String("compression", "", "artifact compression (gzip)")
+	compression := fs.String("compression", "", "artifact packaging (gzip or split)")
 	maxArchiveBytesRaw := fs.String("max-archive-bytes", "", "maximum finalized archive bytes")
 	consumeSource := fs.Bool("consume-source", false, "consume an exclusively owned, backed-up database instead of copying it")
 	jsonOut := fs.Bool("json", false, "write JSON output")
@@ -89,8 +89,8 @@ func (a *App) runPortableExport(ctx context.Context, args []string) error {
 		}
 		maxArchiveBytes = &parsed
 	}
-	if *compression != "" && *compression != portableexport.CompressionGzip {
-		return usageErr(fmt.Errorf("--compression must be %q", portableexport.CompressionGzip))
+	if *compression != "" && *compression != portableexport.CompressionGzip && *compression != portableexport.CompressionSplit {
+		return usageErr(fmt.Errorf("--compression must be %q or %q", portableexport.CompressionGzip, portableexport.CompressionSplit))
 	}
 	if maxArchiveBytes != nil && *compression == "" {
 		return usageErr(fmt.Errorf("--max-archive-bytes requires --compression"))

internal/cli/portable_manifest.go

@@ -2,9 +2,11 @@ package cli
 
 import (
 	"context"
+	"crypto/sha256"
 	"encoding/json"
 	"errors"
 	"fmt"
+	"io"
 	"os"
 	"path/filepath"
 	"strings"
@@ -26,6 +28,73 @@ type portableDBManifest struct {
 	ArchivePath       string `json:"archivePath,omitempty"`
 	ArchiveBytes      int64  `json:"archiveBytes,omitempty"`
 	ArchiveSHA256     string `json:"archiveSha256,omitempty"`
+	// Split archives list ordered uncompressed parts beside the manifest.
+	ArchiveParts    []portableArchivePart `json:"archiveParts,omitempty"`
+	MaxArchiveBytes int64                 `json:"maxArchiveBytes,omitempty"`
+}
+
+type portableArchivePart struct {
+	Path   string `json:"path"`
+	Bytes  int64  `json:"bytes"`
+	SHA256 string `json:"sha256"`
+}
+
+const (
+	portableMaxArchiveParts = 64
+	// Each committed part stays below the hosting per-file limit.
+	portableMaxArchivePartBytes = 99999999
+)
+
+// portableArchivePartPaths returns the ordered part paths of a split archive
+// whose first part is firstPart (as returned by portableSourceArtifact).
+func portableArchivePartPaths(firstPart string, manifest portableDBManifest) []string {
+	paths := make([]string, 0, len(manifest.ArchiveParts))
+	for _, part := range manifest.ArchiveParts {
+		paths = append(paths, filepath.Join(filepath.Dir(firstPart), part.Path))
+	}
+	return paths
+}
+
+func portableSplitArchive(manifest portableDBManifest) bool {
+	return strings.TrimSpace(manifest.Compression) == portableexport.CompressionSplit
+}
+
+func validatePortableSplitMetadata(manifest portableDBManifest) error {
+	if strings.TrimSpace(manifest.ArchivePath) != "" {
+		return fmt.Errorf("portable manifest mismatch: split archive must not name archivePath")
+	}
+	if len(manifest.ArchiveParts) == 0 || len(manifest.ArchiveParts) > portableMaxArchiveParts {
+		return fmt.Errorf("portable manifest mismatch: split archive must have 1..%d parts", portableMaxArchiveParts)
+	}
+	if manifest.OutputBytes <= 0 || manifest.ArchiveBytes != manifest.OutputBytes ||
+		!strings.EqualFold(strings.TrimSpace(manifest.ArchiveSHA256), strings.TrimSpace(manifest.SHA256)) ||
+		strings.TrimSpace(manifest.SHA256) == "" {
+		return fmt.Errorf("portable manifest mismatch: split archive must equal the database size and sha256")
+	}
+	var total int64
+	seen := map[string]bool{}
+	for index, part := range manifest.ArchiveParts {
+		name := part.Path
+		if name == "" || name == "." || name == ".." || strings.ContainsAny(name, "/\\\x00:") || seen[name] {
+			return fmt.Errorf("portable manifest mismatch: archive part %d path must be a unique basename", index)
+		}
+		seen[name] = true
+		if part.Bytes <= 0 || part.Bytes > portableMaxArchivePartBytes ||
+			manifest.MaxArchiveBytes > 0 && part.Bytes > manifest.MaxArchiveBytes {
+			return fmt.Errorf("portable manifest mismatch: archive part %d size is outside its budget", index)
+		}
+		if len(strings.TrimSpace(part.SHA256)) != 64 {
+			return fmt.Errorf("portable manifest mismatch: archive part %d sha256 missing", index)
+		}
+		if total > manifest.OutputBytes-part.Bytes {
+			return fmt.Errorf("portable manifest mismatch: archive parts exceed outputBytes")
+		}
+		total += part.Bytes
+	}
+	if total != manifest.OutputBytes {
+		return fmt.Errorf("portable manifest mismatch: archive parts total %d != %d", total, manifest.OutputBytes)
+	}
+	return nil
 }
 
 func portableDBManifestPath(dbPath string) string {
@@ -153,8 +222,21 @@ func portableSourceArtifact(dbPath string) (string, portableDBManifest, bool, er
 		return "", portableDBManifest{}, false, fmt.Errorf("portable manifest mismatch: %w", err)
 	}
 	if !ok || strings.TrimSpace(manifest.Compression) == "" {
+		if ok && len(manifest.ArchiveParts) > 0 {
+			return "", portableDBManifest{}, false, fmt.Errorf("portable manifest mismatch: archiveParts require split compression")
+		}
 		return dbPath, manifest, false, nil
 	}
+	if portableSplitArchive(manifest) {
+		if err := validatePortableSplitMetadata(manifest); err != nil {
+			return "", portableDBManifest{}, false, err
+		}
+		// Callers receive the first part; portableArchivePartPaths lists all.
+		return filepath.Join(filepath.Dir(manifestPath), manifest.ArchiveParts[0].Path), manifest, true, nil
+	}
+	if len(manifest.ArchiveParts) > 0 {
+		return "", portableDBManifest{}, false, fmt.Errorf("portable manifest mismatch: archiveParts require split compression")
+	}
 	if strings.TrimSpace(manifest.Compression) != "gzip" {
 		return "", portableDBManifest{}, false, fmt.Errorf(
 			"portable manifest mismatch: unsupported compression %q",
@@ -200,6 +282,9 @@ func validatePortableArchive(path string, manifest portableDBManifest) error {
 }
 
 func validatePortableArchiveContext(ctx context.Context, path string, manifest portableDBManifest) error {
+	if portableSplitArchive(manifest) {
+		return validatePortableSplitArchiveContext(ctx, path, manifest)
+	}
 	info, err := os.Stat(path)
 	if err != nil {
 		return err
@@ -225,3 +310,39 @@ func validatePortableArchiveContext(ctx context.Context, path string, manifest p
 	}
 	return nil
 }
+
+// validatePortableSplitArchiveContext checks every part's size and digest and
+// the digest of their ordered concatenation.
+func validatePortableSplitArchiveContext(ctx context.Context, firstPart string, manifest portableDBManifest) error {
+	if err := validatePortableSplitMetadata(manifest); err != nil {
+		return err
+	}
+	whole := sha256.New()
+	for index, path := range portableArchivePartPaths(firstPart, manifest) {
+		part := manifest.ArchiveParts[index]
+		info, err := os.Stat(path)
+		if err != nil {
+			return err
+		}
+		if !info.Mode().IsRegular() || info.Size() != part.Bytes {
+			return fmt.Errorf("portable manifest mismatch: archive part %d size %d != %d", index, info.Size(), part.Bytes)
+		}
+		file, err := os.Open(path)
+		if err != nil {
+			return err
+		}
+		hash := sha256.New()
+		written, copyErr := io.Copy(io.MultiWriter(hash, whole), portableContextReader{ctx: ctx, reader: io.LimitReader(file, part.Bytes+1)})
+		closeErr := file.Close()
+		if err := errors.Join(copyErr, closeErr); err != nil {
+			return err
+		}
+		if written != part.Bytes || !strings.EqualFold(fmt.Sprintf("%x", hash.Sum(nil)), strings.TrimSpace(part.SHA256)) {
+			return fmt.Errorf("portable manifest mismatch: archive part %d sha256 mismatch", index)
+		}
+	}
+	if sum := fmt.Sprintf("%x", whole.Sum(nil)); !strings.EqualFold(sum, strings.TrimSpace(manifest.ArchiveSHA256)) {
+		return fmt.Errorf("portable manifest mismatch: archive sha256 %s != %s", sum, manifest.ArchiveSHA256)
+	}
+	return nil
+}

internal/cli/portable_recovery.go

@@ -173,8 +173,12 @@ func preserveMalformedPortableDB(root, dbPath string) (string, error) {
 		dbPath + "-shm",
 		dbPath + ".manifest.json",
 	}
-	if archivePath, _, compressed, err := portableSourceArtifact(dbPath); err == nil && compressed {
-		paths = append(paths, archivePath)
+	if archivePath, manifest, compressed, err := portableSourceArtifact(dbPath); err == nil && compressed {
+		if portableSplitArchive(manifest) {
+			paths = append(paths, portableArchivePartPaths(archivePath, manifest)...)
+		} else {
+			paths = append(paths, archivePath)
+		}
 	}
 	for _, path := range paths {
 		if _, err := os.Stat(path); err != nil {