feat(storage): migrate control plane to SQLite

[PascaleBeier]

Summary

• make the 2.13 default-tenant split mandatory and move all tenant analytics into the shared attached-catalog DuckDB data plane
• introduce a pure-Go SQLite control plane in internal/controlstore using modernc.org/sqlite
• convert compacted legacy DuckDB control databases through a verified, crash-safe publication protocol
• preserve the configured HITKEEP_DB_PATH while retaining the post-split DuckDB source as migration recovery evidence
• update control backups, recovery, storage reporting, runtime types, dashboard copy, documentation, and acceptance workflows

Architecture and migration safety

The default tenant DuckDB file remains the root of the single tenant-data DuckDB instance. Non-default tenant files continue to use deterministic attached catalogs under the existing global memory and connection gates.

The new SQLite control store owns control-only schema, queries, migrations, health, takeout, and backup behavior. It uses WAL mode, full synchronization, foreign-key enforcement, bounded connections, a busy timeout, file-backed temporary storage, and no CGO dependency.

Legacy conversion:

1. identifies the configured database format without modifying it
2. finishes the durable default-tenant split and cleanup when the source is DuckDB
3. classifies and copies every legacy control table into a SQLite work file
4. verifies counts, canonical row fingerprints, foreign keys, integrity, and migration markers
5. atomically retains the DuckDB source and publishes SQLite
6. resumes safely across partial publication states

Control backups now use validated SQLite snapshots, manifests, checksums, and completion markers. Tenant backups remain catalog-scoped DuckDB snapshots. Incomplete backup cycles are not considered restorable.

Operator impact

• 2.13 is a mandatory, non-downgradable storage migration without restoring a complete pre-upgrade backup.
• HITKEEP_DB_PATH now points to the SQLite control database.
• DuckDB memory and thread settings apply only to the tenant data plane.
• Mutable snapshot publishing is reduced; destructive/private migration acceptance remains manual and on demand.
• The former control DuckDB is retained as <db-path>.pre-sqlite-2.13.0 and is never automatically overwritten or deleted.

Validation completed

• go test -p 2 ./... -count=1
• go test -p 2 -tags billing ./... -count=1
• targeted race suites for controlstore, worker, command startup, and attached tenant lifecycle
• CGO_ENABLED=0 go test ./internal/controlstore -count=1
• private-acceptance build-tag compilation
• source diff checks, module tidiness, and locale JSON validation

Required before release

This draft must not be released until all manual gates pass:

• client, EU, and US production-copy migration fixtures
• exact control and tenant fingerprints plus representative analytics queries
• regional S3 backup and clean-directory restore
• idempotent second startup
• peak RSS at or below 1 GiB and temporary-disk limits
• PR-parity, exhaustive, image, race, and cloud-tagged release profiles

No client database, production identifier, credential, or private diagnostic is included in this PR.

[PascaleBeier]

This PR is AI generated and a POC