JITSU-227: Object-backed Reverse ETL persistence

[absorbb]

Stack / merge order

Stacked on #1525 (feat/jitsu-227-console-ui). Merge #1525 first, then retarget
this PR to newjitsu before merging. This branch now contains both the console UI
and object-backed runner for combined local testing; this PR's diff includes persistence/deployment, delivery observability, and the
opt-in Google full-audience replacement feature described below.

Restacked on the recovered #1525 tip d55901592, including its 14 previously
unpushed local commits. Preserves editor/preview, consent, scheduling, navigation,
ClickHouse spooling and snapshot-progress fixes. Conflict resolution retains the
object-backed backend, both safe failure paths, controller overrides and the
existing MicroTime fix; the obsolete SQL snapshot writer is not restored.
Includes safe startup diagnostics and the dev syncctl tunnel fix. Local ignored
configuration and dev-only Helm edits remain separate. No deployment was performed
as part of this recovery/restack.

Summary

JITSU-227

Replaces Reverse ETL per-row PostgreSQL persistence with local SQLite indexing and
GCS/S3 immutable snapshots, batches and receipts. PostgreSQL retains only control,
audience ownership, checkpoints and task metadata.

• Removes six payload tables: reverse_sync_batch, reverse_sync_operation,
  reverse_sync_generation, reverse_sync_source_key, reverse_sync_desired,
  reverse_sync_membership.
• Removes five obsolete control accounting columns, the unused action enum, the
  legacy backend/fallback and its SQL retention sweeper.
• Object storage is required. No per-source-row/member/operation PostgreSQL writes.
• One small artifact_head pointer commits the immutable manifest together with
  lifecycle/store/checkpoint transitions. No object I/O inside SQL transactions.
• New control rows atomically receive a previously uploaded empty manifest pointer.
  Legacy null-head rows fail closed even in phase new; interrupted new startup
  remains recoverable.
• Exact batches are durable before provider calls. Recovery reconstructs SQLite
  from baseline + receipts, preserving partial success and sequence tombstones.
  Unknown provider outcomes still require reconciliation, not blind replay.
• Bounded, checksummed, scope-bound artifacts; conditional immutable GCS/S3 writes;
  byte-bounded snapshot/baseline files; cancellation and operation-ID guards.
• Destination interfaces and syncctl recovery flow stay intact. Lease renewal
  covers restoration and compaction.
• Existing lifecycle/mirror tests now run against the sole artifact backend.
  Parity fixes cover omitted checkpoint cursors, interrupted finish resolution,
  cancelled staging and competing phase transitions.
• Helm deployment wiring included: opt-in reverseEtl values for runner image,
  existing runtime Secret, dedicated runner service account and cloud identity
  annotations; namespace-scoped lease-only RBAC, separate from connector jobs.
• Configurable CPU/memory/ephemeral-storage requests and limits plus disk-backed
  /tmp size limit for scheduled, manual and recovery runs. syncctl validates
  settings on startup; required store/bucket Secret keys prevent missing-storage starts.
• Helm value validation, offline render tests (Helm 3/4), and syncctl unit tests in
  a dedicated CI workflow. Cached generated Zod output is cleared before regeneration.

Google full audience replacement and delivery observability

• New-sync setup offers Mirror · full replacement alongside existing snapshot-diff
  mirroring. Supported for managed and existing owned audiences, with explicit exclusive
  management/takeover confirmation. Existing sync configurations are not converted.
• Capture and durably save Google's cutoff before uploads; upload every unique desired
  member, including unchanged members; wait for all uploads to be accepted before
  submitting audienceMembers:removeAll with that original removeAsOfTime.
• Persist the cleanup receipt and poll it on status-check jobs without re-extracting or
  re-uploading. Missing/ambiguous cleanup receipts block replay. Promote and prune the
  durable baseline only after confirmed cleanup.
• Replacement is asynchronous, not atomic. A successful empty snapshot clears the
  audience; no other tools/users may upload to an exclusively managed target.
  Failed extraction or rejected uploads never authorize cleanup.
• No additional schema changes for replacement. Google does not expose a removed-member
  count for cleanup; logs explicitly distinguish uploaded/accepted records from cleanup
  status and matched/targetable audience size.
• Adds source/deduplication/comparison counts, short delivery-stage updates, one cumulative
  delivery summary per attempt, and actionable redacted failures. Pending Google work is
  presented as a status refresh rather than recovery.

Latest validation

• Runner: 265 passed, one opt-in scale test skipped; typecheck and bundled build pass.
• Destination functions: 209 tests passed; Google replacement coverage uses mocked HTTP.
• Console: 33 focused Reverse ETL integration tests and 136 unit tests passed; typecheck passes.
• Covers all-upload gating, empty replacement, lost cleanup receipts, status-only retries,
  pruning/restoration, configuration confirmation and unchanged default behavior.
• No live audience changes or deployment performed for this feature; deployment remains manual.

Rollout — destructive schema cutover

• Manual deployment. No deploy:console label. Runner deploys separately.
• Pause schedules and drain old Reverse ETL workers before applying Prisma.
  Back up the database; reconcile pending provider requests with the old
  runner/schema; explicitly retire/reset test syncs and audiences first.
• Applying the schema drops the six old tables and five columns. Prisma may require
  explicit data-loss confirmation. Do not bypass that in unattended deployment.
• Explicit test-sync reset, no migration or automatic deletion, as requested.
  Prefer a fresh disabled test sync with a new empty Jitsu-managed audience.
  Deleting old state alone loses the removal baseline and is not a safe reset.
• Configure RETL_OBJECT_STORE / RETL_OBJECT_BUCKET, then deploy the new runner
  and verify storage before enabling new syncs. Node >=22.13; Node24 recommended.
• Old workers cannot run after the schema cutover. Rollback requires coordinated
  backup restoration and provider reconciliation, not just unsetting storage.
• No automatic object GC yet. Do not use age-only bucket deletion; referenced
  baselines and unresolved batches must survive. Monitor orphan/bucket growth.
• Existing checkouts may retain six ignored generated Zod files for removed models;
  cleanup instructions are in services/retl-runner/src/artifacts/README.md.
• This restack does not deploy or change live state. Earlier explicitly authorized
  local diagnostics rebuilt the runner and reset the paused test sync; they are
  not an automatic migration/deployment step provided by this PR.

See services/retl-runner/src/artifacts/README.md for detailed cutover, permissions,
storage configuration and retention precautions. helm/REVERSE_ETL.md documents
GCS/EKS/Minikube authentication, runtime Secret keys, resources, and the pre-upgrade
schema hook hazard: pause/drain before invoking Helm, not as part of its upgrade.

Verification

• Recovery/restack validation: console and runner TypeScript checks pass; runner
  bundle builds; 229 runner tests pass (1 opt-in scale test skipped), 90 console
  integration tests pass with the reduced schema, and all 136 console unit tests
  pass. Warehouse-query: 170 tests pass. The 9 committed Helm tests and strict lint
  pass. Scoped runtime/Helm reviews, verification of the corrected storage quickstart,
  and fresh final review cover the recovered changes and conflict resolutions.
• 222 runner tests passed, including the opt-in million-identity SQLite test.
• Runner TypeScript check and bundled build pass.
• Console TypeScript check passes after Prisma regeneration and removal of obsolete
  ignored generated model files.
• Tests use disposable PostgreSQL with the reduced Prisma schema, local SQLite,
  immutable test objects, the real S3 SDK against a local HTTP service, and GCS doubles.
• Coverage includes missing legacy tables/counters, legacy reset rejection in every
  phase, lost startup/commit responses, restart, partial failure, staged finish
  continuation, phase conflicts, async mirrors/removals, corruption, duplicate
  operation IDs and snapshots larger than one artifact.
• Incremental local runtime/schema reviews and a fresh final pre-push review
  completed clean for 6061f083..18ce2f97.
• Current deployment follow-up: full syncctl unit suite passes (go test -vet=off ./... -count=1; pre-existing fmt.Errorf vet error is not changed).
• Strict Helm lint and all 7 offline chart contract tests pass on both Helm 3.18.4
  and Helm 4.2.2. Coverage includes disabled defaults, required configuration,
  GKE/EKS annotations, external service accounts, least-privilege RBAC, resource
  propagation, duplicate/conflicting env rejection and no automatic data-loss flag.
• Previous revision also passed the Node24 bundled-SQLite smoke test.

No live GCS/S3 bucket or advertising API was called during validation.

[jitsu-code-review]

Reviewed the object-backed Reverse ETL persistence implementation, including artifact integrity/scoping, storage transports, lifecycle transitions, recovery, and runtime secret wiring. No actionable correctness or security findings.

[jitsu-code-review]

Reviewed the object-storage artifact layer, recovery lifecycle, runner configuration, and Prisma schema cutover.

Finding:

• Existing Reverse ETL control rows are left permanently rejected after the schema update because they have no artifact head.

[jitsu-code-review]

Reviewed the object-backed Reverse ETL persistence cutover, recovery/state transitions, cloud storage adapters, sync-controller pod settings, Helm wiring, and Prisma schema changes.

No new actionable correctness, security, or user-visible regression findings. Validation run: Helm lint and contract tests; sync-controller Go tests; retl-runner typecheck, build, and test suite (221 passed, 1 optional scale test skipped).

[jitsu-code-review]

Reviewed the object-backed Reverse ETL persistence cutover, recovery/snapshot lifecycle, artifact transport safeguards, runner deployment wiring, and Helm validation. No new actionable bugs, security issues, or user-visible regressions found.

[jitsu-code-review]

Reviewed the Reverse ETL object-backed persistence, controller/Helm deployment wiring, and console setup/run-management paths.

No actionable bugs, security issues, or user-visible regressions found. I also checked the existing resolved review thread and did not re-raise it. Local Helm lint and the Reverse ETL Helm contract suite pass.

[jitsu-code-review]

Reviewed the object-backed Reverse ETL persistence transition, recovery and artifact integrity paths, runtime storage configuration, and Helm/controller deployment wiring. No additional actionable findings.

[jitsu-code-review]

Reviewed the object-backed Reverse ETL persistence cutover, recovery and full-replacement paths, plus sync-controller/Helm runtime wiring.

No new actionable findings found. Existing resolved discussion about legacy control rows was considered and is not repeated.