test(kubernetes): verify external-worker restart and quiesced state restore

[yanpgwang]

Problem and result

Operators need evidence that the control-plane chart can preserve work and state before a Kubernetes alpha is published. This delivery installs the real chart in isolated Kubernetes 1.37.0, connects an external operator Docker worker, replaces API/orchestration during a pending confirmation, and proves same-release quiesced recovery into independent PostgreSQL/S3 stores.

The system test preserves public IDs, File/Skill bytes and checksums, Memory Versions and original encryption keys. Read-only Temporal queries establish the original Run ID and complete history prefix after restore, then prove advancement of that same execution after the original custom-tool result; duplicate submission returns 409. Reconstructing a new Workflow from Mango rows cannot pass. Existing Vault GET verifies/decrypts the restored envelope before returning secret-free metadata. The stateless Messages endpoint is explicitly simulated; actual Mango HTTP, adapter, state services and sandbox execution are exercised.

Fixtures own unique projects, cluster, images and workspace volumes and use an explicit kubeconfig. Supervisor receives an Environment-scoped key; no provider credential is needed in CI. Shutdown covers the full worker grace; fallback removal rechecks exact container identity and an uncertain stop aborts backup. A paused orphan/unrelated-item probe verifies cleanup safety. Read-only transport failures during single-replica replacement are retried within a bound; mutation/admission is never automatically retried. Production runtime, persistence schema, HTTP/OpenAPI and SDK surfaces are unchanged.

Scope and evidence

• Add required Kubernetes lifecycle CI with verified pinned kind 0.33.0, kubectl/Kubernetes 1.37.0 and Helm 4.3.0. The existing Required gate now includes it.
• Document exact exercised topology, consistent backup boundaries, original key preservation and independently owned sandbox workspace storage. Clarify existing Skill/read prerequisite.
• No published image/chart claim, general HA, live snapshots, cross-version upgrades/rollback or cloud sandbox. Immutable publication remains the last alpha delivery.
• Design and rationale: docs/design/kubernetes-recovery.md, docs/design/kubernetes-alpha.md, docs/provenance.md. Operator recovery is Mango-owned; CMA hosted rollout policy is not inherited.

Validation

• make test-kubernetes KIND=... KUBECTL=... HELM=... — corrected full acceptance passed290.34s, including original Temporal execution/history continuation and paused orphan/unrelated-item cleanup. Final CI runs the complete tier at this PR head.
• make test-service-core MANGO_TEST_S3_ENDPOINT=http://localhost:19000 — passed, isolated schemas and temporary SeaweedFS; existing local stack unchanged.
• make verify — lint/unit/race/vet passed; make chart-check — six contracts passed; Actions validation passed; make docs-check —62pages/links/anchors/exports passed.
• scripts/with-dev-env make test-self-hosted-live — actual model → authenticated HTTP → external Docker worker passed5.04s, no credential output.
• Fixtures observed RED before missing model/cluster/restore implementations; actual HTTP tests caught and corrected fixture assumptions about Skill prerequisites, CLI tabular output, event pagination/status shapes and endpoint convergence.
• One final build attempt failed before cluster creation at a local Debian HTTP mirror/proxy timeout; unchanged retry passed. No daemon/proxy settings or runtime Dockerfile change.
• Two independent read-only whole-PR reviewers approved after fixes for the Temporal proof gap and failure cleanup budget; offline adversarial identity/history regressions passed with race detection. All required exact-head CI remains the merge gate.