Skip to content

ci: promote a released main image with the reviewed N0 compose template - #315

Merged
samgbafa merged 5 commits into
mainfrom
skgbafa/tc-959-n0-release-template
Oct 18, 2026
Merged

samgbafa merged 5 commits into
mainfrom
skgbafa/tc-959-n0-release-template

Conversation

@samgbafa

@samgbafa samgbafa commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Tracking: TC-959 (https://linear.app/tinycloud-labs/issue/TC-959), TC-1012 (body cap)

Scope and merge order

Rebased onto main 97a2136e (includes #313, #309, #346, #349, #343, #318). Two commits: release-image template lock and the 95m body cap with template rebind. Re-pushing invalidates any earlier approval, so exact-head review is needed. No merge or deployment authorization is requested here.

This PR's own changes: .github/workflows/docker.yml (the Validate and pin deployment image step), scripts/ci/test-deploy-recovery.py, scripts/ci/fixtures/tc959-n0-compose.yaml, docs/releasing.md, a single row in docs/runbooks/tc-959-n0-log-shipping.md, and one line of docker-compose.dstack-postgres.yaml: CLIENT_MAX_BODY_SIZE=32m → 95m. That turns #309's approved raw main's #349 compose c33c23021589163b62a71ef245b7194717711922027629f155bdd5cff806b8c7 into d95bfb707871c1d626c4a8f2b8d32d354103edbb5077847c6a21fe5f7f6cfa50 (2 bytes, offsets 5454–5455). No app/Rust changes. Node and ingress-image body-limit changes are in TC-1012's separate PR.

Change

#313 locks the N0 compose to its raw bytes, so it can only deploy the 1.20.6 image. This PR locks it as a template instead: the final compose bytes with the one node digest normalized to 64 zeros, SHA256 51f8e9e93edc0e16025d27ee42abe003ed5d1cf26ba8f437d5120894aca64bdb. The sed command to reproduce it is in docs/releasing.md.

  • Released image: a node digest other than 1.20.6 fills the template only for a recorded digest promotion on main. The substituted digest must equal deploy_image_digest. Labels must be X.Y.Z-dstack / R, the recorded inputs must equal those labels, tag vX.Y.Z must point at R, Cargo.toml at R must report X.Y.Z, and R must be in main history (existing guard). An untagged, mislabelled, feature-variant or freshly built image is refused before mutation.
  • Deployed compose: must re-normalize to the template, with exactly one node pin, equal to the selected digest. Ingress, socat, Vector and profile bytes stay exact. Add a compose-only node log sidecar #309's approved pre-cap bytes (ca110b0b…) no longer satisfy the template.
  • Unchanged: same-image 1.20.6 recovery (the current compose, or the pre-N0 template/pinned 28302d67…/df239749…); captured pre-N0 rollback (df239749… plus the explicit privacy mode, still 32m); the forward non-N0 route; sealed preflight; token sealing with the allowed_envs sync; general quota reconciliation; dispatch-only and main-only guards.

This also fixes main's current version-guard failures: on main 199a9e3b the recovery suite fails 2/17 (test_future_digest_…, test_normal_build_deployment_…), because those tests deploy main's root compose, which is now N0, with a non-1.20.6 image. They now use an explicit pre-N0 template fixture. Main's failure blocks the tag image build.

One-dispatch release command (NOT executed)

Run this once this lineage and chore(release): 1.21.0 (tagged v1.21.0) are on main:

gh workflow run docker.yml -R TinyCloudLabs/tinycloud-node --ref main \
  -f deploy_phala=true \
  -f authorize_quota_values=true \
  -f image_version=1.21.0 \
  -f deploy_image_digest=sha256:<release-dstack-digest> \
  -f rollback_image_version=1.21.0 \
  -f rollback_image_revision=<release-commit-sha> \
  -f sql_identity_fence=false

Rolling the release back = DB snapshot restore (a separate decision) + the documented captured pre-N0 rollback to 1.20.6, with the explicit privacy mode.

Evidence (tests first)

  • Red before green: the new release tests failed against ci: preserve recovery routes and reconcile quota authority #313's workflow (3 failures). The 95m fixture with the old hash failed 5 tests, then passed after the rebind.
  • On head 970a34f: recovery workflow suite 20/20 (adds release substitution; truthful-release refusals for untagged / tag elsewhere / crate mismatch / build path; template-drift refusals for profile / ingress digest / approved pre-cap bytes / CRLF / duplicate node / quoted node); quota authority 20/20; Docker tags 8/8; deploy-ref 16/16; Policy probes 13 scenarios; scripts/tc959-check.mjs pass; log-shipper/tc959 node tests 32/32; actionlint v1.7.12 clean (shellcheck/pyflakes unavailable/disabled).
  • The root compose equals the fixture copy, byte for byte. Main alone: recovery 2/17 FAIL (reproduced).

No production/Phala/billing request, credential read, workflow dispatch, merge, or push to #309/#313.

Rendered release compose (harness, release digest substituted): differs from main's compose only by the node image line (the release digest, no 4bdae4fc) and CLIENT_MAX_BODY_SIZE 95m. TINYCLOUD_EPOCH_HEAD_BACKFILL__ENABLED: "false" and #346's Vector config are retained.

@samgbafa
samgbafa force-pushed the skgbafa/tc-959-n0-release-template branch 2 times, most recently from ea5d7c0 to 970a34f Compare October 17, 2026 04:15
samgbafa added a commit that referenced this pull request Oct 17, 2026
Same bytes (sha256 28302d67...) as the frozen copy, without adding a fixture
file; matches the derivation used by #315.
samgbafa added a commit that referenced this pull request Oct 17, 2026
* test(TC-959): use pre-N0 template fixture in recovery tests

Future-digest and normal-build recovery cases ran against the repository's
current compose. Since #309 that file is the reviewed N0 compose, which the
workflow intentionally refuses to pair with any image other than the reviewed
1.20.6 digest. Run these cases against a frozen copy of the pre-N0 template
(sha256 28302d67...) instead, and assert the N0 compose refuses a normal build
image as it already refuses a future digest.

* test(TC-959): derive pre-N0 template from the rollback artifact

Same bytes (sha256 28302d67...) as the frozen copy, without adding a fixture
file; matches the derivation used by #315.
Lock #309's compose as a template (raw bytes with the single node digest
normalized) so the main release and the sidecar ship in one dispatch.
Only a recorded released image (labels, vX.Y.Z tag and crate version
agree) may fill the pin; same-image recovery and the captured pre-N0
rollback are unchanged.

Refs TC-959
Raise CLIENT_MAX_BODY_SIZE 32m -> 95m in the root compose (#309's
approved ca110b0b… becomes b0cd8248…, 2 bytes) and lock the release
template over those bytes: 92641ed5…. #309's approved pre-cap bytes no
longer satisfy the template. Pre-N0 template and captured rollback
artifacts are unchanged.

Refs TC-959, TC-1012
@samgbafa
samgbafa force-pushed the skgbafa/tc-959-n0-release-template branch from 970a34f to f70f2b9 Compare October 18, 2026 10:22
@samgbafa

Copy link
Copy Markdown
Contributor Author

REQUEST_CHANGES: see TC-1176 (reviewed exact head f70f2b9)

Head f70f2b9 / base 97a2136 verified. CI is terminal and green: 24 check runs, all success or skipped. Local test-deploy-recovery.py (20), test-quota-authority.py (25), test-docker-tags.py and test-check-deploy-ref.sh all pass.

Blocker: require_release() admits any tagged release, including downgrades. It checks that the labels, the vX.Y.Z tag, the crate version and main history all agree. It never checks that the promoted revision descends from the running production revision. The "Guard production ancestry" step only checks running → HEAD, the main checkout, so it doesn't cover the promoted image either. Concrete case: ghcr.io/tinycloudlabs/tinycloud-node@sha256:ec555ec7ac7e98b05cd793a6d1a2b1a098a2653587c5564c171d3c87008a0dec has labels 1.20.5-dstack / f50a9b3b…, which is the v1.20.5 commit on main with crate 1.20.5. It passes every check and would deploy 1.20.5 under the N0 template, below the running 1.20.6. The test harness confirms this: 1.20.5 and 1.0.0 inputs reach cvm-deploy. Fix: in the release branch, require git merge-base --is-ancestor <running revision> <release revision> with the two revisions unequal (export RUNNING_REVISION from the ancestry step), or pin the version to 1.21.0. Add a refusal test for an older tagged release.

Verified OK:

  • Hashes recomputed. Head compose raw is d95bfb70…fa50, byte-identical to the fixture. Normalized it is 51f8e9e9…4bdb, matching the workflow constant, releasing.md and the runbook. Main's raw is c33c2302….
  • A 1.21.0 render differs from main only in the node image line and CLIENT_MAX_BODY_SIZE 32m→95m. TINYCLOUD_EPOCH_HEAD_BACKFILL__ENABLED: "false", the Vector service and config, ingress, and socat are all byte-identical.
  • The following are refused: template drift, a duplicate or quoted pin, CRLF, a feature-variant label, a missing or moved tag, a crate mismatch, the build path, and a digest whose labels disagree.
  • 1.20.6 same-image recovery and rollback_pre_n0 still pass.

Non-blocking:

  1. Image identity is bound by OCI labels, not by a recorded digest. Anyone who can push to the GHCR package can forge the labels. Consider pinning the 1.21.0 digest once it is released, or checking it against the 1.21.0-dstack tag.
  2. Once 1.21.0 is live, the 12c same-image command still accepts 1.20.6. That would be a cross-migration downgrade with no guard.
  3. The template comment still says "unchanged production 1.20.6 binary".
  4. With main's authority: node-override (ops(TC-1164): flip quota authority to billing #323 is still open), authorize_quota_values=true PUTs node overrides after deploy. ops(TC-1164): flip quota authority to billing #323 must land first if billing should be authoritative. It also edits docs/releasing.md, so expect a rebase.

…e recovery

The promoted revision must strictly descend from the running production
revision; 12c same-image recovery refuses an image older than running.
The captured pre-N0 rollback stays the intentional downgrade route.

Refs TC-959
samgbafa added a commit that referenced this pull request Oct 18, 2026
* TC-1012 cap /invoke upload bodies at 95 MiB with 413

* TC-1012 CHANGELOG: release compose line rides with #315
@samgbafa

Copy link
Copy Markdown
Contributor Author

APPROVE: see TC-1176 (reviewed exact head 62e07e8)

@samgbafa
samgbafa merged commit e6ccb0a into main Oct 18, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant