diff --git a/docs/content/hive/release-channels.md b/docs/content/hive/release-channels.md index 3c1f724..e0b6dba 100644 --- a/docs/content/hive/release-channels.md +++ b/docs/content/hive/release-channels.md @@ -10,7 +10,7 @@ Hive publishes three **release channels** — moving GHCR image tags an operator | `candidate` | A build believed good, awaiting soak before promotion to stable. | | `edge` | The newest good build, with no soak period. | -> **Promotion policy:** the channels diverge by release line and maturity. Every green merge to **`v5`** retags **`candidate`** (and `:latest`); **`stable`** advances later by digest through the scheduled/manual stable-promotion workflow after the [stable soak and promotion policy](https://github.com/hivecommons/hive/blob/v5/src/docs/stable-soak-policy.md) passes. Merges to **`v6`** retag **`edge`**, so `edge` is an active-development v6 build, not a synonym for `stable`. **`v4`** is a maintenance line: its builds publish only `v4-latest` and short-SHA tags, no channel (#7721 Phase 1). +> **Promotion policy:** the channels diverge by release line and maturity. Every green merge to **`v5`** retags **`candidate`** (and `:latest`); **`stable`** advances later by digest through the scheduled/manual stable-promotion workflow after the [stable soak and promotion policy](/docs/hive/stable-soak-policy) passes. Merges to **`v6`** retag **`edge`**, so `edge` is an active-development v6 build, not a synonym for `stable`. **`v4`** is a maintenance line: its builds publish only `v4-latest` and short-SHA tags, no channel (#7721 Phase 1). The hub's release-channel block also shows the stable auto-promotion state. Hub admins see a play/pause control on the `stable` row: play (the default) @@ -23,9 +23,9 @@ a read-only badge. Channels are **retags, not rebuilds**. Each release line's `docker.yml` workflow adds fast-moving channels as extra tags in the same `docker buildx imagetools create` call that publishes the branch's `-latest` and immutable short-SHA tags, so a channel always points at an already-built, multi-arch digest. Builds of branch `v5` publish `candidate`; the separate stable-promotion workflow later retags `stable` to the digest for the newest build that crossed the 24-hour line after the soak gate passes. Builds of branch `v6` publish `edge`. All three images get their line's channels in both published orgs: -- `ghcr.io/hivecommons/hive` and `ghcr.io/hivecommons/hive` -- `ghcr.io/hivecommons/hive-contributor` and `ghcr.io/hivecommons/hive-contributor` -- `ghcr.io/hivecommons/hive-hub` and `ghcr.io/hivecommons/hive-hub` +- `ghcr.io/hivecommons/hive` and `ghcr.io/kubestellar/hive` +- `ghcr.io/hivecommons/hive-contributor` and `ghcr.io/kubestellar/hive-contributor` +- `ghcr.io/hivecommons/hive-hub` and `ghcr.io/kubestellar/hive-hub` The `hivecommons` packages are mirror tags of the same manifest digest as the native `hivecommons` packages during the org transfer, so operators can verify or pin the digest against either registry. Only builds of the release branches (`v5`, `v6`) publish channels — a feature-branch build can never move a production channel. diff --git a/docs/content/hive/stable-soak-policy.md b/docs/content/hive/stable-soak-policy.md new file mode 100644 index 0000000..c5a450c --- /dev/null +++ b/docs/content/hive/stable-soak-policy.md @@ -0,0 +1,268 @@ +> **Synced from Hive.** This page is pulled from [hivecommons/hive@v5](https://github.com/hivecommons/hive/blob/v5/src/docs/stable-soak-policy.md) during the docs build. Edit the canonical source in the Hive repository. + +# Stable chases candidate and is always 24 hours behind it (v5 line) + +Stable chases candidate and is always 24 hours behind it: at each evaluation, +`stable` is the newest `v5` build whose `docker.yml` run completed at least +`SOAK_HOURS` ago — the build `candidate` pointed at 24 hours ago — regardless of +how many newer candidate builds exist. + +## Goals + +- Keep `candidate` fast: it should move on every green `v5` release build. +- Keep `stable` predictable: when `v5` is busy and the promotion workflow runs + hourly, `stable` lags `candidate` by 24 to 25 hours, not by a quiet period. +- Make longer lag explicit: green evidence, `release-blocker`, and smoke-signal + gates are the only normal reasons `stable` stays farther behind, and the + workflow names the gate and build holding it. +- Preserve rollback safety with immutable short-SHA tags and digest evidence. + +## Promotion rule + +Stable chases candidate and is always 24 hours behind it. In this document a +*build* is one successful `docker.yml` run on `v5`; its *generation* is that +run's number and its *digest* is the image manifest list it published for each +image (`hive`, `hive-contributor`, and `hive-hub`). `candidate` is the newest +build pointer. `stable` promotes an already-built digest; it does not rebuild. + +At each hourly evaluation, the workflow finds the newest build newer than the +current `stable` generation whose own `docker.yml` completion time is at least +`SOAK_HOURS` old. That is the build `candidate` pointed at 24 hours ago. No quiet +period is ever required: if `v5` keeps merging, newer builds may keep moving +`candidate`, but the build that just crossed the 24-hour line remains the one +considered for `stable`. + +The workflow moves `stable` to that build only when these hard gates pass: + +1. **Digest integrity:** all three image digests for the chosen generation still + exist in GHCR and carry matching `org.opencontainers.image.revision` and + `io.hivecommons.hive.github-actions-run-number` labels. +2. **Green release evidence:** build, lint, unit tests, changelog/release guards, + and non-flaky required checks are passing or skipped by policy for the chosen + build's commit. +3. **No open blocker:** no open issue label explicitly marks the candidate + digest, release tag, or included fix set as a `release-blocker` for the stable + (v5) line. +4. **Maintained-hive smoke signal:** at least one maintained hive has reported a + healthy heartbeat with zero crash restarts in the soak window. The preferred + evidence is a maintained hive on the exact build being promoted. When `v5` is + busy and the exact build has already been superseded, the workflow may use a + healthy maintained hive on the current, later `candidate` in the same + monotonic `v5` docker.yml lineage: surviving a later build is conservative + smoke evidence for an older build in the same line, and it is never used to + justify a younger build. If the hub is reachable but returns no maintained + hive summaries, the smoke-signal gate holds the selected build until evidence + is available. + +If any hard gate fails, the run exits successfully without moving `stable` and +prints which gate is holding which build. Those gates are the only normal reasons +`stable` can lag more than 24 to 25 hours on the hourly cadence. + +Worked example: `stable` is B1. During the next day, `docker.yml` publishes B2, +B3, … B52 and `candidate` ends at B52. At time T, B52 is only minutes old, but +B37 is the newest build whose own `docker.yml` completion time is at least 24 +hours old. The workflow evaluates B37. If B37's digests, evidence, blockers, and +smoke all pass, `stable` moves to B37, even though `candidate` is B52. On each +later hourly run, `stable` moves again whenever another newer build crosses the +24-hour line. + +Two promotion runs never execute at the same time: `promote-stable.yml` declares +a GitHub Actions `concurrency` group (`stable-promotion-v5`, +`cancel-in-progress: false`), so a scheduled or manually dispatched run waits +for any in-progress run to finish before it starts. GitHub serialises the +runs; the workflow itself does not need an atomic primitive. As a second, +independent guard, immediately before publishing the run re-reads each image's +current `stable` generation and requires it to be unchanged from the decision +read and lower than the chosen build's generation. That re-read is not atomic +with the tag move and is not relied on to be; it only catches a `stable` +generation that moved between the decision and the publish step (for example a +manual retag). If the re-read fails, the run exits successfully without moving +`stable`; the next scheduled run re-evaluates from the new `stable` generation. + +The release captain records the promoted digest, selected build, stable tag, soak +start/end time, and smoke evidence in the workflow summary or promotion PR. + +## CI enforcement + +The existing release build continues to publish immutable short-SHA tags and move +`candidate`; it no longer moves `stable` on every `v5` merge. The +`Promote Stable Channel` workflow runs hourly and can also be manually dispatched. +It: + +1. reads the current `stable` generation for `hive`, `hive-contributor`, and + `hive-hub`; +2. lists successful `docker.yml` runs on `v5` newest-to-oldest; +3. skips builds whose own `docker.yml` completion time has not yet crossed the + configured soak line, remembering the newest unsoaked build's `eligible_at`; +4. selects the first build that has crossed the line — the newest build + `candidate` pointed at 24 hours ago; +5. resolves that build's three short-SHA image tags by digest and verifies their + revision and generation labels match that run; +6. checks required release evidence, open `release-blocker` issues, and + maintained-hive smoke evidence for that build; and +7. retags all three `stable` images to the chosen build's digests only after the + stable-generation compare-and-set passes. + +The hub dashboard's release-channel block has a play/pause control on the +`stable` row for hub admins. Play is the default: scheduled runs advance `stable` +to the newest build that crossed the 24-hour line, so when `v5` is busy and the +cron is hourly, `stable` stays 24 to 25 hours behind `candidate`. No quiet period +is required. Pause records the admin and timestamp, shows "paused" next to any +behind count, and stops scheduled and manual stable-promotion runs until an admin +resumes. + +The public GET endpoint exposes only non-secret channel state, `eligible_at`, an +`eligible_build` object (`sha`, `generation`, `built_at`, and digest when known), +and maintained-hive smoke summaries; the PUT toggle is hub-admin gated and audit +logged. Stable-channel hive rows and spoke dashboards use this same `eligible_at` +calculation for their "Next update" ETA. When there is no eligible or soaking +build ahead of stable, they say that no update is queued instead of hiding the +field. + +Hives on the `stable` channel also receive the hub's expected time of the next +promotion as `next_update_at` in the heartbeat upgrade policy +([#10256](https://github.com/hivecommons/hive/issues/10256)). It is omitted +(unknown) while stable is paused, when there is no build newer than `stable`, or +when the channels have not resolved. + +The workflow writes the selected build digest, SHA, generation, build completion +time and age, checks consulted, blocker count, smoke evidence, decision, and any +exception note to the GitHub Actions step summary. If no build has crossed the +line, the workflow leaves `stable` unchanged and exits successfully with a reason +such as `no eligible build has completed the 24h soak yet; newest unsoaked build + generation completed