diff --git a/CHANGELOG.md b/CHANGELOG.md index ac2fdbc..15ce3c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,40 @@ Versions are git tags (`vX.Y.Z`); projects install one with `uv tool install git+https://github.com/hybridinteract/deployctl@vX.Y.Z`. +## Unreleased + +**Added** +- **A release branch: build on one branch, deploy from another.** `BUILD_BRANCH` (`ci init + --build-branch main`, or Configure → *Build branch*) says images are built on `main`, and + `DEPLOY_BRANCH=prod` that merging into `prod` ships. `build-image.yml` then publishes from + `main`, a new `ci.yml` checks and builds without deploying, and a third managed workflow, + `release.yml`, deploys the `main` commit `prod` matches once that commit's CI has passed. + This is herbally's hand-written `release.yml`, made a template — with it, the local edit to + `deploy.yml`'s branch check that `ci init --force` would drop is no longer needed. Empty + (the default), nothing changes. + +**Faster** (both from herbally's own edits, measured on its production deploys) +- **CI deploys over one ssh connection per host.** `deploy.yml` sets `DEPLOYCTL_SSH_MUX=1`: + the engine opens a `ControlMaster` per host in a private `/tmp/dctl-mux.*` directory and + closes it on exit, error or cancel. Each handshake from a runner costs seconds and an + update makes dozens; herbally's deploy step went from ~350s to ~170s. herbally did this by + writing `~/.ssh/config` and `pkill`-ing the masters; the engine keeps it to its own + connections, so a self-hosted runner's ssh config is never touched. +- **A new `ci.yml` builds the image while the checks run** (no `needs: checks` on `publish`); + the deploy job waits for both. An existing `ci.yml` is yours and keeps its order. + +**Fixed** +- **CI/CD said "Finish CI/CD: .github/workflows/ci.yml" for a pipeline that deployed.** + `ci doctor` looked for the text `deploy.yml` in `ci.yml` alone; herbally calls it from + `release.yml`, so the panel's stepper never reached *Live*. Every workflow is now scanned + for the job that calls `deploy.yml`, and that job is the one checked for `packages: read` — + which now also accepts the inline `permissions: { contents: read, packages: read }` that + doctor's own fix suggests, `read-all`, and a workflow-level grant the job inherits. +- **Deploys on merge were missing from `ci runs` and the panel's history** unless `ci.yml` + called the deploy. They are read from whichever workflow does, and the tag shown is the + image deployed (from the deploy job's name), not the commit pushed — the two differ for a + release, whose merge commit never had an image. + ## 0.14.1 Fixes from taking a second project (herbally) to its first server: things that failed late, diff --git a/docs/15-FIRST-DEPLOYMENT.md b/docs/15-FIRST-DEPLOYMENT.md index 4936b84..cb1060a 100644 --- a/docs/15-FIRST-DEPLOYMENT.md +++ b/docs/15-FIRST-DEPLOYMENT.md @@ -148,7 +148,7 @@ validation re-runs after every Save. | **PostgreSQL** | Single: `container` (default; the password is generated once into `config/secrets..env` and reused forever), or `external` with `POSTGRES_HOST`, `POSTGRES_PASSWORD` and `DB_SSL_MODE=require`. Cluster: the managed database's host, port, name, user, password and SSL mode — and every host must be in the database's trusted sources. | | **Redis** | Single: `container`. Cluster: the managed instance's host, port and password, over TLS (`REDIS_SSL`). | | **Runtime** | `API_WORKERS` and `CELERY_WORKERS` (processes), their memory limits, `LOG_LEVEL`, and `ENABLE_DOCS` (the API docs, behind a login — see [30-OPERATIONS.md](30-OPERATIONS.md#docs-access)). Validation warns when the processes cannot fit in the memory limit. | -| **Continuous deployment** | `DEPLOY_BRANCH`: the branch whose pushes deploy, once CI/CD is on. | +| **Continuous deployment** | `DEPLOY_BRANCH`: the branch whose pushes deploy, once CI/CD is on. `BUILD_BRANCH`, only when images are built on another branch that is then merged into it (see [a release branch](35-CONTINUOUS-DEPLOYMENT.md#a-release-branch)). | | **Application secrets** | Your application's own keys (API keys, object storage, mail credentials…): one field per key in `project/fields.toml`. They are saved to `config/app..env`, which appears on the first Save, and written into the app's `.env` on every deploy. Validation names the keys your `.env.example` leaves blank that the deployed `.env` would not set. | Password fields are never sent back to the page: a saved one shows a *saved* badge, and @@ -303,8 +303,8 @@ each row with the button that fixes it. In order: | Prove it | `deployctl ci deploy --env production` | Redeploys what runs, through GitHub, end to end. | | Deploy on merge | `deployctl ci auto-deploy on --env production` | Every push to the deploy branch now builds and deploys. | -CI pulls the image with its own short-lived token. `ci.yml`'s `deploy:` job must grant -`packages: read` for that; `ci doctor` checks it. Why each step, and what to do when GitHub +CI pulls the image with its own short-lived token. The job that calls the deploy (`ci.yml`'s +`deploy:`, or `release.yml`'s) must grant `packages: read` for that; `ci doctor` checks it. Why each step, and what to do when GitHub is the thing that is broken: [35-CONTINUOUS-DEPLOYMENT.md](35-CONTINUOUS-DEPLOYMENT.md). Once CI/CD is on, the panel opens on **Operate**: what runs and who shipped it, deploying or diff --git a/docs/35-CONTINUOUS-DEPLOYMENT.md b/docs/35-CONTINUOUS-DEPLOYMENT.md index 7a59e57..4459a46 100644 --- a/docs/35-CONTINUOUS-DEPLOYMENT.md +++ b/docs/35-CONTINUOUS-DEPLOYMENT.md @@ -86,7 +86,7 @@ Two conditions, both checked by `ci doctor`: - **The job that calls the deploy grants `packages: read`.** A called workflow gets no more than its caller allows, and `ci.yml` is your file, so `ci init` never rewrites it — add - under its `deploy:` job: + under its `deploy:` job (or whichever job of yours calls `deploy.yml`): ```yaml permissions: @@ -112,13 +112,49 @@ GitHub's runners connect from a large, changing set of addresses. Three ways in: connection goes through it (ssh `ProxyJump`); `ci setup-key` installs the CI key there too and `ci pin-hosts` pins its host key. The usual shape for a fleet on a private network. +However it gets in, the deploy job makes **one connection per host** and runs every ssh and +rsync of the run through it (`DEPLOYCTL_SSH_MUX=1`, ssh's `ControlMaster`). From a runner each +new handshake costs seconds and an update makes dozens: herbally's deploy step went from +~350s to ~170s. The sockets sit in a directory of the run's own and are closed when the +engine exits; nothing is written to `~/.ssh/config`. A laptop can opt in the same way +(`DEPLOYCTL_SSH_MUX=1 deployctl deploy update …`). + +**Checks and build run side by side.** The generated `ci.yml` builds the image while the +checks run, so they cost no time unless they are the slower; the deploy waits for both. A +commit that fails its checks still gets an image — never deployed on a merge, but listed by +`deployctl image tags`, so a hand-picked `ci deploy --tag` should be one whose checks passed. +`ci.yml` is yours: an existing one keeps its order (`needs: [checks]` on `publish` puts it back). + +### A release branch + +Some projects build on one branch and release from another: `main` collects merged work and +builds an image per commit; merging `main` into `prod` ships. Say so with a build branch: + +```bash +deployctl ci init --env production --branch prod --build-branch main +``` + +``` +push to main ci.yml: your checks + build-image.yml ──► (no deploy) +push to prod release.yml: the main commit prod's code matches + └─ waits for that commit's ci.yml run to pass + └─► deploy.yml with its tag (with AUTO_DEPLOY on) +``` + +`release.yml` is managed like the other two. A prod whose code matches no `main` commit — a +commit made on prod, a conflict resolved in the merge — is refused rather than built: prod +never has an image of its own, so what ships is always what the checks ran against. `ci deploy` +dispatches from `DEPLOY_BRANCH` (prod), and `deploy.yml` still refuses runs from any other +branch. The panel's *Configure → Build branch* is the same setting; leave it empty when one +branch does both. + --- ## Day to day | You want to | Command | Panel (Operate tab) | |---|---|---| -| Ship code | Merge into the deploy branch. | — | +| Ship code | Merge into the deploy branch (with a build branch: merge it into the deploy branch). | — | | Change config (a key, a worker count) | edit `config/` → `deployctl ci sync-config` → `deployctl ci deploy --allow-config-change` (redeploys the running tag with the new config) | Configure → Save → *Apply a config change*: Sync config → Deploy the change | | Ship code that needs a new key | The config change first (row above), then merge. | the same | | Deploy a specific tag | `deployctl ci deploy --tag T` | *Deploy a version* → Deploy | diff --git a/docs/60-COMMAND-REFERENCE.md b/docs/60-COMMAND-REFERENCE.md index 777f37b..5527613 100644 --- a/docs/60-COMMAND-REFERENCE.md +++ b/docs/60-COMMAND-REFERENCE.md @@ -210,7 +210,7 @@ private repository) or `repository` (every plan). A private repository on GitHub plan this account cannot see, gets `repository`, with a warning that repository secrets are readable from any branch. `--branch` saves the deploy branch as `DEPLOY_BRANCH`. -### `deployctl ci init --env E [--branch B] [--force]` +### `deployctl ci init --env E [--branch B] [--build-branch B] [--force]` **Touches: local.** Writes `.github/workflows/`: `build-image.yml` (builds and pushes the image, and hands its tag on as the `tag` output) and `deploy.yml` (installs this version of deployctl, pinned, and deploys), both **managed** — regenerated from this version's @@ -220,6 +220,12 @@ replaced with `--force`. `ci.yml` is written only when there is none — it is t own (its checks go there); an existing one is left alone, and if it never calls the two workflows, the jobs to add are printed. `--branch` saves `DEPLOY_BRANCH` first. +With a **build branch** (`BUILD_BRANCH`, saved by `--build-branch`; `''` clears it), images +are built on that branch and deployed from `DEPLOY_BRANCH`: `build-image.yml` publishes from +the build branch, a new `ci.yml` checks and builds there without deploying, and a third +managed workflow, `release.yml`, runs on a push to `DEPLOY_BRANCH`, finds the build-branch +commit it matches, waits for that commit's CI to pass, and deploys its image. + > The build workflow bakes `IMAGE_REPO` in at generation time. Change `IMAGE_REPO` later and > the two silently disagree — `validate` checks for this. @@ -266,8 +272,9 @@ Existing files with different contents are only replaced with `--force`. ### `deployctl ci doctor --env E [--json] [--repo-level]` **Touches: GitHub (read).** Checks everything continuous deployment needs — `gh`, the -secrets' scope, the workflows (present, current for this version, `ci.yml` calling -`deploy.yml`), `DEPLOY_SSH_KEY`, `DEPLOY_KNOWN_HOSTS` covering every host and the jump host, +secrets' scope, the workflows (present, current for this version, some workflow calling +`deploy.yml` — `ci.yml`, `release.yml`, or one of the project's — from a job that grants +`packages: read`), `DEPLOY_SSH_KEY`, `DEPLOY_KNOWN_HOSTS` covering every host and the jump host, the config in GitHub matching this machine's, the install token while deployctl's repository is private, and whether `AUTO_DEPLOY` is on — and prints the command that fills each gap. `--json` is what the control panel reads. Exit `0` when nothing is missing, `1` otherwise. @@ -283,12 +290,14 @@ migrates. Runs on `DEPLOY_BRANCH`. ### `deployctl ci runs --env E [--limit N] [--json]` **Touches: GitHub (read).** Recent deploys through GitHub, newest first — manual runs and the deploys merges triggered (skipped ones, with `AUTO_DEPLOY` off, are left out) — with action, -tag, outcome and a link. +tag, outcome and a link. Merges are read from whichever workflow calls `deploy.yml`, on +`DEPLOY_BRANCH`. ### `deployctl ci auto-deploy --env E` **Touches: GitHub.** Sets the repository variable `AUTO_DEPLOY`: on, every push to `DEPLOY_BRANCH` that passes its checks deploys itself; off, deploys are `ci deploy`. Always -at repository level — `ci.yml` reads it before any environment is entered. +at repository level — the workflow calling the deploy (`ci.yml`, or `release.yml`) reads it +before any environment is entered. --- diff --git a/src/deployctl/cli/commands/ci.py b/src/deployctl/cli/commands/ci.py index 58ddad1..e1eed9c 100644 --- a/src/deployctl/cli/commands/ci.py +++ b/src/deployctl/cli/commands/ci.py @@ -20,11 +20,13 @@ import difflib import json import os +import pathlib import re import subprocess import sys import tempfile import time +from dataclasses import dataclass import jinja2 import typer @@ -59,6 +61,9 @@ #: Regenerated by `ci init` — edits are overwritten, so they carry a header saying so. MANAGED = ("build-image.yml", "deploy.yml") +#: Managed too, when images are built on another branch than the one that deploys: +#: a push to DEPLOY_BRANCH releases the image BUILD_BRANCH built and checked. +RELEASE = "release.yml" #: Written once and then the project's own: its checks are its business. OWNED = "ci.yml" _MANAGED_HEADER = re.compile(r"^# Managed by deployctl (\S+)") @@ -97,6 +102,85 @@ def workflows_dir(): return paths.REPO_ROOT / ".github" / "workflows" +def build_branch(cfg: Config) -> str: + """The branch whose pushes run the checks and publish the image.""" + return cfg.raw["BUILD_BRANCH"] or cfg.raw["DEPLOY_BRANCH"] + + +def releases(cfg: Config) -> bool: + """Whether images are built on one branch and deployed from another (release.yml).""" + return build_branch(cfg) != cfg.raw["DEPLOY_BRANCH"] + + +def managed(cfg: Config) -> tuple[str, ...]: + """The workflows `ci init` regenerates for this configuration.""" + return MANAGED + ((RELEASE,) if releases(cfg) else ()) + + +@dataclass(frozen=True) +class DeployCaller: + """A job, in one of the project's workflows, that calls deploy.yml.""" + + workflow: str # its file in .github/workflows + job: str # its id + name: str # what GitHub shows for it: its `name:`, or the id + can_pull: bool # whether it lets the called deploy pull the image (packages: read) + + +#: A top-level key and its block: every line after it that is indented deeper, or blank. +#: Under `jobs:` that is one job each; other top-level blocks never call deploy.yml. +_JOB = re.compile(r"^ ([A-Za-z0-9_-]+):[ \t]*(?:#.*)?\n((?:(?: .*|[ \t]*)\n)*)", re.M) +_CALLS_DEPLOY = re.compile(r"^ uses:\s*['\"]?\./\.github/workflows/deploy\.yml\b", re.M) + + +def _permissions(text: str, indent: str) -> str | None: + """The `permissions:` at this indent — inline or as a block, comments dropped — or None.""" + match = re.search(rf"^{indent}permissions:(.*)\n((?:{indent}[ \t]+.*\n|[ \t]*\n)*)", text, re.M) + if not match: + return None + # A comment is prose: "# … packages: read …" grants nothing. + return re.sub(r"#.*", "", match.group(1) + "\n" + match.group(2)) + + +def _grants_packages(job: str, workflow: str) -> bool: + """Whether a job's token can read packages: its own permissions, else the workflow's. + + With neither, the token has the repository's default, which reads packages on + both of GitHub's settings. Any `permissions:` replaces it, and only what it names + is granted. + """ + perms = _permissions(job, " ") + if perms is None: + perms = _permissions(workflow, "") + if perms is None: + return True + return re.search(r"\b(read-all|write-all|packages:\s*(read|write))\b", perms) is not None + + +def deploy_callers(wf: pathlib.Path) -> list[DeployCaller]: + """Every job in .github/workflows that calls deploy.yml — a text scan, no YAML parser. + + ci.yml's `deploy:` job when one branch builds and deploys; release.yml's when a + build branch is merged into the deploy branch. Every workflow is read, so a + project that wires the deploy its own way is seen as it is. + """ + found: list[DeployCaller] = [] + if not wf.is_dir(): + return found + for path in sorted([*wf.glob("*.yml"), *wf.glob("*.yaml")]): + if path.name == "deploy.yml": + continue + text = path.read_text() + "\n" + for match in _JOB.finditer(text): + job, body = match.group(1), match.group(2) + if not _CALLS_DEPLOY.search(body): + continue + name = re.search(r"^ name:[ \t]*(.+?)[ \t]*$", body, re.M) + found.append(DeployCaller(path.name, job, name.group(1).strip("'\"") if name else job, + _grants_packages(body, text))) + return found + + def deploy_timeout(hosts: int) -> int: """Minutes for the deploy job: a few per host, so a fleet is never cut off mid-roll.""" return max(20, min(360, 10 + 4 * max(1, hosts))) @@ -129,6 +213,8 @@ def _context(cfg: Config) -> dict: "TOOL_REF": f"v{__version__}", "ENV": cfg.env, "DEPLOY_BRANCH": cfg.raw["DEPLOY_BRANCH"], + "BUILD_BRANCH": build_branch(cfg), + "RELEASES": releases(cfg), "IMAGE_REPO": cfg.raw["IMAGE_REPO"], "REGISTRY_HOST": cfg.derived["REGISTRY_HOST"] or "ghcr.io", "CI_BUILD_CONTEXT": cfg.derived["CI_BUILD_CONTEXT"], @@ -210,13 +296,18 @@ def init( branch: str = typer.Option( None, "--branch", "-b", help="The deploy branch, saved as DEPLOY_BRANCH. Default: DEPLOY_BRANCH (main)." ), + build: str = typer.Option( + None, "--build-branch", help="The branch images are built on, when not the deploy branch — saved as " + "BUILD_BRANCH. Merging it into the deploy branch then releases (release.yml). '' clears it." + ), force: bool = typer.Option(False, "--force", "-f", help="Overwrite workflows that differ from the templates."), ) -> None: """Write .github/workflows/build-image.yml and deploy.yml, and ci.yml if there is none. build-image.yml and deploy.yml are managed: regenerated from this version's templates, and a file that differs is shown as a diff and needs --force. - ci.yml is written once and is then the project's own — its checks. + ci.yml is written once and is then the project's own — its checks. With a + build branch (BUILD_BRANCH), release.yml is managed too. """ cfg = load_config(env, require_valid=False) if not cfg.raw["IMAGE_REPO"] or "your-org" in cfg.raw["IMAGE_REPO"]: @@ -225,12 +316,15 @@ def init( if branch: patch_env_file(paths.COMMON_CONFIG, {"DEPLOY_BRANCH": branch}) cfg.raw["DEPLOY_BRANCH"] = branch + if build is not None: + patch_env_file(paths.COMMON_CONFIG, {"BUILD_BRANCH": build}) + cfg.raw["BUILD_BRANCH"] = build ctx = _context(cfg) target = workflows_dir() target.mkdir(parents=True, exist_ok=True) blocked, foreign = [], [] - for name in MANAGED: + for name in managed(cfg): path = target / name rel = path.relative_to(paths.REPO_ROOT) content = _render(name, ctx) @@ -267,11 +361,14 @@ def init( owned.write_text(_render(OWNED, ctx)) ui.ok(f"wrote {rel_owned} — yours from now on: put your checks in its `checks` job") else: - text = owned.read_text() - if "build-image.yml" not in text or "deploy.yml" not in text: - ui.warn(f"{rel_owned} is your own and was left alone — but it does not call " - f"{'build-image.yml' if 'build-image.yml' not in text else 'deploy.yml'}.") - ui.hint("add these two jobs to it, after your checks (and list your checks in `needs`):") + # The deploy may be called from any workflow (release.yml, with a build + # branch); the build is always ci.yml's, after its checks. + missing = [] if "build-image.yml" in owned.read_text() else ["build-image.yml"] + if not deploy_callers(target): + missing.append("deploy.yml") + if missing: + ui.warn(f"{rel_owned} is your own and was left alone — but it does not call {' or '.join(missing)}.") + ui.hint("add these jobs to it, after your checks (and list your checks in `needs`):") skeleton = _render(OWNED, ctx) print(skeleton[skeleton.index(" publish:"):]) @@ -635,7 +732,7 @@ def doctor_items(cfg: Config, repo_level: bool = False) -> list[dict]: scope = scope_env(cfg, repo_level) wf = workflows_dir() - for name in MANAGED: + for name in managed(cfg): path = wf / name if not path.is_file(): _check(items, f"workflow:{name}", "todo", f".github/workflows/{name}", "missing", @@ -651,18 +748,25 @@ def doctor_items(cfg: Config, repo_level: bool = False) -> list[dict]: f"move it aside, then: deployctl ci init --env {env}", value="hand-written") else: _check(items, f"workflow:{name}", "ok", f".github/workflows/{name}", "current") - ci_text = (wf / OWNED).read_text() if (wf / OWNED).is_file() else "" - if "deploy.yml" in ci_text: - _check(items, "workflow:ci", "ok", f".github/workflows/{OWNED}", "calls the deploy workflow") - else: - _check(items, "workflow:ci", "todo", f".github/workflows/{OWNED}", - "missing" if not ci_text else "never calls deploy.yml", f"deployctl ci init --env {env}") - if "deploy.yml" in ci_text and not deploy_job_can_pull(ci_text): - # ci.yml is the project's own file, so this is said, never rewritten: a called - # workflow gets no more permission than the job calling it grants. - _check(items, "workflow:ci-packages", "todo", f"{OWNED}: the deploy job may pull images", - "the deploy job does not grant packages: read — CI's own token cannot pull the image", - "add under the `deploy:` job in .github/workflows/ci.yml: " + # Whichever workflow calls the deploy — ci.yml's `deploy:` job, or release.yml + # with a build branch — not the file the template would have put it in. + callers = deploy_callers(wf) + expected = RELEASE if releases(cfg) else OWNED + if callers: + _check(items, "workflow:ci", "ok", f".github/workflows/{', '.join(sorted({c.workflow for c in callers}))}", + "calls the deploy workflow") + elif expected == OWNED or (wf / RELEASE).is_file(): + # (A missing release.yml has its own row above.) + _check(items, "workflow:ci", "todo", f".github/workflows/{expected}", + "never calls deploy.yml" if (wf / expected).is_file() else "missing", + f"deployctl ci init --env {env}") + blocked = next((c for c in callers if not c.can_pull), None) + if blocked: + # The caller may be the project's own file, so this is said, never rewritten: + # a called workflow gets no more permission than the job calling it grants. + _check(items, "workflow:ci-packages", "todo", f"{blocked.workflow}: the deploy job may pull images", + f"the `{blocked.job}:` job does not grant packages: read — CI's own token cannot pull the image", + f"add under the `{blocked.job}:` job in .github/workflows/{blocked.workflow}: " "permissions: { contents: read, packages: read }") personal = sorted({key for path in bundle.members(env) for key, value in read_env_file(path).items() @@ -729,19 +833,13 @@ def doctor_items(cfg: Config, repo_level: bool = False) -> list[dict]: "token set" if "DEPLOYCTL_INSTALL_TOKEN" in names else f"{TOOL_REPO} is public") auto = repo_values.get("AUTO_DEPLOY", "") - _check(items, "auto-deploy", "ok", "Automatic deploys", - f"on — every push to {cfg.raw['DEPLOY_BRANCH']} deploys" if auto == "true" else "off — deploy with `ci deploy`", + on = (f"on — merging {build_branch(cfg)} into {cfg.raw['DEPLOY_BRANCH']} deploys" if releases(cfg) + else f"on — every push to {cfg.raw['DEPLOY_BRANCH']} deploys") + _check(items, "auto-deploy", "ok", "Automatic deploys", on if auto == "true" else "off — deploy with `ci deploy`", value="on" if auto == "true" else "off") return items -def deploy_job_can_pull(ci_text: str) -> bool: - """Whether ci.yml's `deploy:` job grants packages: read (a text check — no YAML parser).""" - match = re.search(r"^ deploy:\n((?:(?: .*|\s*)\n)*)", ci_text + "\n", re.M) - # A line of its own, not a mention in a comment: "# … packages: read …" is prose. - return bool(match) and re.search(r"^\s+packages:\s*read\b", match.group(1), re.M) is not None - - def _safe_variables(scope: str | None) -> dict[str, str]: try: return github.variables(scope) @@ -859,6 +957,22 @@ def deploy( raise typer.Exit(github.watch(run_id)) +def _deploy_job(jobs: list[dict], callers: list[DeployCaller]) -> dict | None: + """The run's job that is the deploy: a caller's own (skipped), or " / ". + + A name built from an expression is only known once evaluated; then any job + named like a deploy is taken, as before callers were known. + """ + names = [c.name.lower() for c in callers if "${{" not in c.name] + for job in jobs: + name = job.get("name", "").lower() + if any(name == n or name.startswith(f"{n} / ") for n in names): + return job + if len(names) < len(callers) and name.startswith("deploy"): + return job + return None + + def deploy_runs(cfg: Config, limit: int = 10) -> list[dict]: """Deploys through GitHub, newest first: manual ones, and those a push to the deploy branch ran.""" out = [] @@ -870,15 +984,23 @@ def deploy_runs(cfg: Config, limit: int = 10) -> list[dict]: "action": parts[0] if parts else "update", "tag": parts[1] if len(parts) > 1 else "", "status": item.get("conclusion") or item.get("status", ""), "url": item["url"], }) - for item in github.runs(OWNED, limit=limit, branch=cfg.raw["DEPLOY_BRANCH"], event="push"): - job = next((j for j in github.run_jobs(item["databaseId"]) - if j.get("name", "").lower().startswith("deploy")), None) - if job is None or job.get("conclusion") == "skipped": - continue - out.append({ - "at": item["createdAt"], "trigger": "merge", "action": "update", "tag": item["headSha"][:7], - "status": job.get("conclusion") or job.get("status", ""), "url": item["url"], - }) + by_workflow: dict[str, list[DeployCaller]] = {} + for caller in deploy_callers(workflows_dir()): + by_workflow.setdefault(caller.workflow, []).append(caller) + for workflow, callers in by_workflow.items(): + for item in github.runs(workflow, limit=limit, branch=cfg.raw["DEPLOY_BRANCH"], event="push"): + job = _deploy_job(github.run_jobs(item["databaseId"]), callers) + if job is None or job.get("conclusion") == "skipped": + continue + # The called job is named " → ". The commit pushed is + # not the tag when a release ships an image built from another branch. + named = re.search(r" / (update|rollback) (\S+) → ", job.get("name", "")) + out.append({ + "at": item["createdAt"], "trigger": "merge", + "action": named.group(1) if named else "update", + "tag": named.group(2) if named else item["headSha"][:7], + "status": job.get("conclusion") or job.get("status", ""), "url": item["url"], + }) return sorted(out, key=lambda r: r["at"], reverse=True)[:limit] @@ -915,13 +1037,16 @@ def auto_deploy( cfg = load_config(env, require_valid=False) _github() try: - # Repository level whatever CI_SCOPE says: ci.yml reads it in a job's `if`, - # before any environment is entered. + # Repository level whatever CI_SCOPE says: the workflow calling the deploy + # (ci.yml, or release.yml) reads it in a job's `if`, before any environment + # is entered. github.set_variable("AUTO_DEPLOY", "true" if state == "on" else "false", None) except github.GitHubError as exc: ui.error(str(exc)) raise typer.Exit(1) from None - if state == "on": + if state == "on" and releases(cfg): + ui.ok(f"on — merging {build_branch(cfg)} into {cfg.raw['DEPLOY_BRANCH']} deploys the image its checks passed") + elif state == "on": ui.ok(f"on — every push to {cfg.raw['DEPLOY_BRANCH']} that passes its checks deploys itself") else: ui.ok("off — deploy with: deployctl ci deploy") diff --git a/src/deployctl/cli/config.py b/src/deployctl/cli/config.py index d3a59bc..956dd2d 100644 --- a/src/deployctl/cli/config.py +++ b/src/deployctl/cli/config.py @@ -114,6 +114,10 @@ "CI_SCOPE": "", # The branch whose pushes build and, with AUTO_DEPLOY on, deploy. "DEPLOY_BRANCH": "main", + # Set when images are built on another branch than the one that deploys: pushes + # here run the checks and publish, and merging it into DEPLOY_BRANCH releases + # that image (release.yml). Empty: DEPLOY_BRANCH does both. + "BUILD_BRANCH": "", "WITH_BEAT": "true", "ENABLE_DOCS": "false", "LOG_LEVEL": "WARNING", @@ -415,6 +419,8 @@ def warn(message: str, hint: str = "") -> None: "deployctl ci connect detects the right one") if not re.fullmatch(r"[A-Za-z0-9._/-]+", raw["DEPLOY_BRANCH"]): err(f"DEPLOY_BRANCH {raw['DEPLOY_BRANCH']!r} is not a branch name") + if raw["BUILD_BRANCH"] and not re.fullmatch(r"[A-Za-z0-9._/-]+", raw["BUILD_BRANCH"]): + err(f"BUILD_BRANCH {raw['BUILD_BRANCH']!r} is not a branch name") if raw["REVERT_SCOPE"] not in ("fleet", "host"): err(f"REVERT_SCOPE must be 'fleet' or 'host' (got {raw['REVERT_SCOPE']!r})") if raw["SSH_JUMP_HOST"] and not re.fullmatch(r"[A-Za-z0-9@._:\[\]-]+(,[A-Za-z0-9@._:\[\]-]+)*", raw["SSH_JUMP_HOST"]): diff --git a/src/deployctl/scripts/common/remote.sh b/src/deployctl/scripts/common/remote.sh index 9e47f62..4d178d4 100755 --- a/src/deployctl/scripts/common/remote.sh +++ b/src/deployctl/scripts/common/remote.sh @@ -40,8 +40,22 @@ _JUMP_OPT="" if [[ -n "${SSH_JUMP_HOST:-}" ]]; then _JUMP_OPT=" -o ProxyJump=${SSH_JUMP_HOST}" fi +# DEPLOYCTL_SSH_MUX=1 makes one connection per host and runs every ssh and rsync +# of the run through it (ControlMaster). CI sets it: from a GitHub runner each +# handshake costs seconds and an update makes dozens — herbally's deploy step went +# from ~350s to ~170s. The sockets live in a directory of this run's own, under +# /tmp because a socket path is capped near 104 bytes (macOS's $TMPDIR alone is +# ~50); engine_cleanup closes them, and ControlPersist ends any it never reaches. +# Never ~/.ssh/config: that is the machine's, and on a self-hosted runner it +# outlives the job. +MUX_DIR="" +_MUX_OPT="" +if [[ "${DEPLOYCTL_SSH_MUX:-}" == "1" ]]; then + MUX_DIR="$(mktemp -d /tmp/dctl-mux.XXXXXX)" + _MUX_OPT=" -o ControlMaster=auto -o ControlPath=${MUX_DIR}/%C -o ControlPersist=60" +fi # SSH_BASE_OPTS reaches the jump host itself; ProxyJump through it would loop. -readonly SSH_BASE_OPTS="-o StrictHostKeyChecking=${_HOST_KEY_POLICY} -o ConnectTimeout=10 -o ServerAliveInterval=15 -o ServerAliveCountMax=4" +readonly SSH_BASE_OPTS="-o StrictHostKeyChecking=${_HOST_KEY_POLICY} -o ConnectTimeout=10 -o ServerAliveInterval=15 -o ServerAliveCountMax=4${_MUX_OPT}" readonly SSH_OPTS="${SSH_BASE_OPTS}${_JUMP_OPT}" # Artifacts are scoped per environment, and generated// mirrors REMOTE_DIR @@ -245,6 +259,18 @@ engine_cleanup() { registry_logout "$h" done release_deploy_lock + close_ssh_mux # last: the two above still go through it +} + +# The shared connections DEPLOYCTL_SSH_MUX opened, closed, and their directory gone. +close_ssh_mux() { + [[ -n "$MUX_DIR" && -d "$MUX_DIR" ]] || return 0 + local sock + for sock in "$MUX_DIR"/*; do + [[ -S "$sock" ]] || continue + ssh -o ControlPath="$sock" -O exit mux >/dev/null 2>&1 || true + done + rm -rf "$MUX_DIR" } arm_cleanup() { @@ -254,6 +280,8 @@ arm_cleanup() { trap 'exit 130' INT trap 'exit 129' HUP } +# A run that shares connections closes them however it ends, lock or not. +[[ -z "$MUX_DIR" ]] || arm_cleanup # Log the host's docker in to the registry, when credentials are configured. ensure_registry_login() { diff --git a/src/deployctl/templates/ci/build-image.yml.j2 b/src/deployctl/templates/ci/build-image.yml.j2 index f55ad24..7db4878 100644 --- a/src/deployctl/templates/ci/build-image.yml.j2 +++ b/src/deployctl/templates/ci/build-image.yml.j2 @@ -1,7 +1,7 @@ # Managed by deployctl «VERSION» — regenerate with: deployctl ci init --force # # Build and publish the application image. ci.yml calls this after its checks -# pass on a push to «DEPLOY_BRANCH»; a manual run or a v* tag builds without them. +# pass on a push to «BUILD_BRANCH»; a manual run or a v* tag builds without them. # # «IMAGE_REPO»: the api image {% if WORKER_BUILD_TARGET %} @@ -49,7 +49,7 @@ jobs: with: persist-credentials: false - # Pushes to «DEPLOY_BRANCH» and v* tags publish the plain short SHA — the form + # Pushes to «BUILD_BRANCH» and v* tags publish the plain short SHA — the form # `deployctl image tags` lists. Any other branch gets -dev appended: pullable # by name, never offered as a release. - name: Compute the image tag @@ -57,7 +57,7 @@ jobs: run: | short="$(git rev-parse --short=7 HEAD)" case "$GITHUB_REF" in - refs/heads/«DEPLOY_BRANCH»|refs/tags/*) echo "short=$short" >> "$GITHUB_OUTPUT" ;; + refs/heads/«BUILD_BRANCH»|refs/tags/*) echo "short=$short" >> "$GITHUB_OUTPUT" ;; *) echo "short=$short-dev" >> "$GITHUB_OUTPUT" ;; esac @@ -81,7 +81,7 @@ jobs: push: true tags: | «IMAGE_REPO»:${{ steps.sha.outputs.short }} - ${{ github.ref == 'refs/heads/«DEPLOY_BRANCH»' && '«IMAGE_REPO»:latest' || '' }} + ${{ github.ref == 'refs/heads/«BUILD_BRANCH»' && '«IMAGE_REPO»:latest' || '' }} cache-from: type=gha,scope=api cache-to: type=gha,mode=max,scope=api # A provenance attestation turns the tag into a manifest index with an @@ -102,7 +102,7 @@ jobs: push: true tags: | «IMAGE_REPO»:${{ steps.sha.outputs.short }}-worker - ${{ github.ref == 'refs/heads/«DEPLOY_BRANCH»' && '«IMAGE_REPO»:latest-worker' || '' }} + ${{ github.ref == 'refs/heads/«BUILD_BRANCH»' && '«IMAGE_REPO»:latest-worker' || '' }} cache-from: | type=gha,scope=worker type=gha,scope=api diff --git a/src/deployctl/templates/ci/ci.yml.j2 b/src/deployctl/templates/ci/ci.yml.j2 index 5182d3d..7c7b2e2 100644 --- a/src/deployctl/templates/ci/ci.yml.j2 +++ b/src/deployctl/templates/ci/ci.yml.j2 @@ -1,17 +1,24 @@ # Written once by deployctl «VERSION» (`deployctl ci init`) — this file is YOURS: # ci init never overwrites it. Put your project's checks in the `checks` job. # -# A push to «DEPLOY_BRANCH» runs the checks, then builds and publishes the image -# (build-image.yml), then — with the repository variable AUTO_DEPLOY set to -# 'true' — deploys that exact image (deploy.yml). Nothing is deployed that did -# not pass the checks. +{% if RELEASES %} +# A push to «BUILD_BRANCH» runs the checks while it builds and publishes the image +# (build-image.yml). It does not deploy: merging «BUILD_BRANCH» into «DEPLOY_BRANCH» does +# (release.yml), and only an image whose run here passed. Pull requests run the +# checks only. +{% else %} +# A push to «DEPLOY_BRANCH» runs the checks while it builds and publishes the image +# (build-image.yml), then — once both pass, and with the repository variable +# AUTO_DEPLOY set to 'true' — deploys that exact image (deploy.yml). Nothing is +# deployed that did not pass the checks. +{% endif %} name: CI on: pull_request: push: - branches: ["«DEPLOY_BRANCH»"] + branches: ["«BUILD_BRANCH»"] permissions: contents: read @@ -34,16 +41,20 @@ jobs: publish: name: Build & push image - needs: [checks] - if: github.event_name == 'push' && github.ref == 'refs/heads/«DEPLOY_BRANCH»' + # Not `needs: checks`: the image builds while the checks run, so the checks + # cost no time unless they are the slower. A commit that fails its checks + # still gets an image, which is never deployed on its own: the deploy waits + # for both{% if RELEASES %} (release.yml, for this whole run){% endif %}. + if: github.event_name == 'push' && github.ref == 'refs/heads/«BUILD_BRANCH»' permissions: contents: read packages: write uses: ./.github/workflows/build-image.yml +{% if not RELEASES %} deploy: name: Deploy to «ENV» - needs: publish + needs: [checks, publish] if: github.event_name == 'push' && github.ref == 'refs/heads/«DEPLOY_BRANCH»' && vars.AUTO_DEPLOY == 'true' # A called workflow gets no more than this job grants: packages: read lets the # deploy pull the image with the run's own token. @@ -55,3 +66,4 @@ jobs: environment: «ENV» tag: ${{ needs.publish.outputs.tag }} secrets: inherit +{% endif %} diff --git a/src/deployctl/templates/ci/deploy.yml.j2 b/src/deployctl/templates/ci/deploy.yml.j2 index fb5ba6c..ebc16b1 100644 --- a/src/deployctl/templates/ci/deploy.yml.j2 +++ b/src/deployctl/templates/ci/deploy.yml.j2 @@ -3,8 +3,14 @@ # Deploys an already-published image with deployctl — the same engine, deploy # lock and guards as a deploy from a laptop, run from here. # +{% if RELEASES %} +# automatic release.yml calls this after a push to «DEPLOY_BRANCH» — merging +# «BUILD_BRANCH» into it — with the image «BUILD_BRANCH»'s checks passed, +# when the repository variable AUTO_DEPLOY is 'true' +{% else %} # automatic ci.yml calls this after a push to «DEPLOY_BRANCH» passed its checks # and published, when the repository variable AUTO_DEPLOY is 'true' +{% endif %} # manual Actions → deploy → Run workflow from «DEPLOY_BRANCH», or # `deployctl ci deploy`: a tag, a config change, or a rollback # @@ -146,6 +152,7 @@ jobs: - name: Deploy env: DEPLOYCTL_SSH_STRICT: "1" # only the pinned host keys, never trust-on-first-use + DEPLOYCTL_SSH_MUX: "1" # one ssh connection per host for the whole run DEPLOYCTL_CONFIG_STRICT: "1" # a value the servers run with changes only on purpose DEPLOYCTL_NO_MINT: "1" # a secret missing from the bundle fails, never regenerates # The hosts log in to the registry with this run's token: short-lived, diff --git a/src/deployctl/templates/ci/release.yml.j2 b/src/deployctl/templates/ci/release.yml.j2 new file mode 100644 index 0000000..abaf6d2 --- /dev/null +++ b/src/deployctl/templates/ci/release.yml.j2 @@ -0,0 +1,116 @@ +# Managed by deployctl «VERSION» — regenerate with: deployctl ci init --force +# +# «DEPLOY_BRANCH» is the release branch. «BUILD_BRANCH» only checks and builds (ci.yml); a +# push to «DEPLOY_BRANCH» — merging «BUILD_BRANCH» into it — deploys «ENV»: +# +# push to «BUILD_BRANCH» ci.yml: checks + image «IMAGE_REPO»: +# push to «DEPLOY_BRANCH» here: find the «BUILD_BRANCH» commit «DEPLOY_BRANCH» now matches, wait for its +# CI run to pass, deploy that commit's image (deploy.yml) +# +# «DEPLOY_BRANCH» never builds an image of its own. A PR merge commit has a new SHA, +# but its code is identical to the «BUILD_BRANCH» commit it merged, so it ships that +# commit's image — the one the checks ran against. A «DEPLOY_BRANCH» whose code matches +# no «BUILD_BRANCH» commit (a commit made on «DEPLOY_BRANCH» directly, a conflict resolved +# in the merge) is refused: land the change on «BUILD_BRANCH» and merge again. +# +# The repository variable AUTO_DEPLOY must be 'true'. Set it to anything else +# to pause releases: merges then deploy nothing. + +name: release + +on: + push: + branches: ["«DEPLOY_BRANCH»"] + +permissions: + contents: read + +# Merges in quick succession: one release at a time, the newest one next. +# deploy.yml also takes a per-environment lock of its own. +concurrency: + group: release-«DEPLOY_BRANCH» + cancel-in-progress: false + +jobs: + resolve: + name: Find the image + if: vars.AUTO_DEPLOY == 'true' + runs-on: ubuntu-24.04 + timeout-minutes: 25 + permissions: + contents: read + actions: read # gh run list: «BUILD_BRANCH»'s CI run for the commit + outputs: + tag: ${{ steps.source.outputs.tag }} + steps: + - uses: actions/checkout@«ACTIONS.checkout» # «ACTIONS.checkout_version» + with: + # Full history and every branch: needs origin/«BUILD_BRANCH» and the merge's parents. + fetch-depth: 0 + persist-credentials: false + + # Candidates: HEAD itself (a fast-forward), a PR merge's second parent + # («BUILD_BRANCH»'s tip when merged), «BUILD_BRANCH»'s tip (a squash merge). The first + # that is on «BUILD_BRANCH» and has exactly «DEPLOY_BRANCH»'s code wins. HEAD goes + # first: on a fast-forward it is «BUILD_BRANCH»'s own merge commit, whose second + # parent is a feature branch tip that never got an image. + # + # The tag is the first 7 characters, as build-image.yml computes it in a + # one-commit checkout, where `--short=7` is in practice never longer. + - name: Match «DEPLOY_BRANCH» to a «BUILD_BRANCH» commit + id: source + env: + BUILD_REF: origin/«BUILD_BRANCH» + run: | + for ref in HEAD HEAD^2 "$BUILD_REF"; do + sha="$(git rev-parse -q --verify "$ref^{commit}")" || continue + git merge-base --is-ancestor "$sha" "$BUILD_REF" || continue + git diff --quiet "$sha" HEAD || continue + echo "sha=$sha" >> "$GITHUB_OUTPUT" + echo "tag=${sha:0:7}" >> "$GITHUB_OUTPUT" + echo "«DEPLOY_BRANCH» $(git rev-parse --short HEAD) has the code of «BUILD_BRANCH» ${sha:0:7}" + exit 0 + done + echo "::error::«DEPLOY_BRANCH»'s code matches no commit on «BUILD_BRANCH» — commit the change to «BUILD_BRANCH», then merge «BUILD_BRANCH» into «DEPLOY_BRANCH»" + exit 1 + + # Nothing ships that did not pass its checks and publish. Merging right + # after «BUILD_BRANCH» means that run is often still going: wait for it. + - name: Wait for «BUILD_BRANCH»'s CI on that commit + env: + GH_TOKEN: ${{ github.token }} + SHA: ${{ steps.source.outputs.sha }} + run: | + while :; do + run="$(gh run list --repo "$GITHUB_REPOSITORY" --workflow ci.yml --event push \ + --branch "«BUILD_BRANCH»" --commit "$SHA" --limit 1 --json status,conclusion,url --jq '.[0] // {}')" + status="$(jq -r '.status // "missing"' <<< "$run")" + url="$(jq -r '.url // ""' <<< "$run")" + if [[ "$status" == completed ]]; then + conclusion="$(jq -r .conclusion <<< "$run")" + [[ "$conclusion" == success ]] && { echo "«BUILD_BRANCH»'s CI passed: $url"; exit 0; } + echo "::error::«BUILD_BRANCH»'s CI for ${SHA:0:7} ended '$conclusion' — fix «BUILD_BRANCH», then merge again: $url" + exit 1 + fi + if [[ "$status" == missing ]] && (( SECONDS > 120 )); then + echo "::error::no CI run on «BUILD_BRANCH» for ${SHA:0:7}, so no image — was it pushed to «BUILD_BRANCH»?" + exit 1 + fi + (( SECONDS < 1200 )) || { echo "::error::«BUILD_BRANCH»'s CI for ${SHA:0:7} still '$status' after 20 minutes: $url"; exit 1; } + echo "«BUILD_BRANCH»'s CI for ${SHA:0:7}: $status — waiting" + sleep 20 + done + + deploy: + name: Deploy to «ENV» + needs: resolve + # A called workflow gets no more than this job grants: packages: read lets the + # deploy pull the image with the run's own token. + permissions: + contents: read + packages: read + uses: ./.github/workflows/deploy.yml + with: + environment: «ENV» + tag: ${{ needs.resolve.outputs.tag }} + secrets: inherit diff --git a/src/deployctl/webui/fields/core.toml b/src/deployctl/webui/fields/core.toml index 6797292..0f083ca 100644 --- a/src/deployctl/webui/fields/core.toml +++ b/src/deployctl/webui/fields/core.toml @@ -149,3 +149,10 @@ label = "Deploy branch" target = "common" placeholder = "main" help = "A merge here builds the image and, with automatic deploys on, ships it. Generate workflows again after changing it." + +[[section.field]] +key = "BUILD_BRANCH" +label = "Build branch" +target = "common" +placeholder = "same as the deploy branch" +help = "Only when images are built on another branch: a merge there checks and builds, and merging it into the deploy branch releases that image (release.yml). Generate workflows again after changing it." diff --git a/src/deployctl/webui/panel/actions.py b/src/deployctl/webui/panel/actions.py index 45131dd..c97957a 100644 --- a/src/deployctl/webui/panel/actions.py +++ b/src/deployctl/webui/panel/actions.py @@ -137,11 +137,12 @@ def shown(self, env: str) -> str: "Print every ssh, rsync and compose command an update to this tag would run.", "Touches nothing — not even ssh.", params=(TAG,)), Action("ci-init", "Generate workflows", ("ci", "init", "--env", "{env}"), - "Write build-image.yml and deploy.yml into .github/workflows, and ci.yml if there is none.", + "Write build-image.yml and deploy.yml (and release.yml, with a build branch) into " + ".github/workflows, and ci.yml if there is none.", "Writes files in this repository. Commit and push them — GitHub runs what is on the branch."), Action("ci-init-force", "Regenerate workflows", ("ci", "init", "--env", "{env}", "--force"), - "Replace the two managed workflows with this version's. ci.yml is yours and never touched.", - "Overwrites build-image.yml and deploy.yml. Review with git diff, then commit and push."), + "Replace the managed workflows with this version's. ci.yml is yours and never touched.", + "Overwrites build-image.yml and deploy.yml (and release.yml). Review with git diff, then commit and push."), # -- every host, reading -------------------------------------------------- Action("doctor", "Doctor", ("deploy", "doctor", "--env", "{env}"), "Per host: ssh, docker without sudo, writable directory, the running image pullable.", @@ -334,7 +335,7 @@ def ci_fix(item: Mapping) -> str | None: return "ci-rotate-key" if status == "ok" else "ci-setup-key" if status == "ok" or item.get("value") == "hand-written": return None # nothing to press: a workflow deployctl did not write is moved aside by a person - if key in ("workflow:build-image.yml", "workflow:deploy.yml"): + if key in ("workflow:build-image.yml", "workflow:deploy.yml", "workflow:release.yml"): return "ci-init-force" if status == "warn" else "ci-init" return _CI_FIXES.get(key) diff --git a/src/deployctl/webui/panel/live.py b/src/deployctl/webui/panel/live.py index 89da303..4a91eff 100644 --- a/src/deployctl/webui/panel/live.py +++ b/src/deployctl/webui/panel/live.py @@ -101,6 +101,8 @@ def local_facts(cfg: Config) -> dict: "workflows": (paths.REPO_ROOT / ".github" / "workflows" / "deploy.yml").is_file(), "ci_scope": bool(cfg.raw.get("CI_SCOPE")), "branch": cfg.raw.get("DEPLOY_BRANCH") or "main", + # Where images are built, when that is not the deploy branch (release.yml). + "build_branch": cfg.raw.get("BUILD_BRANCH") or cfg.raw.get("DEPLOY_BRANCH") or "main", "ssh_user": cfg.raw.get("SSH_USER") or "deploy", } @@ -177,6 +179,9 @@ def _next(stage: dict[str, tuple[str, str]], local: dict) -> dict: return {"text": f"Finish CI/CD: {stage['cicd'][1]}.", "tab": "cicd"} if all(stage[s][0] == "done" for s in ("server", "first", "cicd")): # Nothing to go and do: no button, just the everyday rule. + build = local.get("build_branch") or local["branch"] + if build != local["branch"]: + return {"text": f"Live. Ship by merging {build} into {local['branch']}.", "tab": ""} return {"text": f"Live. Ship by merging to {local['branch']}.", "tab": ""} return {"text": "Checking the servers and GitHub…", "tab": ""} diff --git a/src/deployctl/webui/templates/_tab_cicd.html b/src/deployctl/webui/templates/_tab_cicd.html index 93075c4..6b67459 100644 --- a/src/deployctl/webui/templates/_tab_cicd.html +++ b/src/deployctl/webui/templates/_tab_cicd.html @@ -9,9 +9,16 @@

Continuous deployment

+ {% if local.build_branch and local.build_branch != local.branch %} +

A merge to {{ local.build_branch }} checks and builds the image; merging + {{ local.build_branch }} into {{ local.branch }} — once automatic deploys are on — + deploys that image: deployctl itself, run in GitHub Actions with a CI-only key, pinned host keys and a copy + of this machine's config. Work down the list; each row re-checks after its button runs.

+ {% else %}

A merge to {{ local.branch }} builds the image and — once automatic deploys are on — deploys it: deployctl itself, run in GitHub Actions with a CI-only key, pinned host keys and a copy of this machine's config. Work down the list; each row re-checks after its button runs.

+ {% endif %}
asking GitHub…
diff --git a/tests/test_ci_commands.py b/tests/test_ci_commands.py index d5baf06..d26154f 100644 --- a/tests/test_ci_commands.py +++ b/tests/test_ci_commands.py @@ -389,6 +389,8 @@ def test_deploy_follows_the_run_it_started(repo, monkeypatch): def test_runs_merges_manual_deploys_and_deploys_on_merge(repo, monkeypatch): + _cli("init", "--env", "production") + def fake_runs(workflow, **kwargs): if workflow == "deploy.yml": return [{"databaseId": 1, "displayTitle": "rollback aaa1111 → production", "status": "completed", @@ -490,7 +492,25 @@ def test_the_generated_pipeline_lets_the_deploy_pull(repo): _cli("init", "--env", "production") pipeline = yaml.safe_load(_workflow(repo, "ci.yml")) assert pipeline["jobs"]["deploy"]["permissions"] == {"contents": "read", "packages": "read"} - assert ci.deploy_job_can_pull(_workflow(repo, "ci.yml")) + assert ci.deploy_callers(repo / ".github" / "workflows") == [ + ci.DeployCaller("ci.yml", "deploy", "Deploy to production", True)] + + +def test_ci_deploys_over_one_connection_per_host(repo): + _cli("init", "--env", "production") + step = next(s for s in yaml.safe_load(_workflow(repo, "deploy.yml"))["jobs"]["deploy"]["steps"] + if s.get("name") == "Deploy") + assert step["env"]["DEPLOYCTL_SSH_MUX"] == "1" + + +@pytest.mark.parametrize("build", ["", "main"]) +def test_the_image_builds_while_the_checks_run(repo, build): + """herbally's ci.yml: the checks cost no time unless they are the slower.""" + _cli("init", "--env", "production", "--branch", "prod", "--build-branch", build) + jobs = yaml.safe_load(_workflow(repo, "ci.yml"))["jobs"] + assert "needs" not in jobs["publish"] + if not build: # one branch: the deploy still waits for both + assert jobs["deploy"]["needs"] == ["checks", "publish"] def test_every_job_runs_on_a_pinned_ubuntu(repo): @@ -564,3 +584,156 @@ def logged_out(): monkeypatch.setattr(github, "require", logged_out) item = json.loads(_cli("doctor", "--env", "production", "--json").stdout)["items"][0] assert item["value"] == "logged-out" and item["fix"] == "gh auth login" + + +# ---- a build branch: images built on one branch, released from another ----------------- +# +# herbally builds on main and deploys from prod (release.yml). The checks read ci.yml +# alone and said "never calls deploy.yml" about a pipeline that deployed fine. + + +def test_init_with_a_build_branch_writes_a_managed_release_workflow(repo): + result = _cli("init", "--env", "production", "--branch", "prod", "--build-branch", "main") + assert result.exit_code == 0, result.output + assert config.load("production").raw["BUILD_BRANCH"] == "main" + release = _workflow(repo, "release.yml") + assert release.startswith(f"# Managed by deployctl {__version__}") + assert "«" not in release + flow = yaml.safe_load(release) + assert flow[True]["push"]["branches"] == ["prod"] + assert flow["jobs"]["deploy"]["uses"] == "./.github/workflows/deploy.yml" + assert flow["jobs"]["deploy"]["with"]["tag"] == "${{ needs.resolve.outputs.tag }}" + pipeline = yaml.safe_load(_workflow(repo, "ci.yml")) + assert pipeline[True]["push"]["branches"] == ["main"] + assert "deploy" not in pipeline["jobs"] # ci.yml builds; release.yml deploys + assert "refs/heads/main|refs/tags/*" in _workflow(repo, "build-image.yml") + assert 'refs/heads/prod"' in _workflow(repo, "deploy.yml") # deploys still run only from prod + + +def test_without_a_build_branch_there_is_no_release_workflow(repo): + _cli("init", "--env", "production") + assert not (repo / ".github" / "workflows" / "release.yml").exists() + assert "deploy" in yaml.safe_load(_workflow(repo, "ci.yml"))["jobs"] + + +HERBALLY_CI = """\ +name: CI +on: + push: + branches: ["main"] +permissions: + contents: read +jobs: + checks: + runs-on: ubuntu-24.04 + steps: + - run: true + publish: + permissions: + contents: read + packages: write + uses: ./.github/workflows/build-image.yml +""" + +HERBALLY_RELEASE = """\ +# Not generated by deployctl — this file is ours. +name: release +on: + push: + branches: ["prod"] +permissions: + contents: read +jobs: + resolve: + name: Find the image + runs-on: ubuntu-24.04 + steps: + - run: true + deploy: + name: Deploy to production + needs: resolve + # A called workflow gets no more than this job grants: packages: read lets the + # deploy pull the image with the run's own token. + permissions: + contents: read + packages: read + uses: ./.github/workflows/deploy.yml + secrets: inherit +""" + + +def _herbally(repo): + """herbally's shape: managed build + deploy, its own ci.yml and a hand-written release.yml.""" + paths.COMMON_CONFIG.write_text(paths.COMMON_CONFIG.read_text() + "DEPLOY_BRANCH=prod\nBUILD_BRANCH=main\n") + wf = repo / ".github" / "workflows" + wf.mkdir(parents=True) + (wf / "ci.yml").write_text(HERBALLY_CI) + (wf / "release.yml").write_text(HERBALLY_RELEASE) + return wf + + +def test_doctor_finds_the_deploy_in_release_yml(repo, monkeypatch): + _herbally(repo) + assert _cli("init", "--env", "production").exit_code == 1 # release.yml is theirs: refused, not replaced + assert _workflow(repo, "release.yml") == HERBALLY_RELEASE + items = _doctor_items(monkeypatch) + assert items["workflow:ci"]["status"] == "ok" + assert items["workflow:ci"]["title"] == ".github/workflows/release.yml" + assert "workflow:ci-packages" not in items + assert items["workflow:release.yml"]["status"] == "warn" # hand-written: said, never blocking + + +def test_a_deploy_called_from_nowhere_is_still_todo(repo, monkeypatch): + wf = _herbally(repo) + (wf / "release.yml").unlink() + _cli("init", "--env", "production") + (wf / "release.yml").write_text("name: release\non: push\njobs: {}\n") + items = _doctor_items(monkeypatch) + assert items["workflow:ci"]["status"] == "todo" + assert items["workflow:ci"]["detail"] == "never calls deploy.yml" + + +@pytest.mark.parametrize("permissions, can_pull", [ + (" permissions:\n contents: read\n packages: read\n", True), + (" permissions: { contents: read, packages: read }\n", True), # the form doctor's fix suggests + (" permissions: read-all\n", True), + (" permissions:\n contents: read\n # packages: read\n", False), # prose grants nothing + (" permissions:\n contents: read\n", False), + ("", False), # the workflow's `contents: read` replaces the default for every job +]) +def test_the_calling_job_must_let_the_deploy_pull(repo, permissions, can_pull): + wf = repo / ".github" / "workflows" + wf.mkdir(parents=True) + (wf / "release.yml").write_text(HERBALLY_RELEASE.replace( + " permissions:\n contents: read\n packages: read\n", permissions)) + [caller] = ci.deploy_callers(wf) + assert (caller.workflow, caller.job, caller.name, caller.can_pull) == ( + "release.yml", "deploy", "Deploy to production", can_pull) + + +def test_with_no_permissions_anywhere_the_default_token_can_pull(repo): + wf = repo / ".github" / "workflows" + wf.mkdir(parents=True) + (wf / "release.yml").write_text(HERBALLY_RELEASE.replace("permissions:\n contents: read\n", "").replace( + " permissions:\n contents: read\n packages: read\n", "")) + assert ci.deploy_callers(wf)[0].can_pull + + +def test_runs_read_releases_and_name_the_image_not_the_merge_commit(repo, monkeypatch): + """A release deploys main's image; prod's merge commit is not its tag.""" + _herbally(repo) + asked = [] + + def fake_runs(workflow, **kwargs): + asked.append((workflow, kwargs.get("branch"))) + if workflow != "release.yml": + return [] + return [{"databaseId": 9, "headSha": "fff9999aaaa", "createdAt": "2026-10-01T09:00:00Z", "url": "u9"}] + + monkeypatch.setattr(github, "runs", fake_runs) + monkeypatch.setattr(github, "run_jobs", lambda run_id: [ + {"name": "Find the image", "conclusion": "success"}, + {"name": "Deploy to production / update abc1234 → production", "conclusion": "success"}]) + found = ci.deploy_runs(config.load("production")) + assert ("release.yml", "prod") in asked + assert [(r["trigger"], r["tag"], r["status"]) for r in found] == [("merge", "abc1234", "success")] diff --git a/tests/test_ci_deploy.py b/tests/test_ci_deploy.py index 83c6e89..5677000 100644 --- a/tests/test_ci_deploy.py +++ b/tests/test_ci_deploy.py @@ -341,6 +341,41 @@ def test_no_jump_host_means_a_direct_connection(self): assert "ProxyJump" not in out +class TestSharedConnections: + """DEPLOYCTL_SSH_MUX: one handshake per host per run — herbally's deploy step halved.""" + + def test_off_unless_asked(self): + rc, out = _bash('echo "$SSH_OPTS"') + assert rc == 0, out + assert "ControlMaster" not in out + + def test_one_connection_per_host_in_a_directory_of_the_runs_own(self): + rc, out = _bash('echo "$SSH_OPTS"; echo "dir=$MUX_DIR"; ls -ld "$MUX_DIR"', DEPLOYCTL_SSH_MUX="1") + assert rc == 0, out + assert "-o ControlMaster=auto" in out and "-o ControlPersist=60" in out + mux_dir = next(line[4:] for line in out.splitlines() if line.startswith("dir=")) + assert mux_dir.startswith("/tmp/dctl-mux.") and f"-o ControlPath={mux_dir}/%C" in out + assert out.splitlines()[-1].startswith("drwx------") # mktemp: this user's alone + # A unix socket path is capped near 104 bytes; %C expands to 40 characters. + assert len(mux_dir) + 1 + 40 < 100 + + def test_ssh_itself_accepts_the_options(self): + """ssh -G resolves the configuration without connecting.""" + rc, out = _bash('ssh -G $SSH_OPTS deploy@203.0.113.10', DEPLOYCTL_SSH_MUX="1") + assert rc == 0, out + assert "controlmaster auto" in out and "controlpersist 60" in out + + def test_the_directory_is_gone_when_the_run_ends(self): + rc, out = _bash('echo "$MUX_DIR"', DEPLOYCTL_SSH_MUX="1") + assert rc == 0, out + assert not pathlib.Path(out.strip().splitlines()[-1]).exists() + + def test_and_when_it_fails(self): + rc, out = _bash('echo "$MUX_DIR"; exit 3', DEPLOYCTL_SSH_MUX="1") + assert rc == 3 + assert not pathlib.Path(out.strip().splitlines()[-1]).exists() + + @pytest.fixture def history(tmp_path): """A repository with three commits, a deploy directory inside it, and a stand-in host. diff --git a/tests/test_panel_live.py b/tests/test_panel_live.py index 997b60b..a513ca1 100644 --- a/tests/test_panel_live.py +++ b/tests/test_panel_live.py @@ -117,6 +117,11 @@ def test_everything_done_is_live(self): assert stages(journey) == {"server": "done", "first": "done", "cicd": "done", "live": "done"} assert journey["next"] == {"text": "Live. Ship by merging to prod.", "tab": ""} + def test_with_a_build_branch_shipping_is_merging_it_into_the_deploy_branch(self): + local = {**LOCAL, "build_branch": "main"} + journey = live.journey(local, fact({"tag": "8e3e648", "hosts": [host()]}), ci("ok")) + assert journey["next"] == {"text": "Live. Ship by merging main into prod.", "tab": ""} + def test_an_unconfigured_project_starts_at_setup(self): journey = live.journey({**LOCAL, "configured": False}, None, None) assert stages(journey)["server"] == "todo"