Pipeline v2: build once, promote the artifact (development branch) - #427
Draft
Omicron7 wants to merge 43 commits into
Draft
Pipeline v2: build once, promote the artifact (development branch)#427Omicron7 wants to merge 43 commits into
Omicron7 wants to merge 43 commits into
Conversation
Introduce the build-once/promote-the-artifact pipeline v2 actions: - actions/resolve-image: resolve a tag or a running environment to a digest-pinned image reference in the shared registry (cru-shared-artifacts). CloudRun implemented; ecs/lambda stubbed. - actions/deploy: deploy a pre-built digest reference to a target environment, enforcing the digest invariant (tag refs are rejected). CloudRun orchestration ported from v1's deploy-cloudrun (db-migrate job first, refresh other jobs, rewrite only the app container to preserve sidecars, re-attach RUNTIME secrets). ecs/lambda stubbed. New v2 helpers live in src/v2/ (env mapping, shared-registry paths, image-ref parsing, Artifact Registry tag->digest resolution via the REST API + google-auth-library, app-container heuristics). Existing src/gcp.js helpers are imported read-only; no v1 source is modified. esbuild emits dist/resolve-image.js and dist/deploy.js; build-dist.yml now also triggers on the pipeline-v2 branch. Adds vitest coverage for ref parsing/validation, registry paths, container heuristics, tag->digest resolution, and deploy orchestration. Docs in docs/pipeline-v2.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ispatch action Add the four reusable workflow_call workflows that wire the Pass 1 resolve-image/deploy actions into the build-once / promote-the-artifact flow, plus a generic cross-repo dispatch action: - build-candidate.yml: router (setup + cloudrun/ecs/lambda); cloudrun builds candidate-<n> + sha-<gitsha> to the shared registry, with a no-change guard that reuses an already-built git sha. - deploy-candidate.yml: deploy a candidate to release-candidate (no authz). - promote.yml / rollback.yml: production-locked, authz-gated; promote reuses the candidate build number as release-<n>. - actions/dispatch: generic workflow_dispatch trigger (replaces the hardcoded v1 trigger-deploy map); src + esbuild entry + vitest. - docs/pipeline-v2.md: workflow interfaces, app-info + env translation, release numbering, authz contract, locks, grants matrix, Pass 3 sketch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The shared BUILD-secrets store (D2) is not provisioned yet: the cru-shared-artifacts project lacks the Secret Manager API, gcp-secrets has no per-app filtering for a shared project, and build SAs have no scoped IAM there. The first hoax candidate build failed on this step. Gate it behind a new build-secrets input (default false) until the D2 store lands; apps without BUILD secrets are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…der router Fill in the ECS stubs for pipeline v2 (build-once/promote-the-artifact) and restructure the deploy workflows into provider routers. - src/v2/aws.js: ECR tag<->digest resolution, tags-for-digest, manifest re-tag (BatchGetImage -> PutImage), the app-container predicate + compose helper, and the service-match regex. All clients use v1's retry config (now policy). - src/v2/resolve-ecs.js / deploy-ecs.js: ECS keyed on the PROJECT NAME (fixing v1's repo-name mismatch). Deploy composes from the FAMILY'S LATEST task-def revision (Terraform's template), swaps only the app container's image, refreshes RUNTIME secrets, registers a revision, updates every service, and re-points EventBridge scheduled tasks; sidecars pass through untouched. assertDigestRef enforces the digest invariant. No runtime-project (GCP-only). - src/v2/image-ref.js: extract provider-neutral ref helpers (parseImageRef etc.) so aws.js and gcp.js share them without cross-provider imports; gcp.js re-exports. - actions/tag-image: provider-agnostic release tagging (cloudrun -> Artifact Registry REST addTag; ecs/lambda -> ECR manifest re-tag), replacing promote's gcloud CLI step. - workflows: deploy-candidate/promote/rollback become D9 routers — a lookup job (app-info + authz) outputs provider/type/project-ids; gcp/aws jobs gated on provider. GCP keeps today's flow (tag-image swaps the gcloud step); AWS assumes GitHubDeployECS. build-candidate's build-ecs job is implemented (prod-bound, role-suffix GitHubRole). Concurrency locks + authz + datadog conventions kept. Lambda stays stubbed. - docs + tests for every new module. Terraform follow-ups (aws/ecs/app, separate PR): add <project>-<env>-GitHubRole for builds (retire TaskRole's GitHub trust); add ecr:PutImage to GitHubDeployECS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rebuild bundles for the v2 ECS pass: new dist/tag-image.js, and dist/deploy.js / dist/resolve-image.js now include the ECS implementations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Datadog DORA Metrics is a separate per-committer SKU with unclear per-app billing exposure; Cru's telemetry cost model is one CI Visibility committer on cru-deploy. Deploy/promote/rollback now post structured events to the standard Events API (included with the platform): service/environment/action/revision tags, promote carries the candidate, rollback carries rollback:true. A deployments ledger via an app-info extension is the deferred phase 2 for DORA-style math. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tes, multi-provider aws jobs Implements the Lambda runtime for pipeline v2, matching the ECS/Cloud Run contracts: - resolve-lambda: tag->digest via ECR; environment->digest by reading the deployed image off the app's <project>-<nick>* image functions (scratch placeholder skipped, mirroring ECS). - deploy-lambda: v1's ratified selection (Image functions on the app OR scratch ECR repo), digest-pinned UpdateFunctionCode, plus v2 hardening — wait for each function to finish updating (waitUntilFunctionUpdatedV2) so promote/rollback never read back a stale digest. New lambdaWaitForFunctionUpdated helper in src/aws.js. - build-candidate: real build-lambda job mirroring build-ecs, with two deliberate differences (--provenance=false so Lambda accepts the manifest; no PROJECT_NAME/ENVIRONMENT/BUILD_NUMBER build args — v2 candidates are env-neutral). - deploy-candidate/promote/rollback aws jobs: guard extended to ecs|lambda, type-keyed deploy role (GitHubDeployLambda vs GitHubDeployECS), (AWS) job names. tag-image already routes lambda down the shared ECR path. - docs + tests (resolve/deploy lambda, incl. scratch flip, wait, no-match). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two ratified changes ahead of v2.0.0: D10 — candidate/release tags carry the BUILD date: candidate-<yyyy-mm-dd>-<n>, promoted as release-<yyyy-mm-dd>-<n> (full-suffix reuse keeps release == candidate identity). Humans get age-at-a-glance in tag listings, dispatch inputs, and Slack; the build number remains the unique key. Legacy pre-D10 tags stay resolvable everywhere: the reuse guard and promote accept both forms, and rollback normalizes dated or legacy input (bare build numbers resolve only for legacy releases). deploy-candidate is idempotent by default: after resolving the candidate it reads what release-candidate currently runs and no-ops on a digest match, making scheduled app workflows (the RFC's nightly-if-changed cadence) safe to dispatch unconditionally — a quiet night is a true no-op. New force input covers deliberate same-digest redeploys (e.g. picking up an applied Terraform task-definition template). Cadence itself stays the app workflow's choice: per-merge (pilots) or nightly (RFC default) with no reusable-workflow changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first human rollback after D10 typed the natural thing — the bare build number — and failed at resolve: bare numbers normalized to release-<n>, which only exists for pre-D10 releases. Fall back in both providers' tag resolution: when release-<n> misses, search for exactly one matching release-<yyyy-mm-dd>-<n> and resolve that (0 matches rethrows the original error; ambiguity never guesses). Failure was pre-mutation by design — production untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BUILD-time secrets live in the app repo as Actions secrets named BUILD_<NAME>. The repo is the isolation boundary -- matching the build's repo-scoped OIDC identity -- and one mechanism serves all three runtimes, replacing the planned shared Secret Manager store (API enablement, per-app label filtering, IAM name-prefix conditions, and a GCP/AWS asymmetry, all now unnecessary). Each build job exports ONLY BUILD_*-prefixed keys from the inherited secrets context (prefix stripped: BUILD_NPM_TOKEN -> NPM_TOKEN), no-oping when none exist; the build-secrets gate input is removed (no current caller passes it). The prefix filter is load-bearing: secrets:inherit exposes org secrets to the workflow, and nothing outside BUILD_* may reach build.sh. Writes need repo admin: DevOps via gh secret set until cru secret gains a brokered write path through the planned GitHub App. Out of scope for cru app secrets (runtime-only) -- build secrets are part of the build, which is GitHub. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s, event enrichment, GCP retries Implements the memo's deferred telemetry hardening plus two flightdeck-adopted enrichments, all preserving the hard policy that telemetry never fails a deploy. - DD_VERSION injection: new optional `version` input on the deploy action, threaded through the router to all three impls. Cloud Run upserts DD_VERSION on the app container's env (services + jobs); ECS upserts into the composed task def's app-container `environment` array (composeTaskDefinition stays pure); Lambda deliberately ignores it (function env is Terraform-owned — never call UpdateFunctionConfiguration). Existing DD_VERSION entries are replaced, never duplicated; sidecars untouched. Workflows pass it: deploy-candidate -> the candidate tag, promote -> the derived release tag, rollback -> the actual release-* tag on the resolved digest (not the bare-number input). - Visible swallowed-telemetry failures: every Datadog step keeps continue-on-error but now appends a `|| echo "::warning ..."` so a failed tag/event post annotates the run instead of vanishing. - Event enrichment (from flightdeck's event shape): the Events API posts gain `actor:<github.actor>` and, when the resolved digest carries a sha-<gitsha> tag, `sha:<gitsha>` (prefix stripped, omitted cleanly when absent) — the deployments-ledger precursor. The datadog-ci pipeline tag steps also gain `version:<version>` where known. - GCP transient-5xx retries: all three Artifact Registry REST client.request sites in src/v2/gcp.js get shared gaxios retry options (5 attempts, GET/POST, 429 + 5xx) so a transient AR 503 no longer fails a resolve, matching the AWS SDK clients' maxAttempts:5. The retried tag-create POST is safe (alreadyExists is treated as success by the caller). Tests extended (composeTaskDefinition upsert new/replace/no-op, cloud run env injection incl. replace, deploy.js router version passthrough, lambda ignores version, ECS threading, gcp retryConfig passthrough); docs updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dger Append each pipeline-v2 deployment event to the CruDeploymentLedger DynamoDB table (app-info's deployments ledger -- the fleet dashboard / DORA data spine) directly via `aws dynamodb put-item`, in both provider jobs of deploy-candidate, promote, and rollback, immediately after the Datadog event step. - Same telemetry policy as Datadog: continue-on-error + a trailing ::warning so a failed write is visible but NEVER fails the deploy. - GCP (Cloud Run) jobs assume a deploy role (ECS, arbitrarily -- trust is cru-deploy-repo-scoped) via configure-aws-credentials placed AFTER all GCP steps; AWS jobs already carry the grant on their deploy role. - The dynamodb:PutItem grant is added inline to the existing deploy roles in cru-terraform (no dedicated ledger role); until that applies the steps emit benign ::warning annotations and deploys are unaffected. - docs/pipeline-v2.md: new "Deployments ledger" subsection (schema, writers, never-fails policy, GET /deployments read path, v1 deferred to wave-2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All three pilots run it (staggered noon-UTC crons); per-merge remains supported per D11. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…m-owned Caught by review question before any deploy ran with it: the Cloud Run service's ignore_changes covers the app container's IMAGE only; its env is Terraform-managed (secret refs, environment map, cloud_sql vars). A pipeline-injected DD_VERSION would show as drift on every plan and be removed by every apply -- the docker_config disease. Unlike ECS there is no separate Terraform-owned template for the deploy to compose from, and ignore_changes cannot exempt a single env entry, so Cloud Run joins Lambda: version telemetry rides the deployment events. deployCloudRun now accepts and deliberately ignores version; tests assert the non-injection and that a pre-existing (Terraform-owned) DD_VERSION passes through untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Brian's design, replacing deploy-time injection entirely: every candidate build passes --build-arg VERSION=<yyyy-mm-dd>-<n> (the tag family's bare suffix -- identical for candidate and future release under D10), and apps opt in with ARG VERSION + ENV DD_VERSION at the END of the Dockerfile (zero cache cost, graceful degradation without it). Strictly better than injection: one true version per build in every environment (injection made promote look like a version change on identical bytes); zero Terraform drift on ALL runtimes -- image ENV is invisible to TF env management, and function-config/service env never sets DD_VERSION, so the baked value shines through everywhere INCLUDING Lambda (which injection could never reach); the reused-candidate path keeps its original version, which is the artifact's identity. Removes the deploy action's version input, the ECS compose-time upsert, the router threading, and their tests; events and ledger rows still carry the full revision tag per deploy (what-was-deployed-where vs what-is-running). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…promote/rollback results Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…erdicts on promote Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…pdates Add a pre-deploy migration phase to deployEcs, mirroring the Cloud Run db-migrate job. Before any service is touched, DescribeTaskDefinition on the convention family <project>-<nick>-db-migrate decides whether the app has opted in (family present) or not (ClientException -> skip). When present, the deploy composes a revision from the family's latest (release digest + refreshed RUNTIME secrets, identical to the service path), RunTasks it once (startedBy cru-pipeline-v2), waits via waitUntilTasksStopped (900s), and requires the db-migrate container to exit 0. The run configuration (network + launch type/capacity-provider strategy) is borrowed from the first matching service, or from the EventBridge scheduled-task target for a jobs-only app; if neither exists the deploy throws. Any nonzero exit, failure stopCode/stoppedReason, or wait timeout throws and fails the deploy with the running services untouched. The matching service list is now fetched once in deployEcs and shared by the migration phase and updateServices. This replaces the retired ECS sidecar model, which was verified broken: the app container's dependsOn on the db-migrate container used condition=START with essential=false, so the app raced the migration (serving against the un-migrated schema) and a failed migration never blocked the app or the deploy. Migrations also re-ran on every task launch/scale-up; the pre-deploy task applies them once per deploy. The phase runs for every ECS deploy (rc, promote, rollback) since deployEcs is shared, matching Cloud Run (a rollback's older image no-ops against already-applied migrations). New src/aws.js helpers beside the existing ECS ones: ecsDescribeServices, ecsRunTask, ecsDescribeTasks, ecsWaitUntilTasksStopped (all using RETRY_CONFIG). BREAKING CHANGE: ECS deploys now run database migrations to completion as a discrete pre-deploy step keyed on the <project>-<nick>-db-migrate task-definition family, replacing the sidecar model. Requires the v2 aws/ecs/app module's top-level database_migrations object; module and workflows adopt as one unit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…un image
The no-op guard's 'Check image currently on release-candidate' step resolves
the running image to a shared-registry digest. On an app's FIRST v2 deploy
after v1, the service still runs a tag-pinned image from its old per-project
registry (e.g. bills-stage/container/bills:staging-10108); resolving that tag
against the shared registry threw 'Tag not found', surfacing a red failure
annotation on an otherwise-green run (continue-on-error let the deploy
proceed correctly).
A tag ref whose repo is not the shared registry now returns an empty digest
('no comparable deployment') with an info log instead of throwing. Promote's
environment resolve also improves: a pre-v2 release-candidate now fails at
'Determine release number' with its clearer deploy-a-candidate-first error.
ECS is unaffected (v1/v2 share the app ECR repo, old tags resolve); Lambda
always reports digest refs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The `lookup` job of deploy-candidate / promote / rollback fetched per-app metadata by curl-ing the cru-app-info HTTP API. That API is being extracted into an application deployed by this very pipeline, so every deploy -- including the one that would fix a broken cru-app-info, and every rollback -- was gated on it being up. Read the CruApplicationInfo DynamoDB table directly instead: the lookup job now assumes GitHubDeployECS via OIDC and runs `aws dynamodb get-item`, with jq reading the DynamoDB-typed shape (`.Item.Provider.S`). One role for every provider, since the lookup runs before the provider fan-out and the role's trust is cru-deploy-repo-scoped -- the same precedent the ledger writes set. All existing job outputs are preserved unchanged, including the empty-string behavior for absent attributes. A missing row now fails the job with an error naming the project and environment, replacing curl -f's opaque exit 22. Requires CruGlobal/cru-terraform#11432, which grants the role `dynamodb:GetItem` on the table. The /deployments ledger reads elsewhere in these workflows stay on curl: they are best-effort telemetry that already tolerates failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pipeline's Slack messages now name the environment a human can go look at, and link it to the running app when app-info knows the URL: 📦 *myapp `candidate-...` is on <https://myapp-stage.cru.org|stage>* "is on release-candidate" becomes "is on stage" -- the word matches the hostname the link points at. promote and rollback get the same treatment on the word "production", using the production row's AppUrl. With no AppUrl the word stays plain text, so nothing changes for apps that don't have one. The three lookup jobs read `.Item.AppUrl.S // empty` alongside SlackChannel and expose it as an `app-url` job output (promote reads the prod row -- both of its messages announce production). The link is composed inside the jq -n payload, so Slack's <url|text> angle brackets survive JSON encoding and no shell quoting is involved. Every other part of the six messages -- emoji, compare link, promote-dispatch link, rollback-safety line, run URL -- is byte-identical. AppUrl is written by the app modules from the DNS name they already manage (CruGlobal/cru-terraform-modules#710). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…w run The closing line of every pipeline-v2 Slack message is now a link into the deploys dashboard's changelog page (deploys.cru.org/changelog?project=&from=&to=) instead of the bare workflow-run URL. The page takes either pipeline revision names or raw git shas and resolves names through the deployments ledger, so each workflow passes whichever refs it already holds, preferring revision names: - deploy-candidate: from = the sha running in production (the ledger GET that used to power the compare link), to = the candidate tag; label "what's in this candidate". - promote: from = the production baseline the rollback-safety classifier already fetches, to = the new release tag; label "what's in this release". - rollback: from/to INVERTED (from = the release rolled back to, to = what production was running) so the page shows what production is losing; label "what this rollback reverts". The previously-running revision comes from the ledger fetch the rollback-safety advisory already makes -- one round trip, now two uses. Graceful bootstrap: with no baseline (first deploy / first promote, a rollback with no prior production event, or a failed best-effort ledger query) the line falls back to the workflow-run URL it replaced, so it is never a dead link. The candidate message's "changes since production" GitHub compare link is removed -- the changelog supersedes it. One link per message, not three. Every other message line, including the AppUrl-linked environment word, is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The run-URL fallback is gone: with no production baseline (first deploys, failed best-effort ledger queries, self-comparing rollbacks) the message simply has no closing line. The run URL remains reachable from the ledger row (RunUrl) and the Actions tab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The "Notify Slack" steps fired on success only, so a failed candidate deploy, promote, or rollback was silent: the app-repo dispatch does not propagate the cru-deploy result, leaving the app repo green while only cru-deploy's Actions tab told the truth. Weekend nightly deploys failed at the lookup job for two days with zero notification. Each of the six provider jobs now ends with a `Notify Slack (failure)` step gated on `if: failure()` and `continue-on-error: true` -- a notification must never change a run's outcome, in either direction. The message is two plain lines: `:x:` + what failed, then the workflow-run URL (a failure's next step is the logs, so the run link belongs here; the AppUrl env link stays a success-only celebration, and there is no changelog link). Every value printed comes from a workflow input or the `lookup` job rather than an earlier step's output, so the message renders no matter which step died. The one exception, promote's release name, degrades to "the release-candidate image" when the run failed before it was derived. A no-op candidate deploy stays silent: with skip=true its remaining steps are skipped rather than failed, so failure() is false (and a failed continue-on-error step never trips it either). When the `lookup` job itself fails there is no SlackChannel to notify -- accepted, and documented, with the Actions tab as the fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every v2 app now schedules its nightly build at 00:00 UTC; the staggered per-app ~noon-UTC slots are retired. Also notes that GitHub's top-of-hour cron dispatch can start many minutes late under scheduler load — expected, not a failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Investigation findings: the steps succeeded on every run, but nothing consumes the tags -- they exist only in deploy-candidate (promote and rollback never had them, so the dataset answers no coherent question), they send environment: while the provisioned CI dashboards filter on env:, and --no-fail forces exit 0 on send failure so the ::warning suspenders could never fire. Fleet pipeline visibility comes from the CI Visibility GitHub App webhooks; deployment facts live richer in the Events API (server-verified healthy: 27 events/7d, all apps, all actions) and the CruDeploymentLedger. DD_API_KEY stays -- the events curl uses it. Also fixes three stale doc lines still describing the retired 'dora deployment' flow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rations
Apps with no database migrations (cru-app-info, okta-api-keypalive) reported
"rollback safety: unclassified" on every promote -- noise that reads like a gap
in the pipeline rather than the "nothing to check" it actually is. The
classifier could not do better, because an absent MigrationsPath conflates two
situations that deserve opposite answers:
* the app HAS no migrations -- trivially rollback-safe.
* the app has migrations the classifier cannot SEE (migrations enabled with
no path configured) -- genuinely unknown, must stay unclassified.
Absence of evidence is not evidence of absence, so the app modules now state it
positively (CruGlobal/cru-terraform-modules#715) by writing an explicit
Migrations = "none" attribute, and this half turns that declaration into a
verdict:
Migrations = "none" -> safe, "no database migrations"
MigrationsPath, all added EXPAND -> safe (diff classified, unchanged)
MigrationsPath, any CONTRACT -> unsafe (unchanged)
MigrationsPath, no baseline/sha- tag -> unclassified (unchanged)
neither attribute -> unclassified (unchanged)
The declaration SHORT-CIRCUITS ahead of all baseline and diff work: no compare
API call, no contents fetch, and no dependence on a production baseline
existing -- so a migration-less app's FIRST promote reports safe too.
A non-empty migrations-path takes precedence. The modules make the two
attributes mutually exclusive so they should never both arrive; if they do,
classifying the real path is the conservative answer -- a declaration must
never mask migrations that can actually be read.
Slack and the ledger needed no special-casing: the verdict object flows through
identically, so the ledger records RollbackSafe = true as it does for any safe
verdict. The Slack safe line is generalised from a hardcoded parenthetical to
the classifier's own first reason, so it now says WHY a promote is safe:
🛡️ rollback-safe (no database migrations) declared none
🛡️ rollback-safe (migrations additive) diff classified, unchanged
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`db/migrate/*.rb` previously fell straight to "unsupported migration
format" → unsafe, so a Rails app that set `database_migrations.path` got a
useless verdict on every promote. ararat (Rails/ECS) is onboarding now, so
teach the classifier the ActiveRecord DSL.
Same axis as the .sql rules — can the PREVIOUS release's code still operate
against the NEW schema? — so this is NOT a port of strong_migrations, which
answers the complementary lock-safety question. `safety_assured { }` is
therefore transparent here: its contents are classified like anything else.
Node has no Ruby parser, so the implementation is conservative line-oriented
heuristics: strip `#` comments (quote-aware, so `#{}` survives), swallow
heredoc bodies, join continuation lines and split top-level semicolons into
logical statements, drop `def down` bodies (located by indentation; an
unresolvable `def` falls back to classifying the whole file), then look up
each statement's leading call. Calls on a block variable (`t.string` inside
`create_table`, `dir.up` inside `reversible`) are left to the enclosing call.
EXPAND: create_table, create_join_table, add_column, add_index,
add_reference/add_belongs_to, add_timestamps, enable_extension, remove_index.
CONTRACT: remove_column(s), remove_reference/remove_belongs_to,
drop_table/drop_join_table, rename_column/rename_table, change_column,
change_column_null, change_column_default, add_check_constraint,
add_foreign_key, reversible, and anything unrecognized. The `null: false`
without `default:` caveat mirrors the SQL ADD COLUMN NOT NULL rule and reuses
its reason verbatim. `execute`/`connection.execute` extracts a simple string
literal and runs it back through the .sql rule table; interpolation,
concatenation, a bare variable or a heredoc is unsafe-unknown.
.sql classification is untouched and its tests are unmodified. The action's
content fetch now covers added `.rb` files too, or the bodies would never
arrive.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ions The first real ECS pre-deploy migration (ararat stage) failed with "db-migrate family exists but no service or scheduled task to borrow run configuration from". The service was right there -- ararat-stage-app-rzx1 -- but like most legacy Cru ECS apps it runs EC2 capacity providers in bridge mode, so DescribeServices reports no networkConfiguration and migrationRunConfig treated that absence as "nothing to borrow". The absence IS the configuration. A service (or scheduled-task target) without a network configuration is a bridge/EC2 one: borrow its capacityProviderStrategy (preferred) or launchType and send NO networkConfiguration to RunTask, which rejects the parameter on a task definition that isn't awsvpc. runConfigOf now omits the key entirely rather than passing undefined, and the EventBridge fallback is symmetric with the service path -- only when NEITHER lookup finds anything does the "nothing to borrow" error remain. The awsvpc path is unchanged. The companion module change derives the db-migrate task definition's network mode the same way (cru-terraform-modules fix/db-migrate-bridge-mode), so the task def and the borrowed run config always agree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
05:00 UTC is midnight EST / 1am EDT, and it is off-peak on GitHub's shared scheduler — unlike 00:00 UTC, the busiest cron minute there, where nightlies routinely dispatched many minutes late. Still one unstaggered fleet slot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`.github/merge-bot.yml` is the config that enables the "On Staging" merge bot, which only acted on the `staging` branch and the `On Staging` label — neither of which exists under v2. Adopting apps delete it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ceeds Each app's own Releases page now records what reached production and when, alongside the registry release-* tag and the deployments ledger. Both provider jobs (gcp + aws) get the step: git tag = the release name, pointing at the promoted commit (the sha-<gitsha> tag the candidate digest carries), title = the release name, marked latest. The body is deliberately a short header -- image digest, promoted-by, run link, and a deploys.cru.org/changelog link for the range. It does not restate the commit list: the changelog page is the canonical rendering, and a second copy would only drift from it. The changelog line follows the Slack notify's dead-link rule -- omitted when there is no baseline sha rather than emitted broken. Credential: the existing authz-token, already this workflow's app-repo credential (the authz gate reads collaborator permission with it; the rollback-safety classifier fetches the app repo's compare with it), so this adds no new credential surface -- only a new verb needing contents:write on the app repo. The run-scoped GITHUB_TOKEN cannot substitute: it is scoped to cru-deploy, not the app repo. The narrower long-term home is the org-wide "Cru Deploy" GitHub App, which needs Contents bumped read -> write AND its key made reachable from cru-deploy workflows (today it lives only in SSM /ecs/cru-app-info/*). Publishing NEVER fails a promote: production is already updated by the time the step runs, so continue-on-error plus in-step handling turns any non-2xx into a warning naming the likely fix. A re-run of an already-promoted candidate gets 422 already_exists -- the correct end state -- so that degrades to a notice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lane The doc was a design reference with no onboarding section. Adds one, written from the two real onboardings (ararat: ECS/Rails/migrations/master->main rename; bills: Cloud Run/raw SQL) plus the earlier pilots. Checklist, both sides. Terraform: v2 module refs (and the v2-pipeline vs pipeline-v2 branch-name inversion that catches everyone), required_checks with the real per-app values and the unmanaged-classic-branch-protection gotcha, registry wiring, an explicit database_migrations block behind a loud warning that the v1 ECS module ran rake db:migrate by silent default so omitting the block silently stops production migrations, slack_channel as the notification enable switch, the per-module AppUrl derivation conditions, and the ararat params audit (36 prod params -> 17 terraform-managed -> 19 leftover -> 3 truly required; grep before you copy prod). App repo: pipeline-v2.yml (cron '0 5 * * *'), conventional-commits.yml, Dependabot grouping + prefixes + the auto-merge workflow (check-then-add: template-born repos already ship it), parking rather than deleting the v1 workflow, deleting .github/merge-bot.yml, the Dockerfile build-identity pair, the authoring-time migration gate, a staging-readiness audit, and AGENTS.md/CLAUDE.md. Plus the new-repo immutable OIDC subject note. Sequencing gets its own section: the app PR must merge before the app-level apply (the ruleset requires a check that does not exist yet) and before the prod apply (v1 builds lose their OIDC trust), and the main-branch rename needs required-check branch filters at [main, master] FIRST or the repo deadlocks on checks that never report — the trap ararat proved empirically. Migration discipline frames the two gates as complementary: strong_migrations / squawk ask whether a migration disrupts a live database (development time, has teeth), the classifier asks whether the previous image survives the new schema (promote time, advisory). safety_assured affects only the former. Expand ships first and stays rollback-safe; contract ships separately, after the code that stopped using the old shape is in production. The hotfix lane is deliberately not a mechanism: same path, compressed timeline. Notes cru-deploy's Deploy Candidate (v2) force input for out-of-band infra pickup. Required-CI verification is recorded as considered and dropped, closing the last flightdeck adopt-list item: rulesets require PRs and gate merges on required checks, so a main-only build already implies CI passed wherever CI exists, and the gate could only fire for repos with no CI, where there is nothing to verify. The merge-bot.yml line moves here from the Pass 3 sketch rather than being duplicated, and gains the two-sided reasoning (absence does not disable the App; presence is the one thing that could switch it back on). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This repo is public (public app repos can only call public reusable workflows); the design/onboarding doc aggregates internal infrastructure detail and now lives in CruGlobal/cru-deploy/docs. A stub pointer remains for old links. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The v2 pipeline — build once, promote the artifact. Merging this branch cuts v2.0.0 (with a rolling
v2tag); the v1 surface is untouched. Holding open to onboard a few more apps before the cut.Status: fully built and proven in production on 6 apps across all three runtimes (CloudRun: hoax, bills · ECS: tachyon-app, ararat · Lambda: okta-api-keypalive, cru-app-info), including first promotes, rollbacks, and four takes of production-hardening the ECS migration path.
What v2 delivers (brief)
candidate-<yyyy-mm-dd>-<n>), promotion re-tags the same digest asrelease-<…>; releases are permanent (registry claim rules).DD_VERSIONbaked at build.0 5 * * *UTC (unstaggered, off-peak) + on-demand dispatch; idempotent deploys no-op on digest match..sqland Rails migrations; three verdict paths (safe-by-diff, safe-by-declaration viaMigrations: none, honestunclassified), advisory never blocking.:x:failures with run links), auto-created GitHub Releases on promote.deploys.cru.org: fleet dashboard at/(v1/v2 scoreboard, per-app timelines), rendered changelogs, all behind GitHub OAuth (org-gated, "Cru Deploy" App).cru app deploys/status/promote/rollback+cru auth github(device flow), gated per-app byPipelineVersionin AppInfo (absent = v1, old behavior untouched).v2-pipelinebranch) — squash + auto-merge, conventional-commit titles enforced, rulesets with required checks,maindefault branch (auto-rename + legacy-masterguard), grouped dependabot + gated auto-merge, immutable-OIDC-subject trust fix (fleet-wide, incl. v1 backport).Design record: the design memo (decisions D1–D11). Test suites: 306 (this repo), 293 (cru-app-info), full CLI suite.
At merge (the v2.0.0 cut)
Tag
v2.0.0+ rollingv2→ re-pin app workflows from@pipeline-v2to@v2→ re-pin terraform modules from?ref=v2-pipelineto the release tag → workflow template for new apps.Original scope description (Passes 1–2, historical)
Summary
The v2 pipeline — build-once/promote-the-artifact, per the Agentic Development Lifecycle RFC's Flight phase and the design memo's nine decisions. Long-lived development branch; not for merge until the hoax pilot promotes end-to-end. At release this merges as v2.0.0 with a rolling
v2tag; the v1 surface is never modified (verified: the only v1 file touched isbuild-dist.yml, whose push trigger gains this branch sodist/stays built here).What's here (Passes 1–2)
Actions (provider-agnostic interface, per-runtime modules; CloudRun implemented, ECS/Lambda stubbed):
actions/resolve-image— resolve what's running in an environment, or acandidate-<n>/release-<r>tag, to a digestactions/deploy— deploy an explicit digest (hard-fails on tag refs; the invariant is code, not convention), porting v1's CloudRun orchestration (db-migrate first, sidecars preserved, RUNTIME secrets re-attached)actions/dispatch— generic cross-repo workflow_dispatch (replaces trigger-deploy's hardcoded map)Reusable workflows:
build-candidate.yml— env-neutral image from main → shared registry (cru-shared-artifacts/<app>/<app>:candidate-<n>+sha-<gitsha>), router by type, no-change guard reuses existing images for unchanged commitsdeploy-candidate.yml— candidate tag → release-candidate environment (rc lock)promote.yml— authz (actor must have push on the app repo) → resolve the digest running on rc → deploy to production → tagrelease-<n>(n = the candidate's build number) → Datadog deployment mark (production lock)rollback.yml— redeploy anyrelease-<r>digest (shared production lock)Conventions: digests-only deploys; env long names
release-candidate/previewmap to existingstage/labnicknames; releases are permanent (never pruned — the registry module's KEEP policy enforces it); telemetry pinned@datadog/datadog-ci@5withcontinue-on-erroreverywhere. Full detail indocs/pipeline-v2.md, including the grants matrix and cru-deploy wrapper sketches.Validation
76 tests green (vitest), lint clean, 12 action bundles build. Workflow YAML parsed/validated; action input references cross-checked between passes.
Remaining before merge (Pass 3+)
cru-deploy wrappers (+
authz-tokensecret, WIF vars), hoax Terraform (registry module instance via CruGlobal/cru-terraform-modules#688 + prod env + grants), hoax app workflow pinned@pipeline-v2, end-to-end pilot promote/rollback, then ECS/Lambda passes.Supersedes #423.
🤖 Generated with Claude Code