-
Notifications
You must be signed in to change notification settings - Fork 0
feat(release): write Atlas producer receipts after the deploy health gate #254
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -410,6 +410,7 @@ jobs: | |
| # the running build) must equal the release tag, not only the gitSha. | ||
| TAG_VERSION="${TAG_NAME#v}" | ||
| export TAG_VERSION | ||
| last_health_body="" | ||
| read -r -a health_urls <<< "$OCX_HEALTH_URLS" | ||
| if [ "${#health_urls[@]}" -lt 2 ]; then | ||
| echo "::error::OCX_HEALTH_URLS must include loopback and Tailscale" | ||
|
|
@@ -429,6 +430,7 @@ jobs: | |
| all_ok=0 | ||
| break | ||
| fi | ||
| last_health_body="$body" | ||
| root_url="${url%/healthz}/" | ||
| dashboard="$(curl -fsS --max-time "$max_time" "$root_url" || true)" | ||
| if ! printf '%s' "$dashboard" | grep -Fq '<title>opencodex · proxy dashboard</title>'; then | ||
|
|
@@ -449,6 +451,42 @@ jobs: | |
| echo "- image: \`${PINNED_IMAGE}\`" | ||
| echo "- runtime /healthz: version \`${TAG_VERSION}\`, gitSha \`${TAG_SHA}\` on ${OCX_HEALTH_URLS}" | ||
| } >> "$GITHUB_STEP_SUMMARY" | ||
| # Producer receipt for Atlas' publisher. Display identity is | ||
| # scraped from the compiled GUI HTML, not copied from /healthz. | ||
| # Binding IDs come from the admitted host file; no invented scope. | ||
| binding="${STATUS_VERSION_BINDING_FILE:-/opt/chef/state/opencodex/status-version-binding.json}" | ||
| if [ ! -f "$binding" ]; then | ||
| echo "- producer receipt: UNKNOWN (no admitted binding file)" >> "$GITHUB_STEP_SUMMARY" | ||
| exit 0 | ||
| fi | ||
| dashboard_url="${health_urls[0]%/healthz}/" | ||
| dashboard_html="${GITHUB_WORKSPACE}/ocx-dashboard.html" | ||
| curl -fsS --max-time 5 "$dashboard_url" > "$dashboard_html" | ||
| now="$(date -u +%Y-%m-%dT%H:%M:%SZ)" | ||
| fresh="$(date -u -d '+300 seconds' +%Y-%m-%dT%H:%M:%SZ)" | ||
| receipt="${GITHUB_WORKSPACE}/status-version-receipt.json" | ||
| runtime_version="$(HEALTH_BODY="${last_health_body:-}" python3 -c 'import json,os; print(json.loads(os.environ["HEALTH_BODY"])["version"])')" | ||
| runtime_sha="$(HEALTH_BODY="${last_health_body:-}" python3 -c 'import json,os; print(json.loads(os.environ["HEALTH_BODY"])["gitSha"])')" | ||
| python3 scripts/status-version-receipt.py assemble \ | ||
| --binding "$binding" \ | ||
| --now "$now" \ | ||
| --fresh-until "$fresh" \ | ||
| --artifact-version "$TAG_VERSION" \ | ||
| --artifact-source-sha "$TAG_SHA" \ | ||
| --artifact-digest "$PINNED_IMAGE" \ | ||
| --runtime-version "$runtime_version" \ | ||
| --runtime-source-sha "$runtime_sha" \ | ||
| --runtime-digest "$PINNED_IMAGE" \ | ||
| --display-html "$dashboard_html" \ | ||
| --verification-ref "$dashboard_url" \ | ||
| --operation-id "${GITHUB_RUN_ID}" \ | ||
| --operation-owner "fable-opencodex-deploy" \ | ||
| --operation-status "succeeded" \ | ||
| --desired-sha "$TAG_SHA" \ | ||
| --authority "deploy.yml" \ | ||
| --evidence-ref "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" \ | ||
| --out "$receipt" | ||
| echo "- producer receipt: \`${receipt}\` (display from ${dashboard_url} HTML meta, not /healthz)" >> "$GITHUB_STEP_SUMMARY" | ||
|
Comment on lines
+454
to
+489
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win If receipt generation fails, the health gate fails and the healthy deploy is rolled back. Lines 454-489 run inside the Realistic triggers when the binding file exists:
Receipt generation is auxiliary evidence. It should not control the rollback decision. Move it to its own step with 🐛 Proposed fix: isolate receipt generation from the gateIn - name: Producer receipt (non-gating)
if: steps.health.outputs.status == 'ok'
continue-on-error: true
env:
TAG_SHA: ${{ steps.verify.outputs.tag_sha }}
TAG_NAME: ${{ steps.ref.outputs.tag }}
PINNED_IMAGE: ${{ steps.image.outputs.ref }}
run: |
set -euo pipefail
TAG_VERSION="${TAG_NAME#v}"
binding="${STATUS_VERSION_BINDING_FILE:-/opt/chef/state/opencodex/status-version-binding.json}"
if [ ! -f "$binding" ]; then
echo "- producer receipt: UNKNOWN (no admitted binding file)" >> "$GITHUB_STEP_SUMMARY"
exit 0
fi
if [ ! -f scripts/status-version-receipt.py ]; then
echo "- producer receipt: UNKNOWN (deployed tag predates receipt assembler)" >> "$GITHUB_STEP_SUMMARY"
exit 0
fi
read -r -a health_urls <<< "$OCX_HEALTH_URLS"
last_health_body="$(curl -fsS --max-time 5 "${health_urls[0]}")"
dashboard_url="${health_urls[0]%/healthz}/"
# ... remaining receipt lines unchanged ...Update The PR discussion mentions a "configurable required mode" for the binding file. The supplied diff does not contain one. If that mode is added and makes receipt generation required, it recreates this rollback path unless receipt failures stay separate from the gate. 🤖 Prompt for AI Agents |
||
| exit 0 | ||
| fi | ||
| (( SECONDS < deadline )) && sleep 1 | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -74,7 +74,12 @@ token, so nothing downstream would fire on its own: | |||||||||||||||||||||||||||||||
| `true`. Default `false`: the live cutover on `chef-control-az-01` stays a coordinated | ||||||||||||||||||||||||||||||||
| step (`gh workflow run deploy.yml -f ref=v<version>`). Deploy waits for the image, pins | ||||||||||||||||||||||||||||||||
| its digest, and only passes when `/healthz` reports `version` **and** `gitSha` equal to | ||||||||||||||||||||||||||||||||
| the tag; the GUI shows that same `/healthz.version` top-left. | ||||||||||||||||||||||||||||||||
| the tag; the GUI shows that same `/healthz.version` top-left. After a | ||||||||||||||||||||||||||||||||
| successful health gate, `scripts/status-version-receipt.py` writes an Atlas | ||||||||||||||||||||||||||||||||
| CF#665 producer receipt when an admitted binding file is present. Display | ||||||||||||||||||||||||||||||||
| fields are scraped from the compiled GUI `ocx-build-version` meta tag, never | ||||||||||||||||||||||||||||||||
| copied from `/healthz`. `status/version-policy` is not required until the | ||||||||||||||||||||||||||||||||
| publisher has a real passed and blocked canary. | ||||||||||||||||||||||||||||||||
|
Comment on lines
+77
to
+82
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win The docs should say that the runtime section of the receipt is always Lines 78-82 say that display fields come from the
Every deploy receipt therefore has 📝 Proposed wording CF#665 producer receipt when an admitted binding file is present. Display
fields are scraped from the compiled GUI `ocx-build-version` meta tag, never
- copied from `/healthz`. `status/version-policy` is not required until the
- publisher has a real passed and blocked canary.
+ copied from `/healthz`. The GUI does not yet stamp `ocx-build-sha`, so the
+ receipt `runtime` stays `unknown` (`display_source_sha_unobservable`) until it
+ does. The receipt is written to `${GITHUB_WORKSPACE}/status-version-receipt.json`
+ on the runner and is not uploaded. `status/version-policy` is not required
+ until the publisher has a real passed and blocked canary.📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| ## Post-release | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,252 @@ | ||
| #!/usr/bin/env python3 | ||
| """Assemble a ChefStatus producer receipt (Atlas CF#665). | ||
|
|
||
| Artifact, runtime backend, and visible GUI identity are separate reads. | ||
| Display fields come from the compiled app (HTML meta), never from /healthz. | ||
| """ | ||
| from __future__ import annotations | ||
|
|
||
| import argparse | ||
| from datetime import datetime, timedelta, timezone | ||
| import json | ||
| import re | ||
| import sys | ||
| import uuid | ||
| from pathlib import Path | ||
|
|
||
| SHA = re.compile(r"^[0-9a-f]{40}$") | ||
| DIGEST = re.compile(r"^sha256:[0-9a-f]{64}$") | ||
| VERSION = re.compile(r"^\d+\.\d+\.\d+(?:-preview\.\d+)?$") | ||
| TRACE = re.compile(r"^[0-9a-f]{32}$") | ||
| INSTANT = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$") | ||
|
Comment on lines
+17
to
+21
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win Use Lines 17-20 anchor with
The 🐛 Proposed fix-SHA = re.compile(r"^[0-9a-f]{40}$")
-DIGEST = re.compile(r"^sha256:[0-9a-f]{64}$")
-VERSION = re.compile(r"^\d+\.\d+\.\d+(?:-preview\.\d+)?$")
-TRACE = re.compile(r"^[0-9a-f]{32}$")
+SHA = re.compile(r"[0-9a-f]{40}")
+DIGEST = re.compile(r"sha256:[0-9a-f]{64}")
+VERSION = re.compile(r"\d+\.\d+\.\d+(?:-preview\.\d+)?", re.ASCII)
+TRACE = re.compile(r"[0-9a-f]{32}")Then replace each Also applies to: 50-50, 57-57, 64-64, 198-203, 229-230 🤖 Prompt for AI Agents |
||
| BINDING_KEYS = ( | ||
| "tenantId", | ||
| "projectId", | ||
| "repositoryId", | ||
| "connectorId", | ||
| "providerRepositoryId", | ||
| "configurationRevision", | ||
| "environmentId", | ||
| ) | ||
| OPERATION_STATUSES = frozenset( | ||
| {"queued", "claimed", "running", "blocked", "succeeded", "failed"} | ||
| ) | ||
| VERSION_META = re.compile( | ||
| r'<meta\s+name="ocx-build-version"\s+content="([^"]+)"\s*/?>', | ||
| re.I, | ||
| ) | ||
| SHA_META = re.compile( | ||
| r'<meta\s+name="ocx-build-sha"\s+content="([^"]+)"\s*/?>', | ||
| re.I, | ||
| ) | ||
|
|
||
|
|
||
| def fail(message: str) -> None: | ||
| raise SystemExit(message) | ||
|
|
||
|
|
||
| def parse_build_version_meta(html: str) -> str | None: | ||
| match = VERSION_META.search(html) | ||
| if match and VERSION.match(match.group(1)): | ||
| return match.group(1) | ||
| return None | ||
|
|
||
|
|
||
| def parse_build_sha_meta(html: str) -> str | None: | ||
| match = SHA_META.search(html) | ||
| if match and SHA.match(match.group(1)): | ||
| return match.group(1) | ||
| return None | ||
|
|
||
|
|
||
| def parse_digest(value: str, name: str) -> str: | ||
| digest = value[value.rfind("@") + 1 :] if "@" in value else value | ||
| if not DIGEST.match(digest): | ||
| fail(f"{name} must be sha256: plus 64 hex") | ||
| return digest | ||
|
|
||
|
|
||
| def validate_freshness(now: str, fresh_until: str) -> None: | ||
| if not INSTANT.fullmatch(now) or not INSTANT.fullmatch(fresh_until): | ||
| fail("--now and --fresh-until must be UTC YYYY-MM-DDTHH:MM:SSZ") | ||
| try: | ||
| start = datetime.strptime(now, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc) | ||
| expiry = datetime.strptime(fresh_until, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc) | ||
| except ValueError: | ||
| fail("--now and --fresh-until must be valid UTC instants") | ||
| if not start < expiry <= start + timedelta(minutes=10): | ||
| fail("--fresh-until must be later than --now and within ten minutes") | ||
|
|
||
|
|
||
| def load_binding(path: Path) -> dict: | ||
| raw = json.loads(path.read_text(encoding="utf-8")) | ||
| if not isinstance(raw, dict): | ||
| fail("binding must be a JSON object") | ||
| binding = {} | ||
| for key in BINDING_KEYS: | ||
| if key not in raw: | ||
| fail(f"binding missing {key}") | ||
| value = raw[key] | ||
| if key == "configurationRevision": | ||
| if not isinstance(value, int) or isinstance(value, bool) or value < 1: | ||
| fail("binding.configurationRevision must be a positive integer") | ||
| binding[key] = value | ||
| continue | ||
| if not isinstance(value, str) or not value or len(value) > 256: | ||
| fail(f"binding.{key} must be a non-empty string") | ||
| binding[key] = value | ||
| return binding | ||
|
|
||
|
|
||
| def observed(binding: dict, value: dict, source_revision: str, now: str, fresh_until: str, trace_id: str) -> dict: | ||
| for key, entry in value.items(): | ||
| if not isinstance(entry, str) or not entry or len(entry) > 256: | ||
| fail(f"{key} must be a string of 1-256 chars") | ||
| return { | ||
| "kind": "observed", | ||
| "value": value, | ||
| "freshUntil": fresh_until, | ||
| "provenance": { | ||
| "connectorId": binding["connectorId"], | ||
| "providerResourceId": binding["providerRepositoryId"], | ||
| "sourceRevision": source_revision, | ||
| "observedAt": now, | ||
| "ingestedAt": now, | ||
| "traceId": trace_id, | ||
| }, | ||
| } | ||
|
|
||
|
|
||
| def assemble_receipt( | ||
| *, | ||
| binding: dict, | ||
| now: str, | ||
| fresh_until: str, | ||
| trace_id: str, | ||
| artifact: dict, | ||
| runtime_backend: dict, | ||
| display: dict, | ||
| operation: dict, | ||
| ) -> dict: | ||
| if not display.get("independent") and display.get("version"): | ||
| fail("display fields require an independent HTML/browser read; do not copy /healthz") | ||
| if display.get("version") and display.get("source_sha") and display.get("artifact_digest") and display.get("verification_ref"): | ||
| runtime = observed( | ||
| binding, | ||
| { | ||
| "version": runtime_backend["version"], | ||
| "sourceSha": runtime_backend["sourceSha"], | ||
| "artifactDigest": runtime_backend["artifactDigest"], | ||
| "displayVersion": display["version"], | ||
| "displaySourceSha": display["source_sha"], | ||
| "displayArtifactDigest": display["artifact_digest"], | ||
| "verificationRef": display["verification_ref"], | ||
| }, | ||
| runtime_backend["sourceSha"], | ||
| now, | ||
| fresh_until, | ||
| trace_id, | ||
| ) | ||
| else: | ||
| if not display.get("version"): | ||
| reason = "display_version_unobservable" | ||
| elif not display.get("source_sha"): | ||
| reason = "display_source_sha_unobservable" | ||
| elif not display.get("artifact_digest"): | ||
| reason = "display_artifact_digest_unobservable" | ||
| else: | ||
| reason = "display_verification_ref_unobservable" | ||
| runtime = {"kind": "unknown", "reason": reason} | ||
| return { | ||
| "schemaVersion": "1", | ||
| "binding": binding, | ||
| "artifact": observed(binding, artifact, artifact["sourceSha"], now, fresh_until, trace_id), | ||
| "runtime": runtime, | ||
| "operation": observed(binding, operation, operation["desiredSha"], now, fresh_until, trace_id), | ||
| } | ||
|
|
||
|
|
||
| def main(argv: list[str]) -> int: | ||
| parser = argparse.ArgumentParser() | ||
| parser.add_argument("command", choices=["assemble"]) | ||
| parser.add_argument("--binding", required=True) | ||
| parser.add_argument("--now", required=True) | ||
| parser.add_argument("--fresh-until", required=True) | ||
| parser.add_argument("--trace-id") | ||
| parser.add_argument("--artifact-version", required=True) | ||
| parser.add_argument("--artifact-source-sha", required=True) | ||
| parser.add_argument("--artifact-digest", required=True) | ||
| parser.add_argument("--runtime-version", required=True) | ||
| parser.add_argument("--runtime-source-sha", required=True) | ||
| parser.add_argument("--runtime-digest", required=True) | ||
| parser.add_argument("--display-html") | ||
| parser.add_argument("--display-version") | ||
| parser.add_argument("--display-source-sha") | ||
| parser.add_argument("--display-digest") | ||
| parser.add_argument("--verification-ref", required=True) | ||
| parser.add_argument("--operation-id", required=True) | ||
| parser.add_argument("--operation-owner", required=True) | ||
| parser.add_argument("--operation-status", required=True) | ||
| parser.add_argument("--desired-sha", required=True) | ||
| parser.add_argument("--authority", required=True) | ||
| parser.add_argument("--evidence-ref", required=True) | ||
| parser.add_argument("--out", required=True) | ||
| args = parser.parse_args(argv) | ||
| validate_freshness(args.now, args.fresh_until) | ||
| if args.operation_status not in OPERATION_STATUSES: | ||
| fail("--operation-status is not an admitted operation status") | ||
| if not SHA.match(args.artifact_source_sha) or not SHA.match(args.runtime_source_sha) or not SHA.match(args.desired_sha): | ||
| fail("source SHAs must be 40 hex") | ||
| if not VERSION.match(args.artifact_version) or not VERSION.match(args.runtime_version): | ||
| fail("versions must be X.Y.Z or X.Y.Z-preview.N") | ||
| trace_id = args.trace_id or uuid.uuid4().hex | ||
| if not TRACE.match(trace_id): | ||
| fail("--trace-id must be 32 hex") | ||
| display_version = args.display_version | ||
| display_sha = args.display_source_sha | ||
| independent = False | ||
| if args.display_html: | ||
| html = Path(args.display_html).read_text(encoding="utf-8") | ||
| display_version = parse_build_version_meta(html) | ||
| display_sha = parse_build_sha_meta(html) | ||
| independent = True | ||
| receipt = assemble_receipt( | ||
| binding=load_binding(Path(args.binding)), | ||
| now=args.now, | ||
| fresh_until=args.fresh_until, | ||
| trace_id=trace_id, | ||
| artifact={ | ||
| "version": args.artifact_version, | ||
| "sourceSha": args.artifact_source_sha, | ||
| "artifactDigest": parse_digest(args.artifact_digest, "--artifact-digest"), | ||
| }, | ||
| runtime_backend={ | ||
| "version": args.runtime_version, | ||
| "sourceSha": args.runtime_source_sha, | ||
| "artifactDigest": parse_digest(args.runtime_digest, "--runtime-digest"), | ||
| }, | ||
| display={ | ||
| "version": display_version if display_version and VERSION.match(display_version) else None, | ||
| "source_sha": display_sha if display_sha and SHA.match(display_sha) else None, | ||
| "artifact_digest": ( | ||
| parse_digest(args.display_digest, "--display-digest") | ||
| if args.display_digest else None | ||
| ), | ||
| "verification_ref": args.verification_ref, | ||
| "independent": independent, | ||
| }, | ||
| operation={ | ||
| "id": args.operation_id, | ||
| "owner": args.operation_owner, | ||
| "status": args.operation_status, | ||
| "desiredSha": args.desired_sha, | ||
| "authority": args.authority, | ||
| "evidenceRef": args.evidence_ref, | ||
| }, | ||
| ) | ||
| Path(args.out).write_text(json.dumps(receipt, indent=2) + "\n", encoding="utf-8") | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: sed -n '1,20p' scripts/status-version-receipt.py
sed -n '240,252p' scripts/status-version-receipt.pyRepository: GroepOnline/opencodex Length of output: 1151 🏁 Script executed: set -eu
printf '%s\n' '--- references to receipt and output ---'
rg -n -S --hidden --glob '!node_modules' --glob '!dist' \
'status-version-receipt|status.version.receipt|--out|release evidence|release-state|release state|ChefStatus' \
scripts .github README.md docs 2>/dev/null | head -240
printf '%s\n' '--- script argument and receipt construction context ---'
rg -n -C 4 'add_argument|args\.out|receipt|evidenceRef|artifact|runtime|operation' scripts/status-version-receipt.py | head -260
printf '%s\n' '--- changed-file status against supplied revisions ---'
git diff --stat 9b1bdf6f28e97dbd9fd490e60bfdd80b1fce732d 2c831f630d0b3480ea00eeaff97200c04dacb770 -- scripts/status-version-receipt.pyRepository: GroepOnline/opencodex Length of output: 9859 🏁 Script executed: set -eu
sed -n '450,515p' .github/workflows/deploy.ymlRepository: GroepOnline/opencodex Length of output: 4351 Write the receipt atomically. Line 247 can leave a truncated JSON receipt if the process stops during the write. The deploy workflow uses this file as producer release evidence for Atlas' publisher. Write a temporary file in the same directory, then replace the receipt with
🛡️ Suggested fix import json
+import os
import re- Path(args.out).write_text(json.dumps(receipt, indent=2) + "\n", encoding="utf-8")
+ out = Path(args.out)
+ tmp = out.with_name(f".{out.name}.{uuid.uuid4().hex}.tmp")
+ try:
+ tmp.write_text(json.dumps(receipt, indent=2) + "\n", encoding="utf-8")
+ os.replace(tmp, out)
+ finally:
+ tmp.unlink(missing_ok=True)🤖 Prompt for AI Agents |
||
| return 0 | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| raise SystemExit(main(sys.argv[1:])) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bug: The
curlcommand is missing|| true. A transient failure could cause the step to fail and trigger an unnecessary rollback of a successful deployment.Severity: MEDIUM
Suggested Fix
Append
|| trueto thecurlcommand on line 464. This will prevent a transient failure in fetching the dashboard HTML from causing the entire deployment step to fail and trigger an unnecessary rollback.Prompt for AI Agent
Did we get this right? 👍 / 👎 to inform future reviews.