fix(release): the nightly readiness report never published — and could call a release ready it had not read - #433
Merged
Conversation
…d call a release ready it had not read Found while fixing the same shell trap in #432. #423's nightly job has not produced a report once: `shell: bash` runs with -e, so the script's exit 1 ("not ready", the normal state) ended the step before `READY=$?` ran. Its first scheduled run failed with `Process completed with exit code 1` and an empty summary, while the workflow's own comment promised never to fail on "not ready". Reviewing it for that turned up four ways the tool could issue a verdict it had not earned, each now shown with a control against #423's code: - an unparseable artifact file was warned about and skipped. With the release's only blocker in a broken file it printed "1/1 artifacts done (100%)" and exited 0: READY; - no visible tags (failed git, shallow clone) defaulted to an empty set and re-targeted the oldest scope in the tree; - the next release was picked in string order: falcon-v1.100.0 before falcon-v1.99.1; - any error exited 1, identical to "not ready". Unreadable directory entries were skipped the same way. Exit codes are now 0 ready / 1 not ready / 2 could not evaluate, and the workflow captures the code with `|| READY=$?` and fails only on 2+, writing an UNKNOWN summary line. `run_at(root, ...)` lets the tests build fixture repos; three mutants each fail exactly their matching test. Backfills the requirement #423 shipped without: SWREQ-RELAY-READY-P01 + FV-RELAY-READY-001, both `implemented`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
avrabe
enabled auto-merge (squash)
September 17, 2026 02:54
avrabe
added a commit
that referenced
this pull request
Sep 17, 2026
…s summary comment since v1.55 (#153) Same missing binary as #436, second consumer. The "Post sticky PR comment" step is masked (continue-on-error, cosmetic by design) and its log on #433 reads: `skipping PR comment — 'gh' not on PATH (self-hosted runner without gh?)`. #433 has zero bot comments. The verdict was never affected; the summary simply never reached a PR. The step now runs ./.github/actions/setup-gh first (same condition). This PR's own Verification gate run is the proof: the summary comment should appear on it. #153's other half (Node 20) is already handled — FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 is set in the gate's environment. Refs #153, #436 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
avrabe
added a commit
that referenced
this pull request
Sep 17, 2026
…d never measured starvation (#436) (#437) * fix(ci): the fleet monitor installs gh instead of assuming it — it had never measured starvation (#436) The `light` runners do not ship `gh`. The starvation step ran `gh api ... 2>/dev/null || true` for months, so `gh: command not found` became an empty result and "0 queued". It printed `queued-too-long: 0` at 22:23Z on 2026-09-16 while #424's required Verification gate had been queued for 60 minutes (107 in total). #432 removed the mask, and the first dispatched run on main (35176122362) correctly failed both jobs as UNKNOWN. Both jobs now use ./.github/actions/setup-gh: gh v2.100.0 (two weeks old; the two-day-old v2.101.0 passed over on purpose), SHA-256 verified against the release's checksums file and a local download, installed user-space into $RUNNER_TEMP. A wrong checksum aborts before extraction (checked locally). FV-RELAY-FLEET-001: records the vacuous history and the observed failure, pins the install with three steps, and states what is still not claimed — the */15 schedule actually ran every 2.4–5.5 h, so unattended latency is hours (#436). Refs #436, #429 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG * fix(ci): the verification gate installs gh too — no PR has carried its summary comment since v1.55 (#153) Same missing binary as #436, second consumer. The "Post sticky PR comment" step is masked (continue-on-error, cosmetic by design) and its log on #433 reads: `skipping PR comment — 'gh' not on PATH (self-hosted runner without gh?)`. #433 has zero bot comments. The verdict was never affected; the summary simply never reached a PR. The step now runs ./.github/actions/setup-gh first (same condition). This PR's own Verification gate run is the proof: the summary comment should appear on it. #153's other half (Node 20) is already handled — FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 is set in the gate's environment. Refs #153, #436 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
avrabe
added a commit
that referenced
this pull request
Sep 17, 2026
… fleet monitor, readiness report (#439) * verify(batt): promote FV-FALCON-BATT-003 — the absent-battery evidence ran in the merge gate (#413) Code-free promotion under the two-commit rule. The implementation landed in #430 (a1b0e17) at `implemented`. Evidence, observed rather than asserted: - #430's REQUIRED Verification gate (run 35137919049, the run that gated the merge) matched FV-FALCON-BATT-003 and executed all four of its steps: PASS — `cargo test -p falcon-core --release absent_battery_blocks_arming` (4.27 s) and the three source greps (Option<f32> defaults to None for read_battery_v and BatteryDriver::voltage; arming requires batt_present). - Re-run on main a1b0e17: all four steps exit 0, and the test really runs (`1 passed`, not a vacuous zero-match filter). - #430 merged with all 8 required contexts SUCCESS. Scope: this promotes the VERIFICATION artifact. The requirement it verifies, SWREQ-FALCON-ORPHAN-P01, is broader (wiring the orphaned crates) and stays `proposed`. #413 stays open until a tagged release carries the fix. Refs #413 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG * verify(vgate): promote FV-RELAY-VGATE-004 — the required Format gate can fail, and on main it passes for real (#409) Code-free promotion. Implementation merged in #424 (4c5feda). Evidence, observed: - The flag is gone: ci.yml has no continue-on-error on `cargo fmt --all -- --check` (the artifact's first step). - main bcbe93e's Format job (CI run 35185137151): step `success`, 0 `Diff in`, no `Process completed with exit code` line — a real pass, not a masked failure (the 3ec4a2f log before #424 had 893 diffs and exit code 1 under a `success` job). - #424's own gate did NOT execute this artifact (default `falcon` filter, and it carries no `falcon` tag), so this PR's Verify-Filter includes it: its two steps run in this PR's gate. Scope: this verifies the Format-gate instance only. SWREQ-RELAY-VGATE-P04 stays `proposed` (its v1.139 scope is recording dark tracks). Refs #409 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG * verify(fleet): promote FLEET-P01 — the monitor evaluates on the light runners it failed on for months (#429, #436) Code-free promotion. Implementation merged in #432 (7daf818) and #437 (bcbe93e). Evidence, observed: - FV-RELAY-FLEET-001's steps executed in #437's merge gate (run 35182187826): all PASS, including `rust-script --test scripts/ci-wedge-watch.rs` and the setup-gh pins. - A run that EVALUATES on main (the artifact's own promotion condition): fleet-status dispatched on bcbe93e, run 35185164979, both jobs on ci-01-8 — `gh-2.100.0.tar.gz: OK`; starvation query succeeded (its "could not be evaluated" step skipped); wedge detector `10 job(s) running, 0 wedged`. ci-01-8 is the box whose scheduled runs printed a vacuous `queued-too-long: 0` before #437. - The detector's first live catch (2026-09-17 00:53Z): Kani (relay-mix-quad) 125.4 min vs 4.0 min median, flagged; recorded on #429. Not claimed (unchanged in the artifact): detection within 15 minutes — scheduled runs are throttled to every few hours (#436, CIFLOW-P01). Refs #429, #436 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG * verify(readiness): promote READY-P01 — on a not-ready tree the nightly job is green and publishes the report Code-free promotion. Implementation merged in #433 (572ae38). Evidence, observed: - FV-RELAY-READY-001's steps executed in #433's merge gate (run 35176152604): all PASS, including the fixture-repo tests of the four verdict bugs. - The artifact's promotion condition — a run on main, GREEN, with a non-empty summary: Release readiness dispatched on 572ae38 (run 35182551834), `Report` step success, printing "## Release readiness — falcon-v1.139.0 / 0 of 15 artifacts done" and the notice "next release still has blocking artifacts". That is exactly the falsification case: the 2026-09-16 scheduled run under the old code failed with exit code 1 and an empty summary on a not-ready tree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Verify-Filter: (and (has-tag "readiness") (has-tag "release"))
Follow-up to #423. I found this while fixing the same shell trap in #432.
The report has never been published
release-readiness.yml's Report step setsshell: bash, and on GitHub that runs asbash --noprofile --norc -e -o pipefail. The script exits 1 for "not ready", which is the normal state, so under-ethe step ended beforeREADY=$?could run. The first scheduled run (2026-09-16) failed withProcess completed with exit code 1, and the job summary was empty. The workflow's own comment said it would never fail on "not ready".Four ways it could issue a verdict it hadn't earned
Each was reproduced with fixture repos, against #423's code as a control:
falcon-v*tags visible (failedgit, shallow clone)falcon-v1.99.0tagged;v1.99.1andv1.100.0scopedUnreadable directory entries were also skipped silently (
filter_map(|e| e.ok())). That is now an error too.The change
0ready,1not ready,2could not evaluate.|| READY=$?, finishes normally on 0 or 1, and on 2 or more writes an UNKNOWN summary line and fails.|| trueaftercargo install rust-scriptis gone. A missing tool now fails at the install step instead of surfacing later as a meaningless exit 127.run_at(root, …)makes the tool testable. The tests build throwaway git repos in which tags point at a blob, so no commit is made and nothing needs signing.Evidence
rust-script --test scripts/release-readiness.rs: 4 passed.::error::falcon-v1.139.0, 0 of 9 done, exit 1, in both text and--markdownforms.actionlintis clean.rivet validatepasses. TheVerify-Filtermatches exactly 1 artifact (FV-RELAY-READY-001), and all 3 of its steps pass locally.Rivet: #423 shipped without a requirement, which is the untracked-work gap this closes. SWREQ-RELAY-READY-P01 and FV-RELAY-READY-001 are both
implemented.Not claimed: a green scheduled run on GitHub. That becomes observable after merge, and it is the promotion evidence.
Falsification: this is wrong if the nightly job fails or writes no summary on a tree that is merely not ready, or if the tool gives a ready/not-ready verdict while an artifact file is unparseable or no release tags are visible.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG