Skip to content

fix(ci): the required Format gate has never been able to fail - #424

Merged
avrabe merged 2 commits into
mainfrom
fix/format-gate-can-fail
Sep 17, 2026
Merged

fix(ci): the required Format gate has never been able to fail#424
avrabe merged 2 commits into
mainfrom
fix/format-gate-can-fail

Conversation

@avrabe

@avrabe avrabe commented Sep 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Closes #409.

What

Format is one of the required status checks on main. Its only
substantive step carried continue-on-error: true:

- run: cargo fmt --all -- --check
  continue-on-error: true

A failing cargo fmt --check left the job conclusion success. It was
structurally incapable of going red — and was promoted to a required context
while in that state.

git log -S dates the flag to d30a5e3, 2026-05-18 — the v0.1.0 release. It
has been inert for its entire life, about 137 releases. The comment beside it
read "v0.1 tolerates them while v0.2 cleans the workspace up." We are at
v1.138.

What the mask was hiding: a live violation

(This section originally said the opposite. The correction is below.)

CI's own rustfmt on main 3ec4a2f (Format job 104933185124) logged 893 Diff in hunks and Process completed with exit code 1, and the job reported success. The jobs API also reports that step as success, because continue-on-error rewrites a failed step's conclusion; only the log shows the failure. The 106-file reformat in this PR is the debt the mask hid.

Correction. The first version of this PR (and of FV-RELAY-VGATE-004 and the new ci.yml comment) said the mask was not hiding a violation, attributed ~891 local hunks to "the known local-vs-CI rustfmt divergence", and told readers not to run cargo fmt. All three were wrong in the same way: they read the masked job conclusion as the step result. There is no divergence. The reformat generated locally passed this PR's unmasked check, and it passed again when regenerated on the new main. All three texts are corrected in this push.

Rebuilt, not hand-merged. After #430 merged, both commits were rebuilt on a1b0e17: the fix commit was cherry-picked with the corrections, and cargo fmt --all was regenerated. It touched the same 106 files as the first generation, plus #430/#431's new lines in four of them, and cargo fmt --all -- --check exits 0. The previous head, 193e32e, is kept for reference.

Verification

FV-RELAY-VGATE-004 at implemented (promotion to verified is the separate
code-free PR). Its two steps assert the flag is absent and the check
still exists, so the fix cannot be "delete the check". Both were executed
before the artifact was written.

What stays open

The artifact names the other instances of this class rather than implying they
are closed: #405 (Verus cannot find core/std), #418 (Verus PR trigger
filtered on Lean paths), #410 (verification gate has no main backstop).

🤖 Generated with Claude Code

https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

This was referenced Sep 16, 2026
Closed
Closed
Merged
@avrabe
avrabe enabled auto-merge (squash) September 16, 2026 13:08
@avrabe avrabe mentioned this pull request Sep 16, 2026
Open
avrabe and others added 2 commits September 16, 2026 23:23
@avrabe @claude
fix(ci): the required Format gate has never been able to fail (#409)
2a0349d 
`Format` is a required status check. Its only substantive step carried
`continue-on-error: true` from d30a5e3 (2026-05-18, the v0.1.0 release), so a
failing `cargo fmt --all -- --check` left the job `success` — for its entire
life, about 137 releases.

The mask was hiding a LIVE violation, not merely lost potency: on main
3ec4a2f the step logged 893 `Diff in` hunks and `exit code 1` while the job
reported success. The jobs API reports such a step as `success` too; only the
log shows it. An earlier draft of this change (and a months-old working note)
blamed local-vs-CI rustfmt divergence and told readers NOT to run cargo fmt.
There was no divergence — the locally generated reformat passes the unmasked
check — and that advice is removed. The correction is kept in
FV-RELAY-VGATE-004 because it is the same error the mask produces.

Also: FV-RELAY-VGATE-004 `method: test` -> `automated-test` (schema value).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
@avrabe @claude
style: rustfmt the tree — the debt the Format mask was hiding (#409)
fd4a18f 
`cargo fmt --all`, regenerated on main a1b0e17 (after #430 and #431) rather
than hand-merged: the same 106 files as the first generation, plus the new
lines #430/#431 added to four of them. `cargo fmt --all -- --check` exits 0.
No hand edits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
@avrabe
avrabe force-pushed the fix/format-gate-can-fail branch from 193e32e to fd4a18f Compare September 16, 2026 21:23
@avrabe
avrabe merged commit 4c5feda into main Sep 17, 2026
150 of 154 checks passed
@avrabe
avrabe deleted the fix/format-gate-can-fail branch September 17, 2026 01:53
avrabe added a commit that referenced this pull request Sep 17, 2026
@avrabe @claude
fix(ci): cap Kani matrix legs at 60 min — a hang becomes a red gate, …
9d1280f 
…not a 6-hour hold (#429)

The wedge detector's first live catch: `Kani (relay-mix-quad)` on ci-01-10
at 125.4 min against a 4.0-min median, the only check holding #424. Its
fourth hang, all inside the Kani step, on three different runners — a
per-leg intermittent hang, not one bad box. kani.yml set no
`timeout-minutes`, so every hang held its runner for GitHub's 360-min
default while the PR looked merely slow.

60 min is 2.4x the slowest of 180 measured successful leg runs
(relay-notch, 25.1 min). A timed-out leg concludes `cancelled`, and the
`kani-gate` aggregate fails closed on anything but success/skipped, so the
hang turns the required gate red instead of hiding.

FV-RELAY-FLEET-001 gains a step pinning the cap.

Refs #429

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
avrabe added a commit that referenced this pull request Sep 17, 2026
@avrabe @claude
fix(ci): the fleet monitor can see a wedged job — and "could not eval…
7daf818 
…uate" is no longer green (#429) (#432)

* fix(ci): the fleet monitor can see a wedged job — and "could not evaluate" is no longer green (#429)

A job stuck in_progress holds its runner and looks exactly like a slow one.
On 2026-09-16 `Kani (relay-mix-quad)` wedged three times (52, 96, 105 min
against 2m48s–3m21s completions), each found by a human reading durations.
fleet-status.yml watched for QUEUED runs, a shape a wedge never has.

scripts/ci-wedge-watch.rs flags a job running past 5x the median of its own
recent successful runs (floor 10 min, >= 3 samples, otherwise "no baseline").
Exit 0 / 1 / 2 = clear / wedged / could not evaluate. It reports; it cancels
nothing. A new hosted `wedge` job runs it every 15 minutes and keeps one
rolling issue, naming each wedged job once.

Three traps, each measured before it was designed around:
- run status is not job status: a matrix run with queued legs reports
  `queued` while a job runs, so jobs come from queued AND in_progress runs;
- the last 5 COMPLETED Kani runs held one successful relay-mix-quad leg
  (cancel-in-progress), below the sample minimum, which would have made the
  #429 wedge itself "no baseline"; history comes from SUCCESSFUL runs;
- `shell: bash` runs with -e, so reading $? after a bare call that exits 1
  loses the code and turns a wedge into UNKNOWN. Found by executing the
  extracted blocks under GitHub's flags, with a control; fixed with || rc=$?.

The starvation step's `|| true` swallowed API failures into an empty queue.
Both checks now report UNKNOWN and fail the monitor's own run.

SWREQ-RELAY-FLEET-P01 + FV-RELAY-FLEET-001, both `implemented`.

Refs #429

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* fix(ci): run the wedge alarm on `light`, not hosted — and render its output without jq

Reverses this PR's own placement decision on measurement. On 2026-09-16
relay's hosted jobs waited 60–120 min for a runner (hosted concurrency is
shared across the org; Verus and Rocq created 18:55 started 20:53), so a
hosted wedge alarm could arrive about as late as the human it replaces.
The `status` job already runs on `light`, which is three boxes
(ci-01-5/-8/-10), not the single box an earlier comment claimed.

`jq` is not known to be installed on `light`, so the detector now writes the
job-summary markdown (--summary-md) and the alarm rows (--alarm-tsv,
`<job url>\t<row>`) itself; the workflow uses no jq. New test covers both
renderers. The FV step pinning the `|| rc=$?` fix was repointed at the new
line; checked that it fails against a copy without the fix.

Refs #429

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* test(ci): the relay-notch test used invented durations — use the measured ones

`a_leg_running_normally_is_not_flagged` was named for Kani (relay-notch)
but held made-up ~2-minute durations. Its real successful runs on
2026-09-16 took 888, 816 and 656 s (median 13.6 min). That surfaced live:
with #430's relay-notch leg at 18.8 min I suspected a wedge from a
remembered two-minute sighting; the detector, reading the history, did not
flag it, and was right (threshold 68 min). The test now uses the measured
durations and pins the per-job threshold.

Refs #429

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* fix(ci): cap Kani matrix legs at 60 min — a hang becomes a red gate, not a 6-hour hold (#429)

The wedge detector's first live catch: `Kani (relay-mix-quad)` on ci-01-10
at 125.4 min against a 4.0-min median, the only check holding #424. Its
fourth hang, all inside the Kani step, on three different runners — a
per-leg intermittent hang, not one bad box. kani.yml set no
`timeout-minutes`, so every hang held its runner for GitHub's 360-min
default while the PR looked merely slow.

60 min is 2.4x the slowest of 180 measured successful leg runs
(relay-notch, 25.1 min). A timed-out leg concludes `cancelled`, and the
`kani-gate` aggregate fails closed on anything but success/skipped, so the
hang turns the required gate red instead of hiding.

FV-RELAY-FLEET-001 gains a step pinning the cap.

Refs #429

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 17, 2026
Open
Merged
Closed
avrabe added a commit that referenced this pull request Sep 17, 2026
@avrabe @claude
fix(ci): the fleet monitor installs gh instead of assuming it — it ha…
bcbe93e 
…d never measured starvation (#436) (#437)

* fix(ci): the fleet monitor installs gh instead of assuming it — it had never measured starvation (#436)

The `light` runners do not ship `gh`. The starvation step ran
`gh api ... 2>/dev/null || true` for months, so `gh: command not found`
became an empty result and "0 queued". It printed `queued-too-long: 0` at
22:23Z on 2026-09-16 while #424's required Verification gate had been queued
for 60 minutes (107 in total). #432 removed the mask, and the first dispatched
run on main (35176122362) correctly failed both jobs as UNKNOWN.

Both jobs now use ./.github/actions/setup-gh: gh v2.100.0 (two weeks old; the
two-day-old v2.101.0 passed over on purpose), SHA-256 verified against the
release's checksums file and a local download, installed user-space into
$RUNNER_TEMP. A wrong checksum aborts before extraction (checked locally).

FV-RELAY-FLEET-001: records the vacuous history and the observed failure,
pins the install with three steps, and states what is still not claimed —
the */15 schedule actually ran every 2.4–5.5 h, so unattended latency is
hours (#436).

Refs #436, #429

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* fix(ci): the verification gate installs gh too — no PR has carried its summary comment since v1.55 (#153)

Same missing binary as #436, second consumer. The "Post sticky PR comment"
step is masked (continue-on-error, cosmetic by design) and its log on #433
reads: `skipping PR comment — 'gh' not on PATH (self-hosted runner without
gh?)`. #433 has zero bot comments. The verdict was never affected; the
summary simply never reached a PR.

The step now runs ./.github/actions/setup-gh first (same condition). This
PR's own Verification gate run is the proof: the summary comment should
appear on it. #153's other half (Node 20) is already handled —
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 is set in the gate's environment.

Refs #153, #436

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@avrabe avrabe mentioned this pull request Sep 17, 2026
Merged
avrabe added a commit that referenced this pull request Sep 17, 2026
@avrabe @claude
verify(v1.139): promote the four merged fixes — battery, Format gate,…
916fbca 
… fleet monitor, readiness report (#439)

* verify(batt): promote FV-FALCON-BATT-003 — the absent-battery evidence ran in the merge gate (#413)

Code-free promotion under the two-commit rule. The implementation landed in
#430 (a1b0e17) at `implemented`.

Evidence, observed rather than asserted:
- #430's REQUIRED Verification gate (run 35137919049, the run that gated the
  merge) matched FV-FALCON-BATT-003 and executed all four of its steps:
  PASS — `cargo test -p falcon-core --release absent_battery_blocks_arming`
  (4.27 s) and the three source greps (Option<f32> defaults to None for
  read_battery_v and BatteryDriver::voltage; arming requires batt_present).
- Re-run on main a1b0e17: all four steps exit 0, and the test really runs
  (`1 passed`, not a vacuous zero-match filter).
- #430 merged with all 8 required contexts SUCCESS.

Scope: this promotes the VERIFICATION artifact. The requirement it verifies,
SWREQ-FALCON-ORPHAN-P01, is broader (wiring the orphaned crates) and stays
`proposed`. #413 stays open until a tagged release carries the fix.

Refs #413

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* verify(vgate): promote FV-RELAY-VGATE-004 — the required Format gate can fail, and on main it passes for real (#409)

Code-free promotion. Implementation merged in #424 (4c5feda).

Evidence, observed:
- The flag is gone: ci.yml has no continue-on-error on `cargo fmt --all
  -- --check` (the artifact's first step).
- main bcbe93e's Format job (CI run 35185137151): step `success`, 0 `Diff in`,
  no `Process completed with exit code` line — a real pass, not a masked
  failure (the 3ec4a2f log before #424 had 893 diffs and exit code 1 under
  a `success` job).
- #424's own gate did NOT execute this artifact (default `falcon` filter,
  and it carries no `falcon` tag), so this PR's Verify-Filter includes it:
  its two steps run in this PR's gate.

Scope: this verifies the Format-gate instance only. SWREQ-RELAY-VGATE-P04
stays `proposed` (its v1.139 scope is recording dark tracks).

Refs #409

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* verify(fleet): promote FLEET-P01 — the monitor evaluates on the light runners it failed on for months (#429, #436)

Code-free promotion. Implementation merged in #432 (7daf818) and #437
(bcbe93e).

Evidence, observed:
- FV-RELAY-FLEET-001's steps executed in #437's merge gate (run
  35182187826): all PASS, including `rust-script --test
  scripts/ci-wedge-watch.rs` and the setup-gh pins.
- A run that EVALUATES on main (the artifact's own promotion condition):
  fleet-status dispatched on bcbe93e, run 35185164979, both jobs on
  ci-01-8 — `gh-2.100.0.tar.gz: OK`; starvation query succeeded (its
  "could not be evaluated" step skipped); wedge detector `10 job(s)
  running, 0 wedged`. ci-01-8 is the box whose scheduled runs printed a
  vacuous `queued-too-long: 0` before #437.
- The detector's first live catch (2026-09-17 00:53Z): Kani
  (relay-mix-quad) 125.4 min vs 4.0 min median, flagged; recorded on #429.

Not claimed (unchanged in the artifact): detection within 15 minutes —
scheduled runs are throttled to every few hours (#436, CIFLOW-P01).

Refs #429, #436

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* verify(readiness): promote READY-P01 — on a not-ready tree the nightly job is green and publishes the report

Code-free promotion. Implementation merged in #433 (572ae38).

Evidence, observed:
- FV-RELAY-READY-001's steps executed in #433's merge gate (run
  35176152604): all PASS, including the fixture-repo tests of the four
  verdict bugs.
- The artifact's promotion condition — a run on main, GREEN, with a
  non-empty summary: Release readiness dispatched on 572ae38 (run
  35182551834), `Report` step success, printing "## Release readiness —
  falcon-v1.139.0 / 0 of 15 artifacts done" and the notice "next release
  still has blocking artifacts". That is exactly the falsification case:
  the 2026-09-16 scheduled run under the old code failed with exit code 1
  and an empty summary on a not-ready tree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@avrabe avrabe mentioned this pull request Sep 17, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

The required Format check has NEVER been able to fail — continue-on-error since v0.1.0

1 participant

@avrabe