Skip to content

fix(ci): the fleet monitor installs gh instead of assuming it — it had never measured starvation (#436) - #437

Merged
avrabe merged 3 commits into
mainfrom
fix/fleet-monitor-needs-gh
Sep 17, 2026
Merged

fix(ci): the fleet monitor installs gh instead of assuming it — it had never measured starvation (#436)#437
avrabe merged 3 commits into
mainfrom
fix/fleet-monitor-needs-gh

Conversation

@avrabe

@avrabe avrabe commented Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Verify-Filter: (has-tag "fleet")

Refs #436, #429, #153.

What was wrong

gh is not installed on the self-hosted light runners. The fleet monitor's starvation step ran gh api … 2>/dev/null || true from its creation, so gh: command not found turned into an empty result, which it reported as "0 queued". It never measured anything. On 2026-09-16 at 22:23Z it printed queued-too-long: 0 while #424's required Verification gate had been queued for 60 minutes (107 in total). #432 removed the mask, and the first dispatched run on main correctly failed both jobs as UNKNOWN (run 35176122362).

The change

./.github/actions/setup-gh is a local composite action that puts a pinned, checksum-verified gh on PATH. Both fleet-status.yml jobs use it.

  • Version: gh v2.100.0, released 2026-09-03. v2.101.0 was two days old, so I deliberately passed it over.
  • Checksum: SHA-256 e4d4bb44…164be, verified against gh_2.100.0_checksums.txt and a local download. A republished asset fails sha256sum -c.
  • Install: user-space, into $RUNNER_TEMP, with no root.

Evidence: run on the real runners before merge

This workflow can't run from a PR, so I dispatched it on this branch (run 35176388374). success:

job runner what it logged
status ci-01-5 gh-2.100.0.tar.gz: OK · gh version 2.100.0 · queued-too-long: 0; the "could not be evaluated" step was skipped, so the query succeeded
wedge ci-01-10 gh-2.100.0.tar.gz: OK · 5 job(s) running, 0 wedged, 1 without a baseline (itself)

I also checked locally that a wrong checksum stops the install before extraction: FAILED, exit 1, and extraction never runs.

actionlint is clean. rivet validate passes. Verify-Filter matches exactly 1 artifact (FV-RELAY-FLEET-001), and all its steps pass locally, including three new ones that pin the install.

Also: the verification gate's PR comment (#153)

The same missing gh has kept every PR from receiving the verification summary comment since the v1.55 runner migration. The masked step logs skipping PR comment — 'gh' not on PATH, and #433 has zero bot comments. verification-gate.yml now runs the same setup-gh step before posting. This PR's own Verification gate run is the test: the summary comment should appear here. (#153's other half, Node 20, is already handled: FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 is set.)

Not fixed here

The */15 schedule actually ran every 2.4–5.5 h on 2026-09-16, because GitHub throttles scheduled workflows. Unattended detection latency is therefore hours. That stays open on #436.

Falsification: this is wrong if a scheduled or dispatched run of either job on a light runner reports a result without gh having been installed and checksum-verified, or reports green when its API query failed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

avrabe and others added 3 commits September 17, 2026 04:58
@avrabe @claude
fix(ci): the fleet monitor installs gh instead of assuming it — it ha…
ad39e27 
…d never measured starvation (#436)

The `light` runners do not ship `gh`. The starvation step ran
`gh api ... 2>/dev/null || true` for months, so `gh: command not found`
became an empty result and "0 queued". It printed `queued-too-long: 0` at
22:23Z on 2026-09-16 while #424's required Verification gate had been queued
for 60 minutes (107 in total). #432 removed the mask, and the first dispatched
run on main (35176122362) correctly failed both jobs as UNKNOWN.

Both jobs now use ./.github/actions/setup-gh: gh v2.100.0 (two weeks old; the
two-day-old v2.101.0 passed over on purpose), SHA-256 verified against the
release's checksums file and a local download, installed user-space into
$RUNNER_TEMP. A wrong checksum aborts before extraction (checked locally).

FV-RELAY-FLEET-001: records the vacuous history and the observed failure,
pins the install with three steps, and states what is still not claimed —
the */15 schedule actually ran every 2.4–5.5 h, so unattended latency is
hours (#436).

Refs #436, #429

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
@avrabe
Merge remote-tracking branch 'origin/main' into fix/fleet-monitor-nee…
b0f4b26 
…ds-gh
@avrabe @claude
fix(ci): the verification gate installs gh too — no PR has carried it…
a8c460e 
…s summary comment since v1.55 (#153)

Same missing binary as #436, second consumer. The "Post sticky PR comment"
step is masked (continue-on-error, cosmetic by design) and its log on #433
reads: `skipping PR comment — 'gh' not on PATH (self-hosted runner without
gh?)`. #433 has zero bot comments. The verdict was never affected; the
summary simply never reached a PR.

The step now runs ./.github/actions/setup-gh first (same condition). This
PR's own Verification gate run is the proof: the summary comment should
appear on it. #153's other half (Node 20) is already handled —
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 is set in the gate's environment.

Refs #153, #436

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
@avrabe
avrabe enabled auto-merge (squash) September 17, 2026 04:30
@github-actions

github-actions Bot commented Sep 17, 2026

Copy link
Copy Markdown

✅ Rivet verification gate — falcon

1/1 passed

count
Passed 1
Failed 0
Skipped (bench-only — needs hardware / sim) 0
Skipped (no steps) 0

Source of truth: artifacts/verification/FV-FALCON-*.yaml.

@avrabe
avrabe merged commit bcbe93e into main Sep 17, 2026
12 checks passed
@avrabe
avrabe deleted the fix/fleet-monitor-needs-gh branch September 17, 2026 05:17
@avrabe avrabe mentioned this pull request Sep 17, 2026
Merged
avrabe added a commit that referenced this pull request Sep 17, 2026
@avrabe @claude
verify(v1.139): promote the four merged fixes — battery, Format gate,…
916fbca 
… fleet monitor, readiness report (#439)

* verify(batt): promote FV-FALCON-BATT-003 — the absent-battery evidence ran in the merge gate (#413)

Code-free promotion under the two-commit rule. The implementation landed in
#430 (a1b0e17) at `implemented`.

Evidence, observed rather than asserted:
- #430's REQUIRED Verification gate (run 35137919049, the run that gated the
  merge) matched FV-FALCON-BATT-003 and executed all four of its steps:
  PASS — `cargo test -p falcon-core --release absent_battery_blocks_arming`
  (4.27 s) and the three source greps (Option<f32> defaults to None for
  read_battery_v and BatteryDriver::voltage; arming requires batt_present).
- Re-run on main a1b0e17: all four steps exit 0, and the test really runs
  (`1 passed`, not a vacuous zero-match filter).
- #430 merged with all 8 required contexts SUCCESS.

Scope: this promotes the VERIFICATION artifact. The requirement it verifies,
SWREQ-FALCON-ORPHAN-P01, is broader (wiring the orphaned crates) and stays
`proposed`. #413 stays open until a tagged release carries the fix.

Refs #413

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* verify(vgate): promote FV-RELAY-VGATE-004 — the required Format gate can fail, and on main it passes for real (#409)

Code-free promotion. Implementation merged in #424 (4c5feda).

Evidence, observed:
- The flag is gone: ci.yml has no continue-on-error on `cargo fmt --all
  -- --check` (the artifact's first step).
- main bcbe93e's Format job (CI run 35185137151): step `success`, 0 `Diff in`,
  no `Process completed with exit code` line — a real pass, not a masked
  failure (the 3ec4a2f log before #424 had 893 diffs and exit code 1 under
  a `success` job).
- #424's own gate did NOT execute this artifact (default `falcon` filter,
  and it carries no `falcon` tag), so this PR's Verify-Filter includes it:
  its two steps run in this PR's gate.

Scope: this verifies the Format-gate instance only. SWREQ-RELAY-VGATE-P04
stays `proposed` (its v1.139 scope is recording dark tracks).

Refs #409

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* verify(fleet): promote FLEET-P01 — the monitor evaluates on the light runners it failed on for months (#429, #436)

Code-free promotion. Implementation merged in #432 (7daf818) and #437
(bcbe93e).

Evidence, observed:
- FV-RELAY-FLEET-001's steps executed in #437's merge gate (run
  35182187826): all PASS, including `rust-script --test
  scripts/ci-wedge-watch.rs` and the setup-gh pins.
- A run that EVALUATES on main (the artifact's own promotion condition):
  fleet-status dispatched on bcbe93e, run 35185164979, both jobs on
  ci-01-8 — `gh-2.100.0.tar.gz: OK`; starvation query succeeded (its
  "could not be evaluated" step skipped); wedge detector `10 job(s)
  running, 0 wedged`. ci-01-8 is the box whose scheduled runs printed a
  vacuous `queued-too-long: 0` before #437.
- The detector's first live catch (2026-09-17 00:53Z): Kani
  (relay-mix-quad) 125.4 min vs 4.0 min median, flagged; recorded on #429.

Not claimed (unchanged in the artifact): detection within 15 minutes —
scheduled runs are throttled to every few hours (#436, CIFLOW-P01).

Refs #429, #436

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* verify(readiness): promote READY-P01 — on a not-ready tree the nightly job is green and publishes the report

Code-free promotion. Implementation merged in #433 (572ae38).

Evidence, observed:
- FV-RELAY-READY-001's steps executed in #433's merge gate (run
  35176152604): all PASS, including the fixture-repo tests of the four
  verdict bugs.
- The artifact's promotion condition — a run on main, GREEN, with a
  non-empty summary: Release readiness dispatched on 572ae38 (run
  35182551834), `Report` step success, printing "## Release readiness —
  falcon-v1.139.0 / 0 of 15 artifacts done" and the notice "next release
  still has blocking artifacts". That is exactly the falsification case:
  the 2026-09-16 scheduled run under the old code failed with exit code 1
  and an empty summary on a not-ready tree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 17, 2026
Open
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@avrabe