Skip to content

fix(ci): the fleet monitor can see a wedged job — and "could not evaluate" is no longer green (#429) - #432

Merged
avrabe merged 5 commits into
mainfrom
fix/ci-wedge-alarm
Sep 17, 2026
Merged

fix(ci): the fleet monitor can see a wedged job — and "could not evaluate" is no longer green (#429)#432
avrabe merged 5 commits into
mainfrom
fix/ci-wedge-alarm

Conversation

@avrabe

@avrabe avrabe commented Sep 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Verify-Filter: (has-tag "fleet")

Refs #429.

What was wrong

A CI job stuck in_progress holds its runner and blocks its PR while looking exactly like a slow job. On 2026-09-16 Kani (relay-mix-quad) wedged three times: 52, 96 and 105 minutes, against normal completions of 2m48s–3m21s. The same leg finished normally four times that day, twice while a wedged instance was still running. A human found each one by reading durations.

fleet-status.yml could not see this. It watches for runs queued too long (starvation), and a wedged job has a runner and reports itself in_progress.

The change

  • scripts/ci-wedge-watch.rs (rust-script). Flags a job running past 5× the median of its own recent successful runs, with a 10-minute floor. With fewer than 3 samples it reports "no baseline" and does not guess. Exit codes are 0 clear, 1 wedged, 2 could not evaluate. It only reports; it cancels nothing.
  • A new wedge job, on light, alongside status. The detector writes the summary markdown and the alarm rows itself, so the job needs no jq. It keeps one rolling issue and names each wedged job once, not every 15 minutes. (The first draft ran on hosted runners. That was reversed on measurement; see the comments.)
  • The starvation step no longer turns a failed API call into a green result (|| true → UNKNOWN). Both checks now fail the monitor's own run when they cannot evaluate.

Three traps, each measured before designing around it

Trap Evidence Consequence if unhandled
Run status ≠ job status #430's CI run read queued while Format had already completed a wedged leg in a partly-queued matrix run is invisible
Completed runs are mostly cancelled the last 5 completed Kani runs held 1 successful relay-mix-quad leg; the last 5 successful runs held 3 the #429 wedge itself would read "no baseline" and never alarm
shell: bash runs with -e the extracted detect step, run under bash --noprofile --norc -eo pipefail: the old form with detector exit 1 ended the step with nothing recorded (control) a real wedge would report as UNKNOWN and skip the alarm

The third trap was a bug in this PR's first draft. Running the blocks with plain bash had hidden it.

Evidence

  • rust-script --test scripts/ci-wedge-watch.rs: 7 passed. These tests replay CI jobs wedge in_progress for 30x their normal duration, and the fleet monitor cannot see it (watches queued, not stuck) #429's durations: not flagged at 14 min, flagged at 15 and at 105. A mutant (elapsed_s > threshold_s * 10) fails 2 of them.
  • Live against pulseengine/loom (21 jobs running; relay had 0, so a relay-only run would prove nothing):
    • real policy: 0 wedged, 4 without a baseline, exit 0
    • forced to 0.01×: 17 flagged, exit 1
    • nonexistent repo: exit 2, with the 404 in the message
  • The summary and alarm blocks were run under GitHub's flags against that JSON, with a stub gh:
    • the summary rendered the green, wedged and UNKNOWN forms
    • the alarm posted 17 rows with no existing issue, and 16 when an existing issue already named one job
  • actionlint is clean, apart from the pre-existing custom light runner-label notice. shellcheck is not installed locally, which is why the blocks were executed rather than trusted.
  • rivet validate passes. Verify-Filter: (has-tag "fleet") matches exactly 1 artifact (FV-RELAY-FLEET-001), and all 5 of its steps pass locally.

Not claimed: that the scheduled workflow has run on GitHub. That becomes observable after merge, and it is the evidence for promoting FLEET-P01 to verified in a separate change.

Falsification: this is wrong if a job stuck in_progress past 5× its median (with ≥3 successful runs of history) produces no wedge report, or if the monitor reports green on a run where its API queries failed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

@avrabe @claude
fix(ci): the fleet monitor can see a wedged job — and "could not eval…
58d9c35 
…uate" is no longer green (#429)

A job stuck in_progress holds its runner and looks exactly like a slow one.
On 2026-09-16 `Kani (relay-mix-quad)` wedged three times (52, 96, 105 min
against 2m48s–3m21s completions), each found by a human reading durations.
fleet-status.yml watched for QUEUED runs, a shape a wedge never has.

scripts/ci-wedge-watch.rs flags a job running past 5x the median of its own
recent successful runs (floor 10 min, >= 3 samples, otherwise "no baseline").
Exit 0 / 1 / 2 = clear / wedged / could not evaluate. It reports; it cancels
nothing. A new hosted `wedge` job runs it every 15 minutes and keeps one
rolling issue, naming each wedged job once.

Three traps, each measured before it was designed around:
- run status is not job status: a matrix run with queued legs reports
  `queued` while a job runs, so jobs come from queued AND in_progress runs;
- the last 5 COMPLETED Kani runs held one successful relay-mix-quad leg
  (cancel-in-progress), below the sample minimum, which would have made the
  #429 wedge itself "no baseline"; history comes from SUCCESSFUL runs;
- `shell: bash` runs with -e, so reading $? after a bare call that exits 1
  loses the code and turns a wedge into UNKNOWN. Found by executing the
  extracted blocks under GitHub's flags, with a control; fixed with || rc=$?.

The starvation step's `|| true` swallowed API failures into an empty queue.
Both checks now report UNKNOWN and fail the monitor's own run.

SWREQ-RELAY-FLEET-P01 + FV-RELAY-FLEET-001, both `implemented`.

Refs #429

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
@avrabe
avrabe enabled auto-merge (squash) September 16, 2026 19:23
@avrabe
avrabe disabled auto-merge September 16, 2026 19:24
@avrabe

avrabe commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

A limitation of the hosted placement, measured while this PR waited.

At 19:24Z, #430's hosted Test (windows) and Test (macos) legs had been queued 24 minutes. Hosted concurrency is shared across the org, and pulseengine/loom was holding about 15 hosted jobs at the time (the same loom run this PR's live evidence was taken from).

The new wedge job runs from that same pool, so while the org's hosted pool is saturated, the wedge check runs late. It is still correct when it runs; it is delayed, not wrong. The status job (starvation) stays on light and is unaffected.

I'm keeping the placement anyway. The alternative is a self-hosted runner, which would put the monitor back on the fleet it watches (or on light, the one box behind the required Format check). A late wedge alarm is still an alarm, which is more than a monitor that fails along with its fleet gives you. This note is here so nobody later reads a delayed wedge report as a bug in the detector.

🤖 Generated with Claude Code

@avrabe avrabe mentioned this pull request Sep 16, 2026
Merged
@avrabe @claude
fix(ci): run the wedge alarm on light, not hosted — and render its …
942f794 
…output without jq

Reverses this PR's own placement decision on measurement. On 2026-09-16
relay's hosted jobs waited 60–120 min for a runner (hosted concurrency is
shared across the org; Verus and Rocq created 18:55 started 20:53), so a
hosted wedge alarm could arrive about as late as the human it replaces.
The `status` job already runs on `light`, which is three boxes
(ci-01-5/-8/-10), not the single box an earlier comment claimed.

`jq` is not known to be installed on `light`, so the detector now writes the
job-summary markdown (--summary-md) and the alarm rows (--alarm-tsv,
`<job url>\t<row>`) itself; the workflow uses no jq. New test covers both
renderers. The FV step pinning the `|| rc=$?` fix was repointed at the new
line; checked that it fails against a copy without the fix.

Refs #429

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
@avrabe

avrabe commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Reversing my earlier comment: the wedge job now runs on light, not hosted runners.

Earlier I wrote that saturation would only make a hosted job "late" and kept the placement. The same evening measured how late: relay's hosted jobs waited 60–120 minutes for a runner.

hosted job created started
Verus proofs (main) 18:55:35 20:53:44
Rocq + Gappa proofs (main) 18:55:35 20:54:09
Component Model cascade (#430) 19:00:31 20:56:06
Test (macos) (#432) 19:24:00 20:57:49

At that delay, a wedge alarm arrives about as late as the human it was meant to replace.

Commit 942f794:

  • Runner: runs-on: [self-hosted, linux, x64, light], the same as the status job. light is three boxes today (ci-01-5, -8, -10), not the single box my original comment claimed; that came from a stale note.
  • Install cost: a cold rust-script install measured 0–49 s on self-hosted.
  • No jq. I couldn't find any evidence that jq is installed on light, so the detector now writes the job-summary markdown (--summary-md) and the alarm rows (--alarm-tsv, <job url>\t<row>) itself. A new test covers both renderers (8 tests pass).
  • The detect, summary and alarm blocks were re-run under bash --noprofile --norc -eo pipefail, with files the real detector wrote against loom:
    • detector exits 0, 1 and 2 each record the matching rc
    • the UNKNOWN banner appears on exit 2
    • the alarm posts 7 rows, or 6 when an existing issue already names one job
  • FV-RELAY-FLEET-001's step pinning || rc=$? had drifted with the line split, and the gate would have failed on it. It is repointed, and it fails against a copy without the fix.

🤖 Generated with Claude Code

avrabe and others added 2 commits September 16, 2026 23:15
@avrabe @claude
test(ci): the relay-notch test used invented durations — use the meas…
fd2b9e2 
…ured ones

`a_leg_running_normally_is_not_flagged` was named for Kani (relay-notch)
but held made-up ~2-minute durations. Its real successful runs on
2026-09-16 took 888, 816 and 656 s (median 13.6 min). That surfaced live:
with #430's relay-notch leg at 18.8 min I suspected a wedge from a
remembered two-minute sighting; the detector, reading the history, did not
flag it, and was right (threshold 68 min). The test now uses the measured
durations and pins the per-job threshold.

Refs #429

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
@avrabe @claude
fix(ci): cap Kani matrix legs at 60 min — a hang becomes a red gate, …
9d1280f 
…not a 6-hour hold (#429)

The wedge detector's first live catch: `Kani (relay-mix-quad)` on ci-01-10
at 125.4 min against a 4.0-min median, the only check holding #424. Its
fourth hang, all inside the Kani step, on three different runners — a
per-leg intermittent hang, not one bad box. kani.yml set no
`timeout-minutes`, so every hang held its runner for GitHub's 360-min
default while the PR looked merely slow.

60 min is 2.4x the slowest of 180 measured successful leg runs
(relay-notch, 25.1 min). A timed-out leg concludes `cancelled`, and the
`kani-gate` aggregate fails closed on anything but success/skipped, so the
hang turns the required gate red instead of hiding.

FV-RELAY-FLEET-001 gains a step pinning the cap.

Refs #429

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
@avrabe avrabe mentioned this pull request Sep 17, 2026
Open
@avrabe
avrabe enabled auto-merge (squash) September 17, 2026 01:57
@avrabe
avrabe merged commit 7daf818 into main Sep 17, 2026
56 checks passed
@avrabe
avrabe deleted the fix/ci-wedge-alarm branch September 17, 2026 02:28
This was referenced Sep 17, 2026
Open
Merged
avrabe added a commit that referenced this pull request Sep 17, 2026
@avrabe @claude
fix(release): the nightly readiness report never published — and coul…
572ae38 
…d call a release ready it had not read (#433)

Found while fixing the same shell trap in #432. #423's nightly job has not
produced a report once: `shell: bash` runs with -e, so the script's exit 1
("not ready", the normal state) ended the step before `READY=$?` ran. Its
first scheduled run failed with `Process completed with exit code 1` and an
empty summary, while the workflow's own comment promised never to fail on
"not ready".

Reviewing it for that turned up four ways the tool could issue a verdict it
had not earned, each now shown with a control against #423's code:
- an unparseable artifact file was warned about and skipped. With the
  release's only blocker in a broken file it printed "1/1 artifacts done
  (100%)" and exited 0: READY;
- no visible tags (failed git, shallow clone) defaulted to an empty set and
  re-targeted the oldest scope in the tree;
- the next release was picked in string order: falcon-v1.100.0 before
  falcon-v1.99.1;
- any error exited 1, identical to "not ready". Unreadable directory entries
  were skipped the same way.

Exit codes are now 0 ready / 1 not ready / 2 could not evaluate, and the
workflow captures the code with `|| READY=$?` and fails only on 2+, writing
an UNKNOWN summary line. `run_at(root, ...)` lets the tests build fixture
repos; three mutants each fail exactly their matching test.

Backfills the requirement #423 shipped without: SWREQ-RELAY-READY-P01 +
FV-RELAY-READY-001, both `implemented`.


Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@avrabe avrabe mentioned this pull request Sep 17, 2026
Closed
avrabe added a commit that referenced this pull request Sep 17, 2026
@avrabe @claude
fix(ci): the fleet monitor installs gh instead of assuming it — it ha…
bcbe93e 
…d never measured starvation (#436) (#437)

* fix(ci): the fleet monitor installs gh instead of assuming it — it had never measured starvation (#436)

The `light` runners do not ship `gh`. The starvation step ran
`gh api ... 2>/dev/null || true` for months, so `gh: command not found`
became an empty result and "0 queued". It printed `queued-too-long: 0` at
22:23Z on 2026-09-16 while #424's required Verification gate had been queued
for 60 minutes (107 in total). #432 removed the mask, and the first dispatched
run on main (35176122362) correctly failed both jobs as UNKNOWN.

Both jobs now use ./.github/actions/setup-gh: gh v2.100.0 (two weeks old; the
two-day-old v2.101.0 passed over on purpose), SHA-256 verified against the
release's checksums file and a local download, installed user-space into
$RUNNER_TEMP. A wrong checksum aborts before extraction (checked locally).

FV-RELAY-FLEET-001: records the vacuous history and the observed failure,
pins the install with three steps, and states what is still not claimed —
the */15 schedule actually ran every 2.4–5.5 h, so unattended latency is
hours (#436).

Refs #436, #429

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* fix(ci): the verification gate installs gh too — no PR has carried its summary comment since v1.55 (#153)

Same missing binary as #436, second consumer. The "Post sticky PR comment"
step is masked (continue-on-error, cosmetic by design) and its log on #433
reads: `skipping PR comment — 'gh' not on PATH (self-hosted runner without
gh?)`. #433 has zero bot comments. The verdict was never affected; the
summary simply never reached a PR.

The step now runs ./.github/actions/setup-gh first (same condition). This
PR's own Verification gate run is the proof: the summary comment should
appear on it. #153's other half (Node 20) is already handled —
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 is set in the gate's environment.

Refs #153, #436

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@avrabe avrabe mentioned this pull request Sep 17, 2026
Merged
avrabe added a commit that referenced this pull request Sep 17, 2026
@avrabe @claude
verify(v1.139): promote the four merged fixes — battery, Format gate,…
916fbca 
… fleet monitor, readiness report (#439)

* verify(batt): promote FV-FALCON-BATT-003 — the absent-battery evidence ran in the merge gate (#413)

Code-free promotion under the two-commit rule. The implementation landed in
#430 (a1b0e17) at `implemented`.

Evidence, observed rather than asserted:
- #430's REQUIRED Verification gate (run 35137919049, the run that gated the
  merge) matched FV-FALCON-BATT-003 and executed all four of its steps:
  PASS — `cargo test -p falcon-core --release absent_battery_blocks_arming`
  (4.27 s) and the three source greps (Option<f32> defaults to None for
  read_battery_v and BatteryDriver::voltage; arming requires batt_present).
- Re-run on main a1b0e17: all four steps exit 0, and the test really runs
  (`1 passed`, not a vacuous zero-match filter).
- #430 merged with all 8 required contexts SUCCESS.

Scope: this promotes the VERIFICATION artifact. The requirement it verifies,
SWREQ-FALCON-ORPHAN-P01, is broader (wiring the orphaned crates) and stays
`proposed`. #413 stays open until a tagged release carries the fix.

Refs #413

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* verify(vgate): promote FV-RELAY-VGATE-004 — the required Format gate can fail, and on main it passes for real (#409)

Code-free promotion. Implementation merged in #424 (4c5feda).

Evidence, observed:
- The flag is gone: ci.yml has no continue-on-error on `cargo fmt --all
  -- --check` (the artifact's first step).
- main bcbe93e's Format job (CI run 35185137151): step `success`, 0 `Diff in`,
  no `Process completed with exit code` line — a real pass, not a masked
  failure (the 3ec4a2f log before #424 had 893 diffs and exit code 1 under
  a `success` job).
- #424's own gate did NOT execute this artifact (default `falcon` filter,
  and it carries no `falcon` tag), so this PR's Verify-Filter includes it:
  its two steps run in this PR's gate.

Scope: this verifies the Format-gate instance only. SWREQ-RELAY-VGATE-P04
stays `proposed` (its v1.139 scope is recording dark tracks).

Refs #409

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* verify(fleet): promote FLEET-P01 — the monitor evaluates on the light runners it failed on for months (#429, #436)

Code-free promotion. Implementation merged in #432 (7daf818) and #437
(bcbe93e).

Evidence, observed:
- FV-RELAY-FLEET-001's steps executed in #437's merge gate (run
  35182187826): all PASS, including `rust-script --test
  scripts/ci-wedge-watch.rs` and the setup-gh pins.
- A run that EVALUATES on main (the artifact's own promotion condition):
  fleet-status dispatched on bcbe93e, run 35185164979, both jobs on
  ci-01-8 — `gh-2.100.0.tar.gz: OK`; starvation query succeeded (its
  "could not be evaluated" step skipped); wedge detector `10 job(s)
  running, 0 wedged`. ci-01-8 is the box whose scheduled runs printed a
  vacuous `queued-too-long: 0` before #437.
- The detector's first live catch (2026-09-17 00:53Z): Kani
  (relay-mix-quad) 125.4 min vs 4.0 min median, flagged; recorded on #429.

Not claimed (unchanged in the artifact): detection within 15 minutes —
scheduled runs are throttled to every few hours (#436, CIFLOW-P01).

Refs #429, #436

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* verify(readiness): promote READY-P01 — on a not-ready tree the nightly job is green and publishes the report

Code-free promotion. Implementation merged in #433 (572ae38).

Evidence, observed:
- FV-RELAY-READY-001's steps executed in #433's merge gate (run
  35176152604): all PASS, including the fixture-repo tests of the four
  verdict bugs.
- The artifact's promotion condition — a run on main, GREEN, with a
  non-empty summary: Release readiness dispatched on 572ae38 (run
  35182551834), `Report` step success, printing "## Release readiness —
  falcon-v1.139.0 / 0 of 15 artifacts done" and the notice "next release
  still has blocking artifacts". That is exactly the falsification case:
  the 2026-09-16 scheduled run under the old code failed with exit code 1
  and an empty summary on a not-ready tree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@avrabe avrabe mentioned this pull request Sep 17, 2026
Merged
avrabe added a commit that referenced this pull request Sep 17, 2026
@avrabe @claude
fix(soak): the endurance ladder loses its measurement exactly when th…
7dbd42d 
…e hold breaks

`shell: bash` runs with -e, so the rung that FAILS kills the step before
it prints the rung, the reason, or the longest-holding number the job
exists to publish. Both soak runs on main (2026-09-17 08:41 and 21:08)
stopped right after the 40 s rung with `Process completed with exit code
1` and no ::notice::. The ratchet this job is built around — "a
published number that may not get worse" — has therefore never published
one on a breaking run.

Same trap as the wedge detector's (#432): capture the code with
`|| RC=$?` instead of letting -e end the step. The break message now
also says WHICH failure it was — timeout (rc 124), a verdict FAIL, or a
bench that exited without a verdict — so a crash is never read as a hold
that merely broke.

Measured with this fix on the #452/#434 branch: the ladder ran 25, 40,
60, 90, 120 and 300 s, all PASS, and printed "longest holding duration
300s". On main it dies at the 60 s rung.

Refs #403

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
@avrabe avrabe mentioned this pull request Sep 18, 2026
Closed
avrabe added a commit that referenced this pull request Sep 18, 2026
@avrabe @claude
fix(core+bench): the accelerometer is not a gravity reference in flig…
0185e8a 
…ht, and the analytic plant can tilt (#434, #452, #435) (#456)

* fix(core): the accelerometer is not a gravity reference in flight — hold converges at 5 Hz aiding (#434, #452)

Two faults, each measured on its own discriminating run.

THE HARNESS (#452). SimBackend::read_imu returned the gravity reaction
whatever the motors did, so tilt could be read off the accelerometer
exactly and dead-reckoning between fixes was free. It now carries the
non-gravitational acceleration its own physics produced — thrust, wind,
drag, and the ground reaction while in contact — and rotates that into
the body frame: exact for this Euler plant, taken after the ground
clamp, no finite differencing.

THE ESTIMATOR (#434). EstimatorPartition::step fused the accelerometer
as gravity every tick. A multirotor's accelerometer senses specific
force along the thrust axis and shows almost none of the vehicle's
tilt, so the tilt estimate was pulled level exactly while the vehicle
tilted to accelerate; with a realistic accelerometer the hold diverged
at every fix rate, including a fix on every tick. The update now runs
only where the reading really is gravity: on the ground, or when no fix
has been accepted for about a second. FlightCore::step supplies
"flying" from the cascade's own previous collective, so a standalone
estimator and pre-arm on the ground are unchanged.

MEASURED: #434's 5 Hz case (1 m off, gz tuning) holds within 0.5 m at
60 s at both 500 Hz and 100 Hz loop rates, against 75 m at 20 s and
3.4 km at 60 s before. Restoring the unconditional update fails that
test at 5134 m. A 15 s GNSS outage stays within 5 m.

Three harnesses were idealized in the same direction and were fixed
with it, because no vehicle conveys attitude through the accelerometer
alone: TiltBackend and TumbleBackend now ROTATE at a fixed rate and
report that rate, with the accelerometer reading the thrust axis. Both
failsafe campaigns and the runaway point test pass on them.

#404's altitude anti-windup gate also depended on the old estimator: it
read a vehicle stopped 2.15 m short under thrust lapse as "still
converging" on a 0.0005 m/s residual, and only charged before because
the vertical-velocity estimate lagged and dithered in sign. It now also
requires |v_z| > 0.05 m/s.

optimistic_variance_diverges_under_noisy_gps no longer holds: over-trust
peaks at 0.40 m, not the ~2400 m recorded at v1.19. The test is renamed
to what it measures and FV-FALCON-GNSS-001 drops to implemented for
re-verification. Golden log and f100 fixture regenerated deliberately
(LOG-P03) via their own ignored regen tests.

SWREQ-FALCON-HOLD-P01 + FV-FALCON-HOLD-001 (v1.140, implemented): the
requirement's acceptance is on the gz plant, and every number here is
SimBackend, so the gz run is what promotes it.

Refs #434, #452, #403

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* feat(gz): the hold verdict measures horizontal drift, not just altitude (#403)

The flightcore verdict's final_dist is the ALTITUDE error alone, so every
gz hold verdict to date reported horizontal drift as zero by
construction — the same shape as a gate that passes having checked less
than it claims. SWREQ-FALCON-HOLD-P01 names closing this measurement gap
as part of the requirement, not a follow-up to it.

The verdict now also carries final_horiz, peak_horiz and
rms_horiz_steady, and PASS requires final_horiz < 1.0 m and peak_horiz
< 2.0 m alongside the existing altitude terms — HOLD-P01's 1.0 m
horizontal, 0.5 m vertical.

Measured after the change: mock flightcore at 25 s still PASSes
(horizontal 0.00 m — MockPhysics applies no torque, #435), and the gz
leg on this branch held 2.121 m against a 2.00 m command with
n=0.013 e=-0.038 m at 25 s.

Refs #403, #434

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* fix(soak): the endurance ladder loses its measurement exactly when the hold breaks

`shell: bash` runs with -e, so the rung that FAILS kills the step before
it prints the rung, the reason, or the longest-holding number the job
exists to publish. Both soak runs on main (2026-09-17 08:41 and 21:08)
stopped right after the 40 s rung with `Process completed with exit code
1` and no ::notice::. The ratchet this job is built around — "a
published number that may not get worse" — has therefore never published
one on a breaking run.

Same trap as the wedge detector's (#432): capture the code with
`|| RC=$?` instead of letting -e end the step. The break message now
also says WHICH failure it was — timeout (rc 124), a verdict FAIL, or a
bench that exited without a verdict — so a crash is never read as a hold
that merely broke.

Measured with this fix on the #452/#434 branch: the ladder ran 25, 40,
60, 90, 120 and 300 s, all PASS, and printed "longest holding duration
300s". On main it dies at the 60 s rung.

Refs #403

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* docs(hold): the gz ladder confirms it — 40 s on main, 300 s with the fix (#403, #434)

The Soak workflow's gz ladder at the default 5 Hz aiding, each rung on a
fresh server:

  main (35276573490): 25 s PASS (0.01 m) · 40 s PASS (0.24 m, degrading)
                      60 s FAIL — alt err 1.46 m, rms 1.87 m, est_z -0.53 m
                      longest holding duration 40 s
  fix  (35274954189): 25/40/60/90/120/300 s all PASS
                      alt err 0.00 m from 60 s on, horizontal peak <= 0.04 m
                      longest holding duration 300 s (the dispatched ceiling)

Main's vehicle does not drift away, it SAGS: at the 60 s rung it is at
half a metre while commanded to 2 m — the altitude half of #403 following
the horizontal loss.

Every branch rung is measured against the STRICTER verdict from this
branch (horizontal now checked, HOLD-P01's 1.0 m); main's rungs are the
altitude-only verdict, so main's 40 s is the weaker claim.

FV-FALCON-HOLD-001 and SWREQ-FALCON-HOLD-P01 carry the numbers. Both stay
`implemented`: promotion is the code-free PR that follows, per the
two-commit rule.

Refs #403, #434, #452

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* fix(bench): the analytic plant can tilt, and something disturbs it (#435)

MockPhysics::step summed the motor commands into a collective and
applied NO moment — "the mixer's torque is approximated as zero in this
scaffold" — so the vehicle could not tilt whatever the attitude loop
commanded, could not be pushed off its position, and could not be
corrected back. Every closed-loop attitude or horizontal result on the
analytic tier was vacuous, including ENDURANCE-P01's 12 h hold.

It now applies the verified mixer's own allocation
(relay_mix_quad::motors_to_torque_signs scaled by 0.25, matching
falcon-core's SimBackend so both analytic plants respond to a motor
command alike).

THAT ALONE CHANGED NOTHING, which is the more interesting half: the
scenario starts level at the setpoint on a perfectly symmetric plant, so
the horizontal loop is never excited and the excursion stays 0.000 m at
any duration. The bench gained the IMU_NOISE knob #435 used locally and
never committed. gz bakes noise into its own sensors (a no-op there),
and the default 0.0 leaves every existing scenario byte-identical.

Measured (mock, gyro sigma 0.01, 250 Hz, 5 Hz aiding), horizontal
peak / final / altitude:

  120 s   0.24 / 0.11 / 0.01 m
  1 h     0.37 / 0.13 / 0.01 m
  12 h    0.46 / 0.16 / 0.01 m

Every one of those was 0.000 m horizontal before.

And the tier now catches what it used to hide: the same plant against
the pre-#452 estimator gives 1746 m at 120 s and 124 km at 1 h, with the
altitude estimate at 5e22 m — the fault the gz plant showed as a 40 s
ceiling (#403/#434).

SWREQ-FALCON-ENDURANCE-P01 -> implemented, with the numbers.

Refs #435, #403, #434

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* fix(bench): the wasm equivalence crate needs the mixer too (#435)

tests/cascade-sitl-wasm INCLUDES examples/falcon-sitl-gz/src/physics.rs
by path — deliberately, so the two cannot drift — and that file now
applies the verified mixer's torque allocation. The crate is its own
workspace, so `cargo build --workspace` at the root never touches it and
the break only surfaced in the gz job:

  error[E0433]: cannot find module or crate `relay_mix_quad` in this scope

Adding the same path dependency the example carries. Verified by
building tests/cascade-sitl-wasm directly, which is the only thing that
does.

Refs #435

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* fix(soak): run the analytic tier with the noise its claim is measured at (review of #456)

TWO REVIEW FINDINGS, both correct.

1. The nightly analytic tier ran with IMU_NOISE unset, i.e. 0.0 — and
   this branch's own comment says that at 0.0 the plant is perfectly
   symmetric, starts level at the setpoint, and reports a hold of
   0.000 m by construction at any duration. So ENDURANCE-P01 was flipped
   to `implemented` on a table measured at sigma 0.01 that the job named
   as its guardrail could not produce, and the new horizontal PASS bars
   would have been met vacuously. The step now sets IMU_NOISE: '0.01',
   the sigma the artifact's table uses; at it the 12 h rung peaks at
   0.46 m against a 1.0 m bar.

2. `libm_fabsf_core` re-implemented |x| next to `relay_math::fabsf`,
   which this same file already uses (line 2230) — two implementations,
   one bypassing the qualification wrapper. Removed; the anti-windup
   gate calls relay_math::fabsf.

Also adds FV-FALCON-ENDURANCE-001 (implemented): the requirement had no
verification artifact, and its steps pin the torque, the knob and the
job's noise setting.

falcon-core: 63 tests pass. rivet validate PASS.

Refs #435

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* fix(campaign): a rotor-out quad must not CLIMB — the deck tolerated 62 m (#398)

The full-loop motor-out campaign reported 200/200 with `least_descent`
at −62.01 m: a trial that GAINED 62 metres of altitude after losing a
rotor, and nothing looked. The struct field said "must stay > 0" while
the gate a hundred lines below said altitude is deliberately not
asserted — the two contradicted each other, so neither held.

The depth of the descent stays unasserted, for the stated reason (a
rotor-out quad on this plant is near neutrally buoyant, and bringing it
down is the supervisor's LAND job). A CLIMB is different: a vehicle that
has just lost a quarter of its thrust authority must not gain altitude.
The bound is deliberately weak — 1 m — so it catches a runaway without
pretending to gate the profile.

Measured on this deck, 200 dispersed trials:

  pre-#452/#434 estimator: 4 failures — trials CLIMBED 62.01, 47.43,
                           29.90 m; least descent −62.01 m; worst
                           detect latency 5 steps
  with the fix:            0 failures; least descent +1.75 m; worst
                           detect latency 2 steps

So the bound bites exactly where the estimator was wrong, and the fixed
stack passes it.

Refs #398, #434, #452

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* fix(failsafe): a pack that stops REPORTING in flight must raise the failsafe (#413's in-flight half)

Found by the v1.139 candidate review (#459). #413 narrowed the reader to
Option<f32> so a missing battery cannot masquerade as a healthy one, and
that closed the PRE-ARM half: the gate refuses to arm without a pack.

In flight nothing consulted it. On None the supervisor recorded
batt_present = false and did nothing else; batt_present is read only by
update_preflight, and the failsafe evaluates batt_state.low ||
batt_state.critical — latches that advance only when the estimator is
UPDATED. So a vehicle that armed healthy and then lost its ADC, harness
or driver kept a frozen last-known-healthy snapshot, and no low-battery
failsafe could fire while the pack drained to depletion.

Reproduced first: armed, took off, reached Loiter, stopped reporting
voltage — after 4 s the mode was still Loiter. The supervisor now
accumulates batt_missing_s and the failsafe term is
`low || critical || silence > 1 s`: longer than any sane sense interval,
far shorter than the reserve an RTL needs. The sustained-excursion
debounce on low/critical is untouched, so neither a throttle-punch sag
nor a single dropped sample can false-trigger.

SimBackend gained `battery_lost`, because a sense that FAILS mid-flight
is a different case from one absent at boot and the harness could not
express it.

SWREQ-FALCON-BATTERY-P03 + FV-FALCON-BATTERY-003 (v1.140, implemented).
falcon-core: 64 tests pass.

Refs #413, #459

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

* fix(fleet): starvation is a queued JOB, not a queued run (#449)

Pulled into v1.140 by maintainer decision, 2026-09-18, after the third
false alarm.

The check measured RUN status. A matrix run stays `queued` while its
legs are served one after another, so it cried starvation three times
out of three: #447 and #455 were wholly false, and in #462 the CI run
was reported "queued 54 min" while its first job had started four
seconds after creation.

It now flags a JOB queued past its threshold when nothing else in that
run has started since. Hosted runners get their own threshold (120 min
against 30) because org-shared hosted concurrency was measured at
60-120 min: #462's OTHER leg really did wait 67 minutes, on
ubuntu-latest, which is contention, not a dead fleet — and the alarm
text blamed the self-hosted fleet for it. The message now names the
runner labels and which kind it is, because the two need different
operator actions.

REPLAYED against all three alarms with a stub gh, under GitHub's own
`shell: bash` flags: #462's CI run not flagged, #462's hosted Gazebo run
not flagged, a self-hosted leg queued 45 min with nothing started
FLAGGED. Clean against the live API too.

EXECUTION FOUND A BUG READING WOULD NOT HAVE. The first draft emitted an
empty field for a job with no started_at. Tab is an IFS *whitespace*
character, so consecutive tabs collapse: every later field shifted by
one, the parsed "timestamp" was a job name, age came out -1, and every
case was skipped — the check would have reported a healthy fleet
forever. The absent marker is `-` now.

FV-RELAY-FLEET-002 (v1.140, implemented) carries the replay.

Refs #449, #462, #447, #455

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@avrabe