Skip to content

ci: a red scheduled main run must open or update an issue — a gate that fails unwatched is a fourth way to be dark #9830

Description

@proggeramlug

Split out of #9829 because it is independent of every counter question there,
and it is the change that would have saved nineteen days.

The failure mode

CLAUDE.md records three ways a gate can be unable to fail: continue-on-error,
missing from required contexts, and cancelled-shown-as-failed. #7856 added a
fourth, starvation. gc-ratchet has been demonstrating a fifth: failing,
correctly, and being watched by nobody.

Measured in #9829: gc-ratchet has been red on main since 2026-08-18
nineteen days. Of its last 100 scheduled runs, 15 succeeded (all on or before
08-17), 64 failed, 16 were cancelled.
Nothing blocked, nothing notified, and
the failure grew from 19 regression cells to 43 in that window without anyone
noticing the second accrual.

This is worse than the three recorded modes in one specific respect: those
produce a green that is wrong, which is at least invisible. This produced a
red that is right, sitting in plain view in the Actions tab, and it still
told nobody. It also silently disarms the PR arm's usefulness: because the gate
was already red on main, the first PR to add run-extended-tests inherits an
unrelated red, which trains reviewers to treat that gate's failures as noise.

The consequence for anyone reading recent work: for nineteen days the one gate
that watches GC counters could not report a GC counter regression, during a
period when a lot of GC-adjacent code landed. Claims of the form "the collection
schedule is unchanged" made in that window rest on ad-hoc diag counters, not on
this gate — because this gate was already red and therefore uninformative.

The ask

When a scheduled (or tag/dispatch) run of a gate workflow on main
concludes failure, open an issue — or update the existing open one — with the
run URL, the head SHA and the failing rows. Close or comment when it goes green
again.

Properties worth having, learned from #7856 and #9829:

Why not just re-enable push: branches: [main]

Because that is what #7856 removed, for good reasons that still hold: fourteen
workflows × ~29 jobs against ~9 concurrent slots at ~58 merges/day outran drain
and the gate went dark by starvation instead. The six-hourly schedule is the
right cadence. The missing piece is not more runs, it is that a failing run
must produce an artefact a human is obliged to look at.

Suggested check on any gate, worth writing into the contributing docs

Running is not the same as being watched. For every gate, ask: if this fails
on main tonight, who finds out, and how? If the answer is "someone opens the
Actions tab", the gate is dark.

Cross-references #9829 (the nineteen-day red and its bisection), #7856 (the
starvation mode), #9774 (a shared false failure hiding real ones in
check_thread_locals.py).

https://claude.ai/code/session_014UZWia6L37DpA93VLtNK9m

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions