A red nightly is invisible: the deep gate failed two nights running and nothing surfaced it

[OffgridwithJD]

The nightly deep gate was red for two consecutive nights and nothing surfaced it. I found it by going to look, not by being told.

nightly deep gate #64  2026-09-09  success
nightly deep gate #65  2026-09-10  FAILURE
nightly deep gate #66  2026-09-11  FAILURE

Both red runs failed in exactly one job, coverage report (PG 18). Every other job passed in both: suites on 15, 16, 17, 18 and 18/aarch64, the sanitizer gate, the extension-upgrade guard. The regression is fixed in #972 — this issue is about the two nights, not the bug.

Why nobody noticed

Three properties compound, and each is reasonable alone:

1. A scheduled run has no pull request to be red on. There is no "checks failed" against a branch anyone was looking at, and the PR gate stayed green the whole time because the failing suite passes wherever the checkout is traversable by a second user.
2. The failing job is the one that reports rather than gates. A reader glancing at the run sees seven green jobs and one red, and the red one is named "coverage report", which reads like a reporting step rather than a gate.
3. Nothing notifies. GitHub emails the actor of a failed scheduled workflow, and the actor of a schedule event is whoever last touched the workflow file — not whoever broke it, and not a channel anyone reads.

Why it matters more than the two nights

The failure was a suite inside the job: 249 passed, 1 failed ( harness_selftest). So for two nights the deep gate was running and its result was being discarded by nobody reading it, which makes every claim of the form "the nightly covers that" unbacked for that window. It is the same shape as a green CI job that reports PASS for outcomes it cannot see — an aggregate whose consumers are absent is not an aggregate.

This has happened before: the nightly ran red for 25 consecutive nights behind a green PR gate, which is how I know two nights is the lucky version.

What would actually fix it

Not a dashboard. Something that makes a red scheduled run arrive somewhere a human or an agent already looks. Options, cheapest first:

• A failure that opens or updates an issue. One issue per workflow, reopened with the run URL and the failing job names. It appears in the issue list, which is read, and it closes itself on the next green. No new infrastructure.
• A failure that comments on the most recent merged PR, which is the thing whose author is still paying attention. Noisier and attributes blame badly, since the breaking change may be several merges back.
• A status check mirrored onto main, so the default branch carries the nightly's verdict. Visible on the repo front page, but it conflates "main is broken" with "a nightly job is broken".

I would take the first. It needs no secrets, it is a dozen lines of gh in the workflow, and the thing it produces is the thing that was missing: a red nightly that is findable without knowing to look.

The acceptance test

Break a suite deliberately on a branch, run the nightly workflow manually against it, and show that an issue appears naming the run and the job. A notification mechanism that has never been seen to fire is indistinguishable from none — I have filed that exact mistake against my own watchers before.

Filed separately from #972 because #972 fixes the instance and this is the class.

[OffgridwithJD]

The nightly covers a configuration the PR gate cannot reach, which makes a red nightly the only signal for that configuration. I did not know this when I filed the issue and it raises the stakes.

The two-night red (#972) failed in the coverage job and every suites job passed — on 15, 16, 17, 18 and 18/aarch64, in both red runs. I assumed the suites jobs simply got lucky. They did not: they run the suites from a different place.

run_all_versions.sh:671   builddir="$(mktemp -d "/tmp/pgcolumnar-matrix-${major}.XXXXXX")"
run_all_versions.sh:684   cp -a "$SRCDIR/." "$builddir/"

run_coverage.sh:34        SRCDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
                          # and the suites run in place, from the checkout

Both jobs run under sudo, so both are root, so both take the same branch of the arm that broke. The difference is the path: the matrix copies the tree into /tmp, which is world-traversable, so an unprivileged reader can reach lib.sh. The coverage job runs from /home/runner/work/..., which postgres and nobody cannot traverse.

So the defect was not one the PR gate missed. It was one the PR gate could not observe, because the matrix relocates the tree before running anything.

Why that matters for this issue

It changes "the nightly's red is invisible" from a monitoring gap into a coverage gap.

• Anything sensitive to the tree's location or permissions is exercised only by the coverage job. In-place runs, the checkout's own path, and whatever an unprivileged helper can reach from it are covered nowhere else.
• The same is true of the other nightly-only jobs for their own reasons: the sanitizer gate, the extension-upgrade guard, PG15/16/17, and aarch64. Each is the sole observer of something.
• A red that nobody reads in a job that is the sole observer of a configuration means that configuration has no gate at all — not a weaker one. For two nights, "the unprivileged reader can reach the harness" was untested anywhere, and the arm that was supposed to test it had been reporting PASS while proving nothing.

What it does not change

The fix suggested above — a failure that opens or updates an issue — is still the right first move and still the cheapest. This only argues that it is worth more than I said: the acceptance test should confirm a notification fires, and the issue it opens should name the job, because "nightly failed" and "the only job that covers in-place runs failed" are different messages to a reader deciding whether to look.

One measurement anyone implementing this can use

gh api repos/commandprompt/pgcolumnar/actions/workflows/324085087/runs answers with conclusion per run, and the two reds were #65 and #66. Any check that the notification works can be validated against those two runs without breaking anything: a mechanism that would not have fired on #65 is not finished.