Skip to content

Commit 428c04d

Browse files
committed
ci(shard-timings): zero candidate runs is NOT MEASURED, not a finding
The refresh lane read every non-zero from the run selector as a refusal, so an EMPTY candidate list printed "NO ELIGIBLE RUN among the 0 ... every one was censored, failed, or has lost its artifacts" and failed the job. Nothing had been examined, and none of those three causes had occurred. Split into two legs with two exits and two messages: an empty candidate list exits EXIT_PREREQUISITE_NOT_MET (3, the repo-wide code), candidates that existed and were all rejected keep exit 1 and keep the sentence naming the causes that now apply. The workflow reads 3 as NOT MEASURED, leaves the dataset alone, stays green, and says loudly in its annotation and its step summary that a persistent NOT MEASURED is a defect rather than a steady state. The step's own run block is lifted out of the YAML and driven under bash against a stub node on all three exit codes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012GKcPZbMoGq7WPzKLfRBTU
1 parent f6a2737 commit 428c04d

2 files changed

Lines changed: 521 additions & 28 deletions

File tree

.github/workflows/shard-timings-refresh.yml

Lines changed: 73 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -268,8 +268,74 @@ jobs:
268268
# is coverage left on the table; the cost is two API reads per run
269269
# examined and the loop below still stops at the first accumulation
270270
# that covers the workspace.
271+
#
272+
# ⭐ THREE EXITS, THREE READINGS (#16467). The selector answers 0, 1
273+
# or 3, and this step must not flatten them:
274+
#
275+
# 0 eligible runs on stdout — carry on.
276+
# 1 candidates existed and every one was censored, failed or lost
277+
# its artifacts. A FINDING. The job goes red, as it always did.
278+
# 3 EXIT_PREREQUISITE_NOT_MET — the API held NO completed
279+
# `schedule` run of ci.yml at all. NOTHING was measured. The
280+
# dataset is untouched, which is the correct outcome, so the job
281+
# must not go red for it.
282+
#
283+
# Measured, which is why this branch exists: on the pull request that
284+
# landed the hourly run, this lane ran with `event=schedule` before
285+
# that trigger existed on main, printed "NO ELIGIBLE RUN among the 0"
286+
# and named three causes none of which had occurred. It also does not
287+
# clear at merge — there is a bootstrap window of at least an hour
288+
# before the first hourly run finishes.
289+
#
290+
# `|| SELECT_EXIT=$?` and not a bare call: this step runs under
291+
# `bash -e`, where the non-zero would kill it before the code could
292+
# be read at all.
293+
#
294+
# stderr goes to a FILE and is echoed back immediately: the log keeps
295+
# every line it had, and the refusal text becomes quotable into the
296+
# step summary below, where somebody reading a green job will see it.
297+
SELECT_EXIT=0
271298
node scripts/ci/select-shard-timings-run.mjs --candidates --limit 24 \
272-
> "$RUNNER_TEMP/candidates.json"
299+
> "$RUNNER_TEMP/candidates.json" 2> "$RUNNER_TEMP/select-stderr.txt" || SELECT_EXIT=$?
300+
cat "$RUNNER_TEMP/select-stderr.txt"
301+
echo "select_exit=$SELECT_EXIT" >> "$GITHUB_OUTPUT"
302+
if [ "$SELECT_EXIT" -eq 3 ]; then
303+
# ⛔ NEVER QUIET. A 3 that scrolls past in a green job is how this
304+
# lane ends up passing because it never looked — the shape that has
305+
# already cost this repo two cards. It is an annotation, a step
306+
# summary section and a job-level notice, and every one of them says
307+
# that a PERSISTENT 3 is a defect rather than a steady state.
308+
echo "not_measured=true" >> "$GITHUB_OUTPUT"
309+
echo "::warning::Shard timings NOT MEASURED: no completed \`schedule\` run of ci.yml exists on main yet. The dataset was left untouched. If this repeats once the hourly full run has been live for a few hours, the trigger is gone or every hourly run is being cancelled — file it."
310+
{
311+
echo "### Shard timings: NOT MEASURED (exit 3) — nothing was regenerated, and nothing failed"
312+
echo
313+
echo "\`select-shard-timings-run --candidates\` found **no completed \`schedule\` run of"
314+
echo "\`ci.yml\` on \`main\` at all**. That is not \"every candidate was rejected\": there were"
315+
echo "no candidates, so no run was censored, none failed and none lost its artifacts."
316+
echo "\`scripts/test-shard-timings.json\` is untouched, which is the correct outcome for this"
317+
echo "reading, and this job is green because nothing went wrong — not because anything passed."
318+
echo
319+
echo "Expected exactly once: while the hourly full run bootstraps. The trigger has to be on"
320+
echo "\`main\` and one run has to finish, and run-summary artifacts live 1 day."
321+
echo
322+
echo "⛔ **A PERSISTENT NOT MEASURED IS A DEFECT, NOT A STEADY STATE.** If this section is"
323+
echo "still here after the hourly run has been live for a few hours, the \`schedule\` trigger"
324+
echo "has been removed from \`ci.yml\` or every hourly run is being cancelled — and the"
325+
echo "balancing dataset is quietly ageing out while this job reports green. File a card."
326+
echo
327+
echo "The selector's own refusal text, verbatim:"
328+
echo
329+
echo '```'
330+
cat "$RUNNER_TEMP/select-stderr.txt" 2>/dev/null || echo '(the refusal text is in this job log)'
331+
echo '```'
332+
} >> "$GITHUB_STEP_SUMMARY"
333+
exit 0
334+
fi
335+
if [ "$SELECT_EXIT" -ne 0 ]; then
336+
echo "::error::select-shard-timings-run refused with exit $SELECT_EXIT — candidates existed and none was eligible. Nothing was regenerated."
337+
exit "$SELECT_EXIT"
338+
fi
273339
echo "Eligible runs, newest first:"
274340
node -e '
275341
const runs = JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP + "/candidates.json", "utf8"));
@@ -304,6 +370,11 @@ jobs:
304370
# quiet week costs one run's download and a busy one costs a few.
305371
- name: Regenerate the dataset, accumulating runs until the workspace is covered
306372
id: generate
373+
# Skipped on the NOT MEASURED reading: there is nothing to download.
374+
# Every step after `compare` is already gated on
375+
# `steps.compare.outputs.changed`, which is the empty string when
376+
# `compare` itself is skipped — so guarding these two guards the tail.
377+
if: steps.select.outputs.not_measured != 'true'
307378
env:
308379
GITHUB_TOKEN: ${{ github.token }}
309380
run: |
@@ -411,6 +482,7 @@ jobs:
411482
# file is what has to differ for a PR to be worth anyone's attention.
412483
- name: Compare against the committed dataset
413484
id: compare
485+
if: steps.select.outputs.not_measured != 'true'
414486
run: |
415487
if cmp -s scripts/test-shard-timings.json "$RUNNER_TEMP/refresh/refreshed.json"; then
416488
echo "changed=false" >> "$GITHUB_OUTPUT"

0 commit comments

Comments
 (0)