[CI] Fix offline paired-fixture deadline failure

[jjangg96]

Closes #61
Ref #1

Summary

The G01 g01_pair_fixture livecanary collection/listener workload exceeded its existing per-process 120-second deadline because the non-terminal suite had accumulated more work; this PR splits only that workload into two explicit static, exhaustive, disjoint groups. The existing terminal behavior and terminal-storage groups, g01_pair_fixture/live-helper exclusions, Go 1.26.8 pin, race detector, -count=1, 120-second process bounds, sequential failure propagation, and one tagged vet invocation are preserved.

Diagnosis

The reported public CI failure is run 34193163898, job 101955210094, whose metadata shows the reviewed offline experiment step failed after the preceding checks passed. The frozen local Go 1.26.8 race measurements on the pre-fix command were:

• Existing non-terminal collection/listener command: 109.146s.
• Existing terminal behavior group: 63.119s.
• Existing terminal-storage group: 39.347s.
• Cumulative fixture workload: 211.612s.

TestObserveCancellationAndApprovalExpiry completed locally in 0.08s, so the observed CI line showing it running for 0s was not treated as proof of an individual hang.

TDD evidence

Red before implementation

$ GOTOOLCHAIN=go1.26.8 go test -count=1 ./scripts -run '^TestToolingTaggedPairFixturePartitionsRun$' -v
--- FAIL: TestToolingTaggedPairFixturePartitionsRun (3.181s)
FAIL

Exit status was 1 because the generated paired-collection witness still ran through the old unsplit -skip '^TestPairedTerminal' invocation, while the new partition invocation was absent.

Green after implementation

$ GOTOOLCHAIN=go1.26.8 go test -count=1 ./scripts -run '^TestToolingTaggedPairFixturePartitionsRun$' -v
--- PASS: TestToolingTaggedPairFixturePartitionsRun (24.27s)
ok  github.com/1XP-AI/gh-runnerd/scripts 24.635s

$ GOTOOLCHAIN=go1.26.8 go test -race -count=1 ./scripts -run '^TestToolingTaggedPairFixturePartitionsRun$' -v
--- PASS: TestToolingTaggedPairFixturePartitionsRun (23.81s)
ok  github.com/1XP-AI/gh-runnerd/scripts 25.168s

The generated positive witnesses mark and execute each of the four groups exactly once; generated failures stop the corresponding group and propagate a non-zero result.

Group timing and coverage

Focused pinned Go 1.26.8 race runs after the split completed as follows:

• Paired collection: 90.042s package time, 91s wall time.
• Remaining collection/listener: 48.788s package time, 49s wall time.
• Terminal behavior (unchanged): 63.119s package time.
• Terminal storage (unchanged): 39.347s package time.

The final sequential offline check also passed:

$ /usr/bin/time -p bash scripts/check-offline-experiments.sh
ok  github.com/1XP-AI/gh-runnerd/experiments/g01-scaleset/livecanary 72.150s  # paired collection
ok  github.com/1XP-AI/gh-runnerd/experiments/g01-scaleset/livecanary 31.113s  # remaining collection/listener
ok  github.com/1XP-AI/gh-runnerd/experiments/g01-scaleset/livecanary 53.111s  # terminal behavior
ok  github.com/1XP-AI/gh-runnerd/experiments/g01-scaleset/livecanary 38.914s  # terminal storage
offline experiment checks passed: 2 module(s)
real 266.99

Each G01 process stayed below 120s; the G02 default and reviewed command checks also passed (24.732s, 1.480s, and 2.255s, with the existing synthetic command reporting no test files).

Tagged go test -list '^Test' classification is exhaustive and disjoint over 146 top-level tests:

• 19 paired collection tests (^TestPaired excluding ^TestPairedTerminal).
• 106 remaining collection/listener tests (^Test excluding ^TestPaired).
• 15 terminal behavior tests (the terminal prefix excluding the reviewed storage expression).
• 6 terminal-storage tests (the unchanged exact storage expression).

Verification

• GOTOOLCHAIN=go1.26.8 go test -count=1 ./... — exit 0; root packages passed, scripts package 37.870s.
• GOTOOLCHAIN=go1.26.8 go test -race -count=1 ./... — exit 0; root packages passed, scripts package 42.557s.
• GOTOOLCHAIN=go1.26.8 go vet ./... && make fmt-check — exit 0.
• bash -n scripts/check-offline-experiments.sh && git diff --check — exit 0.

Safety and rollback

No tests were dropped, retried dynamically, or given a wider deadline. No live runners, Docker/Lima/context, GitHub App, Keychain, launchd, credentials, or private logs were touched; all fixtures remain local synthetic tests and the existing live-helper exclusions remain in force.

Rollback is source-only: revert commit 513ed858607094226c1482f4615d8c912c6a1720 or close this PR without applying it. The change creates no runtime state, journal migration, runner resource, or online rollback obligation.

Remaining gates

The exact pushed head is 513ed858607094226c1482f4615d8c912c6a1720. Independent review, required hosted CI, and an exact-head GitHub Codex review remain required before merge; this PR does not claim live acceptance, parent #1/#2 completion, or Project Done.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-08T07:48:29.871631Z	b157412	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[jjangg96]

@codex review

[jjangg96]

Independent Luna/max review APPROVE on exact 513ed85, no findings. Reviewer reproduced the head-test/base-script red (exit1), verified exhaustive disjoint 146-test classification (19/106/15/6), ran all four actual race partitions (71.379s/31.295s/53.794s/38.927s), full offline experiment script (4m41.22s), generated positive/failure propagation tooling tests (29.551s), vet/syntax/diff checks, and canonical guide consistency. Checkout stayed clean/head unchanged. Hosted CI run34196653294 job101965765302 succeeded. Fresh Codex review requested; merge remains held until a timestamped exact-current-head completion and all findings are checked.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 513ed85860

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[jjangg96]

P2 resolution for Codex finding on b15741218d18bb7dca449228ecc6f9b26ad7563d.

Red before the script correction: GOTOOLCHAIN=go1.26.8 go test -count=1 ./scripts -run "^TestToolingTaggedPairFixturePartitionsRun$" -v exited 1; with the old -run ^Test remaining group, the generated Example and Fuzz witnesses reported skipped, the intended unrestricted invocation was logged 0 times, and the positive Example sentinel ran 0 times.

Green after the correction: the same test exited 0 in 31.457s; its race form exited 0 in 32.254s. Generated positive Test, Example (with an Output directive), and Fuzz seed sentinels each ran exactly once; generated negative Test, Example output mismatch, and Fuzz seed failure each propagated through one invocation with no retry.

Final verification on the pushed head:

• GOTOOLCHAIN=go1.26.8 go -C experiments/g01-scaleset test -tags=g01_pair_fixture -race -count=1 -timeout=120s ./livecanary -run "^TestPaired" -skip "^TestPairedTerminal" — PASS, 73.035s.
• Remaining unfiltered group with -skip "^TestPaired" — PASS, 32.279s.
• Terminal behavior group — PASS, 54.127s; terminal storage group — PASS, 40.078s.
• Tagged vet — PASS.
• env GOTOOLCHAIN=go1.26.8 GO=go bash scripts/check-offline-experiments.sh — PASS, both modules, real 278.73s.
• GOTOOLCHAIN=go1.26.8 go test -count=1 ./... — PASS; race form — PASS.
• GOTOOLCHAIN=go1.26.8 go -C experiments/g01-scaleset test -tags=g01_pair_fixture -list . ./livecanary inventory — 146 tests, 0 examples, 0 fuzz targets in current source; generated witnesses cover the latter categories.
• bash -n scripts/check-offline-experiments.sh && git diff --check — PASS.

No live resources or credentials were used. Fresh independent review, hosted CI, and exact-head Codex re-review remain coordinator-owned gates.

[jjangg96]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: b15741218d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".