build(workflows): stream PR list when checking for duplicate PRs - #14850
Draft
Planeshifter wants to merge 1 commit into
Draft
build(workflows): stream PR list when checking for duplicate PRs#14850Planeshifter wants to merge 1 commit into
Planeshifter wants to merge 1 commit into
Conversation
The `check_duplicate_prs` workflow has failed on four of the last five nightly runs against `develop`. The "Check for duplicate PRs" step aborts with `The action 'Check for duplicate PRs' has timed out after 15 minutes.` shortly after reporting `Processed 900 PRs...`. The script accumulates every open pull request into a single JSON array and then indexes into that array once per pull request via `jq -c ".[$i]"`. Each of those invocations re-parses the entire array, which is roughly 4 MB at the current ~950 open pull requests, making the loop quadratic. Job logs show a steady 47 seconds per 50 pull requests, which puts a full pass just past the step's 15 minute budget. Stream the array through a single `jq -c '.[]'` invocation and drive the loop with `read` instead, so that the list is parsed once rather than once per pull request. Process substitution keeps the loop body in the current shell, so the accumulated issue-to-PR arrays are unaffected. Against a mocked API and a 122 pull request fixture covering null bodies, empty bodies, a null array element, and a null pull request number, the script's stdout and stderr are byte-identical to before the change. Ref: https://github.com/stdlib-js/stdlib/actions/runs/33352880096
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This pull request:
jqinvocation in.github/workflows/scripts/check_duplicate_prs/run, instead of indexing into the accumulated JSON array once per pull request.Failing run: https://github.com/stdlib-js/stdlib/actions/runs/33352880096 (nightly
check_duplicate_prs,develop).Symptom: the "Check for duplicate PRs" step aborts with
shortly after logging
Processed 900 PRs.... Four of the last five nightly runs have failed this way (runs 503, 502, 500, 499; run 501 passed).Root cause:
main()pages every open pull request into one JSON array and then walks that array with a C-styleforloop over its indices, reading each element back out withjq -c ".[$i]". Every one of thosejqinvocations re-parses the whole array, which is roughly 4 MB at the current ~950 open pull requests, so the loop is quadratic in the number of open pull requests. Job logs show a flat 47 seconds per 50 pull requests from index 650 onward, which puts a full pass just past the step'stimeout-minutes: 15budget. Nothing else in the step is close to the budget; the workflow started flapping as the open pull request count crossed the threshold, and now fails most nights.Fix: drive the loop with
while IFS= read -r prand feed it from a single streamingjq -c '.[]'invocation via process substitution — the same idiom the file already uses for the labeled-PR loop directly above. The array is parsed once rather than once per pull request. Process substitution rather than a pipe keeps the loop body in the current shell, so the accumulatedissue_prs_keys/issue_prs_valuesarrays andprocessed_countare unaffected.timeout-minutesis deliberately left at 15 — raising it would defer the failure rather than remove it.Related Issues
This pull request has no related issues.
Questions
No.
Other
Validation
bash -non the modified script.nullbodies, empty bodies, both theClosesissue-URL form and thefixes #nform, anullarray element, and an element whosenumberisnull. stdout and stderr are byte-identical, including the ordering of the label add/remove calls, and both exit0.Reviewer notes (non-blocking)
jqinvocations per pull request remain (.numberand.body), each parsing a single small object. These are linear overall (~30 seconds at current scale) and were left alone to keep the diff minimal; folding them into the single streaming pass would buy further headroom if ever needed.jqitself fails, the loop yields zero records and the script goes on to strip thePotential Duplicatelabel from every currently-labeled pull request. The pre-fix code had the identical exposure — a failing per-indexjqmade every iterationcontinue— so this is not a regression, but neither version distinguishes "no duplicates found" from "enumeration failed".shellcheckis not available in the environment used to prepare this change, so no shellcheck run was performed.Process disclosure: the routine that produced this change normally validates each fix with three independent reviewer sub-agents. Sub-agent spawning was unavailable in this environment, so the correctness, regression-scope, and style/conventions reviews were instead carried out as three separate self-review passes against the same briefs. All three concluded "approve" with no blocking findings, but they were not independent agents, and reviewers should weight them accordingly.
Checklist
AI Assistance
If you answered "yes" above, how did you use AI assistance?
Disclosure
This pull request was written primarily by Claude Code as part of an automated CI-failure triage routine. The root cause was identified from GitHub Actions job logs, and the fix was validated locally against a mocked GitHub API as described above. See the process disclosure in the "Other" section regarding how the change was reviewed.
@stdlib-js/reviewers