Repository navigation
Conversation
npm test was a && chain of 105 npm run links: a fresh npm per file, strictly serial, dead at the first failure, and a hand-edit in two places for every new test. On Windows it died at link 15 and the other ninety never ran. scripts/test.js discovers test/*.js, runs the files in parallel (they all use listen(0) and mkdtemp, so they don't collide), prints every failure rather than the first, and exits non-zero if any failed. *.test.js files go through node --test, thumbs.js through electron, everything else through plain node. Arguments filter by filename substring; --jobs=N caps the workers. Discovery also picks up doc-defaults.js, a test that existed but had never been added to the chain. The per-file npm scripts remain for running one test at a time.
The only workflow was release.yml, which triggers on v* tags and never runs npm test — nothing between two releases ever executed the suite anywhere but the machine it was written on. On a fresh Windows checkout of main, 10 of 110 test files fail and the repository shows no sign of it, because none of them fail on macOS. ci.yml runs npm ci && npm test on ubuntu, macos and windows for every pull request and every push to main, fail-fast off so one platform's red doesn't hide another's. The Linux leg gets xvfb and lifts Ubuntu 24.04's AppArmor user-namespace restriction, because thumbs.js opens a real Electron window. The Windows leg will be red on day one. That is the finding, not a flaw in the workflow: the ten failures it shows are the ones currently invisible from macOS.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #20 (the first commit here is that PR's test runner; the diff this PR adds is
.github/workflows/ci.ymlalone).The gap
release.ymlis the only workflow, and it triggers onv*tags. It builds and signs; it never runsnpm test. No push or pull-request workflow exists — so between two releases, the suite only ever runs on the machine a change was written on.The cost is concrete: on a fresh Windows checkout, 10 of 110 test files fail and the repository shows no sign of it, because none of them fail on macOS:
(Verified today on Windows 11, Node 22:
100 passed, 10 failed, exactly those files.)The workflow
ci.ymlrunsnpm ci && npm teston ubuntu-latest, macos-latest and windows-latest for every pull request and every push tomain:fail-fast: false, so one platform's red doesn't cancel and hide another's.release.yml.xvfb-runand lifts Ubuntu 24.04's AppArmor user-namespace restriction, becausetest/thumbs.jsopens a real Electron window and captures from it.It sits on #20's runner rather than on the old
&&chain deliberately: the chain dies at the first failure, so a red Windows leg would report one broken test instead of ten. The runner reports all of them, which is what makes the leg diagnostic rather than just red.The Windows leg will be red on day one
That is the finding, not a flaw in the workflow. The ten failures it shows are the ones currently invisible from macOS; each fix for them becomes verifiable by someone other than its author the moment this merges.