diff --git a/e2e/live/ci/backend.env b/e2e/live/ci/backend.env index 09f12d963a..db35d5ef13 100644 --- a/e2e/live/ci/backend.env +++ b/e2e/live/ci/backend.env @@ -7,11 +7,18 @@ # "this console x that published backend" as a matched pair. Testing a # mismatched pair (backend rc.N vs console pinned rc.M) proves nothing — # bump this in the same PR that bumps the console's @objectstack pins. +# Pinned by scripts/__tests__/ci-cd-pipeline-doc.test.ts, which reads this +# value and the version pnpm-lock.yaml resolves and requires them equal. It +# had drifted two minor versions (17.0.0-rc.2 against a 17.2.0 lockfile) +# before anything compared the two numbers — objectui#7689. # # OBJECTSTACK_REF — the objectstack-ai/objectstack commit the showcase app # metadata is checked out from. Always the commit the release tag # `@objectstack/cli@${OBJECTSTACK_VERSION}` points at, so the app source and # the published packages it runs on come from the same tree. (Mirror image # of the framework repo's `.objectui-sha` console pin.) -OBJECTSTACK_VERSION=17.0.0-rc.2 -OBJECTSTACK_REF=89d2a4eb3f3b6b8f8c0fbc4cb3953cbe8218dc66 +# NOT machine-checked, and deliberately so: resolving that tag needs the +# objectstack repository, which the unit lane cannot reach. Move this by +# hand whenever the version above moves, from the tag's own commit. +OBJECTSTACK_VERSION=17.2.0 +OBJECTSTACK_REF=e7d2cc67fdef7fee9d2c6d65d7363fe1c78ce6a4 diff --git a/e2e/live/saved-view-filter.spec.ts b/e2e/live/saved-view-filter.spec.ts index 6b69679d35..44375b7dc9 100644 --- a/e2e/live/saved-view-filter.spec.ts +++ b/e2e/live/saved-view-filter.spec.ts @@ -6,8 +6,15 @@ import { test, expect } from '@playwright/test'; * * This is the empirical pin for the defect, run against the REAL stack because * only the server can answer the question the issue asked: does it accept a - * rule array in `$filter`? It does not. Measured on - * `@objectstack/*@17.0.0-rc.2` with the showcase app: + * rule array in `$filter`? It does not. Measured against the published backend + * pair this lane pins — `OBJECTSTACK_VERSION` and `OBJECTSTACK_REF` in + * `e2e/live/ci/backend.env`, which move together and are now held to the + * lockfile by `scripts/__tests__/ci-cd-pipeline-doc.test.ts` (objectui#7689). + * That pair has moved since this measurement was first taken, which is why it + * is named by FILE and not by value: a version literal written here goes stale + * behind the pin silently, and the whole point of objectui#7689 is that it did + * — for two minor versions, in the pin itself. The showcase app is the one that + * ref checks out: * * GET /api/v1/data/showcase_task * ?$filter=[{"field":"status","operator":"equals","value":"in_progress"}] diff --git a/scripts/__tests__/ci-cd-pipeline-doc.test.ts b/scripts/__tests__/ci-cd-pipeline-doc.test.ts index 85be60d9af..8ad51d2fc3 100644 --- a/scripts/__tests__/ci-cd-pipeline-doc.test.ts +++ b/scripts/__tests__/ci-cd-pipeline-doc.test.ts @@ -1263,3 +1263,174 @@ describe('ci-cd-pipeline.md — contexts that can never be required (#4170)', () ).toEqual([]); }); }); + +/** + * objectui#7689 — the page's live-e2e section states a pin rule that nothing read. + * + * `content/docs/guide/ci-cd-pipeline.md` says the backend pins "must match the + * `@objectstack/spec` version in `pnpm-lock.yaml` — bump both in the same PR, or the + * run proves nothing", and `e2e/live/ci/backend.env` repeats the same MUST in its own + * header. Both were true statements about what a reader should do and false statements + * about what CI checked: `git grep -l backend.env scripts/ .github/ e2e/` reached only + * the workflow that caches on its hash and the script that sources it. Nothing compared + * the two numbers, so the pin sat at `17.0.0-rc.2` against a lockfile resolving `17.2.0` + * for two minor versions while `Live E2E (informational)` reported green. + * + * That is the shape this file exists for, and the worst instance of it yet. The size + * regime in #3197 advertised a guardrail CI did not have; here the lane's OWN contract + * says a mismatched pair "proves nothing", so every green run it produced over those two + * minors was a check whose documentation declared it meaningless. A stale constant is a + * chore; a green light its own spec disclaims is worse than a red one. + * + * ## What this asserts, and the one half it deliberately does not + * + * ASSERTED: `OBJECTSTACK_VERSION` equals the `@objectstack/spec` version the lockfile + * resolves, read two independent ways (the resolution keys under `packages:`/`snapshots:` + * and the `version:` line of every workspace importer) whose union must hold exactly one + * value. Two readings rather than one because "the resolved version" is only a + * well-formed question while the tree agrees with itself: a lockfile carrying two + * `@objectstack/spec` versions has no single number for the pin to match, and naming that + * is more useful than picking one of them and comparing to it. + * + * NOT asserted: the file's other pin, `OBJECTSTACK_REF`, whose stated rule is that it is + * the commit the `@objectstack/cli@${OBJECTSTACK_VERSION}` release tag points at. Reading + * that tag needs the objectstack repository over the network, which this lane has not got, + * so the pairing moves by hand and this file says so rather than implying coverage it has + * not got. What IS checked is the shape a hand move can still get wrong in a way the lane + * only discovers 300 seconds later: `start-backend.sh` fetches the ref with + * `git fetch --depth 1 origin "$OBJECTSTACK_REF"`, which needs a full object name. + * + * ## Anti-vacuity + * + * Each half is floored. The doc sentence is required to still be on the page, because + * this whole block is the enforcement of that sentence and a page that stopped making the + * claim should retire the pin deliberately rather than leave it running on prose nobody + * reads. The lockfile scan is required to have matched something, so a lockfile format + * change turns this red instead of green-with-an-empty-set — the failure mode #3451 keeps + * teaching this page. + */ +const backendEnvPath = path.join(repoRoot, 'e2e/live/ci/backend.env'); +const lockfilePath = path.join(repoRoot, 'pnpm-lock.yaml'); +const backendEnv = fs.readFileSync(backendEnvPath, 'utf8'); + +/** The page's statement of the rule, whitespace-normalised because the source wraps it. */ +const PIN_RULE_SENTENCE = + /Backend pins live in `e2e\/live\/ci\/backend\.env` and must match the `@objectstack\/spec` version in `pnpm-lock\.yaml`/; + +const readEnvKey = (key: string): string | null => + backendEnv.match(new RegExp('^' + key + '=(.+)$', 'm'))?.[1].trim() ?? null; + +/** + * Every `@objectstack/spec` version the lockfile resolves, from both spellings. + * + * The resolution keys carry the peer-suffixed identity (`17.2.0(ai@…)`) under + * `snapshots:` and the bare one under `packages:`; the importer entries carry the same + * value on their `version:` line. Each is truncated at the first `(` so the three + * spellings collapse to one comparable number. + */ +function resolvedSpecVersions(): string[] { + const lock = fs.readFileSync(lockfilePath, 'utf8'); + const found = new Set(); + + for (const m of lock.matchAll(/^\s{0,4}'?@objectstack\/spec@([0-9][^'(\s:]*)/gm)) { + found.add(m[1]); + } + for (const m of lock.matchAll( + /^\s*'@objectstack\/spec':\s*\n\s*specifier:.*\n\s*version:\s*([0-9][^\s(]*)/gm, + )) { + found.add(m[1]); + } + return [...found].sort(); +} + +describe('ci-cd-pipeline.md — live-e2e backend pin (#7689)', () => { + it('still carries the sentence this block enforces', () => { + expect( + PIN_RULE_SENTENCE.test(doc.replace(/\s+/g, ' ')), + 'content/docs/guide/ci-cd-pipeline.md no longer states that the backend pins in ' + + '`e2e/live/ci/backend.env` must match the `@objectstack/spec` version in ' + + '`pnpm-lock.yaml`. Everything below is the enforcement of that sentence, so losing it ' + + 'would leave these assertions running on a rule the page stopped teaching. If the rule ' + + 'was retired, retire this describe with it; if the sentence merely moved or was ' + + 'reworded, update PIN_RULE_SENTENCE.', + ).toBe(true); + }); + + it('resolves exactly one @objectstack/spec version to compare against', () => { + const versions = resolvedSpecVersions(); + + expect( + versions.length, + `Nothing in ${path.relative(repoRoot, lockfilePath)} matched either reading of an ` + + '`@objectstack/spec` resolution. That is a green this test must never report: the ' + + 'comparison below would run against an empty set and pass no matter what ' + + '`e2e/live/ci/backend.env` says. Either the dependency is genuinely gone — in which ' + + 'case the live lane has nothing to pin and this block should go — or the lockfile ' + + 'format moved and the two regexes above need updating.', + ).toBeGreaterThan(0); + + expect( + versions, + 'The lockfile resolves more than one `@objectstack/spec` version:\n' + + versions.map((v) => ` - ${v}`).join('\n') + + '\n\nThere is then no single "the resolved version" for `OBJECTSTACK_VERSION` to ' + + 'match, so the live lane cannot be a matched pair against any of them. Resolve the ' + + 'workspace onto one version first; this test deliberately names the split rather than ' + + 'picking a winner.', + ).toHaveLength(1); + }); + + it('pins OBJECTSTACK_VERSION to the version the lockfile resolves', () => { + const versions = resolvedSpecVersions(); + if (versions.length !== 1) return; // reported by the test above + + const pinned = readEnvKey('OBJECTSTACK_VERSION'); + expect( + pinned, + `${path.relative(repoRoot, backendEnvPath)} declares no OBJECTSTACK_VERSION. ` + + '`start-backend.sh` sources this file and installs published `@objectstack/*` at that ' + + 'value, so an absent key is not a lighter failure than a wrong one.', + ).not.toBeNull(); + + expect( + pinned, + `The live-e2e lane is pinned to an unmatched pair:\n` + + ` ${path.relative(repoRoot, backendEnvPath)} OBJECTSTACK_VERSION=${pinned}\n` + + ` ${path.relative(repoRoot, lockfilePath)} @objectstack/spec ${versions[0]}\n\n` + + 'content/docs/guide/ci-cd-pipeline.md states the rule: "Backend pins live in ' + + '`e2e/live/ci/backend.env` and must match the `@objectstack/spec` version in ' + + '`pnpm-lock.yaml` — bump both in the same PR, or the run proves nothing." The header ' + + 'of backend.env says the same thing in its own words. A run of `Live E2E ' + + '(informational)` against this pair therefore carries no information, green or red — ' + + 'it exercises one published backend against a console built for another.\n\n' + + 'Fix it in whichever direction the change came from: a lockfile bump must move ' + + 'OBJECTSTACK_VERSION (and, by hand, OBJECTSTACK_REF to the commit the matching ' + + '`@objectstack/cli` release tag points at), and a pin bump must be a lockfile bump. ' + + '⛔ Do not resolve this by reverting the pin to whatever was green last: a failing ' + + 'matched pair carries strictly more information than a green mismatched one ' + + '(objectui#7689).', + ).toBe(versions[0]); + }); + + it('keeps OBJECTSTACK_REF in the one shape start-backend.sh can fetch', () => { + const ref = readEnvKey('OBJECTSTACK_REF'); + expect( + ref, + `${path.relative(repoRoot, backendEnvPath)} declares no OBJECTSTACK_REF, which ` + + '`start-backend.sh` needs to sparse-checkout the showcase app.', + ).not.toBeNull(); + + // Which COMMIT it should be is the half this file cannot read (see the header): that + // needs the objectstack release tag. The shape it must have is readable here, and it + // is the one a hand move gets wrong — `git fetch --depth 1 origin ` wants a full + // object name, and an abbreviated one fails 300 seconds into the lane, in a log nobody + // reads until the job goes red. + expect( + ref, + `OBJECTSTACK_REF is ${JSON.stringify(ref)}. start-backend.sh fetches it with ` + + '`git fetch --depth 1 origin "$OBJECTSTACK_REF"`, which needs a full 40-character ' + + 'commit sha — an abbreviated one, a branch name or a tag is refused by the remote and ' + + 'the lane only says so once the backend fails to boot.', + ).toMatch(/^[0-9a-f]{40}$/); + }); +});