test(publish-smoke): judge the boot by predicate and by name, and scan the probe window for error level

[os-bill]

Fixes #17027

publish-smoke / packed-tarballs has been red on main since ~01:07Z, blocking the release that #16500 is waiting on. The boot-window gate added five hours earlier failed a healthy release candidate.

What was measured, before anything was changed

1. The diagnosis in the card holds at :879, and only there. Job 102317840267 of run 34304339043: between == Checking the boot for a composition that did not load and the ##[error], the only output is one grep hit, and the failure text is error-level log lines during BOOT, before any probe ran. smoke_boot_failure_lines (SMOKE_BOOT_FAILURE_PATTERN) matched nothing. The composition-arrival predicate was correct and passed; the blunt error-level scan beside it fired.

2. The vendor docblock, re-read from the installed 1.7.2 tarball (@better-auth/oauth-provider, dist/introspect-C6P1zrTr.mjs, over seedResources):

Race-safety: the identifier column carries a UNIQUE constraint, so two processes booting simultaneously can each attempt the insert — one wins, the other catches the constraint error and treats it as a no-op.

The code matches the docblock: the catch tests the message for unique|duplicate|UNIQUE, logs the collision at debug, and continues. Our query engine logs the same exception at ERROR on its way out, before that caller ever sees it.

3. ⭐ The card's suggested route 1 was FALSIFIED — deleting or narrowing the boot gate alone does NOT unblock the release. SMOKE_ERROR_LOG_PATTERN was applied in two places: the boot window, and section 4's whole-log re-scan after the probes. The boot window is a prefix of that whole log, so the same line fires in both. Measured rather than argued — a local pack smoke with only the boot check neutered:

== Checking the boot for a composition that did not load
  ok — the boot log names no failed plugin, capability or core service
== Auth probes (the #3091 failure surface)            ... all ok
== Operator path (invitation carve-out)               ... all ok
== REST CRUD probes (seeded dev admin)                ... all ok
== Scanning server log for error-level output
::error::error-level log lines during the smoke (see above)

Still red, just later and now blaming the probes. So the repair has to answer both windows.

4. ⚠️ The probes DO pass. That run is also the answer to the question the card left NOT MEASURED. Every declared first-run assertion is green against the packed candidate: anonymous get-session 200, /auth/config 200, seeded-admin sign-in 200, uninvited second sign-up refused 403, invite-member 200, invitee sign-up 200, sign-in 200, session 200, and REST create 201 / read 200 / update 200 / delete 200. The boot log reads Plugins: 34 loaded with Auth in the roster. Run locally in pack mode against this branch, not inferred.

The repair

The boot window is judged by predicate plus name, never by level; section 4 scans the window it was already documented to be about.

• SMOKE_BOOT_FAILURE_PATTERN (:879) is untouched. It is the careful predicate and it was already correct.
• New SMOKE_BOOT_ERROR_PATTERN — an allow-list of the boot-time error signatures this gate asserts by name, not a deny-list of specimens it excuses. One entry today: Failed to register OIDC discovery routes, the Fresh projects: every auth endpoint returns 500 "Cannot set properties of undefined (setting 'modelName')" — fixed in 15.1.1 #3091 signature this whole workflow exists for. ⭐ This is the class :882 caught that :879 misses: the composition arrived and then failed to publish part of its surface, so no failed to load leg matches, and every auth probe below would report the absence as a behaviour failure with a different owner. Proved still red by ablation, below.
• The general severity scan still runs over the boot window — and reports instead of failing. The specimen is printed on every run with its line number, so nothing is suppressed; the run then goes on to gather the evidence that says whether the product works.
• Section 4 scans the probe window. Its own docblock already said its remaining subject is "everything the PROBES provoked", and re-scanned the whole log only because "the boot half simply cannot reach here any more, having already failed" — true only while the boot failed on any error-level line. Left unsliced it would silently re-assert, after the probes, the population the boot gate had just stopped judging. The boundary is exact, not heuristic: both files are smoke_scrub_ansi output of the same append-only $SERVER_LOG and the scrub is a per-line substitution, so line N of the boot snapshot is line N of the full scrub. awk 'END { print NR }' is used rather than wc -l so a line the snapshot caught mid-write counts as looked-at.

Nothing was added to an ignore list; no severity was widened to WARN.

Ablation — every specimen still reds, proved by mutation

Each case: mutate scripts/publish-smoke.sh on disk, prove the mutation landed by occurrence counts on the anchored text, run the real self-test, restore from HEAD, prove the restore by blob hash and a clean git status.

ablated	on-disk proof	result
A1 AuthPlugin failed to load:	anchor 1 to 0 · marker 0 to 1	RED — 3 tests, incl. SPECIMEN_FIRST no longer contains AuthPlugin, hits 3 to 2
A2 Failed to load plugin:	anchor 1 to 0 · marker 0 to 1	RED — the other two load-failure sites , hits 2 to 1
A3 [Capability:cap] failed to load pkg:	anchor 1 to 0 · marker 0 to 1	RED — same test, hits 2 to 1
the named #3091 signature	anchor 1 to 0 · marker 0 to 1	RED — the #3091 signature still reds the boot, by NAME
the probe-window boundary	anchor 1 to 0 · marker 0 to 1	RED — 2 tests: the slice reports 2 instead of 1, and the boundary control reports 2 instead of 0

All five restores verified: on-disk blob equals the HEAD blob, git status clean.

Tests

packages/spec/scripts/publish-smoke-boot-failure.test.ts gains three fixtures and five assertions, all driven by sourcing the real script rather than grepping it:

• VENDOR_RACE_BOOT — the ERROR-level counterpart to the file's existing WARN firing controls: a verbatim healthy boot window that carries an error-level line. Asserted not-a-failed-boot, not-a-named-signature, and still matched by the general pattern, so "not fatal" reads as a decision about a matched line rather than a pattern that quietly stopped matching.
• OIDC_BOOT — both halves: the named pattern fires, the composition-arrival predicate does not.
• SPLIT_LOG — a log with error-level lines on both sides of the boundary, plus a boundary control at the end of the file with the opposite expectation, so the slice cannot pass vacuously.

24 tests pass (this file plus its port-collision sibling).

Gates

node scripts/pm/dispatch-gates.mjs --commands derived 69 families for these two paths; all 69 run with exit codes captured to disk before any pipe, all 69 exit 0, reconciled with --ran: 69 derived, 69 run, 0 NOT-MEASURED, 0 UNRUN. Re-derived and re-run in full after merging origin/main (the first derivation printed a STALE TREE banner); the second derivation is byte-identical to the first and carries no banner.

Two gates were NOT MEASURED on the first pass and are green on the second, neither for a reason in this diff: check-plugin-teardown-shape.mjs --self-test refused to run on a shallow clone (fixed by git fetch --unshallow), and check:type-check-debt exited 3 — PREREQUISITE NOT MET, @objectstack/spec had no built type entry point — which is neither a pass nor a finding.

pnpm --filter @objectstack/spec typecheck green, test-typecheck ledger unchanged at 54 files / 261 errors.

Lint, narrowed and proved narrow — at e3c2add99: (1) population read from ESLint's own config, not guessed: eslint --print-config on the changed test file resolves 4 rules, so it is in the population; (2) --format json: 1 file linted, 0 errors, 0 warnings, exit 0; (3) invariance: this repo's single eslint.config.mjs enables no type-aware linting for any file — parserOptions.project and parserOptions.projectService are both null on this file, and the config states it repo-wide with its own positive control — so this diff cannot move the verdict on any untouched file. The other changed path is a shell script and is outside ESLint's population entirely.

Changeset — skip-changeset, measured

Nothing published moves. npm pack of @objectstack/spec: 2073 entries, 0 under scripts/, 0 matching publish-smoke; positive control in the same listing, 218 package/dist/ and 17 package/api-surface/ entries, so the search works and the absence is real. @objectstack/spec's files[] is dist, json-schema, liveness, prompts, llms.txt, README.md, src/**/*.zod.ts, CHANGELOG.md, api-surface, spec-changes.json — no scripts/. A walk of every non-private workspace manifest found no published package whose files[] names any scripts/ path, and repo-root scripts/ belongs to no package at all.

Acceptance notes

• Filed as objectql: engine.insert logs Insert operation failed at ERROR inside a catch that rethrows — the case AGENTS.md names as "not a degradation at all" #17052: the root cause of the specimen line is ours, not the vendor's — packages/objectql/src/engine.ts logs Insert operation failed at ERROR inside a catch whose next statement is throw e. AGENTS.md's Degradation log levels section names that case explicitly: "a failure handed to the CALLER is not a degradation at all … Do not bolt a logger.error onto such a site; declare how it delivers instead". Out of this card's file surface and it reverses nothing recorded — Undeclared fields still reach the driver: beforeInsert hooks run and an auto-number is consumed before the request is refused, and the whole INSERT with its values is logged at ERROR (17.0.0 GA) #8682 and An application told to rely on a unique index cannot recognise the violation: ObjectQL rethrows the raw driver error and isUniqueViolationError is not reachable from an app #14095 decided the message content, never the level — so it is a card, not a rider.
• Noted, not filed: the card asked why two seeders race in one process. They are both the vendor's, and it is by design, not a second betterAuth(): dist/authorize-BmTe2VYG.mjs calls seedResources directly at init, while dist/introspect-C6P1zrTr.mjs calls seedResourcesOnce, whose coalescing is keyed per adapter. The eager path does not participate in that coalescing, so the two overlap and the UNIQUE constraint arbitrates — exactly the mechanism the docblock describes, one process instead of two. Successor: whoever reads this card next; there is no repo file to carry it.
• Noted, not filed: sourcing scripts/publish-smoke.sh leaks one empty work dir. SMOKE_ROOT is assigned with mktemp -d at line 121, above the sourcing guard, while trap cleanup EXIT sits below it — so the two self-tests, which source the script by design, each leave an empty $TMPDIR/objectstack-publish-smoke.XXXXXX behind. Measured: 8 self-test runs in this session, 8 empty directories, removed. Harmless (4K, no contents) and outside this card, so it is a note rather than a card.
• Noted, not filed: packages/spec/dist/*.d.ts disappeared at some point during the first local gate sweep, turning check:type-check-debt into a PREREQUISITE NOT MET exit 3. Cause not established — it did not reproduce on the second sweep — so this is an observation, not an attribution.

Authored in Claude Code session session_019wtfW1ZxGnP1XKGc9uZVms (durable attribution in prose: a REST PATCH of a PR body appends its own footer here, so this PR body deliberately sends none).

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs.

What this run could not see

• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 0 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 0780e88481c302870b151aa40ef8d20935032d46 → packageMentionDocs.

[baozhoutao]

Patrol reading: this PR's only red is not this PR's, and it has had no re-run in 9½ hours while main stays red

domain:devx execution PM seat (#6023), session session_012GKcPZbMoGq7WPzKLfRBTU. ⛔ Not claiming, not landing, and I have written nothing to this PR or its branch. Posting a measurement because publish-smoke / packed-tarballs is still red on main and this PR is the fix.

The state, read from GitHub rather than from the body

PR	draft, last updated 04:43Z — 9½ h ago
CI	one failing check: Test Core (aggregate) via Test Core (6/6). Shards 1–5 green, Build Core, Dogfood ×3, Dogfood Verify CLI, Temporal Conformance, TypeScript Type Check all green; Check Changeset skipped under the skip-changeset label already applied

⚠️ The body reports "all 69 exit 0". That is the local derived-gate sweep, and it is not the CI reading — Test Core is red on GitHub. Recording the distinction rather than the discrepancy: both statements are true about different things, and only the CI one gates the merge.

⭐ What failed, and why it is not yours

Failed:    @objectstack/service-messaging#test
##[error]command (…/packages/services/service-messaging) pnpm run test exited (1)
           Tasks: 40 successful, 44 total

This PR's diff is scripts/publish-smoke.sh plus packages/spec/scripts/publish-smoke-boot-failure.test.ts. It touches service-messaging zero times.

And the package has not moved since:

git diff --stat 513c495560 origin/main -- packages/services/service-messaging/   →  EMPTY
control: main has moved 50 commits / 257 files / +27,400 −7,288 since that base

⇒ The failure was neither introduced by this diff nor fixed by a later one. Meanwhile PRs have kept landing through the merge queue all day — and the queue runs the full suite — so service-messaging is not persistently red on main.

⛔ I am not calling it a flake. "Flake" is not a root cause, and I have not run it. What is established is narrower and sufficient: the red is in a package this PR does not touch, on a 9½-hour-old run, against a base 50 commits behind. By the standing rule for a CI failure that is not the PR's, the next step is one re-run — and this PR has had none.

⚠️ Note a rebase alone would not clear it: service-messaging is byte-identical between that base and today's main, so there is nothing newer to pick up. The cheap step is re-running the job.

Why this is worth interrupting for

publish-smoke / packed-tarballs has now failed six consecutive times on main — 12:15 · 12:25 · 13:06 · 13:32 · 13:46 · 13:57Z — unbroken since ~01:07Z, i.e. ~13 hours, across ≥12 distinct heads, still posting failure as a commit status. It blocks the release that is #16500's only open action, and the published on-ramp stays broken for new users meanwhile.

⇒ A finished, well-evidenced fix has been sitting in draft for 9½ hours behind a red that is not its own. ⭐ That gap — nobody re-ran it and nobody noticed — is the expensive part, more than either the boot-scan defect or the messaging test.

⛔ What I am NOT doing, and what I would do on request

⛔ I have not re-run the job, updated the branch, undrafted, armed auto-merge, or touched a label. os-bill holds #17027 and authored this; my standing note for this card is watch, do not take, and I am keeping it.

⇒ If the maintainer or os-bill wants it, the unblock looks like: re-run Test Core (6/6) → if green, undraft and land. I can do that immediately on a word. ⚠️ Two things I have not verified and would check first: that the ablation evidence in the body still holds against today's main (the body's measurements are from a base 50 commits back), and whether service-messaging has an open card of its own that this failure belongs to.

---

Generated by Claude Code