diff --git a/.claude/skills/indexing-diagnostics/SKILL.md b/.claude/skills/indexing-diagnostics/SKILL.md index 5487d68db5f..7e351b96838 100644 --- a/.claude/skills/indexing-diagnostics/SKILL.md +++ b/.claude/skills/indexing-diagnostics/SKILL.md @@ -90,20 +90,24 @@ If you pick the wrong category you waste a day. The diagnostic fields in the [Cl ## Prerender priorities -Every prerender request — visit, module, run-command — carries a numeric `priority` that flows from the originating worker job all the way to the per-tab queue, the per-affinity file-admission semaphore, and the per-server render semaphore. Two priorities are in production today: +A prerender request a worker job spawns carries a numeric `priority` — threaded from that job through the per-tab queue, the per-affinity file-admission semaphore, and the per-server render semaphore (a request with no originating job priority omits the field). The value is the worker-job priority of the job that produced the render, so it splits by both initiator (user vs. system) and job family (indexing vs. prerender-html): -- **`0` — `systemInitiatedPriority`**. Background indexing work: scheduled full-reindex sweeps, `_full-reindex` runs, the worker's continuous reindex queue. The default for any code path that doesn't explicitly opt in. -- **`10` — `userInitiatedPriority`**. Anything a user kicked off: the `_reindex` endpoint, ad-hoc card publishes, manual UI-driven reindex actions. +- **`10` — `userInitiatedPriority`**. A user-initiated index visit: the `_reindex` endpoint, ad-hoc card publishes, manual UI-driven reindex actions. +- **`9` — `userInitiatedPrerenderHtmlPriority`**. The HTML render a user-initiated index pass spawns — one tier below the index visit so it stays off the indexing hot path. **Exception:** a render a publish is awaiting (published-realm readiness gates on it) is lifted to `10`, co-equal with indexing, so the prerender server admits it ahead of ordinary user renders. +- **`1` — `systemInitiatedPriority`**. A system-initiated index visit: scheduled full-reindex sweeps, `_full-reindex` runs, the worker's continuous reindex queue. The default worker-job priority for any code path that doesn't explicitly opt in. +- **`0` — `systemInitiatedPrerenderHtmlPriority`**. The HTML render a system-initiated index pass spawns — the background floor only the all-priority worker pool takes. -Higher priority dequeues first; FIFO is preserved within a priority bucket. There is **no preemption** — an in-flight low-priority render runs to completion. The next free slot goes to the highest-priority queued waiter. +The two job families land on different diagnostics tables: an index-visit row on `boxel_index.diagnostics` carries 10 or 1; the prerender-html render it spawns, on `prerendered_html.diagnostics`, carries 9 or 0 — one notch below its initiator — except a publish-awaited render, which carries 10. So `priority=10` can show up on either table: an index visit, or a publish's HTML render. + +Higher priority dequeues first; FIFO is preserved within a priority bucket. There is **no preemption** — an in-flight lower-priority render runs to completion. The next free slot goes to the highest-priority queued waiter. Why this matters for triage: -1. **Reading a stuck render**: a `priority=10` row is a user request. If it's stuck on `waits.tabQueueMs` or `waits.semaphoreMs`, that's the UX-visible saturation event the priority routing was designed to mitigate. A `priority=0` row stuck on the same wait is background work — operationally less urgent and often expected during a deliberate reindex burst. +1. **Reading a stuck render**: a user-initiated render (10 on an index visit or a publish-awaited render, 9 on an ordinary user render) is a UX-facing request. If it's stuck on `waits.tabQueueMs` or `waits.semaphoreMs`, that's the saturation event priority routing was designed to mitigate. A system-initiated render (1 or 0) stuck on the same wait is background work — operationally less urgent and often expected during a deliberate reindex burst. -2. **Distinguishing capacity issues from priority misrouting**: a `priority=10` row that waited >1s in `tabQueueMs` while the affinity's `prerender-queue-snapshot` shows `priorities=tab:10:N` (queued behind other priority-10 work) is a **capacity** problem — the user-priority workload exceeded the fleet. A `priority=10` row queued behind `priorities=tab:0:N` (queued behind background work, with manager-side priority routing live in the build) is a **routing** failure — the manager picked the wrong server, or the file render the row was queued behind isn't releasing. These need different fixes. +2. **Distinguishing capacity issues from priority misrouting**: a user-priority row that waited >1s in `tabQueueMs` while the affinity's `prerender-queue-snapshot` shows it queued behind same-or-higher-priority work is a **capacity** problem — the user-priority workload exceeded the fleet. The same row queued behind strictly lower-priority (background) work, with manager-side priority routing live in the build, is a **routing** failure — the manager picked the wrong server, or the file render the row was queued behind isn't releasing. These need different fixes. -3. **Confirming priority routing actually fired**: if a known-user `_reindex` shows up in `diagnostics` with `priority=0`, the producer-side threading (job → IndexRunner → `prerenderVisit`) regressed somewhere. Most-likely place is a new task type that didn't pick up `jobInfo.priority`. +3. **Confirming priority routing actually fired**: if a known-user `_reindex` shows up in `diagnostics` with a system-tier priority (1 or 0), the producer-side threading (job → IndexRunner → `prerenderVisit`) regressed somewhere. Most-likely place is a new task type that didn't pick up `jobInfo.priority`. 4. **Sharpening the deadlock fingerprint**: `affinitySnapshot.sameAffinityActivity[*].priority` lets you tell a self-referential prerender deadlock apart from priority-driven queuing. Same-priority queued module sub-render on a stuck same-priority file render → deadlock. Higher-priority queued sibling → priority routing working as intended. @@ -1337,12 +1341,15 @@ WHERE realm_url = 'https://localhost:4201/user/your-realm/' "requestId": "b14e…", // single ID across client/manager/prerender-server "invalidationId": "a3e1…", // single ID across every row written by the same Batch.invalidate() "indexedAt": 1776964391615, // wall-clock ms when IndexWriter.updateEntry ran - "priority": 10, // worker-job priority that produced this render. - // 0 = system-initiated background (default); 10 = - // userInitiatedPriority. Read in post-mortems alongside - // `tabQueueMs` to tell whether priority routing put a - // high-priority render at the head of the queue. May be - // absent on older rows that predate the threading. + "priority": 10, // worker-job priority that produced this render. Index + // visits carry 10 (userInitiatedPriority) or 1 + // (systemInitiatedPriority); the prerender-html render + // they spawn carries 9 or 0 — one tier below its + // initiator — except a publish-awaited render, lifted to + // 10. Read in post-mortems alongside `tabQueueMs` to tell + // whether priority routing put a high-priority render at + // the head of the queue. May be absent on older rows that + // predate the threading. "tabReused": false, // did this render land on a warm same-affinity tab (true) // or a freshly spawned / commandeered tab (false)? // Triage signal: a slow render with `tabReused: false` diff --git a/packages/matrix/support/isolated-realm-server.ts b/packages/matrix/support/isolated-realm-server.ts index 820a44f50db..fa640553dd0 100644 --- a/packages/matrix/support/isolated-realm-server.ts +++ b/packages/matrix/support/isolated-realm-server.ts @@ -464,18 +464,15 @@ export async function startServer({ // workers (fullyParallel) funnel their realm provisioning into one // worker manager, and each new workspace enqueues a from-scratch index // plus a prerender-html job whose realm-wide module pre-warm sweep runs - // for tens of seconds. User indexing and prerender-html are co-equal - // (both priority 10 — see runtime-common/queue.ts: a published realm's - // rendered HTML is as first-class as its search index), so priority - // alone cannot keep sweeps from occupying every high-tier worker while - // a user write's incremental index job sits queued. That queue wait is - // what test assertions feel: createRealm blocks on the from-scratch - // index, and read-your-writes endpoints (card GET, _publishability) - // drain in-flight incremental indexing before responding. The - // user-index worker claims only indexing job types, so an index job - // always has a lane no render sweep can hold; the two high-priority - // workers carry the prerender-html sweeps alongside any other - // user-initiated work. + // for tens of seconds. Prerender-html sits one tier below indexing (see + // runtime-common/queue.ts), so the user-index worker — flooring at the + // indexing tier and further restricted to indexing job types — can never + // be held by a render sweep. That matters here because test assertions + // feel the queue wait: createRealm blocks on the from-scratch index, and + // read-your-writes endpoints (card GET, _publishability) drain in-flight + // incremental indexing before responding. The two high-priority workers + // carry the prerender-html sweeps alongside any other user-initiated + // work. `--userIndexCount=1`, `--highPriorityCount=2`, diff --git a/packages/realm-server/handlers/handle-publish-realm.ts b/packages/realm-server/handlers/handle-publish-realm.ts index 2cb42f5750a..a7a24394919 100644 --- a/packages/realm-server/handlers/handle-publish-realm.ts +++ b/packages/realm-server/handlers/handle-publish-realm.ts @@ -606,7 +606,10 @@ export default function handlePublishRealm({ queue, dbAdapter, userInitiatedPriority, - { clearLastModified: true }, + // This publish is blocked on the published realm's HTML (readiness + // gates on it), so the prerender-html job this pass spawns runs + // co-equal with indexing rather than one tier below. + { clearLastModified: true, awaitedByPublish: true }, ); return { lastPublishedAt, publishedRealmId, isNewRealm }; @@ -654,7 +657,12 @@ export default function handlePublishRealm({ (await reconciler.lookupOrMount(publishedRealmURL)); if (publishedRealm) { void publishedRealm - .fullIndex(userInitiatedPriority, { clearLastModified: true }) + .fullIndex(userInitiatedPriority, { + clearLastModified: true, + // Republish is awaiting this HTML for readiness, so its render + // runs co-equal with indexing (see prerenderHtmlPriority). + awaitedByPublish: true, + }) .catch((err: unknown) => { log.error( `background publish reindex failed for ${publishedRealmURL}: ${ diff --git a/packages/realm-server/tests/prerender-html-split-test.ts b/packages/realm-server/tests/prerender-html-split-test.ts index 13e4eea915f..ea5edf2064a 100644 --- a/packages/realm-server/tests/prerender-html-split-test.ts +++ b/packages/realm-server/tests/prerender-html-split-test.ts @@ -6,6 +6,10 @@ import { IndexWriter, VirtualNetwork, getQueueJobCoalesceHandler, + systemInitiatedPrerenderHtmlPriority, + systemInitiatedPriority, + userInitiatedPrerenderHtmlPriority, + userInitiatedPriority, type DefinitionLookup, type Diagnostics, type IndexingProgressEvent, @@ -13,8 +17,14 @@ import { type QueueCoalesceCandidate, type QueueCoalesceContext, type QueueJobSpec, + type QueuePublisher, type Reader, } from '@cardstack/runtime-common'; +import { + enqueuePrerenderHtmlJob, + prerenderHtmlPriority, + type PrerenderHtmlEnqueueArgs, +} from '@cardstack/runtime-common/jobs/prerender-html'; import { runPrerenderHtmlPass } from '@cardstack/runtime-common/index-runner/prerender-html-visit'; // Registers the `prerender_html` coalesce handler at load time. import '@cardstack/runtime-common/tasks/prerender-html'; @@ -409,6 +419,122 @@ module(basename(import.meta.filename), function () { }); }); + // Prerender-html normally floors one tier below its initiator so a worker + // pool that floors at the indexing tier reserves itself to indexing and + // never carries a render sweep (see queue.ts and the user-index lane in + // worker-manager). The two initiator tiers mirror each other. The sole + // co-equal case is a publish-awaited render (a publish blocks on its HTML). + // Guard both the gap and the exception so neither drifts. + module('priority tiers', function () { + test('prerender-html floors strictly below its initiator in both tiers', function (assert) { + assert.ok( + userInitiatedPrerenderHtmlPriority < userInitiatedPriority, + 'user-initiated prerender-html floors below user-initiated indexing', + ); + assert.ok( + systemInitiatedPrerenderHtmlPriority < systemInitiatedPriority, + 'system-initiated prerender-html floors below system-initiated indexing', + ); + }); + + test('user-initiated prerender-html outranks every system-initiated job', function (assert) { + // The high-priority pool floors at the user prerender-html tier; + // keeping that tier above every system tier is what lets a floor-9 pool + // serve user renders while excluding system indexing (1) and system + // renders (0). (The all-priority pool still serves user renders too; + // this is about which pools the floor lets in, not exclusivity.) + assert.ok( + userInitiatedPrerenderHtmlPriority > systemInitiatedPriority, + 'user-initiated prerender-html floors above system-initiated indexing', + ); + }); + + test('prerenderHtmlPriority maps an index pass to the render tier one notch below it', function (assert) { + assert.strictEqual( + prerenderHtmlPriority(userInitiatedPriority), + userInitiatedPrerenderHtmlPriority, + 'a user-initiated index pass spawns user-initiated prerender-html', + ); + assert.strictEqual( + prerenderHtmlPriority(systemInitiatedPriority), + systemInitiatedPrerenderHtmlPriority, + 'a system-initiated index pass spawns system-initiated prerender-html', + ); + }); + + test('a publish-awaited render runs co-equal with indexing, but only for user work', function (assert) { + assert.strictEqual( + prerenderHtmlPriority(userInitiatedPriority, { + awaitedByPublish: true, + }), + userInitiatedPriority, + 'a publish-awaited user render is lifted to the indexing tier', + ); + assert.strictEqual( + prerenderHtmlPriority(userInitiatedPriority, { + awaitedByPublish: false, + }), + userInitiatedPrerenderHtmlPriority, + 'a non-publish user render stays one tier below indexing', + ); + assert.strictEqual( + prerenderHtmlPriority(systemInitiatedPriority, { + awaitedByPublish: true, + }), + systemInitiatedPrerenderHtmlPriority, + 'the publish exception never lifts system-initiated work', + ); + }); + + // Guards the hop that actually delivers the priority: enqueuePrerenderHtmlJob + // must thread awaitedByPublish into prerenderHtmlPriority. A stub publisher + // captures the enqueued spec, so this needs no DB. + test('enqueuePrerenderHtmlJob prices the render from its publish flag', async function (assert) { + let enqueueArgs = ( + overrides: Partial, + ): PrerenderHtmlEnqueueArgs => ({ + realmURL: testRealm, + realmUsername: 'test_realm', + changes: [{ url: `${testRealm}1.json`, operation: 'update' }], + generation: 1, + loaderEpoch: 'epoch-a', + spawningJobId: 100, + spawningPriority: userInitiatedPriority, + timeoutSec: 60, + preWarm: false, + ...overrides, + }); + let captured: { priority?: number } | undefined; + let stubPublisher = { + publish: (spec: { priority?: number }) => { + captured = spec; + return Promise.resolve(undefined); + }, + destroy: () => Promise.resolve(), + } as unknown as QueuePublisher; + + await enqueuePrerenderHtmlJob( + stubPublisher, + enqueueArgs({ awaitedByPublish: true }), + ); + assert.strictEqual( + captured?.priority, + userInitiatedPriority, + 'a publish-awaited render is enqueued co-equal with indexing', + ); + + await enqueuePrerenderHtmlJob( + stubPublisher, + enqueueArgs({ awaitedByPublish: false }), + ); + assert.strictEqual( + captured?.priority, + userInitiatedPrerenderHtmlPriority, + 'an ordinary user render is enqueued one tier below indexing', + ); + }); + }); + module('prerenderHtmlOnly batch', function (hooks) { let adapter: PgAdapter; let indexWriter: IndexWriter; diff --git a/packages/realm-server/worker-manager.ts b/packages/realm-server/worker-manager.ts index 86593959f6c..b63b3166f43 100644 --- a/packages/realm-server/worker-manager.ts +++ b/packages/realm-server/worker-manager.ts @@ -671,22 +671,25 @@ let adapter: PgAdapter; // is set. eventSink.setAdapter(adapter); - // Each pool's minimum priority is a dequeue floor: its workers only - // claim jobs at or above it, oldest-first among those. User-initiated - // indexing and prerender-html are co-equal (both `userInitiatedPriority` - // — a published realm's rendered HTML is as first-class as its search - // index), so a priority floor alone cannot keep the two kinds of work - // from occupying the same workers. What separates them is the claim - // query's job-type filter: a worker only dequeues job types it has - // registered handlers for, and the user-index pool starts with - // `--indexJobsOnly` so it registers exactly the indexing job types. - // That makes it a lane a prerender-html sweep can never hold — indexing - // gates realm provisioning and every write's read-your-writes drain, so - // it must stay responsive even when long render sweeps saturate the - // high-priority pool. The high-priority pool serves all user-initiated - // work, indexing and prerender-html alike; the all-priority pool floors - // at the lowest tier and serves everything, including system-initiated - // prerender-html. + // Each pool's minimum priority is a dequeue floor: its workers only claim + // jobs at or above it, oldest-first among those. The user-index pool is + // opt-in — it exists only when `--userIndexCount` > 0 (0 by default). When + // run, it floors at the user-initiated indexing tier and registers only the + // indexing job types (`--indexJobsOnly`), giving index jobs a lane no + // prerender-html sweep can hold; both guards carry weight there — the floor + // keeps out ordinary user prerender-html (one tier below), and the job-type + // filter keeps out a publish-awaited render (co-equal at the indexing tier, + // which the floor alone would admit). The contended matrix test stack runs + // this lane to keep realm provisioning responsive under indexing + // back-pressure; deployments that leave `--userIndexCount` at 0 rely on + // high-priority-pool capacity, plus the render-tier gap that keeps HTML + // rendering behind indexing in the prerender server's own admission queue. + // + // The high-priority pool floors at the user-initiated prerender-html tier + // (one below indexing), so it serves all user-initiated work — user + // indexing, publish-awaited renders, and ordinary user renders alike — and + // never system-tier jobs. The all-priority pool floors at the lowest tier + // and serves everything, including system-initiated prerender-html. for (let i = 0; i < userIndexCount; i++) { await startWorker(userInitiatedPriority, urlMappings, { indexJobsOnly: true, diff --git a/packages/runtime-common/jobs/prerender-html.ts b/packages/runtime-common/jobs/prerender-html.ts index cb154546d00..fce6a224a4f 100644 --- a/packages/runtime-common/jobs/prerender-html.ts +++ b/packages/runtime-common/jobs/prerender-html.ts @@ -11,14 +11,31 @@ import { Deferred } from '../deferred.ts'; import type { IncrementalChange } from '../tasks/indexer.ts'; import type { PrerenderHtmlArgs } from '../tasks/prerender-html.ts'; -// User-initiated HTML work shares its initiator's tier — for a published -// realm the rendered HTML is a first-class artifact, as important as the -// search index (see the tier table in queue.ts). System-initiated HTML work -// still drops to the background tier only the all-priority pool takes. -export function prerenderHtmlPriority(spawningPriority: number): number { - return spawningPriority >= userInitiatedPriority - ? userInitiatedPrerenderHtmlPriority - : systemInitiatedPrerenderHtmlPriority; +// A prerender-html job normally floors one tier below the index pass that +// spawned it — a user-initiated index (userInitiatedPriority) yields +// userInitiatedPrerenderHtmlPriority, anything lower yields +// systemInitiatedPrerenderHtmlPriority — which holds HTML rendering off the +// indexing hot path (see the tier table in queue.ts for why the gap is +// load-bearing). +// +// The exception is a render a publish is waiting on: a publish does not +// report the realm ready until its HTML exists, so that render is on the +// publish's critical path rather than a background follow-on. It runs +// co-equal with indexing (userInitiatedPriority) so the prerender server +// admits it ahead of ordinary user renders. Where a dedicated index lane is +// configured (worker-manager's opt-in `--indexJobsOnly` pool), that lane's +// job-type filter — not the priority floor — is what keeps a co-equal publish +// render out of it. +export function prerenderHtmlPriority( + spawningPriority: number, + opts?: { awaitedByPublish?: boolean }, +): number { + if (spawningPriority < userInitiatedPriority) { + return systemInitiatedPrerenderHtmlPriority; + } + return opts?.awaitedByPublish + ? userInitiatedPriority + : userInitiatedPrerenderHtmlPriority; } export interface PrerenderHtmlEnqueueArgs { @@ -30,6 +47,11 @@ export interface PrerenderHtmlEnqueueArgs { spawningJobId: number | null; spawningPriority: number; timeoutSec: number; + // True only when a publish is awaiting this HTML. Lifts the job to the + // indexing tier (co-equal) instead of one notch below it, so the publish's + // critical-path render is admitted ahead of ordinary user renders. See + // prerenderHtmlPriority. + awaitedByPublish?: boolean; // True when a from-scratch index pass spawned this job. The realm-wide // module pre-warm sweep — O(realm module count) — runs at the start of the // job only when set; incremental spawns leave it false. @@ -181,6 +203,7 @@ export async function enqueuePrerenderHtmlJob( spawningPriority, timeoutSec, preWarm, + awaitedByPublish, }: PrerenderHtmlEnqueueArgs, ): Promise> { let args: PrerenderHtmlArgs = { @@ -198,7 +221,7 @@ export async function enqueuePrerenderHtmlJob( // Separate from `indexing:${realmURL}` so HTML work never blocks // indexing. concurrencyGroup: prerenderHtmlConcurrencyGroup(realmURL), - priority: prerenderHtmlPriority(spawningPriority), + priority: prerenderHtmlPriority(spawningPriority, { awaitedByPublish }), timeout: timeoutSec, args, }); diff --git a/packages/runtime-common/jobs/reindex-realm.ts b/packages/runtime-common/jobs/reindex-realm.ts index 03141425b15..964688dfb21 100644 --- a/packages/runtime-common/jobs/reindex-realm.ts +++ b/packages/runtime-common/jobs/reindex-realm.ts @@ -7,6 +7,12 @@ import { interface EnqueueReindexRealmJobOptions { clearLastModified?: boolean; + // Set only by the publish flow. A publish blocks on the published realm's + // HTML before reporting it ready, so the prerender-html job this index pass + // spawns must run co-equal with indexing rather than one tier below it (see + // prerenderHtmlPriority). Surfaced in the from-scratch args, and carried + // only when true so ordinary index args keep their existing shape. + awaitedByPublish?: boolean; } export async function enqueueReindexRealmJob( @@ -25,6 +31,7 @@ export async function enqueueReindexRealmJob( realmURL: realmUrl, realmUsername, clearLastModified, + ...(opts?.awaitedByPublish ? { awaitedByPublish: true } : {}), }; if (clearLastModified) { await query(dbAdapter, [ diff --git a/packages/runtime-common/queue.ts b/packages/runtime-common/queue.ts index 326e85fa422..4b0b7e0c70b 100644 --- a/packages/runtime-common/queue.ts +++ b/packages/runtime-common/queue.ts @@ -8,23 +8,39 @@ import type { Deferred } from './deferred.ts'; // // The tiers: // -// | priority | job | -// | -------- | -------------------------------------------- | -// | 10 | any user-initiated job, incl. prerender-html | -// | 1 | system-initiated job (non-prerender-html) | -// | 0 | system-initiated prerender-html | +// | priority | job | +// | -------- | ------------------------------------------------------- | +// | 10 | user indexing (+ any user job); publish-awaited render | +// | 9 | user prerender-html (the common case) | +// | 1 | system indexing (+ any system job) | +// | 0 | system prerender-html | // -// User-initiated prerender-html is co-equal with user indexing (both 10): -// for a published realm the rendered HTML is the deliverable served to -// visitors — as important as the search index — so it is NOT deprioritized -// below its initiator tier. System-initiated prerender-html stays one notch -// below system work (background); boot rendering is gated separately and must -// not crowd out user-tier jobs. The high-priority pool floors at -// `userInitiatedPrerenderHtmlPriority` (serving all user-initiated work and -// never system-tier jobs); the all-priority pool floors at -// `systemInitiatedPrerenderHtmlPriority` and serves everything. +// Prerender-html floors one tier below its initiator. The two initiator tiers +// mirror each other: user prerender-html (9) is to user indexing (10) as +// system prerender-html (0) is to system indexing (1). The gap does two +// things. Everywhere, the same value rides to the prerender server as the +// render's in-render admission priority (see prerender-headers), where higher +// is dequeued first — so an index visit's render (its initiator tier) outranks +// the slower prerender-html renders, keeping HTML rendering off the indexing +// hot path at the render layer. And it lets a worker pool that floors at the +// indexing tier reserve itself to indexing; that reserved lane is opt-in via +// `--userIndexCount` + `--indexJobsOnly` (the contended matrix test stack runs +// it to keep provisioning responsive — see worker-manager), and deployments +// without it lean on high-priority-pool capacity instead. +// +// One render escapes the gap: a publish does not report its realm ready until +// the published HTML exists, so a publish-awaited render is on the publish's +// critical path and runs co-equal with indexing (10), admitted ahead of +// ordinary user renders. Where the index lane is configured, its +// `--indexJobsOnly` filter (not the priority floor) is what keeps a co-equal +// publish render out of it. +// +// The high-priority pool floors at `userInitiatedPrerenderHtmlPriority` (9), +// serving all user-initiated work — both render tiers and user indexing — and +// never system-tier jobs; the all-priority pool floors at +// `systemInitiatedPrerenderHtmlPriority` (0) and serves everything. export const userInitiatedPriority = 10; -export const userInitiatedPrerenderHtmlPriority = 10; +export const userInitiatedPrerenderHtmlPriority = 9; export const systemInitiatedPriority = 1; export const systemInitiatedPrerenderHtmlPriority = 0; diff --git a/packages/runtime-common/realm-index-updater.ts b/packages/runtime-common/realm-index-updater.ts index 02362b25f2b..5996eb91509 100644 --- a/packages/runtime-common/realm-index-updater.ts +++ b/packages/runtime-common/realm-index-updater.ts @@ -144,7 +144,7 @@ export class RealmIndexUpdater { publishFullIndex( priority = systemInitiatedPriority, - opts?: { clearLastModified?: boolean }, + opts?: { clearLastModified?: boolean; awaitedByPublish?: boolean }, ): { published: Promise>; completed: Promise; @@ -163,6 +163,7 @@ export class RealmIndexUpdater { priority, { clearLastModified: opts?.clearLastModified, + awaitedByPublish: opts?.awaitedByPublish, }, ); return job; diff --git a/packages/runtime-common/realm.ts b/packages/runtime-common/realm.ts index 44c1779ded0..f585154f728 100644 --- a/packages/runtime-common/realm.ts +++ b/packages/runtime-common/realm.ts @@ -1601,7 +1601,10 @@ export class Realm { await this.#startedUp.promise; } - async fullIndex(priority?: number, opts?: { clearLastModified?: boolean }) { + async fullIndex( + priority?: number, + opts?: { clearLastModified?: boolean; awaitedByPublish?: boolean }, + ) { // Clear the realmInfo cache before re-indexing so cards rendered // during this pass read /realm.json from the now-populated index // rather than a stale "Unnamed Workspace" cached during an earlier @@ -1612,7 +1615,10 @@ export class Realm { this.invalidateCachedRealmInfo(); let { completed } = this.#realmIndexUpdater.publishFullIndex( priority ?? systemInitiatedPriority, - { clearLastModified: opts?.clearLastModified }, + { + clearLastModified: opts?.clearLastModified, + awaitedByPublish: opts?.awaitedByPublish, + }, ); await completed; // The from-scratch swap has landed in boxel_index: drop searchCards diff --git a/packages/runtime-common/tasks/indexer.ts b/packages/runtime-common/tasks/indexer.ts index 480e26236ec..b3a83afc3f4 100644 --- a/packages/runtime-common/tasks/indexer.ts +++ b/packages/runtime-common/tasks/indexer.ts @@ -320,6 +320,14 @@ function incomingClearsLastModified(args: unknown): boolean { return isObjectLike(args) && args.clearLastModified === true; } +// A publish sets this on its from-scratch args so the prerender-html job the +// pass spawns runs co-equal with indexing (the publish blocks on that HTML) +// rather than one tier below. Absent on every other index path. Read loosely +// so a job enqueued before this field existed reads as false. +function argsAwaitedByPublish(args: unknown): boolean { + return isObjectLike(args) && args.awaitedByPublish === true; +} + registerQueueJobDefinition({ jobType: 'incremental-index', coalesce: chooseIncrementalCoalesceDecision, @@ -383,6 +391,10 @@ const fromScratchIndex: Task = ({ spawningJobId: jobInfo?.jobId ?? null, spawningPriority: jobInfo?.priority ?? systemInitiatedPriority, timeoutSec: FROM_SCRATCH_JOB_TIMEOUT_SEC, + // A publish awaits this HTML, so its render is on the publish's + // critical path and runs co-equal with indexing (see + // prerenderHtmlPriority). Only the publish flow sets this. + awaitedByPublish: argsAwaitedByPublish(args), // From-scratch: the prerender job runs the realm-wide module // pre-warm sweep before its format renders. preWarm: true,