Floor user-initiated prerender-html one tier below indexing

.claude/skills/indexing-diagnostics/SKILL.md

@@ -90,20 +90,24 @@ If you pick the wrong category you waste a day. The diagnostic fields in the [Cl
 
 ## Prerender priorities
 
-Every prerender request — visit, module, run-command — carries a numeric `priority` that flows from the originating worker job all the way to the per-tab queue, the per-affinity file-admission semaphore, and the per-server render semaphore. Two priorities are in production today:
+A prerender request a worker job spawns carries a numeric `priority` — threaded from that job through the per-tab queue, the per-affinity file-admission semaphore, and the per-server render semaphore (a request with no originating job priority omits the field). The value is the worker-job priority of the job that produced the render, so it splits by both initiator (user vs. system) and job family (indexing vs. prerender-html):
 
-- **`0` — `systemInitiatedPriority`**. Background indexing work: scheduled full-reindex sweeps, `_full-reindex` runs, the worker's continuous reindex queue. The default for any code path that doesn't explicitly opt in.
-- **`10` — `userInitiatedPriority`**. Anything a user kicked off: the `_reindex` endpoint, ad-hoc card publishes, manual UI-driven reindex actions.
+- **`10` — `userInitiatedPriority`**. A user-initiated index visit: the `_reindex` endpoint, ad-hoc card publishes, manual UI-driven reindex actions.
+- **`9` — `userInitiatedPrerenderHtmlPriority`**. The HTML render a user-initiated index pass spawns — one tier below the index visit so it stays off the indexing hot path. **Exception:** a render a publish is awaiting (published-realm readiness gates on it) is lifted to `10`, co-equal with indexing, so the prerender server admits it ahead of ordinary user renders.
+- **`1` — `systemInitiatedPriority`**. A system-initiated index visit: scheduled full-reindex sweeps, `_full-reindex` runs, the worker's continuous reindex queue. The default worker-job priority for any code path that doesn't explicitly opt in.
+- **`0` — `systemInitiatedPrerenderHtmlPriority`**. The HTML render a system-initiated index pass spawns — the background floor only the all-priority worker pool takes.
 
-Higher priority dequeues first; FIFO is preserved within a priority bucket. There is **no preemption** — an in-flight low-priority render runs to completion. The next free slot goes to the highest-priority queued waiter.
+The two job families land on different diagnostics tables: an index-visit row on `boxel_index.diagnostics` carries 10 or 1; the prerender-html render it spawns, on `prerendered_html.diagnostics`, carries 9 or 0 — one notch below its initiator — except a publish-awaited render, which carries 10. So `priority=10` can show up on either table: an index visit, or a publish's HTML render.
+
+Higher priority dequeues first; FIFO is preserved within a priority bucket. There is **no preemption** — an in-flight lower-priority render runs to completion. The next free slot goes to the highest-priority queued waiter.
 
 Why this matters for triage:
 
-1. **Reading a stuck render**: a `priority=10` row is a user request. If it's stuck on `waits.tabQueueMs` or `waits.semaphoreMs`, that's the UX-visible saturation event the priority routing was designed to mitigate. A `priority=0` row stuck on the same wait is background work — operationally less urgent and often expected during a deliberate reindex burst.
+1. **Reading a stuck render**: a user-initiated render (10 on an index visit or a publish-awaited render, 9 on an ordinary user render) is a UX-facing request. If it's stuck on `waits.tabQueueMs` or `waits.semaphoreMs`, that's the saturation event priority routing was designed to mitigate. A system-initiated render (1 or 0) stuck on the same wait is background work — operationally less urgent and often expected during a deliberate reindex burst.
 
-2. **Distinguishing capacity issues from priority misrouting**: a `priority=10` row that waited >1s in `tabQueueMs` while the affinity's `prerender-queue-snapshot` shows `priorities=tab:10:N` (queued behind other priority-10 work) is a **capacity** problem — the user-priority workload exceeded the fleet. A `priority=10` row queued behind `priorities=tab:0:N` (queued behind background work, with manager-side priority routing live in the build) is a **routing** failure — the manager picked the wrong server, or the file render the row was queued behind isn't releasing. These need different fixes.
+2. **Distinguishing capacity issues from priority misrouting**: a user-priority row that waited >1s in `tabQueueMs` while the affinity's `prerender-queue-snapshot` shows it queued behind same-or-higher-priority work is a **capacity** problem — the user-priority workload exceeded the fleet. The same row queued behind strictly lower-priority (background) work, with manager-side priority routing live in the build, is a **routing** failure — the manager picked the wrong server, or the file render the row was queued behind isn't releasing. These need different fixes.
 
-3. **Confirming priority routing actually fired**: if a known-user `_reindex` shows up in `diagnostics` with `priority=0`, the producer-side threading (job → IndexRunner → `prerenderVisit`) regressed somewhere. Most-likely place is a new task type that didn't pick up `jobInfo.priority`.
+3. **Confirming priority routing actually fired**: if a known-user `_reindex` shows up in `diagnostics` with a system-tier priority (1 or 0), the producer-side threading (job → IndexRunner → `prerenderVisit`) regressed somewhere. Most-likely place is a new task type that didn't pick up `jobInfo.priority`.
 
 4. **Sharpening the deadlock fingerprint**: `affinitySnapshot.sameAffinityActivity[*].priority` lets you tell a self-referential prerender deadlock apart from priority-driven queuing. Same-priority queued module sub-render on a stuck same-priority file render → deadlock. Higher-priority queued sibling → priority routing working as intended.
 
@@ -1337,12 +1341,15 @@ WHERE realm_url = 'https://localhost:4201/user/your-realm/'
   "requestId": "b14e…",          // single ID across client/manager/prerender-server
   "invalidationId": "a3e1…",     // single ID across every row written by the same Batch.invalidate()
   "indexedAt": 1776964391615,    // wall-clock ms when IndexWriter.updateEntry ran
-  "priority": 10,                // worker-job priority that produced this render.
-                                 // 0 = system-initiated background (default); 10 =
-                                 // userInitiatedPriority. Read in post-mortems alongside
-                                 // `tabQueueMs` to tell whether priority routing put a
-                                 // high-priority render at the head of the queue. May be
-                                 // absent on older rows that predate the threading.
+  "priority": 10,                // worker-job priority that produced this render. Index
+                                 // visits carry 10 (userInitiatedPriority) or 1
+                                 // (systemInitiatedPriority); the prerender-html render
+                                 // they spawn carries 9 or 0 — one tier below its
+                                 // initiator — except a publish-awaited render, lifted to
+                                 // 10. Read in post-mortems alongside `tabQueueMs` to tell
+                                 // whether priority routing put a high-priority render at
+                                 // the head of the queue. May be absent on older rows that
+                                 // predate the threading.
   "tabReused": false,            // did this render land on a warm same-affinity tab (true)
                                  // or a freshly spawned / commandeered tab (false)?
                                  // Triage signal: a slow render with `tabReused: false`

packages/matrix/support/isolated-realm-server.ts

@@ -464,18 +464,15 @@ export async function startServer({
     // workers (fullyParallel) funnel their realm provisioning into one
     // worker manager, and each new workspace enqueues a from-scratch index
     // plus a prerender-html job whose realm-wide module pre-warm sweep runs
-    // for tens of seconds. User indexing and prerender-html are co-equal
-    // (both priority 10 — see runtime-common/queue.ts: a published realm's
-    // rendered HTML is as first-class as its search index), so priority
-    // alone cannot keep sweeps from occupying every high-tier worker while
-    // a user write's incremental index job sits queued. That queue wait is
-    // what test assertions feel: createRealm blocks on the from-scratch
-    // index, and read-your-writes endpoints (card GET, _publishability)
-    // drain in-flight incremental indexing before responding. The
-    // user-index worker claims only indexing job types, so an index job
-    // always has a lane no render sweep can hold; the two high-priority
-    // workers carry the prerender-html sweeps alongside any other
-    // user-initiated work.
+    // for tens of seconds. Prerender-html sits one tier below indexing (see
+    // runtime-common/queue.ts), so the user-index worker — flooring at the
+    // indexing tier and further restricted to indexing job types — can never
+    // be held by a render sweep. That matters here because test assertions
+    // feel the queue wait: createRealm blocks on the from-scratch index, and
+    // read-your-writes endpoints (card GET, _publishability) drain in-flight
+    // incremental indexing before responding. The two high-priority workers
+    // carry the prerender-html sweeps alongside any other user-initiated
+    // work.
     `--userIndexCount=1`,
     `--highPriorityCount=2`,
 

packages/realm-server/handlers/handle-publish-realm.ts

@@ -606,7 +606,10 @@ export default function handlePublishRealm({
             queue,
             dbAdapter,
             userInitiatedPriority,
-            { clearLastModified: true },
+            // This publish is blocked on the published realm's HTML (readiness
+            // gates on it), so the prerender-html job this pass spawns runs
+            // co-equal with indexing rather than one tier below.
+            { clearLastModified: true, awaitedByPublish: true },
           );
 
           return { lastPublishedAt, publishedRealmId, isNewRealm };
@@ -654,7 +657,12 @@ export default function handlePublishRealm({
           (await reconciler.lookupOrMount(publishedRealmURL));
         if (publishedRealm) {
           void publishedRealm
-            .fullIndex(userInitiatedPriority, { clearLastModified: true })
+            .fullIndex(userInitiatedPriority, {
+              clearLastModified: true,
+              // Republish is awaiting this HTML for readiness, so its render
+              // runs co-equal with indexing (see prerenderHtmlPriority).
+              awaitedByPublish: true,
+            })
             .catch((err: unknown) => {
               log.error(
                 `background publish reindex failed for ${publishedRealmURL}: ${

packages/realm-server/tests/prerender-html-split-test.ts

@@ -6,15 +6,25 @@ import {
   IndexWriter,
   VirtualNetwork,
   getQueueJobCoalesceHandler,
+  systemInitiatedPrerenderHtmlPriority,
+  systemInitiatedPriority,
+  userInitiatedPrerenderHtmlPriority,
+  userInitiatedPriority,
   type DefinitionLookup,
   type Diagnostics,
   type IndexingProgressEvent,
   type Prerenderer,
   type QueueCoalesceCandidate,
   type QueueCoalesceContext,
   type QueueJobSpec,
+  type QueuePublisher,
   type Reader,
 } from '@cardstack/runtime-common';
+import {
+  enqueuePrerenderHtmlJob,
+  prerenderHtmlPriority,
+  type PrerenderHtmlEnqueueArgs,
+} from '@cardstack/runtime-common/jobs/prerender-html';
 import { runPrerenderHtmlPass } from '@cardstack/runtime-common/index-runner/prerender-html-visit';
 // Registers the `prerender_html` coalesce handler at load time.
 import '@cardstack/runtime-common/tasks/prerender-html';
@@ -409,6 +419,122 @@ module(basename(import.meta.filename), function () {
     });
   });
 
+  // Prerender-html normally floors one tier below its initiator so a worker
+  // pool that floors at the indexing tier reserves itself to indexing and
+  // never carries a render sweep (see queue.ts and the user-index lane in
+  // worker-manager). The two initiator tiers mirror each other. The sole
+  // co-equal case is a publish-awaited render (a publish blocks on its HTML).
+  // Guard both the gap and the exception so neither drifts.
+  module('priority tiers', function () {
+    test('prerender-html floors strictly below its initiator in both tiers', function (assert) {
+      assert.ok(
+        userInitiatedPrerenderHtmlPriority < userInitiatedPriority,
+        'user-initiated prerender-html floors below user-initiated indexing',
+      );
+      assert.ok(
+        systemInitiatedPrerenderHtmlPriority < systemInitiatedPriority,
+        'system-initiated prerender-html floors below system-initiated indexing',
+      );
+    });
+
+    test('user-initiated prerender-html outranks every system-initiated job', function (assert) {
+      // The high-priority pool floors at the user prerender-html tier;
+      // keeping that tier above every system tier is what lets a floor-9 pool
+      // serve user renders while excluding system indexing (1) and system
+      // renders (0). (The all-priority pool still serves user renders too;
+      // this is about which pools the floor lets in, not exclusivity.)
+      assert.ok(
+        userInitiatedPrerenderHtmlPriority > systemInitiatedPriority,
+        'user-initiated prerender-html floors above system-initiated indexing',
+      );
+    });
+
+    test('prerenderHtmlPriority maps an index pass to the render tier one notch below it', function (assert) {
+      assert.strictEqual(
+        prerenderHtmlPriority(userInitiatedPriority),
+        userInitiatedPrerenderHtmlPriority,
+        'a user-initiated index pass spawns user-initiated prerender-html',
+      );
+      assert.strictEqual(
+        prerenderHtmlPriority(systemInitiatedPriority),
+        systemInitiatedPrerenderHtmlPriority,
+        'a system-initiated index pass spawns system-initiated prerender-html',
+      );
+    });
+
+    test('a publish-awaited render runs co-equal with indexing, but only for user work', function (assert) {
+      assert.strictEqual(
+        prerenderHtmlPriority(userInitiatedPriority, {
+          awaitedByPublish: true,
+        }),
+        userInitiatedPriority,
+        'a publish-awaited user render is lifted to the indexing tier',
+      );
+      assert.strictEqual(
+        prerenderHtmlPriority(userInitiatedPriority, {
+          awaitedByPublish: false,
+        }),
+        userInitiatedPrerenderHtmlPriority,
+        'a non-publish user render stays one tier below indexing',
+      );
+      assert.strictEqual(
+        prerenderHtmlPriority(systemInitiatedPriority, {
+          awaitedByPublish: true,
+        }),
+        systemInitiatedPrerenderHtmlPriority,
+        'the publish exception never lifts system-initiated work',
+      );
+    });
+
+    // Guards the hop that actually delivers the priority: enqueuePrerenderHtmlJob
+    // must thread awaitedByPublish into prerenderHtmlPriority. A stub publisher
+    // captures the enqueued spec, so this needs no DB.
+    test('enqueuePrerenderHtmlJob prices the render from its publish flag', async function (assert) {
+      let enqueueArgs = (
+        overrides: Partial<PrerenderHtmlEnqueueArgs>,
+      ): PrerenderHtmlEnqueueArgs => ({
+        realmURL: testRealm,
+        realmUsername: 'test_realm',
+        changes: [{ url: `${testRealm}1.json`, operation: 'update' }],
+        generation: 1,
+        loaderEpoch: 'epoch-a',
+        spawningJobId: 100,
+        spawningPriority: userInitiatedPriority,
+        timeoutSec: 60,
+        preWarm: false,
+        ...overrides,
+      });
+      let captured: { priority?: number } | undefined;
+      let stubPublisher = {
+        publish: (spec: { priority?: number }) => {
+          captured = spec;
+          return Promise.resolve(undefined);
+        },
+        destroy: () => Promise.resolve(),
+      } as unknown as QueuePublisher;
+
+      await enqueuePrerenderHtmlJob(
+        stubPublisher,
+        enqueueArgs({ awaitedByPublish: true }),
+      );
+      assert.strictEqual(
+        captured?.priority,
+        userInitiatedPriority,
+        'a publish-awaited render is enqueued co-equal with indexing',
+      );
+
+      await enqueuePrerenderHtmlJob(
+        stubPublisher,
+        enqueueArgs({ awaitedByPublish: false }),
+      );
+      assert.strictEqual(
+        captured?.priority,
+        userInitiatedPrerenderHtmlPriority,
+        'an ordinary user render is enqueued one tier below indexing',
+      );
+    });
+  });
+
   module('prerenderHtmlOnly batch', function (hooks) {
     let adapter: PgAdapter;
     let indexWriter: IndexWriter;

packages/realm-server/worker-manager.ts

@@ -671,22 +671,25 @@ let adapter: PgAdapter;
   // is set.
   eventSink.setAdapter(adapter);
 
-  // Each pool's minimum priority is a dequeue floor: its workers only
-  // claim jobs at or above it, oldest-first among those. User-initiated
-  // indexing and prerender-html are co-equal (both `userInitiatedPriority`
-  // — a published realm's rendered HTML is as first-class as its search
-  // index), so a priority floor alone cannot keep the two kinds of work
-  // from occupying the same workers. What separates them is the claim
-  // query's job-type filter: a worker only dequeues job types it has
-  // registered handlers for, and the user-index pool starts with
-  // `--indexJobsOnly` so it registers exactly the indexing job types.
-  // That makes it a lane a prerender-html sweep can never hold — indexing
-  // gates realm provisioning and every write's read-your-writes drain, so
-  // it must stay responsive even when long render sweeps saturate the
-  // high-priority pool. The high-priority pool serves all user-initiated
-  // work, indexing and prerender-html alike; the all-priority pool floors
-  // at the lowest tier and serves everything, including system-initiated
-  // prerender-html.
+  // Each pool's minimum priority is a dequeue floor: its workers only claim
+  // jobs at or above it, oldest-first among those. The user-index pool is
+  // opt-in — it exists only when `--userIndexCount` > 0 (0 by default). When
+  // run, it floors at the user-initiated indexing tier and registers only the
+  // indexing job types (`--indexJobsOnly`), giving index jobs a lane no
+  // prerender-html sweep can hold; both guards carry weight there — the floor
+  // keeps out ordinary user prerender-html (one tier below), and the job-type
+  // filter keeps out a publish-awaited render (co-equal at the indexing tier,
+  // which the floor alone would admit). The contended matrix test stack runs
+  // this lane to keep realm provisioning responsive under indexing
+  // back-pressure; deployments that leave `--userIndexCount` at 0 rely on
+  // high-priority-pool capacity, plus the render-tier gap that keeps HTML
+  // rendering behind indexing in the prerender server's own admission queue.
+  //
+  // The high-priority pool floors at the user-initiated prerender-html tier
+  // (one below indexing), so it serves all user-initiated work — user
+  // indexing, publish-awaited renders, and ordinary user renders alike — and
+  // never system-tier jobs. The all-priority pool floors at the lowest tier
+  // and serves everything, including system-initiated prerender-html.
   for (let i = 0; i < userIndexCount; i++) {
     await startWorker(userInitiatedPriority, urlMappings, {
       indexJobsOnly: true,