Floor user-initiated prerender-html one tier below indexing#5573
Open
habdelra wants to merge 5 commits into
Open
Floor user-initiated prerender-html one tier below indexing#5573habdelra wants to merge 5 commits into
habdelra wants to merge 5 commits into
Conversation
User-initiated prerender-html had been raised to priority 10 — co-equal with user indexing — on the premise that worker capacity, not a dedicated lane, keeps indexing responsive. A dedicated user-index lane (--indexJobsOnly) was then reintroduced precisely because render sweeps were occupying the workers index jobs need, which leaves co-equal-at-10 resting solely on that lane's job-type filter: the priority floor no longer separates indexing from rendering. Restore userInitiatedPrerenderHtmlPriority to 9 so the floor separates them again, mirroring the system tier (system prerender-html 0 sits one below system indexing 1). The dedicated lane now excludes render work by both its floor and its job-type filter — defense-in-depth. This is behavior-preserving for the current pool configuration: the high-priority pool still floors at the prerender-html tier and serves all user work, and the dedicated index lane still excludes prerender-html. Harden the rationale across the queue tier table, prerenderHtmlPriority, the worker-manager pool setup, and the matrix worker-tier harness, and add a guard test asserting prerender-html floors strictly below its initiator in both tiers. Bring the indexing-diagnostics skill's priority section up to the four-tier scheme (index visits carry 10/1, the prerender-html renders they spawn carry 9/0). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request restores the intended 4-tier worker priority “floor” separation between indexing and the downstream prerender-html job family by lowering user-initiated prerender-html from priority 10 to 9, and reinforces the contract with updated rationale and a guard test.
Changes:
- Lower
userInitiatedPrerenderHtmlPriorityfrom10→9and update tier rationale to explicitly preserve the “one notch below indexing” gap. - Clarify the
prerenderHtmlPriority()mapping contract and worker-manager pool/lane behavior in comments/docs. - Add a focused test module that guards the tier relationships and the
prerenderHtmlPriority()mapping.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| packages/runtime-common/queue.ts | Updates the tier table/rationale and sets userInitiatedPrerenderHtmlPriority = 9. |
| packages/runtime-common/jobs/prerender-html.ts | Documents and enforces the “one notch below indexing” priority mapping for prerender-html jobs. |
| packages/realm-server/worker-manager.ts | Updates pool/lane documentation to match the restored tier separation and keeps --indexJobsOnly as defense-in-depth. |
| packages/realm-server/tests/prerender-html-split-test.ts | Adds guard tests that lock in the intended tier ordering and mapping behavior. |
| packages/matrix/support/isolated-realm-server.ts | Updates the test-harness explanation to reflect the restored priority gap and lane behavior. |
| .claude/skills/indexing-diagnostics/SKILL.md | Updates operational diagnostics guidance to the four-tier scheme (10/9/1/0) and table semantics. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b0e33aadad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
Flooring all user-initiated prerender-html at 9 put it one tier below indexing unconditionally. That is right for the common case — an edit's HTML render is a background follow-on, off the indexing hot path — but wrong for a publish: a publish does not report its realm ready until the published HTML exists, so that render is on the publish's critical path and should be admitted as promptly as indexing. Thread an `awaitedByPublish` signal from the publish handler's index enqueues (the durability reindex and the republish fullIndex) through the from-scratch args to the prerender-html enqueue, where it lifts the job from userInitiatedPrerenderHtmlPriority (9) to userInitiatedPriority (10). The flag is carried in the from-scratch args only when true, so every other index path's args keep their existing shape; the task reads it loosely, matching the clearLastModified pattern. Every non-publish user render stays at 9; system renders stay at 0. At 10 a publish render meets the user-index lane's floor, so the lane's --indexJobsOnly job-type filter — not the priority floor — is what keeps it out of indexing's reserved workers. The high-priority pool floors at 9 and serves all three: user indexing (10), publish renders (10), and ordinary user renders (9). Comments (queue tier table, prerenderHtmlPriority, worker-manager, guard test) and the indexing-diagnostics skill updated for the exception. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Wrap the inline `{ awaitedByPublish: true }` in the priority-tiers guard
test so it satisfies Prettier's print width (was failing lint:js).
- Fix the guard-test comment that claimed priority 9 puts user renders "out
of reach of system-tier workers": the all-priority pool (floor 0) still
serves them; what the tier relationship guarantees is that a floor-9 pool
excludes system work.
- Soften the indexing-diagnostics claim that every prerender request carries
a priority — the visit-arg builders include it only when the originating
job has a priority.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The user-index --indexJobsOnly lane is enabled only by --userIndexCount>0 (0 by default); the contended matrix test stack runs it for provisioning back-pressure, while the staging/production entrypoints leave it at 0. The comments in queue.ts, prerender-html.ts, and worker-manager overstated it as the mechanism that keeps renders off indexing everywhere. Reframe: the render-tier gap's portable effect is the prerender server's in-render admission ordering (an index visit's render outranks the slower prerender-html renders); the reserved worker lane is an opt-in add-on on top of that. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The guard test covered the pure prerenderHtmlPriority function, but nothing asserted the wiring that actually delivers the priority. Add a hermetic test (stub QueuePublisher, no DB) that enqueuePrerenderHtmlJob threads awaitedByPublish into the enqueued job's priority: co-equal with indexing (userInitiatedPriority) when a publish awaits it, one tier below otherwise. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lukemelia
approved these changes
Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Worker jobs run under a four-tier priority scheme where priority is a worker-reservation floor: a pool's workers claim only jobs at or above their floor, so a higher-priority job is reachable by more pools, and a pool that floors at a given tier reserves itself to that tier and above. Indexing and its downstream HTML rendering are separate job families, and rendering is meant to stay off the indexing hot path — indexing gates realm provisioning and every write's read-your-writes drain, so it has to stay responsive even when long render sweeps are in flight.
The scheme keeps prerender-html one tier below its initiator so a pool can reserve itself to indexing by flooring at the indexing tier. The two initiator tiers mirror each other:
User-initiated prerender-html had drifted up to 10 — co-equal with user indexing — as a blanket rule, collapsing that gap. This restores prerender-html to 9 so the floor separates indexing from rendering again, mirroring the system tier (0 below 1).
The publish exception
Co-equal-at-10 was only ever meant for one case: a render a publish is waiting on. A publish does not report its realm ready until the published HTML exists (readiness gates on it), so that render is on the publish's critical path, not a background follow-on — it should be admitted as promptly as indexing. So a publish-awaited render stays co-equal with indexing (10); every other user render is 9, and system renders are 0.
An
awaitedByPublishsignal is threaded from the publish handler's index enqueues (the durability reindex and the republishfullIndex) through the from-scratch args to the prerender-html enqueue, where it lifts the job from 9 to 10. The flag is carried in the from-scratch args only when set, so every other index path's args keep their existing shape; the task reads it loosely (matching the existingclearLastModifiedpattern), so a job enqueued before the field existed reads as non-publish.The tier gap has two effects. Everywhere, the same value rides to the prerender server as the render's in-render admission priority (higher dequeues first), so an index visit's render outranks the slower prerender-html renders — keeping HTML rendering off the indexing hot path at the render layer. Separately, a worker pool that floors at the indexing tier can reserve itself to indexing: that lane is opt-in via
--userIndexCount+--indexJobsOnlyand is run by the contended matrix test stack (which hit indexing back-pressure); the staging/production entrypoints leave--userIndexCountat 0 and rely on high-priority-pool capacity, which is intentional. Where the lane is configured, a publish render (co-equal at 10) is kept out of it by the--indexJobsOnlyjob-type filter, not the floor.The high-priority pool floors at 9 and serves all user-initiated work — user indexing (10), publish renders (10), and ordinary user renders (9) — and never system-tier jobs; the all-priority pool floors at 0 and serves everything.
What's here
userInitiatedPrerenderHtmlPriorityback to 9, with the publish-awaited exception lifting a publish's render to 10.awaitedByPublishthreaded: publish handler →enqueueReindexRealmJob/Realm.fullIndex→publishFullIndex→ from-scratch args → the from-scratch spawn →prerenderHtmlPriority.prerenderHtmlPriority, the worker-manager pool setup, and the matrix worker-tier harness, so the gap, the mirror, and the exception are explicit for future maintainers.prerenderHtmlPrioritymaps each index tier one notch below, and that the publish exception lifts a user render to 10 (but never lifts system work) — so neither the gap nor the exception can quietly drift.boxel_index.diagnostics; the prerender-html renders they spawn carry 9/0 onprerendered_html.diagnostics, except a publish-awaited render, which carries 10 (sopriority=10can appear on either table).Test plan
priority tiersunit module in the prerender-html split tests (passes in the realm-server suite).🤖 Generated with Claude Code