Merge pull request #35 from cardmagic/fix/polling-query-performance

Optimize polling query plans

Author: cardmagic

CHANGELOG.md

@@ -1,5 +1,27 @@
 # Changelog
 
+## 0.14.6 - 2026-09-03
+
+- Poll effects, reminders, and broadcasts through ordered indexes installed by
+  schema migration 8. PostgreSQL and MySQL claim one row with
+  `FOR UPDATE SKIP LOCKED`; candidate probes avoid joins and load actor identity
+  by primary key after the claim. SQLite keeps its serialized transaction path.
+  The broadcast revision guard has its own
+  `(instance_id, state_revision, status)` index instead of rescanning the outbox
+  for every candidate. Claim transactions use read committed isolation on
+  PostgreSQL and MySQL so preceding recovery work cannot turn row skipping into
+  InnoDB gap-lock contention.
+- Replace bulk broadcast and reminder recovery updates with separate available
+  and stale probes. Compare category heads without locking them, lock only the
+  oldest candidate by primary key, and retry past work already locked by another
+  claimant. This preserves global delivery order, concurrent progress,
+  per-instance broadcast revision order, stale-process recovery, and
+  at-least-once delivery.
+- On 50,000 production-shaped rows, SQLite, PostgreSQL 18, and MySQL 8.4 all
+  move from scans and explicit sorts to ordered index probes. The worst SQLite
+  broadcast probe fell from 944 ms to 0.018 ms; PostgreSQL's probes finish in
+  0.02-0.09 ms and MySQL's in 0.10-1.2 ms.
+
 ## 0.14.5 - 2026-08-29
 
 - Record the Ruby state warning in the parity ledger. The row said the Ruby gem

docs/api.md

@@ -206,8 +206,12 @@ rows for `preview()` and rows actually deleted for `prune()`.
 `InstrumentationEvent` type the host integration contract. `JsonPrimitive`,
 `JsonValue`, `JsonObject`, `DeepReadonly`, `ActorIdentifier`, `MessageContext`,
 `MessageStatus`, and `Logger` are shared types. `Database`,
-`DatabaseConnection`, `DatabaseFamily`, and `RunResult` support custom database
-and commit-action integration.
+`DatabaseConnection`, `DatabaseFamily`, `DatabaseTransactionOptions`, and
+`RunResult` support custom database and commit-action integration. A custom
+PostgreSQL or MySQL adapter must honor
+`transaction(callback, { isolationLevel: "read_committed" })`; outbox claiming
+uses that isolation with row locks to avoid InnoDB gap-lock contention. SQLite
+adapters may treat the option as their ordinary serialized transaction.
 
 `BroadcastEvent.observables` contains changed value-broadcast projections.
 `BroadcastEvent.invalidations` contains changed invalidation-only names. The

docs/parity.md

@@ -80,15 +80,15 @@ such boundary between a gem and its dependents.
 
 ## Databases and wake-up
 
-| Capability               | Status | TypeScript shape or remaining work                                                                                                                                                                                                                                                    |
-| ------------------------ | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| SQLite                   | Native | Uses built-in `node:sqlite`, serialized process-local access, bounded transient writer retries, foreign keys, strict tables, database time, and deadline-bounded access and lock waits.                                                                                               |
-| PostgreSQL               | Native | Optional `pg` 8.23 peer, bounded pooling, 64-bit schema, row-locked sequences, server checks, and deadline-bounded pool, statement, and lock waits.                                                                                                                                   |
-| MySQL                    | Native | Optional `mysql2` 3.23 peer, bounded pooling, InnoDB schema, row-locked sequences, scoped deadlock retry, and deadline-bounded pool, query, and lock waits. Ruby also tests a second client, `trilogy`; Node has no comparable second MySQL client, so only `mysql2` is tracked here. |
-| Durable polling fallback | Native | Every role progresses without a notification service.                                                                                                                                                                                                                                 |
-| In-process wake-up       | Native | A generation-based default adapter prevents claim-to-wait signal loss; commits wake role-specific waiters and polling remains the fallback.                                                                                                                                           |
-| PostgreSQL wake-up       | Native | `database.wakeUp()` uses one dedicated event-driven client, role-specific `LISTEN/NOTIFY`, generation fencing, reconnectable listeners, and durable polling fallback.                                                                                                                 |
-| Redis wake-up            | Native | An optional `redis` peer provides role-specific Pub/Sub over separate lazy publisher/subscriber connections, with bounded failures and durable polling fallback.                                                                                                                      |
+| Capability               | Status | TypeScript shape or remaining work                                                                                                                                                                                                                                                                                                                                                    |
+| ------------------------ | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| SQLite                   | Native | Uses built-in `node:sqlite`, serialized process-local access, bounded transient writer retries, foreign keys, strict tables, database time, and deadline-bounded access and lock waits.                                                                                                                                                                                               |
+| PostgreSQL               | Native | Optional `pg` 8.23 peer, bounded pooling, 64-bit schema, row-locked sequences, server checks, and deadline-bounded pool, statement, and lock waits.                                                                                                                                                                                                                                   |
+| MySQL                    | Native | Optional `mysql2` 3.23 peer, bounded pooling, InnoDB schema, row-locked sequences, scoped deadlock retry, and deadline-bounded pool, query, and lock waits. Ruby also tests a second client, `trilogy`; Node has no comparable second MySQL client, so only `mysql2` is tracked here.                                                                                                 |
+| Durable polling fallback | Native | Every role progresses without a notification service. Effects, reminders, and broadcasts use canonical ordered polling indexes; PostgreSQL and MySQL lock only the selected row with `FOR UPDATE SKIP LOCKED`. Broadcasts and reminders compare separate available and stale-recovery probes, preserve the oldest-first choice, and retry past candidates locked by another claimant. |
+| In-process wake-up       | Native | A generation-based default adapter prevents claim-to-wait signal loss; commits wake role-specific waiters and polling remains the fallback.                                                                                                                                                                                                                                           |
+| PostgreSQL wake-up       | Native | `database.wakeUp()` uses one dedicated event-driven client, role-specific `LISTEN/NOTIFY`, generation fencing, reconnectable listeners, and durable polling fallback.                                                                                                                                                                                                                 |
+| Redis wake-up            | Native | An optional `redis` peer provides role-specific Pub/Sub over separate lazy publisher/subscriber connections, with bounded failures and durable polling fallback.                                                                                                                                                                                                                      |
 
 Every wake-up adapter above is opt-in. Neither runtime selects one
 automatically. An application that configures nothing keeps polling. Each

package.json

@@ -1,6 +1,6 @@
 {
   "name": "solid-objects",
-  "version": "0.14.5",
+  "version": "0.14.6",
   "description": "Race-free realtime state per application identity, backed by your SQL database",
   "type": "module",
   "license": "MIT",

src/application-database.ts

@@ -1,5 +1,5 @@
 import { applicationWritesForbidden } from "./context.js"
-import type { Database, DatabaseConnection } from "./database/types.js"
+import type { Database, DatabaseConnection, DatabaseTransactionOptions } from "./database/types.js"
 import { ApplicationWriteForbidden } from "./errors.js"
 
 export function guardApplicationDatabase(database: Database): Database {
@@ -27,8 +27,9 @@ class GuardedApplicationDatabase implements Database {
 
   transaction<Result>(
     callback: (connection: DatabaseConnection) => Promise<Result>,
+    options: DatabaseTransactionOptions = {},
   ): Promise<Result> {
-    return this.database.transaction((connection) => callback(guardConnection(connection)))
+    return this.database.transaction((connection) => callback(guardConnection(connection)), options)
   }
 
   close(): Promise<void> {

src/database/mysql.ts

@@ -6,7 +6,12 @@ import mysqlDriver, {
   type RowDataPacket,
 } from "mysql2/promise"
 import type { ExecuteValues } from "mysql2"
-import type { Database, DatabaseConnection, RunResult } from "./types.js"
+import type {
+  Database,
+  DatabaseConnection,
+  DatabaseTransactionOptions,
+  RunResult,
+} from "./types.js"
 import {
   acquireBeforeDatabaseDeadline,
   databaseDeadlineRemainingMilliseconds,
@@ -152,6 +157,7 @@ export class MySQLDatabase implements Database {
 
   async transaction<Result>(
     callback: (connection: DatabaseConnection) => Promise<Result>,
+    options: DatabaseTransactionOptions = {},
   ): Promise<Result> {
     return withDatabaseTransaction(this, async () => {
       const deadlineActive = databaseDeadlineRemainingMilliseconds() !== undefined
@@ -160,6 +166,9 @@ export class MySQLDatabase implements Database {
         (lateConnection) => lateConnection.release(),
       )
       try {
+        if (options.isolationLevel === "read_committed") {
+          await connection.query("SET TRANSACTION ISOLATION LEVEL READ COMMITTED")
+        }
         await connection.beginTransaction()
         const remaining = requireDatabaseDeadlineRemaining()
         if (remaining !== undefined) {

src/database/postgresql.ts

@@ -1,7 +1,12 @@
 import "../platform/node.js"
 import { Pool, TypeOverrides, types, type PoolClient, type PoolConfig } from "pg"
 import { postgresqlSql } from "./postgresql-sql.js"
-import type { Database, DatabaseConnection, RunResult } from "./types.js"
+import type {
+  Database,
+  DatabaseConnection,
+  DatabaseTransactionOptions,
+  RunResult,
+} from "./types.js"
 import { PostgreSQLWakeUpAdapter, type PostgreSQLWakeUpFailure } from "../wake-up/postgresql.js"
 import {
   acquireBeforeDatabaseDeadline,
@@ -154,6 +159,7 @@ export class PostgreSQLDatabase implements Database {
 
   async transaction<Result>(
     callback: (connection: DatabaseConnection) => Promise<Result>,
+    options: DatabaseTransactionOptions = {},
   ): Promise<Result> {
     return withDatabaseTransaction(this, async () => {
       const deadlineActive = databaseDeadlineRemainingMilliseconds() !== undefined
@@ -162,7 +168,11 @@ export class PostgreSQLDatabase implements Database {
         (lateClient) => lateClient.release(),
       )
       try {
-        await client.query("BEGIN")
+        await client.query(
+          options.isolationLevel === "read_committed"
+            ? "BEGIN ISOLATION LEVEL READ COMMITTED"
+            : "BEGIN",
+        )
         const remaining = requireDatabaseDeadlineRemaining()
         if (remaining !== undefined) {
           await applyPostgreSQLDeadline({ client, milliseconds: remaining, scope: "transaction" })

src/database/types.ts

@@ -5,6 +5,10 @@ export interface RunResult {
   lastInsertId?: string
 }
 
+export interface DatabaseTransactionOptions {
+  isolationLevel?: "read_committed"
+}
+
 export interface DatabaseConnection {
   run(sql: string, parameters?: readonly unknown[]): Promise<RunResult>
   get<Row extends object>(sql: string, parameters?: readonly unknown[]): Promise<Row | undefined>
@@ -19,6 +23,7 @@ export interface Database {
   connection<Result>(callback: (connection: DatabaseConnection) => Promise<Result>): Promise<Result>
   transaction<Result>(
     callback: (connection: DatabaseConnection) => Promise<Result>,
+    options?: DatabaseTransactionOptions,
   ): Promise<Result>
   close(): Promise<void>
 }

src/doctor.ts

@@ -202,11 +202,11 @@ export class Doctor {
           message: `incompatible schema identity ${wrongIdentity.schema_identity}`,
         })
       }
-      if (versions.join(",") !== "1,2,3,4,5,6,7") {
+      if (versions.join(",") !== "1,2,3,4,5,6,7,8") {
         return check({
           name: "schema",
           status: "fail",
-          message: `expected schema migrations 1, 2, 3, 4, 5, 6; found ${versions.join(", ")}`,
+          message: `expected schema migrations 1, 2, 3, 4, 5, 6, 7, 8; found ${versions.join(", ")}`,
         })
       }
       return check({

src/index.ts

@@ -133,7 +133,13 @@ export type {
   MessageStatus,
   SnapshotOptions,
 } from "./types.js"
-export type { Database, DatabaseConnection, DatabaseFamily, RunResult } from "./database/types.js"
+export type {
+  Database,
+  DatabaseConnection,
+  DatabaseFamily,
+  DatabaseTransactionOptions,
+  RunResult,
+} from "./database/types.js"
 export {
   ApplicationWriteForbidden,
   ActorCallCycle,