Skip to content

Route index and page-image WAL from its header - #225

Merged
serprex merged 1 commit into
mainfrom
skip-index-wal
Oct 22, 2026
Merged

serprex merged 1 commit into
mainfrom
skip-index-wal

Conversation

@iskakaushik

@iskakaushik iskakaushik commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Every user-relation record took the full path in the pump: parse every block reference and payload, decide_record, into_owned (copying page images and block data), then dispatch. For index, sequence, generic and page-image records on user relations nothing reads past the header: they route ToDecoder, the decoder and reorder stages skip them by RecordSink::wants, and the shadow gets a NOOP. Index maintenance is most of the WAL in an index-heavy workload, so the pump spent most of its parsing and copying on records it discards.

Walk the header area only (fixed header, origin, toplevel xid, block locators) and, when every block is a user relation the catalog tracker does not know, that shadow_rels does not keep, and whose image teaches no pg_class slot, hand the sinks a header-only record. The pump records the xid ceiling, the subxact link and the filter stats as decide_record would, and rewrites the bytes to the same NOOP.

The sink stack opts in through RecordSink::header_only: the queueing sink answers from its inner wants, and the boundary hold refuses while a database is pending, since attaching one reads block locators. Header-only records still dispatch, so the metrics per rmgr and the idle ack position see every record as before. Anything else, including malformed records, takes the full parse. With [stream] skip_attribution on, every record takes the full parse, so the skip ledger still attributes dropped index records.

The walk rejects every record parse_record_from_bytes rejects. A fixture test checks it reads the same header, xids and locators as the parser on every captured record, and that a stream taking the fast path emits the same bytes, stats, xid ceilings and verdicts. The classify workload gains every index access method and a subtransaction so the corpus covers them.

Based on main, which now includes #222.

Benchmark: a write-heavy repro, one wide table with 5 btree indexes including two partial, insert plus two non-HOT updates per transaction (32 clients, 10 s, 200k seed rows). The daemon is stopped while the load runs, then the drain rate over the fixed backlog is walshadow's ceiling for that WAL mix. About 57% of records are btree. Sides alternated, 4 runs each for drain (min to max, mean), 2 runs each for the pump thread's CPU per record.

drain MiB/s rec/s pump ns/rec
#222, standard 193 to 222 (207) 1.80M to 2.08M (1.93M) 439, 441
this, standard 174 to 224 (203) 1.62M to 2.10M (1.90M) 343, 362
#222, CHECKPOINT_S=5 209 to 234 (224) 1.77M to 1.99M (1.90M) 460, 428
this, CHECKPOINT_S=5 217 to 233 (228) 1.84M to 1.98M (1.93M) 340, 338

The pump thread spends about 22% less CPU per record. End-to-end drain does not move beyond run-to-run noise: the pump runs at 60 to 85% of a core, so the queue worker and the decode pool bound the drain on this mix, not the pump.

🤖 Generated with Claude Code

WAL pump fully parsed and copied block data and page images for user
relation records that decoder and reorder stages skip, then rewrote
those records as NOOPs for shadow output.

Walk headers and block locators for index, sequence, generic and
page-image records. Skip full parsing only when every block references
a nonzero relation outside tracked catalogs, pg_class image tracking
and retained shadow relations. Preserve xid ceilings, subtransaction
links, filter stats and NOOP output. Continue dispatching header-only
records so metrics and acknowledgement positions advance as before.

Require explicit filter opt-in for sinks that need no payload from
these records. Enable routing in stream sessions only when no database
is pending, since database attachment needs block locators. Fall back
to full parsing for skip attribution, rejected headers and all other
records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@serprex
serprex merged commit 151fd47 into main Oct 22, 2026
20 checks passed
@serprex
serprex deleted the skip-index-wal branch October 22, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants