Repository navigation
Route index and page-image WAL from its header - #225
Merged
Merged
Conversation
This was referenced Oct 19, 2026
iskakaushik
force-pushed
the
skip-index-wal
branch
from
October 20, 2026 03:19
5e40fc7 to
87476ae
Compare
This was referenced Oct 20, 2026
serprex
force-pushed
the
skip-index-wal
branch
2 times, most recently
from
October 22, 2026 13:56
f46ec10 to
ed11493
Compare
serprex
force-pushed
the
skip-index-wal
branch
from
October 22, 2026 14:47
ed11493 to
2be2c83
Compare
serprex
approved these changes
Oct 22, 2026
harshil-goel
approved these changes
Oct 22, 2026
serprex
force-pushed
the
skip-index-wal
branch
from
October 22, 2026 14:51
2be2c83 to
fff38ab
Compare
WAL pump fully parsed and copied block data and page images for user relation records that decoder and reorder stages skip, then rewrote those records as NOOPs for shadow output. Walk headers and block locators for index, sequence, generic and page-image records. Skip full parsing only when every block references a nonzero relation outside tracked catalogs, pg_class image tracking and retained shadow relations. Preserve xid ceilings, subtransaction links, filter stats and NOOP output. Continue dispatching header-only records so metrics and acknowledgement positions advance as before. Require explicit filter opt-in for sinks that need no payload from these records. Enable routing in stream sessions only when no database is pending, since database attachment needs block locators. Fall back to full parsing for skip attribution, rejected headers and all other records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
serprex
force-pushed
the
skip-index-wal
branch
from
October 22, 2026 14:56
fff38ab to
6dbe67e
Compare
harshil-goel
approved these changes
Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every user-relation record took the full path in the pump: parse every block reference and payload, decide_record, into_owned (copying page images and block data), then dispatch. For index, sequence, generic and page-image records on user relations nothing reads past the header: they route ToDecoder, the decoder and reorder stages skip them by RecordSink::wants, and the shadow gets a NOOP. Index maintenance is most of the WAL in an index-heavy workload, so the pump spent most of its parsing and copying on records it discards.
Walk the header area only (fixed header, origin, toplevel xid, block locators) and, when every block is a user relation the catalog tracker does not know, that shadow_rels does not keep, and whose image teaches no pg_class slot, hand the sinks a header-only record. The pump records the xid ceiling, the subxact link and the filter stats as decide_record would, and rewrites the bytes to the same NOOP.
The sink stack opts in through RecordSink::header_only: the queueing sink answers from its inner wants, and the boundary hold refuses while a database is pending, since attaching one reads block locators. Header-only records still dispatch, so the metrics per rmgr and the idle ack position see every record as before. Anything else, including malformed records, takes the full parse. With
[stream] skip_attributionon, every record takes the full parse, so the skip ledger still attributes dropped index records.The walk rejects every record parse_record_from_bytes rejects. A fixture test checks it reads the same header, xids and locators as the parser on every captured record, and that a stream taking the fast path emits the same bytes, stats, xid ceilings and verdicts. The classify workload gains every index access method and a subtransaction so the corpus covers them.
Based on main, which now includes #222.
Benchmark: a write-heavy repro, one wide table with 5 btree indexes including two partial, insert plus two non-HOT updates per transaction (32 clients, 10 s, 200k seed rows). The daemon is stopped while the load runs, then the drain rate over the fixed backlog is walshadow's ceiling for that WAL mix. About 57% of records are btree. Sides alternated, 4 runs each for drain (min to max, mean), 2 runs each for the pump thread's CPU per record.
CHECKPOINT_S=5CHECKPOINT_S=5The pump thread spends about 22% less CPU per record. End-to-end drain does not move beyond run-to-run noise: the pump runs at 60 to 85% of a core, so the queue worker and the decode pool bound the drain on this mix, not the pump.
🤖 Generated with Claude Code