Short-lived double write buffer: torn-page protection without full-page writes - #3
Short-lived double write buffer: torn-page protection without full-page writes#3vbp1 wants to merge 52 commits into
Conversation
Introduce the pg_dwb subsystem: an alternative torn-page protection to full-page writes, shaped after InnoDB's doublewrite buffer. A DWB slot lives for the duration of one flush batch and is reused only after the data-file fsync covering its page has become durable, per-batch order: XLogFlush -> batch write + fdatasync -> smgrwrite -> segment fsync. This stage brings the standalone skeleton, not yet wired into FlushBuffer: * storage/dwb module: the batch state machine (FREE -> ALLOCATED -> SEALED -> WRITTEN -> FSYNCED -> DATA_WRITTEN -> RETIRING -> FREE), slot reservation via a 31-bit index with a SEAL_BIT sentinel, seal/leader election, the leader batch write (contiguous image stream, then the meta region, then fdatasync) and abandoned-slot cleanup on process exit. Retirement is synchronous for now; the retire worker pool arrives with the FlushBuffer integration. * On-disk ring under $PGDATA/pg_dwb/: zero-preallocated batch files (fixed size, so fdatasync suffices for durability, the WAL-segment contract) and an atomically replaced control file carrying the geometry and a durable generation, bumped on every start before the ring opens. Slot validity is locally verifiable: meta_crc rejects a torn meta write (including old/new field mixes on slot reuse), image_crc rejects a torn image. * Shared-memory state with an IO-aligned staging pool: writers publish page images by memcpy only, all pg_dwb I/O is done by leaders. * GUCs (io_torn_pages_protection plus the dwb_* family), enforcement of data checksums in double_writes mode, wait events and LWLocks. * test_dwb module: pg_regress tests for write cycles and on-disk CRC validation, TAP coverage for concurrent writers over a small ring, the generation bump across restarts, geometry mismatch and the checksum requirement.
Make the leader write crash-consistent and close the liveness holes found in review: * Run the leader span (seal win -> DWB_FSYNCED broadcast) as a critical section: an I/O error there cannot be unwound - the seal winner is the only process able to advance the batch - so it now escalates to PANIC instead of leaving the batch, its waiters and its staging buffer wedged forever. Everything that may fail harmlessly (the one-time leader allocations, the batch file VFD, the backend's condition- variable wait event set) moves into a pre-seal prepare phase, where an ERROR leaves the batch ALLOCATED for another writer to seal later. * Pin the batch with a leader ref from SEAL to FSYNCED: a sealed batch can no longer drop to zero refs before it is durable, so the FSYNCED -> RETIRING hand-off always has exactly one owner, even when every writer exits during the write. * Guard the seg_set dedup scan with the batch's publish_lock LWLock instead of a spinlock (the O(n) scan far exceeds the spinlock hold-time rule); drop the seg_lock field. * Reserve the staging buffer before taking DWBRingOpenLock, leaving no sleeping or interruptible point while a batch is held out of DWB_FREE. * Assorted hardening: ConditionVariableCancelSleep after wait loops, a hard ERROR on pendingRefs overflow, batch_id assertions in the writer-side API, uint16 slot_flags to match the on-disk width, StaticAssertDecl pins on the on-disk struct layout, canonical read-error reporting, ssize_t for FileWrite results, a symbolic GUC maximum for dwb_batch_pages, and comments aligned with the Stage 1 reality. * test_dwb: prove CRC rejection by flipping bytes in a batch file, exercise both DWBProcExit paths with a backend dying mid-batch (the orphan scenario is a direct regression test for the leader pin), refuse-start coverage for dwb_batch_pages, and tighter bounds.
The "has the open batch already been replaced" guard compared only the ring index. Indexes are reused quickly on a small ring, so a slow opener - one that bounced off a sealed batch and then slept, e.g. in the pre-lock staging-buffer wait - could arrive while open_batch_idx named the SAME index again, now holding a NEW live incarnation. The index-only comparison passed, the opener repointed open_batch_idx to a fresh batch, and the live incarnation was orphaned in ALLOCATED together with its staging buffer: nothing ever seals an unreachable batch, writers waiting on it hang, and a writer blocked there while holding an unpublished reservation in the next batch eventually drives that batch's leader into the coverage-wait PANIC. The concurrent TAP stress hit this in the majority of runs. Replace the guard with the exact criterion, evaluated under DWBRingOpenLock: the open batch needs replacing if and only if its SEAL_BIT is set. SEAL_BIT is set at SEAL and stays set through FREE; only the reopen re-initialization - under this same lock - clears it. A live reopened incarnation at the same index therefore reads as "bit clear" and the stale opener backs off; if the new incarnation has itself been sealed meanwhile, replacing it is correct no matter which incarnation the caller bounced off. Comparing batch ids instead would require capturing a non-atomic uint64 outside the lock (a torn-read window on 32-bit platforms) and plumbing it through the API. Add test_dwb_open_stale(): a deterministic single-backend regression that replays the losing interleaving without any timing dependence - cycle once so open_batch_idx names a sealed-and-retired index, reopen the same index as a live incarnation by acquiring one slot, then call DWBOpenNewBatch with the stale index and assert the pointer is untouched. With the index-only guard the test fails every run; the organic stress race fires only probabilistically. DWBOpenNewBatch is exported non-static solely for this test.
No caller in any planned stage: the apply-pass and ring re-creation run in the startup process before any backend exists, and the batch-file VFD caches of leaders and retire workers are torn down by fd.c at process exit anyway.
FlushBuffer now stages every BM_PERMANENT page into the double write buffer before smgrwrite: DWBStagePageWrite makes the private copy durable in pg_dwb/, DWBFinishPageWrite releases the batch ref afterwards, with an optional smgrwriteback in between (dwb_writeback) so the retirement fsync becomes a cheap barrier. Writers are classified by process role into the eviction and background (checkpointer, bgwriter) classes with separate open batches; background opens keep DWB_EVICT_RESERVE free batches for user evictions. Retirement is driven by the new segment->batch back-reference hash (DWSegmentHash): the last batch ref publishes the seg_set under the batch's publish_lock, and any segment fsync clears the bits it had snapshotted before starting, guarded against ring-index reuse by re-checking batch_id under the same lock. Fsyncs come from three independent sources sharing that accounting: a pool of dwb_retire_workers background workers (partitioned proactive sweeps, force-seal of timed-out batches), ProcessSyncRequests in the checkpointer (wrapped around each md segment fsync), and writers stuck on a full ring sweeping synchronously as self-service. A hash overflow degrades to a synchronous per-batch retire instead of wedging the ring. Sealing is decentralized: a writer whose ref is the only one on a still-open batch seals immediately (a lone sequential flush stream would otherwise pay dwb_batch_timeout_ms per page), a waiter seals after that timeout, and the workers back-stop batches whose writers all died. This keeps the write path fully functional with no worker pool at all (dwb_retire_workers = 0, single-user mode, shutdown checkpoint after the workers are gone). Refs are now attached to the ResourceOwner and released before the buffer-IO cleanup: an abandoned ref whose batch is already durable repairs its data page from the batch file while BM_IO_IN_PROGRESS is still held, closing the window where a half-done smgrwrite could leave a torn page behind a recycling batch, without racing any concurrent flush. Backpressure escalates in two stages clocked by retirement progress (freed_events): a WARNING plus bgwriter pause, then dwb_on_stall, where the checkpointer and the startup process always PANIC by explicit policy. The dwb-force-stall injection point drives both TAP scenarios: a stalled backend eviction fails with an ERROR while the cluster stays up, and a stalled checkpointer PANICs into crash recovery. pg_stat_io gains the "dwb" object (batch writes with bytes, fdatasyncs); new wait events cover the worker main loop and batch-file reads.
…-up) A review round over the Stage 2 commit surfaced one real corruption path and two accounting leaks, all rooted in the assumption that a retirement fsync failure is always a PANIC. That only holds with the default data_sync_retry = off; with data_sync_retry = on the process survives the error and the bookkeeping used to rot: * DWBSegmentFsyncBegin now unconditionally drops a leftover snapshot before deciding whether to take a new one. Previously, after an fsync ERROR between Begin and End, the checkpointer kept the stale snapshot armed; the End of the next successful fsync of an unrelated non-MD tag (CLOG/SLRU share pendingOps) consumed it, decrementing the failed segment's back-references and freeing its batches without their data being durable -- silent corruption after a crash. * DWBRetireSyncSegment no longer throws on a soft failure: it returns false (WARNING), the segment's bits stay for a later retry, and the caller's fsync_in_progress claim is always released. The OOM retire path escalates to PANIC instead: nothing ever revisits a DWB_OOM_RETIRING batch, and the path can run inside a ResourceOwner release callback, which must not fail. The proc-exit backstop now performs the abandoned-slot repair only for refs that were attached to a ResourceOwner. An owned ref alive at proc-exit implies the same-owner buffer-IO resource is alive too (it releases after the DWB ref within one owner, and DWBFinishPageWrite precedes TerminateBufferIO on the success path), so BM_IO_IN_PROGRESS is still ours and the repair is race-free; ownerless test refs never had the interlock, so writing would race concurrent flushes. Hardening: DWB_NUM_BATCHES_MAX ties the GUC maximum to the retire-side snapshot array; static asserts pin the no-padding requirement of the DWSegRef hash key and the semantic ordering of DWBatchState; DWBAcquireSlot memsets the seg ref it publishes; DWBPublishBatchSegSet asserts a non-empty seg_set. Comment fixes: the DWBTrySealBatch header no longer claims every seal funnels through it (the overflow writer calls DWBSealBatch directly), and the FlushBuffer comment no longer overstates PageSetChecksumCopy for all-zero new pages. New test coverage: * test_dwb_torn_repair stages a real table's block, tears it on disk and aborts: the ResourceOwner release must repair the block from the batch copy, verified by reading it back after a restart (001). * 004_retire_paths.pl proves a CHECKPOINT alone retires a parked batch through the DWBSegmentFsyncBegin/End wrap (fsync = on is required: with fsync = off ProcessSyncRequests skips the whole per-file block), and overflows DWSegmentHash to exercise the synchronous OOM retire. * 002 gains a dwb_retire_workers = 0 phase over a real eviction workload and an unlogged-bypass check that reads the ring content itself (test_dwb_ring_rel_slots) -- a normal CHECKPOINT skips unlogged buffers, so the shutdown checkpoint provides the flush. * 003 walks a victim through Stage A into Stage B on the real stall clocks, with no injection point. * 001 covers the DWB_EVICT_RESERVE boundary: a background-class fill stops with the reserve left free, an eviction-class fill drains it. * meson.build now runs all four TAP files (002/003 were missing).
…ollow-up) A cross-agent review round over the Stage 2 commits surfaced three process-lifecycle and accounting defects, closed here together with the regression coverage it asked for. The exit backstop moves from on_proc_exit to before_shmem_exit. Dropping the last ref of a durable batch publishes its seg_set under publish_lock and DWBSegHashLock, which is only legal while the PGPROC is alive -- on_proc_exit callbacks run after ProcKill, and LWLockAcquire asserts a live PGPROC under a postmaster. Because the backstop may now run before the ResourceOwner release of the same refs, DWBAbandonRef became idempotent: whichever side runs second finds in_use = false and returns. test_dwb_leak_fsynced pins the scenario: a session exits holding the LAST ref of a DWB_FSYNCED batch, so the FSYNCED -> RETIRING hand-off executes inside the exit callback itself. The abandoned-slot repair path is now allocation-free. It runs from release callbacks that must not fail, but OpenTransientFile can ERROR in reserveAllocatedDesc and PathNameOpenFile in its VFD bookkeeping. DWBReadSlotImage opens the batch file itself with BasicOpenFile (no allocations; recovers from EMFILE by closing LRU VFDs) and reads with raw pg_pread under the DWB_BATCH_READ wait event; DWBRewriteAbandonedSlot likewise switches to BasicOpenFile. Every failure on the path, opens and closes included, is a PANIC. The retire worker pool now verifies that its workers actually fit. RegisterBackgroundWorker only LOGs "too many background workers", so a pool registered after the logical replication launcher could silently come up short. bgworker.c exposes GetNumRegisteredBackgroundWorkers() -- counting only successful registrations; the previous pre-increment also overcounted failed attempts -- and DWBRetireWorkersRegister FATALs when dwb_retire_workers exceeds the slots that remain free. A TAP test asserts the refusal at max_worker_processes = 1. pgstat_count_io_op_time() feeds pg_stat_database's blk_write_time / blk_read_time only for relation and temp-relation objects now: the old "anything but WAL" condition also counted DWB batch writes, doubling the apparent block write time of a page that goes through the ring. New data_sync_retry regression coverage in 004_retire_paths.pl: test_dwb_stale_snapshot replays a checkpointer fsync ERROR between DWBSegmentFsyncBegin/End followed by a successful non-MD sync -- the parked batch must stay RETIRING; and a directory planted at a fake segment's path makes the retire fsync fail with EISDIR under data_sync_retry = on -- the sweep must not throw, the batch must stay, and the segment must be coverable by the next sweep after the obstacle is removed, proving the advisory claim was released. All DWB typedefs are added to typedefs.list and the touched files are pgindent-clean.
A complexity-focused review pass over the Stage 2 commits; behavior is unchanged and the full test_dwb suite stays green. * Drop the nPendingRefs counter: the free-slot scan in DWBAcquireSlot already knows whether the array is full, so the overflow guard moves after the scan and the counter bookkeeping disappears. * Replace the hand-rolled bubble sort of RETIRING batches in DWBRetireSweep with qsort over a named DWBRetiringBatch struct and a pg_cmp_u64 comparator; batch ids are unique, so stability is not needed, and O(n^2) was pointless on 1024-batch rings. * Deduplicate the test harness: make_tag() replaces nine copies of the RelFileLocator/InitBufferTag boilerplate, stage_one_page() is the shared acquire-publish-seal-wait prologue of the five single-page scenarios, and leak_refs() is the shared body of the leak and abort-release scenarios. Upcoming checkpoint and apply-pass tests can reuse these instead of growing new copies. Deliberately kept: the dwb-after-batch-fsynced injection point (the hook for future apply-pass crash tests between the batch fsync and smgrwrite) and DWBWriterClass() (the seam where flush-source classification will grow).
io_torn_pages_protection now drives WAL full-page images through a single derivation point, EffectiveFullPageWrites(): "double_writes" and "off" force FPIs off, while "full_pages" keeps the legacy full_page_writes GUC meaningful (external consumers such as pg_upgrade still pass it). Online backups keep forcing page images through the vanilla runningBackups term of doPageWrites; repairing a page copied mid-write into a backup is FPI replay's job, not the ring's. The pg_dwb ring is local to an instance, so its contents are excluded from base backups via excludeDirContents, which also converts a symlinked pg_dwb into an empty real directory. This is mirrored in pg_rewind's filemap so a rewind never copies the source's ring over; the target's own leftover ring stays inert thanks to the generation bump on every start. Checkpoints take no DWB barrier by design and needed no code; a new 004 scenario pins that a CHECKPOINT completes while an open ALLOCATED batch is live and leaves it alone. New TAP coverage: 005_standby.pl (a standby cold-starts a fresh ring from a base backup, runs its own ring during replay with the retire worker alive in recovery, survives crashes on either side, promotes with a replay backlog and advances minRecoveryPoint past replayed flushes) and 006_backup.pl (pg_waldump shows no FPW outside a backup window and FPWs inside one; the backup keeps pg_dwb as an empty directory with no warnings, including the symlink layout; a restored cluster cold-starts a fresh generation).
…3 follow-up) The per-class open batch pointer could keep naming a freed batch index after the other class reopened it: open_batch_idx[] is only replaced when its batch is sealed, and reservations did not check who owns the current incarnation. A checkpointer flush could then join a live EVICTION batch and, on the no-pool path, seal it under the holder. next_slot_idx now carries a writer-class bit stamped at re-init under DWBRingOpenLock, reservations use a CAS that validates the seal and class bits atomically with the increment (a blind fetch_add into a foreign batch would leave a never-published slot and hang the leader's coverage wait), and DWBOpenNewBatch treats a foreign-class incarnation as needing replacement. The checkpoint scenario in 004 reproduces the aliasing deterministically and no longer needs its warmup choreography. The rest closes gaps around WAL that carries no full-page images: * pg_rewind now requires io_torn_pages_protection = "full_pages" (and full_page_writes = on) on a live source: pages read from a running server can be torn and only FPI replay repairs them, which double_writes WAL cannot provide. Rewinding from a stopped source stays available. New 007_rewind.pl also pins that the rewind wipes the target's own ring and the rewound node cold-starts a fresh one. * A mode change through a restart emits no XLOG_FPW_CHANGE (the end-of-recovery UpdateFullPageWrites call is gated on recovery being finished), so the standby backup guards never saw the transition and a backup taken before the next restartpoint silently spanned image-less WAL. Replayed checkpoint records declaring full_page_writes = off now advance lastFpwDisableRecPtr too, and the same helper warns once when a server replays image-less WAL without running a double write buffer of its own. * The do_pg_backup_start/stop refusals name io_torn_pages_protection and the workable paths instead of a full_page_writes hint that this mode ignores; 005 tests the refusal. Mode "off" logs its state at startup. pg_dwb is documented in protocol.sgml, backup.sgml and storage.sgml. Test hardening: 005 fits the runtime budget, asserts minRecoveryPoint advance from replay-driven flushes alone, verifies page contents, and promotes with the replay pause still in effect so the backlog is real; 006 checks pg_waldump errors and matches the forced FPI by file node; new 008_modes.pl pins mode "off", the legacy GUC under "full_pages" and the SIGHUP no-op under "double_writes".
pg_rewind now supports a ring living behind a symlinked pg_dwb: the target traversal follows the link so the exclusion-driven removal wipes the ring through it, the entry itself is never created, removed or type-checked (the server recreates it lazily), and sync_pgdata() syncs the linked directory like a symlinked pg_wal, making the wipe durable. The source's link is deliberately not followed: its ring is never used, so a broken link there must not fail the rewind. A new up-front checkTargetDwb() runs before the target is touched in any way -- in particular before the single-user recovery run -- and rejects garbage the file-list traversal cannot see: a non-directory entry, a symlink not pointing at an accessible directory, or anything but regular files inside the flat ring. A restart transition into double_writes emits no XLOG_FPW_CHANGE, so the new 009 test pins the replayed-checkpoint tracking end to end: an online backup opened on a standby before the transition must fail pg_backup_stop(), and a standby without a ring of its own warns exactly once per startup. 007 grew refusal scenarios for every garbage pg_dwb layout, asserted against a crashed target via pg_controldata to prove the refusal precedes recovery.
The replace-guard in DWBOpenNewBatch computes staleness into a flag instead of duplicating the release-and-return sequence per branch. decide_file_action() drops the refusal of a regular-file pg_dwb on the source: the source's ring is never read or copied, so the check protected nothing (the target side is covered by checkTargetDwb()). The rewind test hoists the six copies of the pg_rewind invocation into one array. No behavior change on any path a server can reach.
Before WAL replay, DWBStartup now runs an eager apply-pass over the ring unless it was cleanly closed: candidate slots (valid batch header, meta_crc, current generation, image_crc, not ABORTED) are deduplicated per page keeping the highest LSN, and a data page is rewritten from its slot copy when it fails verification or is older than the copy; repaired forks are fsynced before the generation bump. This is the only repair path, and it covers pages replay never reads, such as hint-bit-only pages logged as XLOG_FPI_FOR_HINT without an image. The ring's control file gains a RING_CLEAN marker, written at the tail of a clean shutdown once the ring is fully retired and cleared when the ring reopens. The apply-pass keys on pg_control OR the marker: a standby's shutdown restartpoint can be skipped entirely, leaving segment fsyncs pending behind a clean pg_control. The same marker guards mode downgrades: with an uncleanly closed ring, a start under full_pages or off is refused until one clean double_writes run applies it. A start from a restored base backup (backup_label present, or backupStartPoint still set after a crash mid-backup-recovery) never applies a ring shipped by a third-party backup tool: the generation and LSN defences both pass honestly there, and an apply would push pages from the backup's future into a PITR target. The ring contents are discarded and recreated cold, in the non-DWB modes too. A geometry change now applies the old ring first and recreates it under the new geometry instead of refusing to start. io_torn_pages_protection is recorded in pg_control and in XLOG_PARAMETER_CHANGE, like wal_level: a server expecting full-page protection refuses to replay WAL generated without page images (CheckRequiredParameterValues FATAL, replacing the once-per-startup WARNING), the standby backup guards name the primary's actual mode, and pg_rewind reads a live source's mode from its pg_control up front, before ensureCleanShutdown can touch the target. On a crashed standby the apply-pass raises minRecoveryPoint to the highest applied LSN as a belt-and-braces enforcement of what the write path already guarantees. Tests: new t/010_recovery.pl (torn hint page, stale-but-valid page repaired by LSN, idempotent re-apply, clean-start skip, past-generation slot never applied); 009 re-pointed at the legacy full_page_writes=off transition plus the incompatible-standby FATAL; 006 plants a foreign ring into a restore; 007 rewinds a crashed target through the single-user run; 008 covers the downgrade guard; 001 the geometry recreate.
…w-up) Key the apply-pass decision on the RING_CLEAN marker alone and drop the unclean_start parameter of DWBStartup. The marker is the exact certificate: it is set only after full retirement and cleared before the ring reopens, while pg_control diverges from the ring in both directions. Keying on pg_control let a crash under an interim full_pages/off run (which touches neither the marker nor the generation) re-arm a fully retired ring: its slots still matched the current generation, and the pass would resurrect ancient page images over blocks torn long after the ring was closed, turning a loud checksum failure into silent corruption. Close the remaining error-path holes around the pass: - Fail closed when backup_label cannot be probed: any errno other than ENOENT is now FATAL instead of reading as "not restoring a backup", which would have applied a ring shipped inside a base backup. - Skip candidates whose on-disk page is new (empty header). A zeroed header proves the covered write's first sector never landed on a block that never held an initialized page, so replay recreates it; repairing would also let a pre-truncate slot image resurrect on a re-extended zero page whose LSN 0 loses the staleness comparison. - Make DWBWipeRing remove the control file first, durably, and let the cold-create path sweep leftovers: a crash mid-wipe used to leave a readable control beside missing batch files, and every retry then died on the ENOENT until manual intervention. - Re-verify the slot image CRC after the repair loop's second read; divergent reads mean failing storage and abort startup instead of writing unverified bytes over a data page. - Cover DB_SHUTDOWNED_IN_RECOVERY in the minRecoveryPoint raise: the pass also runs on a cleanly stopped standby with an unretired ring. - Refuse an unreadable pg_dwb/control in full_pages/off modes with the downgrade guard's message and removal hint instead of a bare low-level FATAL that named no way out. - Warn when crash recovery replays WAL generated under io_torn_pages_protection=off with a different local mode: pages torn by that crash cannot be repaired by any local mechanism. Place io_torn_pages_protection next to wal_level in ControlFileData and xl_parameter_change (both formats were already bumped on this branch), deduplicate applied forks with a hash table, reuse DWBBatchFilePath instead of a second spelling of the batch path, log the ring wipe in non-DWB restore starts, use ereport(LOG) for the minRecoveryPoint raise, and pin down invariants at their definition sites: the enum values are on-disk facts and must not be renumbered, batch_id monotonicity within a run is what makes the equal-LSN tie-break valid, and the flags field fills a former CRC-covered padding hole, which is why version-1 control files read back compatibly. New and extended tests: t/011_geometry_recovery.pl pins that a crashed ring is applied with its recorded geometry before being recreated under new GUCs; t/010 pins the marker-only trigger (clean pg_control, unclean ring) and the dropped-relation skip; t/008 pins that reopening the ring re-arms the downgrade guard and that a corrupt control refuses non-ring modes with the hinted recipe actually working; t/006 pins the planted ring wipe under full_pages; t/005 pins a torn standby page repaired from the standby's own ring.
The verification round over the previous commit confirmed every original finding fixed and surfaced a handful of new ones, all closed here. Re-bump PG_CONTROL_VERSION (1802) and XLOG_PAGE_MAGIC (0xD11A): the previous commit moved io_torn_pages_protection next to wal_level in ControlFileData and xl_parameter_change without changing the version constants it had already claimed, so a data directory initialized on the previous commit would be reinterpreted field-by-field with a matching CRC and no complaint. Limit the crash-under-"off" warning to servers whose crashed run owned the pg_control mode field: on a standby the field describes the primary, and a local double_writes standby of an "off" primary would have drawn the warning on every crash while its own ring repaired the pages the message declared unrepairable. Make DWBWipeRing report whether it removed anything and log the "discarding ring contents" message only then — a restored base backup normally ships pg_dwb/ empty, and every ordinary restore start claimed to have discarded contents that never existed. The empty-directory early return also makes the second wipe of a double_writes restore a no-op. Keep a too-new ring format version on its own FATAL instead of folding it into the corrupt-control refusal: that ring is intact and the "remove pg_dwb" hint would invite discarding it. The remaining corrupt paths now log the specific low-level cause (errno, short read, bad checksum) before the caller's summary FATAL, and the previously unchecked close in the short-read branch is reported too. Comment accuracy, per the re-review: the batch_id tie-break is a deterministic pick among hint-bit-equivalent copies, not a strict later-copy guarantee (ids are assigned at batch open and two writer classes can invert the order); the PageIsNew-skip rationale now covers the all-zero staged image FlushBuffer can produce; the RING_CLEAN contract states that any future ring-opening path must clear the marker in the same control write; DWBWipeRing's header names all its callers; the file header no longer claims non-ring modes never touch the ring. New tests: t/008 pins the suppressing direction of the marker keying — a crash under an interim full_pages run must not re-arm the apply-pass — and the warning after a crash under "off"; t/010 pins that an all-zero on-disk page is never repaired from a slot, bytewise.
DWBReadControlFile duplicated every failure as a LOG-when-tolerated / FATAL-otherwise pair; a single ereport per failure with a variable elevel says the same thing in half the lines. The two restored-backup wipe branches of DWBStartup were byte-identical; hoist one conditional wipe above the mode split. This also discards a shipped ring before the data-checksums check rather than after — an inconsequential reordering, since such a ring is doomed in every mode. The batch_id tie-break story was told in full at three sites, and the last review round had to correct all three in lockstep; the dedup comment in DWBApplyPass keeps the full argument, the two field comments now just point there while retaining the local fact (assignment at batch open, monotonic per run). The raw block I/O helpers were copied into three TAP files; they now live in t/DWBTest.pm (following pg_rewind's RewindTest.pm precedent).
Pin down the apply-pass behaviours the runtime write path cannot be steered into: * New test_dwb_craft_batch() writes a synthetic single-slot batch file directly, taking geometry and generation from the ring control; the image is the block's on-disk content with a chosen LSN, a marker in the page hole and a recomputed checksum, so an applied image remains a valid page. 012_apply_crafted.pl uses it to pin the dedup comparator - the higher LSN wins, and an equal-LSN tie goes to the higher batch_id against scan order - and the standby minRecoveryPoint raise, crafting a slot LSN inside the received-but-unreplayed window held open by pg_wal_replay_pause. * 013_backup_start_point.pl pins the backupStartPoint arm of the restored-backup detection end to end: a low-level copy taken inside an open backup window lacks the backup-end WAL, so recovery from it fails deterministically; the first start consumes backup_label, the second start runs on backupStartPoint alone, and both must discard the ring instead of applying it. * 014_pg_upgrade.pl runs a same-version pg_upgrade from a double_writes cluster: nothing of pg_dwb/ transfers, and the upgraded cluster cold-starts a ring of its own. * New test_dwb_set_control_min_version() rewrites the ring control with a chosen min_version and a matching CRC; 008_modes.pl asserts that the format-version refusal names the version gap rather than the corrupt-ring removal advice, and that leftovers of an interrupted wipe (batch files behind a missing control) are swept by the cold-create path. * 009_fpw_transition.pl asserts that a crashed double_writes standby of an "off" primary restarts without the crash-under-off warning, and 006_backup.pl that an ordinary restore does not claim to discard the empty pg_dwb directory it ships.
config.sgml gains io_torn_pages_protection and the dwb_* parameters, ordered as in postgresql.conf.sample, plus a note that full_page_writes is consulted only in full_pages mode. wal.sgml describes the double write buffer next to the discussion of partial page writes. backup.sgml notes that an active base backup forces page images back into WAL under double_writes; pg_basebackup's standby-backup limitations name the full_pages requirement on the primary; pg_rewind documents the full_pages requirement for a live source and the fate of pg_dwb during a rewind. monitoring.sgml adds the dwb object of pg_stat_io.
The double_writes documentation promised that WAL carries no full page images at all; operations registering a block with REGBUF_FORCE_IMAGE still log one regardless of doPageWrites, so speak of the automatic first-modification-after-checkpoint images instead, in config.sgml, wal.sgml and the pg_basebackup standby-backup limitation. The ring also covers only permanent relations, so say "every permanent data page". The pg_stat_io intro in monitoring.sgml now mentions the double write buffer next to relations and WAL. 013_backup_start_point.pl asserted the absence of an apply-pass only after the second failed start; add the same negative check against the first start's log window.
pgindent rewraps two comments in bgwriter.c and postmaster.c that fell short of the 78-column margin. perltidy reformats the 14 test_dwb TAP tests -- argument wrapping, paren tightness and continuation indentation. Run with version 20230309, the one pinned by src/tools/pgindent/README. perlcritic flagged the sysseek() calls in 001_dwb.pl as ignoring their return value. The defined-or guard does check it, but RequireCheckedSyscalls only recognizes "or die", which is the form used throughout the rest of the tree, so switch to it. Formatting only; the test_dwb suite (261 tests) still passes.
The 6-point grid on the SF75K stand showed two structural sink defects: the checkpointer degenerated to one-slot batches (one fdatasync per page via the lone-writer seal), and at >= 2700 connections it starved outright because the one-sided eviction reserve never let the background class open a batch while evicting backends kept the free count at the throttle line. BufferSync now collects permanent buffers into bins of up to dwb_batch_pages (capped at 64 by the content-lock budget) and flushes each bin through the DWB as one batch: a non-blocking gather (conditional content lock plus nowait StartBufferIO; contended buffers fall back to the per-page path), one WAL flush, one staged batch with a single fdatasync, then the data-file writes. New DWBStagePageWriteNoWait and DWBWaitStagedWrites compose the existing write-path primitives for vectored callers. The FREE-batch policy becomes sliced: the bottom DWB_BG_RESERVE batches are for the background class only, the middle DWB_EVICT_RESERVE slice for eviction only, everything above is shared. A leave-some-behind rule alone cannot end the starvation: under saturation the free count hovers at the greedier class's throttle line and the background stream never reaches a batch. test_dwb: 001 covers the sliced boundaries including the background lane at free=1; 003 fills the background lane before the checkpointer stall scenario; 010/011 accept the larger candidate sets of bin-written rings; new 015_vectored_flush.pl pins the multi-slot batch ratio via pg_stat_io and repairs a torn page written by the vectored path.
…lush The bin gather trusted the caller's unlocked BM_PERMANENT pre-check, with only an Assert behind it: a buffer recycled for an unlogged page between the pre-check and the gather would have been staged into the double write buffer and its fake LSN fed to XLogFlush. Re-check under the buffer header lock and route non-permanent buffers to the per-page fallback, whose FlushBuffer skips both the WAL flush and the DWB for them. The deterministic unlogged paths were already pinned by t/002 (a shutdown checkpoint's unlogged buffers never enter the ring); the recycle window itself is instruction-scale and remains guard-only.
…e 5) Under thousands of concurrent writers the single cv_free_batch condition variable collapsed: every batch SEAL pushed all same-class writers into DWBOpenNewBatch at once, each of them had to win one of the four staging buffers just to discover that somebody else had already opened the next batch, and every staging release or batch retirement broadcast to every waiter. A profile at 5000 connections showed ~74% of a 104-core machine spinning in s_lock on the condition variable's spinlock, 817K context switches per second, and the retire sink idle. Restructure the wait path: - cv_free_batch becomes cv_want_batch[writer class]: one queue per class, used by both the staging-pool and the ring-space waits (both conditions are re-checked in the common DWBOpenNewBatch loop). Splitting by class makes a wake-up impossible to lose across the class boundary, where the sliced reserves may forbid the woken class to open a batch. - DWBOpenNewBatch starts with a lock-free staleness check (the same SEAL-bit/class-bit test, authoritatively repeated under DWBRingOpenLock), so the herd that piles in after a SEAL returns to slot reservation without touching the staging pool or waking anyone. The staging reservation itself is now non-blocking; an empty pool is waited out in the same outer loop. - All wake-ups are targeted signals instead of broadcasts: a successful opener wakes up to batch_pages - 1 same-class waiters (the number of slots joiners can still take), and a staging release or a batch retirement wakes one would-be opener per class. The 1s sleep timeout stays as the lost-wakeup backstop, and the stall escalation clock is unchanged. The open-batch sharding idea (K open batches per class) stays shelved: the same profile shows the intra-batch rendezvous is not hot (DWBRingOpen waits ~0, coverage/fsync waits in the single digits out of 5000 waiters). Suite: test_dwb 15 files / 269 tests, core regress 231, concurrency tests looped 3x.
With every DWBStagingRelease signalling cv_want_batch, a full ring turned the paced 1s waits into a busy rotation: a woken prober re-signalled the queue when it returned its unused staging buffer, handing the wake token to the next waiter (or straight back to itself — a timeout wake-up leaves the process queued on the condition variable, so its own signal can pop it), and the token circulated through DWBRingOpenLock acquisitions and ring scans indefinitely while nothing had changed. Make DWBStagingRelease silent and wake explicitly at the real capacity transitions only: a leader finishing its image pwrite (the buffer can serve the next batch), a batch returning to FREE, and a fresh batch opening. A probe-acquired buffer bouncing back unused wakes nobody; the 1s sleep timeout remains the backstop for the rare sleeper that raced against the last buffer. Add a shmem counter of wait-loop iterations that went to sleep (ring_wait_retries) with a test_dwb reader, and pin the pacing in the 003 real-clock scenario: a victim parked on an unchanged full ring for ~1s must accrue a handful of retries, not the thousands a rotating wake token produces. Suite: test_dwb 15 files / 271 tests, 003 looped 5x.
Two sink-side costs measured on the 104-core stand after the wakeup-storm fix (2700 connections, 300s checkpoints): - ~51% of all CPU went to LWLock traffic under DWBRetireSweep: every writer parked on a full ring ran the self-help sweep before sleeping, and hundreds of concurrent sweepers hammered the per-batch publish locks and the segment hash (907 backends queued on DWBPublish) while losing almost every fsync claim to whoever got there first. Gate the self-help with a new DWBSelfSweep LWLock taken conditionally: one sweeper sweeps, trylock losers go straight to sleep and are woken by the winner's frees through cv_want_batch. An LWLock rather than an atomic flag keeps the gate error-safe (released by the unwind). - The bgwriter emitted 178.7K single-slot batches in 900s — one full batch fdatasync per scattered LRU page (199/s) and ~12% of the ring's batch turnover for ~0.4% of the pages. Route the LRU scan through the same bin machinery as the checkpoint flush: BgSyncPeekBuffer (the check half of SyncOneBuffer, same skip-recently-used semantics) classifies buffers, would-write candidates collect into a bin and flush as one batch via FlushBufferBin (renamed from FlushCkptBufferBin, with the bin cap now DWB_FLUSH_BIN_MAX), preserving the bgwriter_lru_maxpages budget and the reusable-buffers estimator. A buffer that changes between the peek and the flush is re-checked under the header lock inside the bin, exactly as for the checkpointer. New test t/016_bgwriter_bin.pl pins the bgwriter batching through pg_stat_io (average batch >= 4 slots under UPDATE passes through a small buffer pool). Suite: test_dwb 16 files / 272 tests, 016 looped 5x, core regress 231/231.
Review follow-ups to the sweep-gate / bgwriter-bin commit: - New 003 scenario proves the single-sweeper property end to end: a dwb-self-sweep injection point sits inside the trylock-guarded section, so a process can only park there after winning DWBSelfSweepLock. With the ring exhausted and two stalled writers, the first victim parks at the point while the second one goes to sleep in the ring wait — the sleep proves it took the trylock-failed path, and pg_stat_activity shows exactly one process at the point. The point is detached before the wakeup: the winner re-enters the gate on its retry loop, and with the point still attached it would park again with no wakeup left to release it. The scenario then lets both victims finish: the ring is not required to go idle while they still write through it. - FlushBufferBin's header no longer claims every caller pre-filters BM_PERMANENT (the bgwriter's peek deliberately does not) and states explicitly that pin and usage counts are not re-checked past the callers' pick — the same benign race the per-page paths have. BgSyncPeekBuffer's comment lists exactly what the bin re-checks. - 016 resets shared io statistics after the data load so only the bgwriter's own writes enter the average, and polls between UPDATE passes instead of sampling once per pass. Suite: test_dwb 16 files / 275 tests, 003+016 looped 5x, pgindent and perltidy clean.
A shared random workload touches nearly every segment of a large table between retire rounds, so the per-segment protocol pays hundreds of fdatasync calls per sweep and a batch waits for the sweep to visit all of its segments before it can be freed. The new dwb_retire_sync_method (default syncfs where available) makes a retire round issue one syncfs() per file system holding data files and then free every batch that was already RETIRING when the round began: their data-file writes preceded the syncfs, so they are durable. The per-segment path stays as the fallback and keeps serving the checkpointer piggyback and the OOM retire in both modes. Also add per-class, per-reason seal counters (overflow, lone-writer, waiter timeout, worker timeout, bin flush, forced) with slot sums, exposed through test_dwb_seal_stats(): client batches average half-full on the perf stand and the counters attribute that turnover to its trigger. New TAP test 017 drives a workload through a worker running syncfs rounds with fsync on and crash-checks the result; 002 asserts the no-pool lone-seal accounting, 004 pins the fsync method to keep the per-segment sweep covered, and the regress test pins the deterministic overflow/forced counts.
Every retire worker wakes on the same cv_retire_wake broadcast, so with a pool larger than one the workers would all run the same global syncfs() round back to back, multiplying whole-file-system syncs for no benefit; the ring-full self-help could pile on top. Guard the round with LWLockConditionalAcquire on the new DWBSyncfsRoundLock: one process collects and syncs, losers return at once and are woken again by the winner's frees or the next publication. The batch collect runs under the lock, so a loser can never free batches some other round's syncfs did not cover. 017 now runs two retire workers and pins the single-admission property deterministically: a dwb-syncfs-round injection point inside the gate parks the winner, the loser must fall back asleep in its main loop, and the resumed round must free a batch parked with test_dwb_park(). While a process is parked inside the gate ALL wholesale retirement is frozen, so the scenario stays ring-quiet — no eviction workload runs until the point is detached (detach strictly before wakeup, as in 003). Also reunite the "change requires restart" comment with dwb_retire_workers in postgresql.conf.sample; the new SIGHUP parameter had been inserted between them.
Seal accounting on the perf stand showed 54% of eviction batches sealed by the lone-writer fast seal at 1.33 slots average, while overflow-sealed batches carry exactly 64: under a dense concurrent stream the FIRST writer of every freshly opened batch publishes within microseconds and reaches the fsync wait while ref_count is still 1, so the fast seal meant for sequential streams halves the ring into one-page batches and doubles the batch-file fsync count. Gate the fast seal on the class being quiet: every overflow seal stamps the class's last-overflow timestamp (twice — at the seal win so concurrent writers see the class as hot during the leader write, and after the write so the window survives a slow batch fdatasync), and a lone writer skips the fast seal while DWBClassIsHot sees the stamp as younger than dwb_batch_timeout_ms. A stamp from the future — a backward system-clock step — reads as quiet, keeping the immediate seal instead of taxing sequential streams until the clock catches up. A wrongly suppressed seal costs at most the window: the waiter's own timeout seal fires after it in every mode, and the retire workers' force-seal backs it up whenever a pool is configured. Sequential streams see no overflow seals and keep the immediate seal, as does the unconditional no-pool seal in DWBStagePageWrite. The regress test pins all three sides deterministically through the seal counters: a quiet class fast-seals one staged page (lone +1); a hot-window driver overflows a batch in the same backend and enters the wait on the next batch's first slot microseconds later — the fast seal must give way to the waiter's timeout seal (overflow +1, wait_timeout +1, lone +0); and a planted future stamp must read as quiet through the same DWBClassIsHot helper and keep the fast seal immediate. dwb_batch_timeout_ms is raised to 200ms in the regress configuration so the in-process stamp-to-check gap has a wide margin over the threshold on slow machines.
With the double write buffer, one bgwriter executing its flush bins serially cannot clean more than ~11K pages/s: each bin waits out a batch fdatasync before the data-file writes. Backends then evict dirty buffers themselves and pay the full batch rendezvous (~2 ms per page against ~50 us of a plain page-cache write) on every eviction. Keep the bgwriter as the only LRU scanner and pacing estimator (the StrategySyncStart allocation counter is consumed on read, so the estimator cannot be split), but hand the bins to a small shared-memory queue served by dwb_cleaner_workers background workers. Queue entries are hints, not obligations: the pool executes them through FlushBufferBin's new opportunistic mode, which reclassifies every member under the buffer header lock with the scan's LRU-candidate predicate and drops - never waits on - members that became hot, went clean, lost BM_PERMANENT or are busy with somebody's I/O. The mandatory mode of the existing callers (checkpointer, bgwriter self-flush) is unchanged. A full or busy queue fails the enqueue and the bgwriter flushes the bin itself, degrading exactly to the pool-less behavior; nothing needs draining on shutdown. The bgwriter_lru_maxpages budget now caps pages issued per round (queued plus self-written) while buf_written_clean keeps counting actual writes: the pool's completions are folded in once per round and the self-flushed bins are counted at the flush, so pg_stat_bgwriter stays "pages written by LRU cleaning" no matter which process wrote them. The counters reconcile exactly in error-free operation; a worker error mid-bin abandons the bin's remainder, which stays dirty for a later scan. The pool lands in the background writer class of the ring (a plain B_BG_WORKER would classify as eviction and fight the clients for the eviction reserve). Workers use the aux-process resource owner so an error mid-bin releases pins and buffer I-O through the existing repair path, exit through die() on SIGTERM - an exit code of 0 would unregister the worker for good and one stray terminate would permanently shrink the pool - and flush their I/O statistics before each queue sleep. t/018_cleaners.pl pins the queue protocol down with one-page claims driven straight into the queue: a cooled dirty page is written, a page pinned by another session and an already-clean duplicate are skipped (the pin helper survives rollbacks via a transaction callback), a real workload flows scan -> queue -> pool with the writes visible in the background DWB class, pg_stat_io and pg_stat_bgwriter, a parked pool lets claims overflow the queue and the bgwriter cleans solo, the drained queue reconciles enqueued = written + skipped, a terminated worker is restarted keeping the pool size, and an oversized pool is refused at startup while free worker slots are counted. Parked workers are woken one wakeup per waiter - a worker left sleeping at a detached injection point would touch the detached segment on exit.
…age 5) Per-interval diagnostics of the tps saw showed a bistable latch: at the pool's drain ceiling the bin queue runs at capacity, and the bgwriter's fallback of flushing refused bins itself stalled its LRU scan behind serial batch fsyncs. The strategy clock hand then caught the scan point, backends started evicting dirty buffers at full double write buffer latency, and the queued hints went stale en masse — a state that sustained itself until the tps sag let the scan break ahead again. Remove the fallback: with an active cleaner pool the bgwriter writes no data pages at all. A bin refused by a full queue is kept in the (now static) bin storage and the round ends early — scanning further ahead would only produce bins nobody can drain — then the carried bin is re-offered at the top of the next round, so a saturated pool is polled at bgwriter_delay cadence and the scan resumes the instant a worker frees queue space. A disabled scan (bgwriter_lru_maxpages = 0) drops the carried bin, and a budget shrunk below an accepted bin's size ends the round with maxwritten_clean counted. Pool-less mode is unchanged. The enqueue now takes the queue lock unconditionally (the critical section is one bin copy), so a refusal means exactly "queue full" and the renamed deferred_bins counter is a pure saturation gauge. The queue capacity becomes a fixed 64 bins instead of scaling with the worker count: it is a burst absorber, and even a full queue drains in tens of milliseconds at observed pool rates. The backpressure test scenario now pins the deferral contract instead: a full queue grows the deferral counter while the bgwriter's own DWB write count stays flat, a disabled scan stops the deferral stream and feeds the drained queue nothing, and re-enabling the scan resumes the feed.
With backend evictions absorbed by the cleaner pool, autovacuum workers were left as the last degenerate ring writers: flushing dirty pages out of their private ring strategy through the per-page path in a cold eviction class, where the lone-writer fast seal fires for every page — one batch and one ring fdatasync per page (142K one-slot batches per 900 s in the r14 verdict run, ~158 wasted syncs/s). Autovacuum is a scheduled sequential writer, not a latency-critical evictor, so classify its workers with the background stream. The class hotness gate then does the right thing in both regimes: under load the background class is hot with the pool's bin batches, the lone seal is suppressed and autovacuum pages ride those batches at their fill; on a quiet system the class is cold and the immediate lone seal keeps today's per-page latency. The launcher and manual VACUUM stay in the eviction class. The new test drives a real autovacuum pass through its ring strategy: the table is deleted from while autovacuum is held off per-table, a clean restart empties shared buffers, and the pass then reads every page through the ring and prunes it dirty, forcing ring-wrap flushes. The walwriter is paced to flush aggressively because the vacuum ring only reuses a dirty buffer whose WAL is already flushed. The background class must grow while the eviction class stays flat, and the autovacuum worker's pg_stat_io ring writes must appear.
The post-promotion window check captures its start LSN with pg_current_wal_lsn() while concurrent imageless FPI_FOR_HINT records are still being inserted; the write position advances in whole pages, so the captured LSN occasionally lands exactly on a WAL page boundary. pg_waldump then skips the page header to the first whole record and reports that with a benign informational line on stderr, which the test treated as a failure. Strip exactly that line before asserting that stderr is empty; anything else on stderr still fails the test. Caught by an instrumented 30-run loop of the full suite (2 hits, both with the start LSN page-aligned); 30/30 clean with the fix.
With io_torn_pages_protection = double_writes the WAL stream carries no full-page images and degenerates into a dense flow of small records, so high-connection workloads hit contention on the eight insertion slots long before the WAL device saturates. Benchmarks of the double-write path on a 104-thread machine at 2700 connections were measured with 32 insertion locks on both the patched and the reference builds; codify that value.
Recommended settings for the primary-replica benchmarkBest-known configuration from the single-node benchmark series (104-thread Double-write bufferPerformance settings the DWB path depends onThe bgwriter triple is critical: at the defaults (200ms / 100 pages) the Non-GUC prerequisites
Generic settings used by the benchmark gridNot DWB-specific, but the published numbers were measured with these on both Autovacuum was left enabled for every measured run, at the stock settings — The only exception is the reference load: the SF75000 base is built with Replication-specific notes
|
Out-of-tree assets: TPS and latency charts of the vanilla vs double-write pgbench series (connection-count axis and the 2700-conn time series), referenced by immutable raw URLs from the pull request comments.
Benchmark: double writes vs full-page images, connection and checkpoint axesSetup
TPS and latency vs connection count (checkpoint_timeout = 300s)
The price of torn-page protection (2700 connections)
On this stand the WAL device is far from saturation, so vanilla's Checkpoint-interval axis (2700 connections)
DWB throughput is flat across a 5x change of checkpoint interval TPS and latency over time (2700 connections, checkpoint_timeout = 300s)
Reproducibility notes
|
The two remaining in-run throughput dips sit exactly in the checkpoint write windows, where the checkpointer's ring traffic rides on top of the cleaner pool's peaks. Teach CheckpointWriteDelay a second, pressure-gated way to reach its nap: when the base schedule check says behind, but the pool's bin queue is at least half full and the lag is still within a bounded margin — Min(0.05, half the slack checkpoint_completion_target leaves), recomputed at every check since the target is SIGHUP-reloadable — the checkpointer naps anyway, so its writes land in the quieter phases of the window. Once the margin is used up, pacing is the stock behavior regardless of pressure, and immediate/shutdown checkpoints bypass the branch as before. IsCheckpointOnSchedule gains a slack parameter applied after the completion-target scaling, so the granted lag is exactly the margin. The checkpointer never touches the queue lock: the queue keeps an atomic depth mirror updated at the two nqueued mutation sites, and the hot check reads only that. A pressure_naps counter (exposed through test_dwb's cleaner counters) counts only margin-granted naps. The new TAP scenario parks the pool at an injection point, fills the queue past the hot threshold, and lets a timed — non-immediate — checkpoint pace over a dirty set sized well past the pacing slot count (at one page per nap a small set never falls behind schedule at an evaluation point and the branch never fires): pressure naps must register, the checkpoint must complete under sustained pressure, and a control checkpoint without pressure must add none. Also add the missing 019/020 entries to the module's meson TAP list.
The 2700-connection time series now shows the double-write build with the checkpointer-yield change: the checkpoint-window dips are gone from the DWB curve (worst 10 s sample 102.8K vs 98.6K before).
Assets for the primary/synchronous-standby comparison: throughput and shipped WAL per client count, and the standby replay backlog over the run.
ec99854 to
5885066
Compare
Primary and a synchronous standby over an emulated 10 GbE linkThe single-host grid measured what the protection modes cost one server. Headline: this is the first configuration where DWB is ahead on tps — SetupOne host, split by socket so replay never competes with the primary's
The 10 GbE hop is a netem qdisc on Everything else follows the established methodology: SF75000 reference Both standbys are configured identically, and generously, so the Results
Reading the numbersThroughput. With a synchronous standby the bottleneck moves into the Wire. DWB ships 2.5× less: 33 MB/s against 83 MB/s. On a 10 GbE link Replay is the price. The DWB standby absorbs ~23 MB/s of WAL against Two observations narrow down where the replay ceiling actually sits:
Work on closing that gap is in progress — a profile of the replay process Caveats, in the interest of fairness
Repro
|
A standby of a cluster running io_torn_pages_protection = double_writes has no full-page images to apply, so replay must fetch every page it modifies. Doing that from the startup process costs about half of its single core: kernel advice per block, the read itself, the checksum, and the search for a victim buffer. On the primary-to-sync-standby benchmark that showed up as replay falling behind without bound while the primary kept up. Add a pool of background workers that read those pages into shared buffers ahead of replay. Where the prefetcher would issue advice, it now publishes the block to a ring in shared memory; a worker reads it and records which buffer it landed in, and replay picks that up through the recent-buffer hint XLogReadBufferExtended() already validates. The redo path itself is unchanged, and nothing here is an obligation: an unanswered request, a failed read or an evicted buffer all mean replay reads the page itself, exactly as with the pool disabled. The pool is off by default and sized by replay_warm_workers, with replay_warm_queue_size for the ring. The workers hold no database connection, so they can serve crash recovery from the first record; having no pg_stat_activity row, they advertise their pids in shared memory. Reading pages ahead of replay means reading them next to relations replay is dropping and truncating, and DropRelationBuffers() requires that nobody else be loading pages of the relation while it works. A worker holds ReplayWarmReadLock shared across its checks and its read; smgrdounlinkall(), smgrtruncate(), dropdb() and dbase_redo() hold it exclusively across the buffer drop and the file operation that follows, so a worker either finishes before the drop scans the pool or starts after the relation is gone. The exclusive side also bumps an epoch, and a worker seeing a new one releases its own smgr state: without a database connection it receives no cache invalidations and would otherwise trust a size from before the truncation.
A worker asked smgr for the relation's size on every request, and smgr answers that question from its cache only for the startup process: smgrnblocks_cached() returns nothing to anybody else, "due to lack of a shared invalidation mechanism for changes in file size". So every request measured afresh, and measuring walks the segment chain from the beginning after smgrexists() has closed it — a thousand file opens on a terabyte relation. On the primary-to-sync-standby stand that was the whole pool: twelve workers each burning a core at 273k file opens a second, 4.6 ms of CPU per request and nothing left for reading pages. The ring stayed full, three quarters of the blocks the prefetcher offered were refused for want of a slot, and replay collected almost nothing — the standby fell behind exactly as it did with no pool at all. The pool does have the invalidation mechanism smgr lacks: the drop epoch, bumped under ReplayWarmReadLock whenever a relation or a database is about to lose its files. A size measured within an epoch can therefore only be too small, never too large, so a worker keeps its own small direct-mapped table of them: the first request for a fork pays for the existence check and the measurement, a block past a remembered end pays for measuring again — the relation may simply have been extended — and everything else is a comparison. The same run then reads 58k pages a second in the workers, hands replay 98.3% of what it publishes, and takes the standby's replay lag from a median of 63 seconds to 9 milliseconds, with the startup process reading 40 thousand pages over the run instead of 43.5 million.
Replay warm pool: the standby now keeps upThe primary → synchronous standby results posted earlier showed the one thing this design could not live with: with no full-page images in the stream, the standby's replay fell behind without bound. Replay had to fetch every page it modified, and it did so from the startup process, one page at a time, on one core. A pool of background workers now does that fetching ahead of replay ( SetupUnchanged from the earlier replication runs, so the numbers line up point for point: both instances on one host split by socket (primary on node 0, standby on 20 cores of node 1, pgbench on the remaining 6), 192 GB shared buffers each, Results at 750 clients
Replay lag drops from a minute to single-digit milliseconds, and throughput is unchanged — 78.9k against 81.0k without the pool and 79.5k for vanilla, which is inside the run-to-run spread on this stand. Worth noting that vanilla's standby does not keep up either: its own replay lag is 3.1 s at the median and 16 s at p90, with up to 2 GB of unapplied WAL, so on this workload the pool leaves the standby closer to the primary than full-page images do. The startup process stops reading (43.5 M pages → 60.6 k) and stops writing (25.8 M pages in vanilla → 3.0 k): both the fetching and the eviction that used to happen inside it now happen elsewhere. Over the run
Ten-second pgbench samples; the vanilla run is 900 s and the pool run 600 s, so the blue line simply ends earlier. Both sides sit at the same level and dip in the same direction — vanilla in narrow single-sample spikes, the pool in wider windows. The dips are on the primary and are under investigation; see the caveats. What the pool itself didOver the 600 s run, from
Two things the measurement foundA worker cannot use smgr's size cache. The workers also evict. They were expected to read; they turn out to write too. A read that misses needs a victim buffer, and when the free list is empty the victim is somebody else's dirty page, which under CaveatsThe pool run reused the copies left by the previous run rather than restoring the reference, so its page-cache state differs from the vanilla and no-pool runs; the throughput comparison above is indicative rather than exact, and a pristine-restore series is the next step. An earlier attempt at the same point produced 35.6k tps — that run spent its whole length catching up on a 7 GB backlog inherited from an interrupted run, and the catch-up reads (1.4 GB/s) saturated the shared array. A standby that starts caught up shows none of that. Throughput over the run is not flat: it plateaus at 81–85k and dips to 69–77k in three windows. The dips are on the primary side — the pool's read rate and the standby's write rate fall with tps rather than ahead of it, the standby's backlog stays at zero throughout, and the standby acknowledges every commit in well under a millisecond even inside a dip, so the synchronous wait is not what the clients are queuing on. Nor is it the eviction saw of the standalone rounds: the same build, the same 750 clients and the same cleaner-pool settings run flat on this stand without a standby (132.5k tps, 130–135k across 900 s). What the replication pair adds is a primary confined to 26 cores instead of 44 and a second instance writing to the same array — its restartpoint alone flushes ~7 M buffers per window. Which of the two the primary is queuing on is being measured now. |
TPS and average latency over the run at 750 connections, vanilla against DWB with the replay warm pool, both against a synchronous standby. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tqXDnyVLb628yQ14HaznA
Throughput per socket count and the two-socket point over its run, DWB with the replay warm pool against vanilla, primary and standby on separate hosts across a 100 GbE link. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tqXDnyVLb628yQ14HaznA
Two hosts, a real 100 GbE hop, and the primary widened one socket at a timeThe replication numbers posted earlier came from one machine split by socket, with the 10 GbE hop emulated by netem. That setup confounds two things: the primary was squeezed into 26 cores, and it shared its storage array with the standby. This round separates them. Setup. The primary runs alone on a 4-socket, 240-thread host (1.9 TB RAM, 63 TB ext4 array measured at 13.7 GiB/s random read and 5.8 GiB/s random write) and owns 1, 2 or 4 whole NUMA nodes; pgbench sits beside it on 16 threads node 3 does not give the server, over the unix socket. The synchronous standby has the previous host entirely to itself (104 threads, Throughput
Both builds peak at two sockets and lose throughput at four. The wait profile is the same at every point — ~300 of the 750 backends sit in
Over the run the two builds track each other closely and dip in the same windows — those dips are the synchronous-commit queue, not either build's page-write path (a run with the standby taken out of the commit path removes them entirely and lifts throughput ~15 %). WALAt the two-socket point, over the same 600 s:
2.4× less WAL, which is the whole point of the design, and it holds at every socket count (the standby applied 26–29 GB per run against vanilla's 51–61 GB). What the standby does with itThis is where the two designs diverge, and neither of them keeps up at the top of the range.
Vanilla replays in real time as long as the primary stays under roughly 105k tps — at four sockets, where its throughput is lowest, its lag is 20 ms. Above that it collapses: 73–88 s behind and 20+ GB of received-but-unapplied WAL. The reason is measurable rather than structural. At the two-socket point vanilla's standby had to chew through 151.7 GB of WAL against DWB's 80.0 GB (15.9 M WAL reads against 8.1 M), and its single startup process also wrote 1 644 667 pages of its own — it evicts while it replays. DWB's startup process wrote 18 410: the reading and the eviction are done by the warm pool and the cleaners, on the cores the host has spare. DWB's own ceiling is the pool, not the design: 12 workers were sized against a 79k-tps primary. They hold replay within 3–5 s up to ~115k tps and fall to 30 s at 137k. The standby host has 104 threads and the pool is using twelve of them, so there is room; sizing the pool against the primary's actual rate is the next measurement. CaveatsWithin a build the three socket points reuse the pair the first point left, since the socket axis changes nothing on disk; each build starts from a fresh restore of the reference and a fresh base backup. The vanilla single-socket point was rerun after its standby lost the WAL it still needed — a slot the runner recreated on a reused pair, since fixed. |
A page staged through the double write buffer asked the kernel to start writing it back the moment its data-file write returned, one sync_file_range per page from whichever process wrote it. On a primary evicting hard that is a syscall per eviction from every backend at once: half-second sampling of a 750-connection run caught 635 of 750 backends parked in IO:DataFileFlush together for a second and a half inside a checkpoint's write phase, with WAL advance down from 67 MB/s to 3 and the enclosing ten seconds 15% off the surrounding throughput. The call blocks whenever the device's request queue is congested, so under a checkpoint they all block together. Route those pages through the writeback accumulation the checkpointer and the background writer already use instead. FlushBuffer stops issuing anything: it queues a staged page into a context the caller names and returns whether it staged, so each caller schedules the page exactly once and picks where it lands. A backend's evictions of staged pages go to a new per-process context paced by dwb_writeback_after, replacing the boolean dwb_writeback; pages the buffer does not stage keep following backend_flush_after, and the checkpointer and background writer keep checkpoint_flush_after and bgwriter_flush_after. The bin-gather path loses its per-page call for the same reason: it already scheduled every page it wrote. This is pacing, not ordering. A batch can now be retired before the kernel has heard about its pages, which the fsync makes correct anyway. The exception is a server without retire workers, where a page write retires its own batch before returning: there the pending pages are issued on the spot, so any positive threshold behaves as one. 022_writeback_pacing.pl covers the parameter's bounds and reload, the threshold page by page through pg_buffercache_evict, the scope against unlogged relations, and — through an injection point handed the context in use, reporting to the server log — that nothing is left queued when a batch is made durable, on both the eviction and the bin-gather paths. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replay signalled the pool once per published block, which is a kill(2) each time, and it decided page residency itself before publishing, which is a buffer-table lookup each time. Together those were 28 % of the process that applies the WAL, as much as applying it. The publisher now signals only when nobody is searching the ring, and a worker that leaves the search hands the ring on in its place; scanners, pending and sleepers are what the two sides read to agree on that. A worker that dies hands it on from its exit path, whether it was searching or asleep. Residency moves into the worker, so every block reference is published rather than only the misses, and the worker counts the hit or the read for pg_stat_recovery_prefetch. Publishing every reference put five times more traffic through the size check each request makes, which turned a latent defect in that check into a fatal one. It kept its own sixteen-entry cache indexed by the low bits of the relation number, and relation numbers are handed out sequentially, so two hot relations sixty-four apart share a row for good. Its miss path asked smgrexists(), which closes the whole fork outside the startup process, and the smgrnblocks() behind it then reopened the entire segment chain: one worker made 572k file opens against a single page read in twelve seconds. The size now comes from smgr_cached_nblocks on the SMgrRelation the worker already opens for the read. That is keyed by the whole relation identity, has no fixed size and no eviction policy to get wrong, and is cleared by the smgrreleaseall() this pool already runs when replay tells it that files have gone. smgrexists() stays for the one cold path that has to tell a vanished relation from a short one.
The ring bounds how far ahead of replay the pool looks, and since every block reference now takes a slot rather than only the misses, the shipped 256 buys a quarter of the read-ahead it used to. Four points on a two-host pair, one after another so the pair ages equally between them, 600 s of 750-client pgbench each against a synchronous standby: ring applied pool CPU tps 256 38.8 MB/s 5.9 cores 136.0k 512 47.8 MB/s 9.7 cores 141.2k 1024 47.0 MB/s 14.3 cores 137.1k 2048 42.7 MB/s 13.3 cores 139.0k 512 is where it stops paying: 1024 buys nothing for half again as much pool CPU, and at 1024 the startup process's wait for pages falls from 15.3 % to 10.3 % without moving throughput, which says the wall past this point is the single replay process itself and not the window. Also adds the scenario for the race the exit-path hand-off exists to close: the worker that dies is the one the publisher has just woken. A signal goes to the head of the wait list and takes it off, so a head on its way out consumes the only wakeup. The head is made certain by replacing one of two workers — the list is joined at the tail — and the race is held open by stopping that process, which keeps it registered and asleep where an injection point could not: preparing to sleep on the point's own variable cancels the registration on the pool's. The evidence is which worker ends up holding the slot, not that the slot was served, because the replacement of the dead worker would serve it either way.
It holds a worker still with SIGSTOP to keep it registered on the wait list while the publisher signals, and Windows has no equivalent of that. 019_replslot_limit.pl steps around the same limitation.
Three assets for the pull request's benchmark comment: what each replay_warm_queue_size buys the standby and costs the pool, and the primary's throughput and latency over each of the four 600 s runs.
Sizing the replay warm ring, and the defect the sizing exposedThe warm pool used to publish only the blocks replay had missed; it now publishes every block a record refers to, because deciding residency in the startup process was costing that process a buffer-table lookup per block. That change moves the question of what Measuring that turned out to answer a different question first. The first series found a defect, not a defaultFour points, 256/512/1024/2048, on the two-host pair (primary alone on a 240-thread host, synchronous standby alone on a second, real 100 GbE hop, 750 clients, sf 75000, 600 s each):
At 1024 the pool served 29 K requests/s where 512 served 716 K, and A syscall summary of one worker for 12 s said what they were doing instead: 572 047 Cause. The miss path is what made that fatal. It asked Publishing every reference is what turned a latent defect into a fatal one: 431 M requests per run against 85 M before, five times more traffic across that path, and a deeper ring interleaves the two relations more finely. The fix: use the size the tree already caches
Invalidation needed no new code: Per worker per second, at ring 1024 — the size that used to collapse the standby:
and the point itself: 5.7 → 47.0 MB/s applied, pool CPU 31.3 → 14.3 cores at 8 % kernel instead of 93 %, ring overflow 23.6 M → 1 725, and 421 M of 427 M published requests collected instead of 1.2 M of 19 M. The series, re-runFour points again, back to back on one pair so ageing is equal between neighbours:
512 is where it stops paying. 1024 buys nothing for half again as much pool CPU, and 2048 gives some back. The profile says why: at 1024 the startup process's wait for pages falls from 15.3 % to 10.3 % without moving throughput, and it spends 73.8 % of its time running on its own single core. Past 512 the wall is single-process replay, not the read-ahead window.
Over the run the four points track each other closely once past the first two minutes; the dips they share are the synchronous-commit queue, as in the earlier series. The chosen point is the steadiest of the four in the second half — 141.2 K tps at 5.18 ms average latency, against 136.0 K at 5.55 ms for 256. The default is now 512, in the GUC, the sample configuration and the documentation. Caveat on comparing across series. A reused pair ages: at one ring size, an hour apart on the same pair, replay went 47.8 → 38.8 MB/s with no change to the build. Only neighbouring points are comparable, which is why the four re-run points ran consecutively and why the before/after above is quoted at one ring size rather than as a series difference. Tests
|
Two assets for the pull request's comment on why the standby falls behind: the backlog it builds over a run at ring 512, and where the single replay process spends its time — waiting against running, and what the running part is made of.
The standby still falls behind, and tuning will not fix itThe ring series above chose 512 because that is where the read-ahead window stops paying. It does not make the standby keep up. At the chosen point the primary writes 67.6 MB/s of WAL and the standby applies 47.8, so the backlog grows about 19 MB/s and ends a 600 s run 11.2 GB behind:
The run starts at zero on purpose — the harness now waits for the standby to drain before the load begins, so the slope is what the load produces and not a leftover. Where the time goesTwo profiles of the startup process on the standby, taken over the same point: 1 648 wait-event samples for its wall clock, and a 60 s DWARF profile (7 985 samples, 120 symbols covering 93.5 %) for the running part.
The wall clock says the process is not blocked on storage in any interesting way: it runs 73.8 % of the time, waits 18.3 % for pages and 5.7 % for its own double-write batch fsync. It occupies 0.75 of one core. That last number is the argument. At 0.75 of a core it applies 47.8 MB/s, so a version of this process that never waited at all would reach about 62 MB/s — still short of the primary's 67.6. Every single-process lever left is worth about 1.3× combined: the publication path's contended counters (11.95 % of the running part is The composition also shows what is not the problem. Keeping standby queries costs 2.0 % — the KnownAssignedXids machinery, CLOG writes and conflict resolution together — so there is nothing to win by giving them up. The WAL allows parallel apply
Dealing pages to workers by a hash of relation and block also spreads the work: the busiest of four workers takes 27.6 % against a fair share of 25 %, and the single busiest page — a visibility map page of the accounts table — carries 4.2 % of all references, bounding any page-based scheme at roughly 23×. Serial work left over is WAL decode (6.9 %), the snapshot machinery (2.0 %) and the dispatch loop (3.4 %) — about 12.4 %, which puts the ceiling near 8× and three workers at 2.4×. Against 1.74× that is margin. Parallel apply would also retire the warm pool: workers fetch their own pages, so the overlap the pool exists to create comes for free, and the 17.8 % the startup process spends feeding it leaves the serial path along with 32 processes. That is not an argument against the pool now — it is what a standby needs on the single-process path, which is the only path that exists today. The open question, stated plainlyPartitioning by page is not obviously correct while the standby serves queries, and the objection is not ours: it was put clearly by Matthias van de Meent in 2024. A transaction inserts a heap tuple on page A and an index entry on page B. Split across workers, the index entry can be applied first. An index-only scan then finds the index entry, consults the visibility map, sees the heap page still marked all-visible — because the record that would clear that bit has not been applied — and returns a row that does not exist yet. No error, no crash. Holding commit records until every worker has drained that transaction does not answer it: an index-only scan never consults transaction status at all. It is worse than the stated example. An index entry points at a specific line pointer on a specific heap page. If the heap change has not been applied, that line pointer may hold an older tuple that is visible, and a plain index scan does not, as a rule, re-check the scan key against the heap tuple. The failure mode is not an empty result but a wrong row. Three shapes of an answer, each with a cost that can be measured rather than argued:
The next measurement is the one that chooses between the first two: the distribution of block references by relation group over a full set of segments, with the relation names resolved. If the largest group is about half, the provably-correct partitioning is enough and the design is small. If it is two thirds, it is not, and the work becomes dependency tracking. Nothing in this comment changes the branch. It is the case for what comes after it. |
CREATE DATABASE fails outright when io_torn_pages_protection is
double_writes:
ERROR: before_shmem_exit callback (0x..., 0x...) is not the latest entry
PG_ENSURE_ERROR_CLEANUP registers a callback and cancels it again when the
command ends, and the cancellation only works while that callback is still
the last one registered. The backstop that hands back a process's staged
writes was registered at the first slot the process took, and for a fresh
connection that first slot is taken while the template database is copied
through the buffer cache -- landing on top of createdb's own callback.
Register it in BaseInit, among the other process-wide registrations, which
also takes a branch out of the slot path. BaseInit runs in every process
that can stage a write: backends, auxiliary processes and background workers
alike.
smgrdounlinkall and smgrtruncate call XLogWarmDropBegin and XLogWarmDropEnd without a declaration in scope, which the compiler accepts with a warning and a guessed prototype. Include the header that declares them.
test_dwb_checkpoint_pending reads and writes a block through smgr, which may reach the file with direct I/O and requires a buffer aligned for it. The buffer was declared with block alignment only, and an assert build dies on the first call -- the standby is killed by the assertion in buffers_to_iovec and the test hangs waiting for it, which takes the whole tree's check-world down with it. Give that one buffer PGIOAlignedBlock. The other page buffers here are read and written through this test's own descriptors, where block alignment is all that is asked for, and the comment says which is which. Nothing outside the test needs changing: every buffer the double write buffer itself hands to smgr is allocated with palloc_aligned, its staging area is aligned where it is carved out of shared memory, and its one raw write opens the file without direct I/O.
1 participant
What
A new torn-page protection mechanism for PostgreSQL 18, selected by the
io_torn_pages_protectionGUC:full_pages(default) — the traditional protection: automatic full-pageimages in WAL, controlled by
full_page_writes, exactly as today.double_writes— a short-lived double write buffer: every permanent datapage leaving shared buffers is first written and fsynced into a small
reusable ring of files in
pg_dwb/, and only then written to its actuallocation. Automatic page images disappear from WAL entirely.
off— no protection (full_page_writesis ignored), for storage withatomic 8kB writes.
The goal is to remove the FPI share of WAL volume and the post-checkpoint
latency spikes, moving the torn-page cost from the WAL device to the
data-file write path.
How it works
batch leader writes the whole batch as one contiguous I/O plus one
metadata write and a single
fdatasync. Two writer classes (eviction vsbackground) keep checkpoint storms from starving user backends, with
sliced reserves of free batches so neither class can starve the other.
The checkpointer and the background writer gather buffers into bins and
flush them as full vectored batches instead of page-at-a-time. A
hotness gate suppresses the lone-writer fast seal while a class is
under concurrent load, so batches fill up instead of sealing at one
page; autovacuum workers ride the background class the same way.
(
dwb_cleaner_workers), the background writer becomes a pure scanner —it walks the LRU and enqueues bins into a small shared-memory queue,
and the pool executes the writes through the background class. A full
queue defers the bin to the next round instead of stalling the scan,
which keeps the scan ahead of the strategy clock hand and takes
eviction writes out of client backends almost entirely.
covers has been made durable — by a pool of retire workers
(
dwb_retire_workers), by piggybacking on the checkpointer'sProcessSyncRequests, or synchronously as a fallback. On Linux theworkers retire whole rounds with a single
syncfs()(
dwb_retire_sync_method, with a per-segmentfdatasyncfallback)instead of hundreds of per-segment fsyncs.
dwb_writebackstartskernel writeback right after the double write so retirement fsyncs act
as cheap barriers.
ring, keep candidates of the current durable generation, dedup per page by
LSN, and rewrite every data page that is torn or older than its ring copy.
This also covers pages replay never reads (hint-bit-only pages logged as
XLOG_FPI_FOR_HINTwithout an image). A durable ring generation plus aRING_CLEANmarker decide exactly when the pass must run; on standbys thepass raises
minRecoveryPointwhen needed.pg_controland inXLOG_PARAMETER_CHANGE, so standbys track the primary's mode and refusereplay they cannot survive; base backups force page images for the backup
window and exclude
pg_dwb/; restored backups (detected viabackup_label/backupStartPoint) discard any shipped ring; pg_rewindrequires
full_pageson a live source; pg_upgrade transfers nothing ofthe ring.
background writers first, then
dwb_on_stall(defaultpanic) afterdwb_write_timeout_ms.Performance
Measured on a 104-core NVMe stand (RAID0, XFS), update-heavy pgbench over
a 1.5 TB cluster, 2700 connections, 900 s runs at
checkpoint_timeout = 300s, against a vanilla baseline of the same treewith data checksums enabled:
(vanilla: ~117K). The performance series in this branch — vectored
checkpoint flush with sliced reserves, targeted per-class ring wakeups,
syncfs-based retirement, the gated self-help sweep, lone-sealsuppression, the cleaner worker pool, the pure-scanner background
writer and autovacuum classing — brings the same point to 112.5K tps,
within 3.6% of vanilla (116.5K). DWB is absent from both the perf
profile and the wait-event top; the residual gap sits in the generic
ProcArray group-commit queue that both builds share.
while vanilla structurally dips to 82% of its mean in post-checkpoint
full-page-image waves.
full-page images) per run — 2.7× less WAL to write, ship and replay.
Testing and docs
src/test/modules/test_dwb: a C test extension driving the batch statemachine directly, plus 19 TAP files / 314 tests covering the write path,
backpressure, retirement, standbys, backups, pg_rewind, mode transitions,
crash recovery (including crafted-batch dedup scenarios), geometry
changes, restored-backup guards, pg_upgrade, the cleaner pool and the
writer-class assignments.
pg_basebackup/pg_rewindlimitations and thedwbobject ofpg_stat_io.