CSN ordinary path prototype

[vbp1]

Purpose

This PR publishes the current CSN prototype branch for review against an up-to-date vbp1/postgres:master.

The work explores replacing PostgreSQL's ordinary primary MVCC hot path with commit-sequence-number based visibility. The immediate operational goal is to reduce contention around ordinary transaction finish and prepare the next step where supported primary snapshot acquisition no longer scans ProcArray.

In user-visible terms, the intended direction is:

• fewer latency spikes when many sessions commit or abort concurrently;
• better scalability with high client counts;
• cheaper snapshot acquisition for read-heavy and mixed OLTP workloads in the next milestone;
• a clearer commit-order visibility model that can later become a foundation for replica consistency, logical decoding, and compatibility work.

What Is Implemented In This Branch

This branch contains the CSN prototype through Stage 3 H1 Milestone A.

Major changes:

• Adds pg_csnlog storage for xid-to-CSN state.
• Adds snapshot_csn to MVCC snapshots.
• Makes supported primary tuple visibility CSN-aware.
• Makes ordinary TransactionIdIsInProgress() prefer CSN status for the supported primary path.
• Publishes ordinary commit/abort completion through lock-free metadata instead of relying on ordinary xid removal under ProcArrayLock.
• Removes supported ordinary primary COMMIT / ABORT from the regular ProcArrayEndTransaction() / ProcArrayGroupClearXid() path.
• Adds injection-point proof coverage for ordinary commit/abort, completion-visible mirror state, backend slot reuse, snapshot/xmax handoff, and negative ProcArrayLock acquisition witnesses.
• Adds CSN-specific regression, isolation, recovery, and TAP coverage.
• Documents the Milestone B and C plan in the local .agents planning notes.

Milestone A Result

Milestone A is locally complete on this branch.

For supported ordinary primary transactions:

• ordinary COMMIT / ABORT no longer acquire ProcArrayLock on the supported path;
• ordinary transaction completion is published before compatibility cleanup;
• supported ordinary readers treat completion-visible ordinary ProcArray mirrors as non-authoritative;
• broad local validation completed successfully with documented CSN-specific validation debt.

This does not mean ProcArray is gone. After Milestone A it is still present as a compatibility and legacy surface. The branch only removes the ordinary transaction-end hot path dependency that was targeted by Milestone A.

Known Limitations

Current limitations are intentional for this prototype stage:

• Snapshot acquisition is not O(1) yet. GetSnapshotData() still scans ProcArray for supported CSN snapshots today.
• Supported CSN snapshots still carry legacy xip / subxip compatibility payload.
• SERIALIZABLE, hot standby, recovery snapshots, and logical decoding are not fully supported CSN targets yet.
• SQL snapshot export/import rejects CSN-sensitive snapshots.
• Parallel pg_dump is skipped under CSN snapshots because synchronized snapshot export is intentionally unsupported at this stage.
• pg_xact_status() retention for old committed prepared xids is unresolved: after truncation pressure, an old prepared xid may become unavailable even though data visibility remains correct.
• CSNLOG does not yet have an upper-bound cache for read-heavy visibility checks.
• CSNLOG durable retention and crash/restart semantics are still prototype-grade.

Validation

The following broad local gate was previously green on this branch:

SKIP_READLINE_TESTS=1 \
PG_TEST_EXTRA='kerberos ldap ssl load_balance libpq_encryption oauth wal_consistency_checking regress_dump_restore saslprep checksum_extended xid_wraparound' \
make -C /home/vbponomarev/postgres check-world

The pass included core regress, isolation, recovery, subscription, contrib, src/bin, checksum extended, xid wraparound, and the adapted pg_upgrade regress dump/restore path.

Validation was not rerun during the PR publishing step.

Validation Debt

The following test coverage is deliberately disabled or weakened and must be revisited before treating the prototype as production-ready:

• src/test/modules/test_plan_advice is disabled as wrapper-level validation debt.
• src/test/recovery/t/031_recovery_conflict.pl is skipped under CSN snapshots.
• src/test/recovery/t/035_standby_logical_decoding.pl is skipped under CSN snapshots.
• src/test/recovery/t/044_invalidate_inactive_slots.pl is skipped under CSN snapshots.
• src/bin/pg_dump/t/004_pg_dump_parallel.pl is skipped under CSN snapshots.
• src/test/recovery/t/053_csnlog_truncate.pl is weakened to allow pg_xact_status(prepared_xid) to become unavailable after truncation pressure.
• src/bin/psql/t/010_tab_completion.pl was skipped in the broad local gate through SKIP_READLINE_TESTS=1 because the local PTY/readline TAP is unstable in this environment.

What Comes Next

Planned Milestone B:

• make supported primary MVCC snapshot acquisition CSN-based and O(1) relative to active backend count;
• remove ProcArrayLock and ProcArray scanning from supported primary CSN snapshot acquisition;
• stop building authoritative xip / subxip payload for supported CSN snapshots;
• replace reader-side ProcArray scan for snapshot xmin / retention with a conservative published horizon contract;
• remove legacy xid-array fallback from supported primary CSN visibility;
• add a positive-only CSNLOG upper-bound cache for read-heavy visibility checks.

Planned Milestone C:

• remove the remaining ordinary primary authoritative dependence on ProcArray;
• leave ProcArray authoritative only for explicit legacy surfaces such as 2PC, recovery, standby, logical decoding, and unsupported snapshot shapes;
• narrow or remove the ordinary compatibility mirror/tail.

Current PR State

The fork master was fast-forwarded to upstream postgres/postgres:master before opening this PR.

GitHub currently reports this PR as conflicting. That is expected to be inspected as part of this publication step: the branch has a substantial prototype delta and upstream has moved forward significantly.