WIP: Explain RDI CDC offsets and source retention

content/integrate/redis-data-integration/_index.md

@@ -102,4 +102,7 @@ RDI can capture data from any of the following sources:
 
 ## Documentation
 
-Learn more about RDI from the other pages in this section:
+Learn why a saved capture position can become unusable in
+[Understand CDC offsets and source retention](/content/integrate/redis-data-integration/cdc-offsets-and-retention.md).
+
+Learn more about RDI from the other pages in this section:

content/integrate/redis-data-integration/architecture/_index.md

@@ -82,6 +82,10 @@ overall state).
 
 ## Checkpointing
 
+The processor checkpoint is separate from the collector's position in source change
+history. See [Understand CDC offsets and source retention](/content/integrate/redis-data-integration/cdc-offsets-and-retention.md)
+for how source retention affects recovery after an interruption.
+
 RDI uses Redis streams to store the sequence of change events
 captured from the sources. The events are then retrieved in order
 from the streams, processed, and written to the target. The stream

content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md

@@ -22,6 +22,10 @@ generally, these mechanisms are not switched on by default.
 RDI's Debezium collector uses these mechanisms for change data capture (CDC),
 so you must prepare each source database before you can use it with RDI.
 
+Plan how much change history your source needs to retain during interruptions.
+See [Understand CDC offsets and source retention](/content/integrate/redis-data-integration/cdc-offsets-and-retention.md)
+for a conceptual guide and examples for each source family.
+
 A pipeline can capture from more than one source database, and you must prepare each of them
 separately. Follow the page for each source's database type. See
 [Multiple sources in one pipeline](/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md)

content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md

@@ -33,7 +33,7 @@ The following table summarizes the considerations to prepare a MongoDB database
 | MongoDB Topology    | Replica Set, Sharded Cluster, or MongoDB Atlas                              |
 | User Roles          | readAnyDatabase, clusterMonitor                                             |
 | Oplog               | Sufficient size for snapshot and streaming                                  |
-| Pre/Post Images     | Enable on collections **only if using a custom key**                        |
+| Pre/Post Images     | Enable when the pipeline needs old document values, including custom keys and set output types |
 | Connection String   | Must include all hosts, replicaSet (if applicable), authSource, credentials |
 | MongoDB Atlas       | **[SSL required](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-property-mongodb-ssl-enabled)**, provide root CA as the source's `CACERT` secret in RDI       |
 | MongoDB mTLS        | X.509 authentication requires source TLS secrets and MongoDB SSL properties |
@@ -48,7 +48,7 @@ complete each step.
 - [ ] [Configure oplog size](#1-configure-oplog-size)
 - [ ] [Create a MongoDB user for RDI](#2-create-a-mongodb-user-for-rdi)
 - [ ] [Connection string format](#3-connection-string-format)
-- [ ] [Enable change streams and pre/post images (only if using a custom key)](#4-enable-change-streams-and-prepost-images-only-if-using-a-custom-key)
+- [ ] [Enable pre-images when required](#4-enable-pre-images-when-required)
 - [ ] [MongoDB Atlas specific requirements](#5-mongodb-atlas-specific-requirements)
 - [ ] [Self-hosted MongoDB mTLS and X.509 authentication](#6-self-hosted-mongodb-mtls-and-x509-authentication)
 - [ ] [Network and security](#7-network-and-security)
@@ -107,13 +107,15 @@ mongodb://${MONGODB_DB_USERNAME}:${MONGODB_DB_PASSWORD}@host:30000
 - For Atlas, adjust the connection string accordingly (see example below).
 - Set `replicaSet` and `authSource` as appropriate for your deployment.
 
-## 4. Enable change streams and pre/post images (only if using a custom key)
+<a id="4-enable-change-streams-and-prepost-images-only-if-using-a-custom-key"></a>
 
-Change Streams are required only if you are using a custom key in your RDI pipeline. Change streams are available by default on replica sets, sharded clusters, and MongoDB Atlas.
+## 4. Enable pre-images when required
 
-If your RDI pipeline uses a custom key, you must enable pre- and post-images on the relevant collections to capture the document state before and after updates or deletes. This allows RDI to access both the previous and updated versions of documents during change events, ensuring accurate synchronization.
+RDI uses MongoDB change streams for ongoing replication, with both default and custom Redis keys. Change streams are available on replica sets, sharded clusters, and MongoDB Atlas.
 
-Use the command below to enable change streams and pre/post images:
+A **pre-image** contains a document's values before an update or deletion. For example, if a custom Redis key uses an email address, changing that address requires RDI to remove the old Redis key and write the new one. The pre-image supplies the old address. A **post-image** contains the document after an insert or update; a deleted document has no post-image.
+
+Jobs that use custom output keys or set output types require pre-images. Enable document pre- and post-images on the relevant collections with this command. It enables the extra document images, not change streams themselves:
 
 ```javascript
 db.runCommand({
@@ -122,6 +124,10 @@ db.runCommand({
 });
 ```
 
+Also set `capture.mode` in the source's advanced settings to a mode ending in `_with_pre_image`, such as `change_streams_update_full_with_pre_image`. Enabling images on the collection alone does not make the collector request them. See the [source configuration reference](/content/integrate/redis-data-integration/reference/config-yaml-reference.md#sourcesadvancedsource-advanced-source-settings).
+
+Keep required pre-images available for the lag and interruptions you need to tolerate. A change event can remain available after its pre-image is gone. See [MongoDB history and old values](/content/integrate/redis-data-integration/cdc-offsets-and-retention.md#mongodb-account-for-write-volume-and-old-values) and [MongoDB pre- and post-images](https://www.mongodb.com/docs/manual/changestreams/#change-streams-with-document-pre--and-post-images).
+
 ## 5. MongoDB Atlas specific requirements
 
 MongoDB Atlas only supports secure connections via SSL.

content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md

@@ -147,18 +147,11 @@ wal_level = logical
 
 You can also set other PostgreSQL streaming replication parameters if you need them.
 For example, you can use `max_wal_senders` and `max_replication_slots` to increase
-the number of connectors that can access the sending server concurrently,
-and `wal_keep_size` to limit the maximum WAL size that a replication slot retains.
+the number of connectors that can access the sending server concurrently.
 The
 [configuration parameters](https://www.postgresql.org/docs/current/runtime-config-replication.html#RUNTIME-CONFIG-REPLICATION-SENDER)
 documentation describes all the parameters you can use.
 
-PostgreSQL’s logical decoding uses replication slots. These are guaranteed to retain all the WAL
-segments that Debezium needs even when Debezium suffers an outage. You should monitor replication
-slots carefully to avoid excessive disk consumption and other conditions such as catalog bloat that can arise
-if a replication slot is used infrequently. See the PostgreSQL documentation about
-[replication slots](https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION-SLOTS)
-for more information.
 If you are using a `synchronous_commit` setting other than `on`, then you should set `wal_writer_delay`
 to a value of about 10 milliseconds to ensure a low latency for change events. If you don't set this then
 the default value of about 200 milliseconds will apply.
@@ -168,6 +161,63 @@ the default value of about 200 milliseconds will apply.
 > recommend you consult the [PostgreSQL write-ahead log](https://www.postgresql.org/docs/current/wal-configuration.html)
 > documentation to get a better understanding.
 
+### Manage WAL retention
+
+Logical replication slots retain write-ahead log (WAL) needed by their readers,
+including during an RDI outage. Retained WAL can grow until it fills the disk.
+Configured limits or slot invalidation can also prevent RDI from resuming.
+See [Understand CDC offsets and source retention](/content/integrate/redis-data-integration/cdc-offsets-and-retention.md#postgresql-preserve-wal-and-slot-state)
+for the conceptual explanation.
+
+Choose controls for your PostgreSQL version:
+
+| PostgreSQL version | Retention controls |
+| :-- | :-- |
+| 10–12 | `wal_keep_segments` specifies a **minimum** number of retained segments, not a maximum for a replication slot. These versions do not have `max_slot_wal_keep_size`. |
+| 13–17 | `wal_keep_size` replaces `wal_keep_segments` and still specifies a **minimum**. `max_slot_wal_keep_size` limits how much WAL a slot can retain at checkpoint time. Its default, `-1`, allows unlimited retention. |
+| 18 | The same size controls apply. `idle_replication_slot_timeout` can also invalidate an inactive slot at a checkpoint. Its default, `0`, disables this timeout. |
+
+See the [PostgreSQL 13 release notes](https://www.postgresql.org/docs/release/13.0/)
+and [PostgreSQL 18 replication settings](https://www.postgresql.org/docs/18/runtime-config-replication.html)
+for these version differences. Managed services can apply additional limits.
+
+Check the settings available on your server:
+
+```sql
+SELECT name, setting, unit
+FROM pg_settings
+WHERE name IN (
+    'wal_keep_segments', 'wal_keep_size',
+    'max_slot_wal_keep_size', 'idle_replication_slot_timeout'
+);
+```
+
+Monitor slot state, retained WAL, and free disk together. The following query works
+across these versions. Status fields that a version does not provide return `NULL`.
+
+```sql
+SELECT slot_name, active, restart_lsn, confirmed_flush_lsn,
+       pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS retained_bytes,
+       to_jsonb(s)->>'wal_status' AS wal_status,
+       to_jsonb(s)->>'invalidation_reason' AS invalidation_reason
+FROM pg_replication_slots AS s
+WHERE slot_type = 'logical';
+```
+
+`retained_bytes` measures the log distance from the slot's oldest required position;
+it is not the total disk usage of the WAL directory. Size retention for peak writes
+and the interruption you need to tolerate. A small `wal_keep_size` does not cap
+slot storage. A finite slot limit can protect disk space at the cost of losing
+required history. Increasing the limit after that history is removed does not
+restore it. Do not drop or recreate an RDI slot as a routine disk-cleanup step.
+
+The slot's `confirmed_flush_lsn` is the position the collector has acknowledged.
+PostgreSQL starts logical streaming at the requested position or
+`confirmed_flush_lsn`, whichever is later. Moving only RDI's saved offset backward
+does not rewind the slot, even if older WAL remains on disk. Keeping unread history
+for normal resume is different from replaying already acknowledged changes.
+See the [PostgreSQL replication protocol](https://www.postgresql.org/docs/17/protocol-replication.html).
+
 ## Set up permissions
 
 The Debezium connector needs a database user that has the REPLICATION and LOGIN roles so that it