From 143558e7ca24725c437cd887b0c2228fda2cff74 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Sun, 18 Oct 2026 02:02:34 +0300 Subject: [PATCH 1/8] Explain RDI CDC offsets and source retention --- .../redis-data-integration/_index.md | 5 +- .../architecture/_index.md | 4 + .../cdc-offsets-and-retention.md | 153 ++++++++++++++++++ .../data-pipelines/prepare-dbs/_index.md | 4 + .../data-pipelines/prepare-dbs/postgresql.md | 59 ++++++- static/images/rdi/cdc-retention/bookmark.svg | 43 +++++ static/images/rdi/cdc-retention/mongodb.svg | 43 +++++ .../rdi/cdc-retention/mysql-mariadb.svg | 43 +++++ static/images/rdi/cdc-retention/oracle.svg | 43 +++++ .../images/rdi/cdc-retention/postgresql.svg | 66 ++++++++ static/images/rdi/cdc-retention/snowflake.svg | 43 +++++ static/images/rdi/cdc-retention/spanner.svg | 44 +++++ .../images/rdi/cdc-retention/sql-server.svg | 44 +++++ 13 files changed, 585 insertions(+), 9 deletions(-) create mode 100644 content/integrate/redis-data-integration/cdc-offsets-and-retention.md create mode 100644 static/images/rdi/cdc-retention/bookmark.svg create mode 100644 static/images/rdi/cdc-retention/mongodb.svg create mode 100644 static/images/rdi/cdc-retention/mysql-mariadb.svg create mode 100644 static/images/rdi/cdc-retention/oracle.svg create mode 100644 static/images/rdi/cdc-retention/postgresql.svg create mode 100644 static/images/rdi/cdc-retention/snowflake.svg create mode 100644 static/images/rdi/cdc-retention/spanner.svg create mode 100644 static/images/rdi/cdc-retention/sql-server.svg diff --git a/content/integrate/redis-data-integration/_index.md b/content/integrate/redis-data-integration/_index.md index 6aeb833ebb..7780e71074 100644 --- a/content/integrate/redis-data-integration/_index.md +++ b/content/integrate/redis-data-integration/_index.md @@ -102,4 +102,7 @@ RDI can capture data from any of the following sources: ## Documentation -Learn more about RDI from the other pages in this section: \ No newline at end of file +Learn why a saved capture position can become unusable in +[Understand CDC offsets and source retention](/content/integrate/redis-data-integration/cdc-offsets-and-retention.md). + +Learn more about RDI from the other pages in this section: diff --git a/content/integrate/redis-data-integration/architecture/_index.md b/content/integrate/redis-data-integration/architecture/_index.md index c55e9db9f5..564ccb1a83 100644 --- a/content/integrate/redis-data-integration/architecture/_index.md +++ b/content/integrate/redis-data-integration/architecture/_index.md @@ -82,6 +82,10 @@ overall state). ## Checkpointing +The processor checkpoint is separate from the collector's position in source change +history. See [Understand CDC offsets and source retention](/content/integrate/redis-data-integration/cdc-offsets-and-retention.md) +for how source retention affects recovery after an interruption. + RDI uses Redis streams to store the sequence of change events captured from the sources. The events are then retrieved in order from the streams, processed, and written to the target. The stream diff --git a/content/integrate/redis-data-integration/cdc-offsets-and-retention.md b/content/integrate/redis-data-integration/cdc-offsets-and-retention.md new file mode 100644 index 0000000000..158a825f71 --- /dev/null +++ b/content/integrate/redis-data-integration/cdc-offsets-and-retention.md @@ -0,0 +1,153 @@ +--- +Title: Understand CDC offsets and source retention +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: Learn why RDI can lose access to source change history during an interruption and how to plan for recovery. +group: di +hideListLinks: false +linkTitle: CDC offsets and retention +summary: Understand saved capture positions, source history, and how retention affects recovery after an interruption. +type: integration +weight: 31 +--- + +Redis Data Integration (RDI) can resume after an interruption only if the source still provides the change history and capture state it needs. Saving a position does not preserve that history. + +Imagine that RDI stops on Friday and starts again on Sunday. Your application keeps writing to Oracle during those two days. RDI remembers where it stopped, but Oracle has removed some of the redo logs needed to continue. The collector reports that its saved system change number (SCN) is no longer available. + +The source tables still exist. RDI has not forgotten its position. What is missing is the history between that position and the present. + +Two days is only an example. There is no universal expiration time for an RDI offset. The outcome depends on the database, its settings, the amount of change, and how far capture had already fallen behind. + +## Think of an offset as a bookmark + +Change data capture (CDC) lets RDI follow changes instead of repeatedly reading the entire source dataset. Four concepts explain how it works: + +- A **snapshot** reads the current dataset to establish a starting point. +- **Change history** records subsequent inserts, updates, and deletes. Depending on the database, this history lives in logs, change tables, or a change stream. +- An **offset** records the collector's progress through that history. Think of it as a bookmark, not a copy of the pages. +- **Retention** determines how long, or how much of, that history remains available. + +The source continues maintaining its history while RDI is disconnected. If the required history remains available, RDI can read the missed changes. If the source removes it, the bookmark alone cannot reconstruct those changes. + +![Two timelines show RDI resuming when its bookmark is inside retained history and failing when required history has been removed.](/images/rdi/cdc-retention/bookmark.svg) +{width="560"} + +*These timelines are illustrations, not database defaults. The important boundary is the oldest history still available when capture resumes.* + +The collector's offset is also different from the processor's **checkpoint**. The offset tracks capture from the source. The checkpoint tracks processing of events already captured into RDI. A processor can be behind even while capture is current. If the backlog eventually stops capture through [backpressure](/content/integrate/redis-data-integration/architecture/_index.md#backpressure-mechanism), source retention becomes important again. + +## See how each source keeps history + +The bookmark idea applies across sources, but the way each database retains history differs. The examples assume capture was working before the interruption. A source restart, replacement, or failover can introduce separate capture-state requirements. + +### Oracle: keep the required redo history + +Oracle records changes in redo logs. With LogMiner, RDI reads online and archived redo logs and tracks progress using SCNs. An SCN identifies a position in database history; it is not a timestamp with a fixed lifetime. + +Online redo logs are reused. Archived logs let a reader reach further back, but only while the required files remain available to the mining process. If cleanup removes a required archive during an outage, the collector can no longer continue from its saved position. Having newer logs does not fill the gap. + +![Oracle timelines show a saved SCN with all required redo available, then a saved SCN whose required archived redo has been removed.](/images/rdi/cdc-retention/oracle.svg) +{width="560"} + +Plan archive retention and storage with your database administrator. Check the archive destination RDI actually reads, not only whether backups exist elsewhere. Long-running transactions can also require older redo than the most recent visible activity suggests. + +XStream uses an Oracle capture process and an outbound server, with its own restart position and capture checkpoints. Preserve the history and state required by that capture process; do not apply a LogMiner error diagnosis unchanged to XStream. See [Prepare Oracle](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md), [Oracle LogMiner](https://docs.oracle.com/en/database/oracle/oracle-database/19/sutil/oracle-logminer-utility.html), and [XStream restartability](https://docs.oracle.com/en/database/oracle/oracle-database/19/xstrm/xstream-out-concepts.html). + +### MySQL and MariaDB: keep the binary logs + +MySQL and MariaDB record changes in binary logs, often called *binlogs*. A resume position identifies the required place in this history. Using transaction identifiers does not preserve the underlying events after their logs are removed. + +Automatic expiration or a manual purge can remove a file that a disconnected collector still needs. Increasing retention after that purge does not bring the file back. + +![Binary-log timelines show successful resume while the required file remains, then failed resume after that file is purged.](/images/rdi/cdc-retention/mysql-mariadb.svg) +{width="560"} + +Set retention for the interruption you need to tolerate and leave enough disk space for peak writes. Coordinate manual cleanup with CDC readers. Managed services can expose different retention controls, so use the instructions for your deployment in [Prepare MySQL and MariaDB](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md). The [MySQL purge reference](https://dev.mysql.com/doc/refman/8.0/en/purge-binary-logs.html) and [MariaDB binary-log guide](https://mariadb.com/docs/server/server-management/server-monitoring-logs/binary-log/using-and-maintaining-the-binary-log) describe the underlying cleanup behavior. + +### PostgreSQL: retain history without filling the disk + +PostgreSQL records changes in its write-ahead log (WAL). RDI reads through a logical replication slot and tracks a log sequence number (LSN). + +A slot can retain WAL that its reader still needs, even while RDI is stopped. This makes PostgreSQL different from a simple rolling time window: the immediate problem may be growing disk usage rather than removed history. + +Retention is not unconditional. A configured slot retention limit can allow required WAL to be removed. Newer PostgreSQL versions also support invalidating slots after an idle timeout. A dropped or invalidated slot cannot be treated as a healthy saved bookmark. + +![PostgreSQL timelines show WAL retained during an outage with growing storage, required WAL removed after a size limit, and an inactive slot invalidated by a timeout.](/images/rdi/cdc-retention/postgresql.svg) +{width="560"} + +Monitor slot health, retained WAL, and free disk together. Choose limits that balance recovery time with database storage capacity. Parameter names and available controls differ across PostgreSQL versions; see [Manage WAL retention](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#manage-wal-retention). + +The same model applies to Supabase, Neon, AlloyDB for PostgreSQL, Amazon Aurora PostgreSQL, and other managed PostgreSQL services. Check provider-specific slot cleanup, storage, and failover policies rather than assuming that every service retains an idle slot indefinitely. + +### SQL Server: retain the captured change rows + +SQL Server CDC captures changes from the transaction log into change tables. RDI reads those captured changes using an LSN-based position. + +The CDC cleanup process removes old entries from the change tables. If RDI returns after its required entries have been cleaned up, keeping transaction-log backups alone does not make those entries available through the CDC reader. + +![SQL Server timelines show an LSN inside retained CDC change-table history, then outside the available range after cleanup.](/images/rdi/cdc-retention/sql-server.svg) +{width="560"} + +Plan CDC change-table retention for consumer interruptions. Monitor both capture progress into those tables and RDI's progress through them. Capture delays and reader delays are different problems. See [Prepare SQL Server](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md) and Microsoft's [CDC administration guide](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/administer-and-monitor-change-data-capture-sql-server). + +### MongoDB: keep enough operation-log history + +MongoDB change streams provide a **resume token** that identifies where reading can continue. Resuming requires enough history in the replica set's operation log, or *oplog*. + +For a size-limited oplog, a busy day can retain fewer hours of history than a quiet day. A collector that resumes successfully after one weekend might fail after another weekend with a large write burst. + +![MongoDB timelines show a resume token whose history remains in the oplog, then history overwritten during a write burst.](/images/rdi/cdc-retention/mongodb.svg) +{width="560"} + +Monitor the actual oplog time window. Size it for peak activity, and consider a minimum retention period where available. A minimum retention period can require the oplog to grow beyond its configured size, so account for storage too. Managed-service controls vary. See [Prepare MongoDB](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md) and the [MongoDB oplog guide](https://www.mongodb.com/docs/manual/core/replica-set-oplog/). + +### Spanner: distinguish two retention windows + +Spanner change streams retain change records for a configured period. RDI needs its streaming resume point to remain within the available change-stream history. Once those records expire, a later restart cannot read them from the stream. + +![Spanner timelines show a streaming resume point within change-stream retention and a point older than the retained records.](/images/rdi/cdc-retention/spanner.svg) +{width="560"} + +Spanner also has **database version retention**. That separate setting matters when RDI reads a consistent snapshot at a particular timestamp. Increasing version retention does not substitute for sufficient change-stream retention during streaming. + +Plan the change-stream window for interruptions and the version window for snapshot duration. See [Prepare Spanner](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md) and Google's [change-stream overview](https://docs.cloud.google.com/spanner/docs/change-streams). + +### Snowflake: consume changes before the stream becomes stale + +Snowflake streams track an offset against source-table history. The stream does not hold an independent, permanent copy of every change. If the history needed by its offset is no longer retained, the stream becomes **stale**. + +![Snowflake timelines show a usable stream offset with retained table history and a stale stream after required history is no longer retained.](/images/rdi/cdc-retention/snowflake.svg) +{width="560"} + +Monitor stream staleness and the reported `STALE_AFTER` time. Account for table retention and Snowflake's retention-extension rules; do not assume that a stream keeps history indefinitely. Reading a stream with a query alone does not necessarily advance its offset: Snowflake distinguishes querying from consuming changes in a committed transaction. + +Snowflake support in RDI is in preview. See [Prepare Snowflake](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md) and Snowflake's [stream retention and staleness guidance](https://docs.snowflake.com/en/user-guide/streams-intro#data-retention-period-and-staleness). + +## Plan for the whole interruption + +RDI does not have to be completely down for history to become unavailable. A network failure, lost source access, or sustained capture lag can leave its position behind. A slow target can build a backlog until backpressure stops capture. Source cleanup continues independently. + +Use a conservative planning budget: + +**Existing capture lag + longest expected interruption + catch-up allowance + safety margin.** + +This is a planning aid, not a universal database setting. For example, a 48-hour outage budget needs more than 48 hours of retained history if capture was already behind. Catch-up also needs spare processing capacity while the application continues writing. + +Before maintenance, check capture progress, available history, and storage. For size-based limits, estimate how much history peak writes produce, not just average writes. Where history is retained by a slot, check how much additional disk an outage could consume. + +Alert while there is still time to act. Combine [RDI monitoring](/content/integrate/redis-data-integration/observability.md) with source-database checks. A running process or a saved offset does not prove that capture is advancing or that its history remains available. Confirm managed-service limits and cleanup policies with the database owner. + +## Recover when the history is gone + +First identify whether required history is missing or temporarily inaccessible. A permissions or archive-access problem needs a different correction from deleted history. + +If the database can make the original history available again through a supported procedure, resuming may still be possible. Otherwise, rebuilding from a fresh snapshot establishes a new starting point. Increasing retention protects future history; it cannot recreate removed records. + +A snapshot reads what exists now. It does not recover every intermediate event. Consider a row that existed in Redis on Friday but was deleted from the source on Saturday: Sunday's snapshot contains no row to overwrite that old Redis key. Recovery must account for such stale target data as well as current source rows. + +Use the [pipeline reset guidance](/content/integrate/redis-data-integration/data-pipelines/deploy.md#reset-a-pipeline) with your source's snapshot configuration and target reconciliation plan. Do not manually move an offset forward to silence an error: that can skip changes. Verify recovered inserts, updates, and deletes before relying on the target again. diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md index 8a2a4e3b2e..816ffea2ca 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md @@ -22,6 +22,10 @@ generally, these mechanisms are not switched on by default. RDI's Debezium collector uses these mechanisms for change data capture (CDC), so you must prepare each source database before you can use it with RDI. +Plan how much change history your source needs to retain during interruptions. +See [Understand CDC offsets and source retention](/content/integrate/redis-data-integration/cdc-offsets-and-retention.md) +for a conceptual guide and examples for each source family. + A pipeline can capture from more than one source database, and you must prepare each of them separately. Follow the page for each source's database type. See [Multiple sources in one pipeline](/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md index 55ae12fc33..3833dbbdf7 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md @@ -147,18 +147,11 @@ wal_level = logical You can also set other PostgreSQL streaming replication parameters if you need them. For example, you can use `max_wal_senders` and `max_replication_slots` to increase -the number of connectors that can access the sending server concurrently, -and `wal_keep_size` to limit the maximum WAL size that a replication slot retains. +the number of connectors that can access the sending server concurrently. The [configuration parameters](https://www.postgresql.org/docs/current/runtime-config-replication.html#RUNTIME-CONFIG-REPLICATION-SENDER) documentation describes all the parameters you can use. -PostgreSQL’s logical decoding uses replication slots. These are guaranteed to retain all the WAL -segments that Debezium needs even when Debezium suffers an outage. You should monitor replication -slots carefully to avoid excessive disk consumption and other conditions such as catalog bloat that can arise -if a replication slot is used infrequently. See the PostgreSQL documentation about -[replication slots](https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION-SLOTS) -for more information. If you are using a `synchronous_commit` setting other than `on`, then you should set `wal_writer_delay` to a value of about 10 milliseconds to ensure a low latency for change events. If you don't set this then the default value of about 200 milliseconds will apply. @@ -168,6 +161,56 @@ the default value of about 200 milliseconds will apply. > recommend you consult the [PostgreSQL write-ahead log](https://www.postgresql.org/docs/current/wal-configuration.html) > documentation to get a better understanding. +### Manage WAL retention + +Logical replication slots retain write-ahead log (WAL) needed by their readers, +including during an RDI outage. Retained WAL can grow until it fills the disk. +Configured limits or slot invalidation can also prevent RDI from resuming. +See [Understand CDC offsets and source retention](/content/integrate/redis-data-integration/cdc-offsets-and-retention.md#postgresql-retain-history-without-filling-the-disk) +for the conceptual explanation. + +Choose controls for your PostgreSQL version: + +| PostgreSQL version | Retention controls | +| :-- | :-- | +| 10–12 | `wal_keep_segments` specifies a **minimum** number of retained segments, not a maximum for a replication slot. These versions do not have `max_slot_wal_keep_size`. | +| 13–17 | `wal_keep_size` replaces `wal_keep_segments` and still specifies a **minimum**. `max_slot_wal_keep_size` limits how much WAL a slot can retain at checkpoint time. Its default, `-1`, allows unlimited retention. | +| 18 | The same size controls apply. `idle_replication_slot_timeout` can also invalidate an inactive slot at a checkpoint. Its default, `0`, disables this timeout. | + +See the [PostgreSQL 13 release notes](https://www.postgresql.org/docs/release/13.0/) +and [PostgreSQL 18 replication settings](https://www.postgresql.org/docs/18/runtime-config-replication.html) +for these version differences. Managed services can apply additional limits. + +Check the settings available on your server: + +```sql +SELECT name, setting, unit +FROM pg_settings +WHERE name IN ( + 'wal_keep_segments', 'wal_keep_size', + 'max_slot_wal_keep_size', 'idle_replication_slot_timeout' +); +``` + +Monitor slot state, retained WAL, and free disk together. The following query works +across these versions. Status fields that a version does not provide return `NULL`. + +```sql +SELECT slot_name, active, restart_lsn, confirmed_flush_lsn, + pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS retained_bytes, + to_jsonb(s)->>'wal_status' AS wal_status, + to_jsonb(s)->>'invalidation_reason' AS invalidation_reason +FROM pg_replication_slots AS s +WHERE slot_type = 'logical'; +``` + +`retained_bytes` measures the log distance from the slot's oldest required position; +it is not the total disk usage of the WAL directory. Size retention for peak writes +and the interruption you need to tolerate. A small `wal_keep_size` does not cap +slot storage. A finite slot limit can protect disk space at the cost of losing +required history. Increasing the limit after that history is removed does not +restore it. Do not drop or recreate an RDI slot as a routine disk-cleanup step. + ## Set up permissions The Debezium connector needs a database user that has the REPLICATION and LOGIN roles so that it diff --git a/static/images/rdi/cdc-retention/bookmark.svg b/static/images/rdi/cdc-retention/bookmark.svg new file mode 100644 index 0000000000..d66da463cc --- /dev/null +++ b/static/images/rdi/cdc-retention/bookmark.svg @@ -0,0 +1,43 @@ + +A bookmark needs its history +Time moves from earlier to later. A vertical bookmark marks where capture stopped. In the first panel the required history is retained. In the second panel required history is removed. + + + +A bookmark needs its history +Illustration · no fixed expiration time + +1 History retained: resume succeeds +Capture stops; source writes continue + + +Saved offset + + +Restart + +Change history +Earlier + +Later +Required changes remain available. +RDI resumes from its saved position. + +2 History removed: cannot resume +Capture stops; source writes continue + + +Saved offset + + +Restart + +Removed +Retained +Earlier + +Later +Required changes have been removed. +The offset cannot recreate the gap. + + diff --git a/static/images/rdi/cdc-retention/mongodb.svg b/static/images/rdi/cdc-retention/mongodb.svg new file mode 100644 index 0000000000..c6638dd62d --- /dev/null +++ b/static/images/rdi/cdc-retention/mongodb.svg @@ -0,0 +1,43 @@ + +MongoDB: operation-log history +Time moves from earlier to later. A vertical bookmark marks where capture stopped. In the first panel the required history is retained. In the second panel required history is removed. + + + +MongoDB: operation-log history +Illustration · no fixed expiration time + +1 History retained: resume succeeds +Capture stops; source writes continue + + +Resume token + + +Restart + +Oplog history +Earlier + +Later +The token still has enough history. +The change stream can resume. + +2 History removed: cannot resume +Capture stops; source writes continue + + +Resume token + + +Restart + +Removed +Retained +Earlier + +Later +A write burst replaces old history. +The token alone cannot recover it. + + diff --git a/static/images/rdi/cdc-retention/mysql-mariadb.svg b/static/images/rdi/cdc-retention/mysql-mariadb.svg new file mode 100644 index 0000000000..0bfafc255a --- /dev/null +++ b/static/images/rdi/cdc-retention/mysql-mariadb.svg @@ -0,0 +1,43 @@ + +MySQL and MariaDB: binary logs +Time moves from earlier to later. A vertical bookmark marks where capture stopped. In the first panel the required history is retained. In the second panel required history is removed. + + + +MySQL and MariaDB: binary logs +Illustration · no fixed expiration time + +1 History retained: resume succeeds +Capture stops; source writes continue + + +Saved position + + +Restart + +Binary logs +Earlier + +Later +The required log file is retained. +Capture can read missed changes. + +2 History removed: cannot resume +Capture stops; source writes continue + + +Saved position + + +Restart + +Removed +Retained +Earlier + +Later +The required log file was purged. +Its events are no longer available. + + diff --git a/static/images/rdi/cdc-retention/oracle.svg b/static/images/rdi/cdc-retention/oracle.svg new file mode 100644 index 0000000000..eb975f07da --- /dev/null +++ b/static/images/rdi/cdc-retention/oracle.svg @@ -0,0 +1,43 @@ + +Oracle: required redo history +Time moves from earlier to later. A vertical bookmark marks where capture stopped. In the first panel the required history is retained. In the second panel required history is removed. + + + +Oracle: required redo history +Illustration · no fixed expiration time + +1 History retained: resume succeeds +Capture stops; source writes continue + + +Saved SCN + + +Restart + +Redo history +Earlier + +Later +Required redo is still available. +LogMiner can continue reading. + +2 History removed: cannot resume +Capture stops; source writes continue + + +Saved SCN + + +Restart + +Removed +Retained +Earlier + +Later +A required archive was removed. +Newer redo cannot fill the gap. + + diff --git a/static/images/rdi/cdc-retention/postgresql.svg b/static/images/rdi/cdc-retention/postgresql.svg new file mode 100644 index 0000000000..6be7311628 --- /dev/null +++ b/static/images/rdi/cdc-retention/postgresql.svg @@ -0,0 +1,66 @@ + +PostgreSQL: slots and WAL +Time moves from earlier to later. A vertical bookmark marks where capture stopped. In the first panel the required history is retained. In the second panel required history is removed. A third panel shows a slot invalidated by an idle timeout even if WAL remains on disk. + + + +PostgreSQL: slots and WAL +Illustration · no fixed expiration time + +1 History retained: resume succeeds +Capture stops; source writes continue + + +Saved position + + +Restart + +WAL retained by slot +Earlier + +Later +The slot keeps the required WAL. +Disk usage grows while RDI is stopped. +Disk used + + + + + + +2 History removed: cannot resume +Capture stops; source writes continue + + +Saved position + + +Restart + +Removed +Retained +Earlier + +Later +A slot limit allows WAL removal. +The invalidated slot cannot resume. + +3 Idle timeout: cannot resume +Capture stops; source writes continue + + +Saved position + + +Restart + +WAL may still exist +Earlier + +Later +PostgreSQL 18: enabled idle timeout +invalidates the slot at a checkpoint. +WAL availability alone is not enough. + + diff --git a/static/images/rdi/cdc-retention/snowflake.svg b/static/images/rdi/cdc-retention/snowflake.svg new file mode 100644 index 0000000000..4297f3c124 --- /dev/null +++ b/static/images/rdi/cdc-retention/snowflake.svg @@ -0,0 +1,43 @@ + +Snowflake: stream staleness +Time moves from earlier to later. A vertical bookmark marks where capture stopped. In the first panel the required history is retained. In the second panel required history is removed. + + + +Snowflake: stream staleness +Illustration · no fixed expiration time + +1 History retained: resume succeeds +Capture stops; source writes continue + + +Stream offset + + +Restart + +Source-table history +Earlier + +Later +Required table history remains. +The stream can provide changes. + +2 History removed: cannot resume +Capture stops; source writes continue + + +Stream offset + + +Restart + +Removed +Retained +Earlier + +Later +Required table history is gone. +The stream is stale. + + diff --git a/static/images/rdi/cdc-retention/spanner.svg b/static/images/rdi/cdc-retention/spanner.svg new file mode 100644 index 0000000000..de5e635cee --- /dev/null +++ b/static/images/rdi/cdc-retention/spanner.svg @@ -0,0 +1,44 @@ + +Spanner: change-stream history +Time moves from earlier to later. A vertical bookmark marks where capture stopped. In the first panel the required history is retained. In the second panel required history is removed. + + + +Spanner: change-stream history +Illustration · no fixed expiration time + +1 History retained: resume succeeds +Capture stops; source writes continue + + +Resume point + + +Restart + +Change records +Earlier + +Later +The resume point is still in range. +Streaming can continue. + +2 History removed: cannot resume +Capture stops; source writes continue + + +Resume point + + +Restart + +Removed +Retained +Earlier + +Later +Required change records expired. +Database version retention +does not extend this stream window. + + diff --git a/static/images/rdi/cdc-retention/sql-server.svg b/static/images/rdi/cdc-retention/sql-server.svg new file mode 100644 index 0000000000..f88562720c --- /dev/null +++ b/static/images/rdi/cdc-retention/sql-server.svg @@ -0,0 +1,44 @@ + +SQL Server: CDC change tables +Time moves from earlier to later. A vertical bookmark marks where capture stopped. In the first panel the required history is retained. In the second panel required history is removed. + + + +SQL Server: CDC change tables +Illustration · no fixed expiration time + +1 History retained: resume succeeds +Capture stops; source writes continue + + +Saved LSN + + +Restart + +Captured change rows +Earlier + +Later +Required change rows remain. +RDI continues reading CDC tables. + +2 History removed: cannot resume +Capture stops; source writes continue + + +Saved LSN + + +Restart + +Removed +Retained +Earlier + +Later +Cleanup removed required rows. +Transaction-log backups alone +do not restore this CDC history. + + From 9cfb187b1d414f9b3f5fb2ef2402f938cfa2da19 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Sun, 18 Oct 2026 02:12:19 +0300 Subject: [PATCH 2/8] Restyle CDC retention timelines with Redis docs design --- static/images/rdi/cdc-retention/bookmark.svg | 116 +++++++---- static/images/rdi/cdc-retention/mongodb.svg | 116 +++++++---- .../rdi/cdc-retention/mysql-mariadb.svg | 116 +++++++---- static/images/rdi/cdc-retention/oracle.svg | 116 +++++++---- .../images/rdi/cdc-retention/postgresql.svg | 183 ++++++++++++------ static/images/rdi/cdc-retention/snowflake.svg | 116 +++++++---- static/images/rdi/cdc-retention/spanner.svg | 118 +++++++---- .../images/rdi/cdc-retention/sql-server.svg | 118 +++++++---- 8 files changed, 654 insertions(+), 345 deletions(-) diff --git a/static/images/rdi/cdc-retention/bookmark.svg b/static/images/rdi/cdc-retention/bookmark.svg index d66da463cc..c437554191 100644 --- a/static/images/rdi/cdc-retention/bookmark.svg +++ b/static/images/rdi/cdc-retention/bookmark.svg @@ -1,43 +1,79 @@ - + A bookmark needs its history -Time moves from earlier to later. A vertical bookmark marks where capture stopped. In the first panel the required history is retained. In the second panel required history is removed. - - - -A bookmark needs its history -Illustration · no fixed expiration time - -1 History retained: resume succeeds -Capture stops; source writes continue - - -Saved offset - - -Restart - -Change history -Earlier - -Later -Required changes remain available. -RDI resumes from its saved position. - -2 History removed: cannot resume -Capture stops; source writes continue - - -Saved offset - - -Restart - -Removed -Retained -Earlier - -Later -Required changes have been removed. -The offset cannot recreate the gap. +Illustrative Friday-to-Sunday interruption. A red bookmark marks the saved capture position. Solid dark blocks show retained history; hatched blocks show removed history. The first timeline can resume. The second cannot resume because required history is missing. + + + + + + +RDI / SOURCE HISTORY +A bookmark needs history +Saved position ≠ saved history + + +01 +History retained +Resume succeeds +Saved offset +Restart + + + + + + + + + + + + + + + + + +Change history + +Friday +Sunday +Source keeps writing + + +Required changes remain available. +RDI resumes from its saved position. + + + + +02 +History removed +Cannot resume +Saved offset +Restart + + + + + + + + + + +Removed +Retained + +Friday +Sunday +Source keeps writing + + +Required changes have been removed. +The offset cannot recreate the gap. + + +Illustrative times · not retention defaults diff --git a/static/images/rdi/cdc-retention/mongodb.svg b/static/images/rdi/cdc-retention/mongodb.svg index c6638dd62d..9002b74f6c 100644 --- a/static/images/rdi/cdc-retention/mongodb.svg +++ b/static/images/rdi/cdc-retention/mongodb.svg @@ -1,43 +1,79 @@ - + MongoDB: operation-log history -Time moves from earlier to later. A vertical bookmark marks where capture stopped. In the first panel the required history is retained. In the second panel required history is removed. - - - -MongoDB: operation-log history -Illustration · no fixed expiration time - -1 History retained: resume succeeds -Capture stops; source writes continue - - -Resume token - - -Restart - -Oplog history -Earlier - -Later -The token still has enough history. -The change stream can resume. - -2 History removed: cannot resume -Capture stops; source writes continue - - -Resume token - - -Restart - -Removed -Retained -Earlier - -Later -A write burst replaces old history. -The token alone cannot recover it. +Illustrative Friday-to-Sunday interruption. A red bookmark marks the saved capture position. Solid dark blocks show retained history; hatched blocks show removed history. The first timeline can resume. The second cannot resume because required history is missing. + + + + + + +RDI / SOURCE HISTORY +MongoDB +Resume tokens and oplog history + + +01 +History retained +Resume succeeds +Resume token +Restart + + + + + + + + + + + + + + + + + +Oplog history + +Friday +Sunday +Source keeps writing + + +The token still has enough history. +The change stream can resume. + + + + +02 +History removed +Cannot resume +Resume token +Restart + + + + + + + + + + +Removed +Retained + +Friday +Sunday +Source keeps writing + + +A write burst replaces old history. +The token alone cannot recover it. + + +Illustrative times · not retention defaults diff --git a/static/images/rdi/cdc-retention/mysql-mariadb.svg b/static/images/rdi/cdc-retention/mysql-mariadb.svg index 0bfafc255a..c28d4c838c 100644 --- a/static/images/rdi/cdc-retention/mysql-mariadb.svg +++ b/static/images/rdi/cdc-retention/mysql-mariadb.svg @@ -1,43 +1,79 @@ - + MySQL and MariaDB: binary logs -Time moves from earlier to later. A vertical bookmark marks where capture stopped. In the first panel the required history is retained. In the second panel required history is removed. - - - -MySQL and MariaDB: binary logs -Illustration · no fixed expiration time - -1 History retained: resume succeeds -Capture stops; source writes continue - - -Saved position - - -Restart - -Binary logs -Earlier - -Later -The required log file is retained. -Capture can read missed changes. - -2 History removed: cannot resume -Capture stops; source writes continue - - -Saved position - - -Restart - -Removed -Retained -Earlier - -Later -The required log file was purged. -Its events are no longer available. +Illustrative Friday-to-Sunday interruption. A red bookmark marks the saved capture position. Solid dark blocks show retained history; hatched blocks show removed history. The first timeline can resume. The second cannot resume because required history is missing. + + + + + + +RDI / SOURCE HISTORY +MySQL and MariaDB +Positions in the binary logs + + +01 +History retained +Resume succeeds +Saved position +Restart + + + + + + + + + + + + + + + + + +Binary logs + +Friday +Sunday +Source keeps writing + + +The required log file is retained. +Capture can read missed changes. + + + + +02 +History removed +Cannot resume +Saved position +Restart + + + + + + + + + + +Removed +Retained + +Friday +Sunday +Source keeps writing + + +The required log file was purged. +Its events are no longer available. + + +Illustrative times · not retention defaults diff --git a/static/images/rdi/cdc-retention/oracle.svg b/static/images/rdi/cdc-retention/oracle.svg index eb975f07da..a891148e09 100644 --- a/static/images/rdi/cdc-retention/oracle.svg +++ b/static/images/rdi/cdc-retention/oracle.svg @@ -1,43 +1,79 @@ - + Oracle: required redo history -Time moves from earlier to later. A vertical bookmark marks where capture stopped. In the first panel the required history is retained. In the second panel required history is removed. - - - -Oracle: required redo history -Illustration · no fixed expiration time - -1 History retained: resume succeeds -Capture stops; source writes continue - - -Saved SCN - - -Restart - -Redo history -Earlier - -Later -Required redo is still available. -LogMiner can continue reading. - -2 History removed: cannot resume -Capture stops; source writes continue - - -Saved SCN - - -Restart - -Removed -Retained -Earlier - -Later -A required archive was removed. -Newer redo cannot fill the gap. +Illustrative Friday-to-Sunday interruption. A red bookmark marks the saved capture position. Solid dark blocks show retained history; hatched blocks show removed history. The first timeline can resume. The second cannot resume because required history is missing. + + + + + + +RDI / SOURCE HISTORY +Oracle +SCNs and required redo history + + +01 +History retained +Resume succeeds +Saved SCN +Restart + + + + + + + + + + + + + + + + + +Redo history + +Friday +Sunday +Source keeps writing + + +Required redo is still available. +LogMiner can continue reading. + + + + +02 +History removed +Cannot resume +Saved SCN +Restart + + + + + + + + + + +Removed +Retained + +Friday +Sunday +Source keeps writing + + +A required archive was removed. +Newer redo cannot fill the gap. + + +Illustrative times · not retention defaults diff --git a/static/images/rdi/cdc-retention/postgresql.svg b/static/images/rdi/cdc-retention/postgresql.svg index 6be7311628..06e5693518 100644 --- a/static/images/rdi/cdc-retention/postgresql.svg +++ b/static/images/rdi/cdc-retention/postgresql.svg @@ -1,66 +1,123 @@ - + PostgreSQL: slots and WAL -Time moves from earlier to later. A vertical bookmark marks where capture stopped. In the first panel the required history is retained. In the second panel required history is removed. A third panel shows a slot invalidated by an idle timeout even if WAL remains on disk. - - - -PostgreSQL: slots and WAL -Illustration · no fixed expiration time - -1 History retained: resume succeeds -Capture stops; source writes continue - - -Saved position - - -Restart - -WAL retained by slot -Earlier - -Later -The slot keeps the required WAL. -Disk usage grows while RDI is stopped. -Disk used - - - - - - -2 History removed: cannot resume -Capture stops; source writes continue - - -Saved position - - -Restart - -Removed -Retained -Earlier - -Later -A slot limit allows WAL removal. -The invalidated slot cannot resume. - -3 Idle timeout: cannot resume -Capture stops; source writes continue - - -Saved position - - -Restart - -WAL may still exist -Earlier - -Later -PostgreSQL 18: enabled idle timeout -invalidates the slot at a checkpoint. -WAL availability alone is not enough. +Illustrative Friday-to-Sunday interruption. A red bookmark marks the saved capture position. Solid dark blocks show retained history; hatched blocks show removed history. The first timeline can resume. The second cannot resume because required history is missing. PostgreSQL also shows growing WAL storage and a third timeline where an enabled idle timeout invalidates the slot even if WAL remains. + + + + + + +RDI / SOURCE HISTORY +PostgreSQL +Replication slots and WAL + + +01 +History retained +Resume succeeds +Saved position +Restart + + + + + + + + + + + + + + + + + +WAL retained by slot + +Friday +Sunday +Source keeps writing + + +The slot keeps the required WAL. +Disk usage grows while RDI is stopped. +WAL storage grows + + + + + + + + + + + +02 +History removed +Cannot resume +Saved position +Restart + + + + + + + + + + +Removed +Retained + +Friday +Sunday +Source keeps writing + + +A slot limit allows WAL removal. +The invalidated slot cannot resume. + + + + +03 +Slot invalidated +Cannot resume +Saved position +Restart + + + + + + + + + + + + + + + + + +WAL may still exist + +Friday +Sunday +Source keeps writing + + +PostgreSQL 18: enabled idle timeout +invalidates the slot at a checkpoint. +WAL availability alone is not enough. + + +Illustrative times · not retention defaults diff --git a/static/images/rdi/cdc-retention/snowflake.svg b/static/images/rdi/cdc-retention/snowflake.svg index 4297f3c124..9bfe75fdfb 100644 --- a/static/images/rdi/cdc-retention/snowflake.svg +++ b/static/images/rdi/cdc-retention/snowflake.svg @@ -1,43 +1,79 @@ - + Snowflake: stream staleness -Time moves from earlier to later. A vertical bookmark marks where capture stopped. In the first panel the required history is retained. In the second panel required history is removed. - - - -Snowflake: stream staleness -Illustration · no fixed expiration time - -1 History retained: resume succeeds -Capture stops; source writes continue - - -Stream offset - - -Restart - -Source-table history -Earlier - -Later -Required table history remains. -The stream can provide changes. - -2 History removed: cannot resume -Capture stops; source writes continue - - -Stream offset - - -Restart - -Removed -Retained -Earlier - -Later -Required table history is gone. -The stream is stale. +Illustrative Friday-to-Sunday interruption. A red bookmark marks the saved capture position. Solid dark blocks show retained history; hatched blocks show removed history. The first timeline can resume. The second cannot resume because required history is missing. + + + + + + +RDI / SOURCE HISTORY +Snowflake +Stream offsets and table history · Preview + + +01 +History retained +Resume succeeds +Stream offset +Restart + + + + + + + + + + + + + + + + + +Source-table history + +Friday +Sunday +Source keeps writing + + +Required table history remains. +The stream can provide changes. + + + + +02 +History removed +Cannot resume +Stream offset +Restart + + + + + + + + + + +Removed +Retained + +Friday +Sunday +Source keeps writing + + +Required table history is gone. +The stream is stale. + + +Illustrative times · not retention defaults diff --git a/static/images/rdi/cdc-retention/spanner.svg b/static/images/rdi/cdc-retention/spanner.svg index de5e635cee..b1ed8427cf 100644 --- a/static/images/rdi/cdc-retention/spanner.svg +++ b/static/images/rdi/cdc-retention/spanner.svg @@ -1,44 +1,80 @@ - + Spanner: change-stream history -Time moves from earlier to later. A vertical bookmark marks where capture stopped. In the first panel the required history is retained. In the second panel required history is removed. - - - -Spanner: change-stream history -Illustration · no fixed expiration time - -1 History retained: resume succeeds -Capture stops; source writes continue - - -Resume point - - -Restart - -Change records -Earlier - -Later -The resume point is still in range. -Streaming can continue. - -2 History removed: cannot resume -Capture stops; source writes continue - - -Resume point - - -Restart - -Removed -Retained -Earlier - -Later -Required change records expired. -Database version retention -does not extend this stream window. +Illustrative Friday-to-Sunday interruption. A red bookmark marks the saved capture position. Solid dark blocks show retained history; hatched blocks show removed history. The first timeline can resume. The second cannot resume because required history is missing. + + + + + + +RDI / SOURCE HISTORY +Spanner +Positions in change-stream history + + +01 +History retained +Resume succeeds +Resume point +Restart + + + + + + + + + + + + + + + + + +Change records + +Friday +Sunday +Source keeps writing + + +The resume point is still in range. +Streaming can continue. + + + + +02 +History removed +Cannot resume +Resume point +Restart + + + + + + + + + + +Removed +Retained + +Friday +Sunday +Source keeps writing + + +Required change records expired. +Database version retention +does not extend this stream window. + + +Illustrative times · not retention defaults diff --git a/static/images/rdi/cdc-retention/sql-server.svg b/static/images/rdi/cdc-retention/sql-server.svg index f88562720c..c8bb1e00d8 100644 --- a/static/images/rdi/cdc-retention/sql-server.svg +++ b/static/images/rdi/cdc-retention/sql-server.svg @@ -1,44 +1,80 @@ - + SQL Server: CDC change tables -Time moves from earlier to later. A vertical bookmark marks where capture stopped. In the first panel the required history is retained. In the second panel required history is removed. - - - -SQL Server: CDC change tables -Illustration · no fixed expiration time - -1 History retained: resume succeeds -Capture stops; source writes continue - - -Saved LSN - - -Restart - -Captured change rows -Earlier - -Later -Required change rows remain. -RDI continues reading CDC tables. - -2 History removed: cannot resume -Capture stops; source writes continue - - -Saved LSN - - -Restart - -Removed -Retained -Earlier - -Later -Cleanup removed required rows. -Transaction-log backups alone -do not restore this CDC history. +Illustrative Friday-to-Sunday interruption. A red bookmark marks the saved capture position. Solid dark blocks show retained history; hatched blocks show removed history. The first timeline can resume. The second cannot resume because required history is missing. + + + + + + +RDI / SOURCE HISTORY +SQL Server +Positions in CDC change tables + + +01 +History retained +Resume succeeds +Saved LSN +Restart + + + + + + + + + + + + + + + + + +Captured change rows + +Friday +Sunday +Source keeps writing + + +Required change rows remain. +RDI continues reading CDC tables. + + + + +02 +History removed +Cannot resume +Saved LSN +Restart + + + + + + + + + + +Removed +Retained + +Friday +Sunday +Source keeps writing + + +Cleanup removed required rows. +Transaction-log backups alone +do not restore this CDC history. + + +Illustrative times · not retention defaults From aa7ce3c8e93d0509cd618a1fb44c7f187fc9faf6 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Sun, 18 Oct 2026 02:29:55 +0300 Subject: [PATCH 3/8] Explain source-specific CDC history and state failures --- .../cdc-offsets-and-retention.md | 91 +++++---- .../data-pipelines/prepare-dbs/postgresql.md | 2 +- .../rdi/cdc-retention/mongodb-pre-images.svg | 33 ++++ static/images/rdi/cdc-retention/mongodb.svg | 119 ++++++------ .../rdi/cdc-retention/mysql-mariadb.svg | 138 +++++++------- .../rdi/cdc-retention/oracle-transactions.svg | 46 +++++ static/images/rdi/cdc-retention/oracle.svg | 124 ++++++------- .../cdc-retention/postgresql-slot-state.svg | 39 ++++ .../images/rdi/cdc-retention/postgresql.svg | 172 +++++++----------- static/images/rdi/cdc-retention/snowflake.svg | 130 ++++++------- static/images/rdi/cdc-retention/spanner.svg | 150 ++++++++------- .../images/rdi/cdc-retention/sql-server.svg | 134 +++++++------- 12 files changed, 641 insertions(+), 537 deletions(-) create mode 100644 static/images/rdi/cdc-retention/mongodb-pre-images.svg create mode 100644 static/images/rdi/cdc-retention/oracle-transactions.svg create mode 100644 static/images/rdi/cdc-retention/postgresql-slot-state.svg diff --git a/content/integrate/redis-data-integration/cdc-offsets-and-retention.md b/content/integrate/redis-data-integration/cdc-offsets-and-retention.md index 158a825f71..5af9fcf005 100644 --- a/content/integrate/redis-data-integration/cdc-offsets-and-retention.md +++ b/content/integrate/redis-data-integration/cdc-offsets-and-retention.md @@ -45,88 +45,103 @@ The collector's offset is also different from the processor's **checkpoint**. Th The bookmark idea applies across sources, but the way each database retains history differs. The examples assume capture was working before the interruption. A source restart, replacement, or failover can introduce separate capture-state requirements. -### Oracle: keep the required redo history +### Oracle: keep a complete, readable redo chain -Oracle records changes in redo logs. With LogMiner, RDI reads online and archived redo logs and tracks progress using SCNs. An SCN identifies a position in database history; it is not a timestamp with a fixed lifetime. +Oracle records changes in redo logs. With LogMiner, RDI reads online and archived redo and tracks system change numbers (SCNs). An SCN does not expire as a number. The problem is whether Oracle can provide the history needed from that position. -Online redo logs are reused. Archived logs let a reader reach further back, but only while the required files remain available to the mining process. If cleanup removes a required archive during an outage, the collector can no longer continue from its saved position. Having newer logs does not fill the gap. +Cleanup can remove an old archive, but a gap in the middle also matters. The file containing the saved SCN and the newest files can all exist while a required file between them is missing. With Oracle Real Application Clusters (RAC), mining needs the required redo from all relevant threads. -![Oracle timelines show a saved SCN with all required redo available, then a saved SCN whose required archived redo has been removed.](/images/rdi/cdc-retention/oracle.svg) +![Oracle archive files form a complete readable chain, or a missing middle file prevents continuation even though the saved SCN's file still exists.](/images/rdi/cdc-retention/oracle.svg) {width="560"} -Plan archive retention and storage with your database administrator. Check the archive destination RDI actually reads, not only whether backups exist elsewhere. Long-running transactions can also require older redo than the most recent visible activity suggests. +Unavailable does not always mean deleted. Archiving might be delayed, or the archive destination might be inaccessible. Check availability before choosing a fresh snapshot. See [Oracle LogMiner requirements](https://docs.oracle.com/en/database/oracle/oracle-database/19/sutil/oracle-logminer-utility.html). -XStream uses an Oracle capture process and an outbound server, with its own restart position and capture checkpoints. Preserve the history and state required by that capture process; do not apply a LogMiner error diagnosis unchanged to XStream. See [Prepare Oracle](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md), [Oracle LogMiner](https://docs.oracle.com/en/database/oracle/oracle-database/19/sutil/oracle-logminer-utility.html), and [XStream restartability](https://docs.oracle.com/en/database/oracle/oracle-database/19/xstrm/xstream-out-concepts.html). +An outage is not the only risk. A long transaction can hold the restart position in older redo while other transactions continue. Quiet captured activity can also leave the saved offset behind. Monitor the saved position, not just recent events or process uptime. Use the [connector's offset and heartbeat guidance](https://debezium.io/documentation/reference/3.5/connectors/oracle.html#low-change-frequency-offset-management) for your Oracle configuration. -### MySQL and MariaDB: keep the binary logs +![A long-running Oracle transaction holds an older restart position while other transactions commit. Removing that older redo makes a later restart unsafe.](/images/rdi/cdc-retention/oracle-transactions.svg) +{width="560"} + +Two other cases need a different diagnosis. During a snapshot, a “snapshot too old” error concerns [undo history for consistent reads](https://docs.oracle.com/en/database/oracle/oracle-database/19/refrn/UNDO_RETENTION.html), not streaming redo retention. With XStream, recovery also depends on Oracle's capture checkpoints and outbound-server state; a saved client position alone is insufficient. See [Prepare Oracle](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md) and [XStream restartability](https://docs.oracle.com/en/database/oracle/oracle-database/19/xstrm/xstream-out-concepts.html). + +### MySQL and MariaDB: preserve files and source continuity -MySQL and MariaDB record changes in binary logs, often called *binlogs*. A resume position identifies the required place in this history. Using transaction identifiers does not preserve the underlying events after their logs are removed. +MySQL and MariaDB record changes in binary logs, often called *binlogs*. A bookmark can name a file and a position inside it. Rotating to a new file is normal. Older files must remain available until capture has read them. -Automatic expiration or a manual purge can remove a file that a disconnected collector still needs. Increasing retention after that purge does not bring the file back. +Expiration or manual purging can remove required events. Resetting the binary logs can remove history immediately, regardless of the planned retention window. The source tables can remain intact throughout. -![Binary-log timelines show successful resume while the required file remains, then failed resume after that file is purged.](/images/rdi/cdc-retention/mysql-mariadb.svg) +![Binary-log file rotation preserves unread events, purging a required file removes them, and switching servers requires checking whether the saved position maps to the new source history.](/images/rdi/cdc-retention/mysql-mariadb.svg) {width="560"} -Set retention for the interruption you need to tolerate and leave enough disk space for peak writes. Coordinate manual cleanup with CDC readers. Managed services can expose different retention controls, so use the instructions for your deployment in [Prepare MySQL and MariaDB](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md). The [MySQL purge reference](https://dev.mysql.com/doc/refman/8.0/en/purge-binary-logs.html) and [MariaDB binary-log guide](https://mariadb.com/docs/server/server-management/server-monitoring-logs/binary-log/using-and-maintaining-the-binary-log) describe the underlying cleanup behavior. +A source switch is a separate risk. File positions belong to a particular server's history. Global transaction identifiers (GTIDs) help identify transactions across servers; they do not recreate deleted events. Before a migration or failover, check that the replacement source has the required history and that your RDI configuration can resume from it. -### PostgreSQL: retain history without filling the disk +Size retention and storage for peak writes. Coordinate purges and log resets with capture. See [Prepare MySQL and MariaDB](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md), [MySQL purging](https://dev.mysql.com/doc/refman/8.0/en/purge-binary-logs.html), [MySQL log resets](https://dev.mysql.com/doc/refman/8.4/en/reset-binary-logs-and-gtids.html), and [MariaDB transaction positioning](https://mariadb.com/docs/server/ha-and-performance/standard-replication/gtid). -PostgreSQL records changes in its write-ahead log (WAL). RDI reads through a logical replication slot and tracks a log sequence number (LSN). +### PostgreSQL: preserve WAL and slot state -A slot can retain WAL that its reader still needs, even while RDI is stopped. This makes PostgreSQL different from a simple rolling time window: the immediate problem may be growing disk usage rather than removed history. +PostgreSQL records changes in its write-ahead log (WAL). RDI uses a logical replication slot and tracks a log sequence number (LSN). A valid slot can retain needed WAL while RDI is stopped. Disk usage can therefore grow throughout an outage. A full disk is a storage failure, not an expired LSN. -Retention is not unconditional. A configured slot retention limit can allow required WAL to be removed. Newer PostgreSQL versions also support invalidating slots after an idle timeout. A dropped or invalidated slot cannot be treated as a healthy saved bookmark. +Two configured limits can break continuity differently. A finite slot size limit can allow required WAL to be removed at a checkpoint. PostgreSQL 18 can also invalidate an inactive slot at a checkpoint after an enabled idle timeout, even if WAL files remain. That timeout is disabled by default. -![PostgreSQL timelines show WAL retained during an outage with growing storage, required WAL removed after a size limit, and an inactive slot invalidated by a timeout.](/images/rdi/cdc-retention/postgresql.svg) +![PostgreSQL shows a slot retaining growing WAL, a size limit followed by checkpoint-time WAL removal, and a separate idle timer that invalidates the slot even when WAL remains.](/images/rdi/cdc-retention/postgresql.svg) {width="560"} -Monitor slot health, retained WAL, and free disk together. Choose limits that balance recovery time with database storage capacity. Parameter names and available controls differ across PostgreSQL versions; see [Manage WAL retention](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#manage-wal-retention). +Keeping WAL is only part of recovery. Dropping and recreating a slot does not restore its old decoding state. After failover, the new primary needs usable slot state and the required history. A routine restart of the same database does not itself remove a persistent slot. -The same model applies to Supabase, Neon, AlloyDB for PostgreSQL, Amazon Aurora PostgreSQL, and other managed PostgreSQL services. Check provider-specific slot cleanup, storage, and failover policies rather than assuming that every service retains an idle slot indefinitely. +![An original PostgreSQL primary has saved slot state, while a replacement primary has WAL but lacks that slot state. Reusing the slot name does not restore it.](/images/rdi/cdc-retention/postgresql-slot-state.svg) +{width="560"} -### SQL Server: retain the captured change rows +Monitor slot health, retained WAL, and free disk together. See [Manage WAL retention](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#manage-wal-retention) for version-specific controls and [PostgreSQL failover guidance](https://www.postgresql.org/docs/18/logical-replication-failover.html) for slot continuity. Supabase, Neon, AlloyDB, Amazon Aurora PostgreSQL, and other managed variants can have additional cleanup, storage, and failover policies. Do not assume they preserve every idle slot. -SQL Server CDC captures changes from the transaction log into change tables. RDI reads those captured changes using an LSN-based position. +### SQL Server: watch both capture stages -The CDC cleanup process removes old entries from the change tables. If RDI returns after its required entries have been cleaned up, keeping transaction-log backups alone does not make those entries available through the CDC reader. +SQL Server first captures transaction-log changes into CDC change tables. RDI then reads those tables using an LSN-based position. If source capture stops, newer changes cannot reach RDI yet and the transaction log can grow. If RDI falls behind while capture continues, CDC cleanup can remove rows it still needs. -![SQL Server timelines show an LSN inside retained CDC change-table history, then outside the available range after cleanup.](/images/rdi/cdc-retention/sql-server.svg) +![SQL Server copies the transaction log into CDC tables before RDI reads them. Cleanup can overtake RDI's saved LSN, or maintenance can delete the old capture instance and its table.](/images/rdi/cdc-retention/sql-server.svg) {width="560"} -Plan CDC change-table retention for consumer interruptions. Monitor both capture progress into those tables and RDI's progress through them. Capture delays and reader delays are different problems. See [Prepare SQL Server](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md) and Microsoft's [CDC administration guide](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/administer-and-monitor-change-data-capture-sql-server). +History can also disappear during maintenance. Disabling a capture instance deletes its change table, even inside the retention window. A new instance does not recreate those old rows. Follow the [schema-change guidance](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md#drop-the-old-capture-table) before removing an old instance. -### MongoDB: keep enough operation-log history +Monitor source capture and RDI reading separately. Keeping transaction-log backups alone does not make cleaned-up CDC rows available to this reader. See Microsoft's [CDC administration](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/administer-and-monitor-change-data-capture-sql-server) and [capture-instance removal](https://learn.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sys-sp-cdc-disable-table-transact-sql) references. -MongoDB change streams provide a **resume token** that identifies where reading can continue. Resuming requires enough history in the replica set's operation log, or *oplog*. +### MongoDB: account for write volume and old values -For a size-limited oplog, a busy day can retain fewer hours of history than a quiet day. A collector that resumes successfully after one weekend might fail after another weekend with a large write burst. +MongoDB change streams provide a **resume token**. Resuming requires enough history in the replica set's operation log, or *oplog*. For a size-limited oplog, a busy day covers fewer hours than a quiet day. The same weekend interruption can therefore succeed once and fail after a large write burst. -![MongoDB timelines show a resume token whose history remains in the oplog, then history overwritten during a write burst.](/images/rdi/cdc-retention/mongodb.svg) +![Two equal-size MongoDB oplogs cover different periods. Quiet writes preserve Friday's token through Sunday; busy writes leave only Sunday's history.](/images/rdi/cdc-retention/mongodb.svg) {width="560"} -Monitor the actual oplog time window. Size it for peak activity, and consider a minimum retention period where available. A minimum retention period can require the oplog to grow beyond its configured size, so account for storage too. Managed-service controls vary. See [Prepare MongoDB](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md) and the [MongoDB oplog guide](https://www.mongodb.com/docs/manual/core/replica-set-oplog/). +Some pipelines also need a document's old values, called a **pre-image**. Custom Redis keys can depend on fields that changed or were deleted. An event can remain in the oplog after its pre-image is gone. A usable resume token does not guarantee that all required old values remain. + +![MongoDB retains a change event in the oplog while its document pre-image is unavailable. This can affect pipelines using custom Redis keys without making the resume token invalid.](/images/rdi/cdc-retention/mongodb-pre-images.svg) +{width="560"} + +Monitor the actual oplog time window and any required pre-images. A minimum oplog retention period can require extra storage. Dropping or renaming watched objects can also [invalidate a stream, depending on its watch scope](https://www.mongodb.com/docs/manual/reference/change-events/invalidate/); this is separate from history expiry. See [Prepare MongoDB](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md), [oplog retention](https://www.mongodb.com/docs/manual/core/replica-set-oplog/), and [pre-image retention](https://www.mongodb.com/docs/manual/changestreams/#change-streams-with-document-pre--and-post-images). ### Spanner: distinguish two retention windows -Spanner change streams retain change records for a configured period. RDI needs its streaming resume point to remain within the available change-stream history. Once those records expire, a later restart cannot read them from the stream. +Spanner has two independent histories. **Database version retention** keeps older row versions for RDI's fixed-timestamp snapshot. A slow or interrupted snapshot can lose access to that timestamp. **Change-stream retention** keeps change records for streaming. Increasing one window does not extend the other. -![Spanner timelines show a streaming resume point within change-stream retention and a point older than the retained records.](/images/rdi/cdc-retention/spanner.svg) +![Spanner compares two separate histories: both can cover the work, snapshot row versions can expire while change records remain, or streaming history can expire after a completed snapshot.](/images/rdi/cdc-retention/spanner.svg) {width="560"} -Spanner also has **database version retention**. That separate setting matters when RDI reads a consistent snapshot at a particular timestamp. Increasing version retention does not substitute for sufficient change-stream retention during streaming. +Changes to the stream can remove history before an outage ends. Reducing retention makes records outside the new window unavailable immediately. Dropping a stream deletes its records; recreating the same name does not restore them. -Plan the change-stream window for interruptions and the version window for snapshot duration. See [Prepare Spanner](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md) and Google's [change-stream overview](https://docs.cloud.google.com/spanner/docs/change-streams). +Also preserve what the stream watches. Suspending capture or excluding a required table or operation means those changes were never recorded. Longer retention cannot recover events that were never captured. -### Snowflake: consume changes before the stream becomes stale +Plan version history for snapshot duration and change-stream history for lag, interruptions, and catch-up. See [Prepare Spanner](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md), [timestamp limits](https://docs.cloud.google.com/spanner/docs/timestamp-bounds#maximum_timestamp_staleness), and [change-stream management](https://docs.cloud.google.com/spanner/docs/change-streams/manage). -Snowflake streams track an offset against source-table history. The stream does not hold an independent, permanent copy of every change. If the history needed by its offset is no longer retained, the stream becomes **stale**. +### Snowflake: preserve table history and identity -![Snowflake timelines show a usable stream offset with retained table history and a stale stream after required history is no longer retained.](/images/rdi/cdc-retention/snowflake.svg) +Snowflake streams track an offset against source-table history. They do not hold a permanent copy of every change. If the required history is no longer retained, the stream becomes **stale**. Replacing the source table can also break continuity immediately: an old stream does not follow a new table that reuses the same name. + +![Snowflake retains table versions for a stream offset, loses required versions through retention, or replaces the source table with a new object that the old stream does not follow.](/images/rdi/cdc-retention/snowflake.svg) {width="560"} -Monitor stream staleness and the reported `STALE_AFTER` time. Account for table retention and Snowflake's retention-extension rules; do not assume that a stream keeps history indefinitely. Reading a stream with a query alone does not necessarily advance its offset: Snowflake distinguishes querying from consuming changes in a committed transaction. +Monitor staleness and the predicted `STALE_AFTER` time. A plain query does not advance a stream's offset. RDI manages its streams; do not manually consume or replace them to clear an error. + +RDI also checks whether streams have data. Snowflake can protect an empty stream from staleness when that check returns false. This does not protect pending changes during a collector outage. -Snowflake support in RDI is in preview. See [Prepare Snowflake](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md) and Snowflake's [stream retention and staleness guidance](https://docs.snowflake.com/en/user-guide/streams-intro#data-retention-period-and-staleness). +Snowflake support is in preview. See [Prepare Snowflake](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md), [stream staleness](https://docs.snowflake.com/en/user-guide/streams-intro#data-retention-period-and-staleness), and [empty-stream checks](https://docs.snowflake.com/en/sql-reference/functions/system_stream_has_data#usage-notes). ## Plan for the whole interruption @@ -138,7 +153,7 @@ Use a conservative planning budget: This is a planning aid, not a universal database setting. For example, a 48-hour outage budget needs more than 48 hours of retained history if capture was already behind. Catch-up also needs spare processing capacity while the application continues writing. -Before maintenance, check capture progress, available history, and storage. For size-based limits, estimate how much history peak writes produce, not just average writes. Where history is retained by a slot, check how much additional disk an outage could consume. +Before maintenance, check capture progress, available history, and storage. After restoring or replacing a source, also verify that the saved offset still belongs to its history and capture state. For size-based limits, estimate how much history peak writes produce, not just average writes. Where history is retained by a slot, check how much additional disk an outage could consume. Alert while there is still time to act. Combine [RDI monitoring](/content/integrate/redis-data-integration/observability.md) with source-database checks. A running process or a saved offset does not prove that capture is advancing or that its history remains available. Confirm managed-service limits and cleanup policies with the database owner. diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md index 3833dbbdf7..171aa1b28c 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md @@ -166,7 +166,7 @@ the default value of about 200 milliseconds will apply. Logical replication slots retain write-ahead log (WAL) needed by their readers, including during an RDI outage. Retained WAL can grow until it fills the disk. Configured limits or slot invalidation can also prevent RDI from resuming. -See [Understand CDC offsets and source retention](/content/integrate/redis-data-integration/cdc-offsets-and-retention.md#postgresql-retain-history-without-filling-the-disk) +See [Understand CDC offsets and source retention](/content/integrate/redis-data-integration/cdc-offsets-and-retention.md#postgresql-preserve-wal-and-slot-state) for the conceptual explanation. Choose controls for your PostgreSQL version: diff --git a/static/images/rdi/cdc-retention/mongodb-pre-images.svg b/static/images/rdi/cdc-retention/mongodb-pre-images.svg new file mode 100644 index 0000000000..aff29561dd --- /dev/null +++ b/static/images/rdi/cdc-retention/mongodb-pre-images.svg @@ -0,0 +1,33 @@ + +MongoDB: A valid token is only one dependency +For custom Redis keys, RDI can need document pre-images. An event can remain in the oplog while its pre-image has expired or was never enabled. Resumable history does not guarantee availability of the old document values. + + + + + + +RDI / SOURCE HISTORY +MongoDB +A valid token is only one dependency + + +01 +Old values have their own lifetime +Custom keys can need document pre-images +Oplog + +Event still available +Pre-images + +Old value no longer available + + + +A usable token does not restore old values. +Plan pre-image retention for custom keys. + + +Illustrations · not retention defaults + + diff --git a/static/images/rdi/cdc-retention/mongodb.svg b/static/images/rdi/cdc-retention/mongodb.svg index 9002b74f6c..a5a8f55b73 100644 --- a/static/images/rdi/cdc-retention/mongodb.svg +++ b/static/images/rdi/cdc-retention/mongodb.svg @@ -1,79 +1,68 @@ - -MongoDB: operation-log history -Illustrative Friday-to-Sunday interruption. A red bookmark marks the saved capture position. Solid dark blocks show retained history; hatched blocks show removed history. The first timeline can resume. The second cannot resume because required history is missing. + +MongoDB: Write volume changes the oplog time window +Two equal-size oplogs at Sunday restart. With quiet writes, history still includes Friday’s resume token. With busy writes, the same capacity covers only Sunday and Friday’s position is no longer available. - - + + - + RDI / SOURCE HISTORY MongoDB -Resume tokens and oplog history +Write volume changes the oplog time window 01 -History retained -Resume succeeds -Resume token -Restart - - - - - - - - - - - - - - - - - -Oplog history - -Friday -Sunday -Source keeps writing - - -The token still has enough history. -The change stream can resume. +Quiet writes +Same example oplog capacity at restart + + + + + + + + + + + + +Friday token + + +Thursday +Sunday + + +Friday’s history still fits; resume is possible. - - + + 02 -History removed -Cannot resume -Resume token -Restart - - - - - - - - - - -Removed -Retained - -Friday -Sunday -Source keeps writing - - -A write burst replaces old history. -The token alone cannot recover it. +A burst of writes +Same example oplog capacity at restart + + + + + + + + + + + + +Friday token: already overwritten +Sunday morning +Sunday now + + +More writes fit fewer hours of history. +The token cannot recover overwritten events. - -Illustrative times · not retention defaults + +Illustrations · not retention defaults diff --git a/static/images/rdi/cdc-retention/mysql-mariadb.svg b/static/images/rdi/cdc-retention/mysql-mariadb.svg index c28d4c838c..2c1d2ea6d8 100644 --- a/static/images/rdi/cdc-retention/mysql-mariadb.svg +++ b/static/images/rdi/cdc-retention/mysql-mariadb.svg @@ -1,79 +1,87 @@ - -MySQL and MariaDB: binary logs -Illustrative Friday-to-Sunday interruption. A red bookmark marks the saved capture position. Solid dark blocks show retained history; hatched blocks show removed history. The first timeline can resume. The second cannot resume because required history is missing. + +MySQL and MariaDB: Files, positions, and source changes +Normal binary-log rotation keeps the saved file position usable while required files remain. Purging the required file removes its events. Switching to another server is a different case: file names and positions belong to that server’s history. - - + + - + RDI / SOURCE HISTORY MySQL and MariaDB -Positions in the binary logs +Files, positions, and source changes 01 -History retained -Resume succeeds -Saved position -Restart - - - - - - - - - - - - - - - - - -Binary logs - -Friday -Sunday -Source keeps writing - - -The required log file is retained. -Capture can read missed changes. +Rotation is normal +A bookmark names a file and a position + +File 041 + + +File 042 + + +File 043 +Position + + + + +Older files still supply the unread events. - - + + 02 -History removed -Cannot resume -Saved position -Restart - - - - - - - - - - -Removed -Retained - -Friday -Sunday -Source keeps writing - - -The required log file was purged. -Its events are no longer available. +A required file is purged +A bookmark names a file and a position + +File 041 + + +File 042 + + +File 043 +Position + + + + +Expiration or purge removes needed events. +A log reset can remove the whole chain. - -Illustrative times · not retention defaults + + + +03 +A different source +Same table rows ≠ same log positions + +Server A + +File 041 + +File 042 + + + + +Server B + +File 007 + +File 008 + + +No automatic mapping + + +Check source history and resume support. +Transaction IDs do not retain lost events. + + +Illustrations · not retention defaults diff --git a/static/images/rdi/cdc-retention/oracle-transactions.svg b/static/images/rdi/cdc-retention/oracle-transactions.svg new file mode 100644 index 0000000000..55891d4275 --- /dev/null +++ b/static/images/rdi/cdc-retention/oracle-transactions.svg @@ -0,0 +1,46 @@ + +Oracle: A running reader can have an old bookmark +A transaction starts on Thursday and stays open through Sunday. Other transactions commit on Friday and Saturday. The restart position can remain in Thursday redo. If cleanup removes that redo before a restart, recent activity does not make the restart safe. + + + + + + +RDI / SOURCE HISTORY +Oracle +A running reader can have an old bookmark + + +01 +An old restart position +Example: a long-running transaction +Transaction stays open + + + + +Thu +Fri +Sat +Sun +Other transactions commit + + + + +Redo available at restart + + + + +Required redo gone +Newer redo remains + + +Recent events do not prove restart safety. + + +Illustrations · not retention defaults + + diff --git a/static/images/rdi/cdc-retention/oracle.svg b/static/images/rdi/cdc-retention/oracle.svg index a891148e09..321e4df596 100644 --- a/static/images/rdi/cdc-retention/oracle.svg +++ b/static/images/rdi/cdc-retention/oracle.svg @@ -1,79 +1,73 @@ - -Oracle: required redo history -Illustrative Friday-to-Sunday interruption. A red bookmark marks the saved capture position. Solid dark blocks show retained history; hatched blocks show removed history. The first timeline can resume. The second cannot resume because required history is missing. + +Oracle: The whole redo chain must be readable +A saved SCN points into archive 101. A complete chain through archives 102 and 103 allows mining. A missing or inaccessible archive 102 interrupts that chain even if archive 101 and newer files exist. - - + + - + RDI / SOURCE HISTORY Oracle -SCNs and required redo history +The whole redo chain must be readable 01 -History retained -Resume succeeds -Saved SCN -Restart - - - - - - - - - - - - - - - - - -Redo history - -Friday -Sunday -Source keeps writing - - -Required redo is still available. -LogMiner can continue reading. +Complete redo chain +Required files must connect to newer redo + +Archive +101 + + +Archive +102 + + +Archive +103 + + +Online +redo +Saved SCN + + + + +Required redo is available for mining. - - + + 02 -History removed -Cannot resume -Saved SCN -Restart - - - - - - - - - - -Removed -Retained - -Friday -Sunday -Source keeps writing - - -A required archive was removed. -Newer redo cannot fill the gap. +A gap in the chain +Required files must connect to newer redo + +Archive +101 + + +Archive +102 + + +Archive +103 + + +Online +redo +Saved SCN + + +Unavailable + + +The saved SCN’s file can still exist. +One missing link stops continuation. - -Illustrative times · not retention defaults + +Illustrations · not retention defaults diff --git a/static/images/rdi/cdc-retention/postgresql-slot-state.svg b/static/images/rdi/cdc-retention/postgresql-slot-state.svg new file mode 100644 index 0000000000..0bd932874c --- /dev/null +++ b/static/images/rdi/cdc-retention/postgresql-slot-state.svg @@ -0,0 +1,39 @@ + +PostgreSQL: Preserve the slot’s state, not just its name +A slot on the original primary has saved decoding state. After a failover without matching slot state, the new primary may have WAL but no usable continuation. Dropping and recreating a slot under the same name also does not restore its old state. + + + + + + +RDI / SOURCE HISTORY +PostgreSQL +Preserve the slot’s state, not just its name + + +01 +Slot state not preserved +Example: failover without matching slot state + +Original primary +Original slot state + +New primary +WAL exists + + + + +Saved state exists + +Slot state missing + + +Reusing a slot name does not restore it. +Check your failover and hosting setup. + + +Illustrations · not retention defaults + + diff --git a/static/images/rdi/cdc-retention/postgresql.svg b/static/images/rdi/cdc-retention/postgresql.svg index 06e5693518..08e7c2d00e 100644 --- a/static/images/rdi/cdc-retention/postgresql.svg +++ b/static/images/rdi/cdc-retention/postgresql.svg @@ -1,123 +1,83 @@ - -PostgreSQL: slots and WAL -Illustrative Friday-to-Sunday interruption. A red bookmark marks the saved capture position. Solid dark blocks show retained history; hatched blocks show removed history. The first timeline can resume. The second cannot resume because required history is missing. PostgreSQL also shows growing WAL storage and a third timeline where an enabled idle timeout invalidates the slot even if WAL remains. + +PostgreSQL: History loss and slot loss are different +A valid replication slot pins WAL while disk use grows. With a configured slot size limit, a checkpoint can allow required WAL removal. Separately, PostgreSQL 18 can invalidate an inactive slot at a checkpoint after its enabled idle timeout, even if WAL files remain. - - + + - + RDI / SOURCE HISTORY PostgreSQL -Replication slots and WAL +History loss and slot loss are different 01 -History retained -Resume succeeds -Saved position -Restart - - - - - - - - - - - - - - - - - -WAL retained by slot - -Friday -Sunday -Source keeps writing - - -The slot keeps the required WAL. -Disk usage grows while RDI is stopped. -WAL storage grows - - - - - - - +A slot holds history +Capture is paused; source writes continue + +Valid slot + + +Required WAL retained +Disk used + + + + + + + +Outage begins +Later + + +Resume remains possible; disk can fill. - - + + 02 -History removed -Cannot resume -Saved position -Restart - - - - - - - - - - -Removed -Retained - -Friday -Sunday -Source keeps writing - - -A slot limit allows WAL removal. -The invalidated slot cannot resume. +A size limit is exceeded +PostgreSQL 13+ · only with a finite slot limit + + + +Configured limit + +Checkpoint + + +Required WAL removed + + +If required WAL is lost, resume fails. - - + + 03 -Slot invalidated -Cannot resume -Saved position -Restart - - - - - - - - - - - - - - - - - -WAL may still exist - -Friday -Sunday -Source keeps writing - - -PostgreSQL 18: enabled idle timeout -invalidates the slot at a checkpoint. -WAL availability alone is not enough. +An idle slot times out +PostgreSQL 18 · disabled by default + +Slot +inactive + + +Idle timeout +passes + + +Checkpoint + +Slot invalid + +WAL may still exist + + +Available WAL alone is not enough. - -Illustrative times · not retention defaults + +Illustrations · not retention defaults diff --git a/static/images/rdi/cdc-retention/snowflake.svg b/static/images/rdi/cdc-retention/snowflake.svg index 9bfe75fdfb..f3c50c58a2 100644 --- a/static/images/rdi/cdc-retention/snowflake.svg +++ b/static/images/rdi/cdc-retention/snowflake.svg @@ -1,79 +1,79 @@ - -Snowflake: stream staleness -Illustrative Friday-to-Sunday interruption. A red bookmark marks the saved capture position. Solid dark blocks show retained history; hatched blocks show removed history. The first timeline can resume. The second cannot resume because required history is missing. + +Snowflake: Table history and identity · Preview +A stream offset references retained versions of its source table. Removing required versions makes the stream stale. Replacing the source table is a separate failure: the old stream does not attach itself to a new table with the same name. - - + + - + RDI / SOURCE HISTORY Snowflake -Stream offsets and table history · Preview +Table history and identity · Preview 01 -History retained -Resume succeeds -Stream offset -Restart - - - - - - - - - - - - - - - - - -Source-table history - -Friday -Sunday -Source keeps writing - - -Required table history remains. -The stream can provide changes. +Table history available +A stream refers to history; it does not copy it + +Version 1 + + +Version 2 + + +Version 3 +Stream offset + + + + +Retained versions can supply changes. - - + + 02 -History removed -Cannot resume -Stream offset -Restart - - - - - - - - - - -Removed -Retained - -Friday -Sunday -Source keeps writing - - -Required table history is gone. -The stream is stale. +Required table history lost +A stream refers to history; it does not copy it + +Version 1 + + +Version 2 + + +Version 3 +Stream offset + + + + +The stream becomes stale. +Increasing retention cannot restore versions. - -Illustrative times · not retention defaults + + + +03 +The source table is replaced +Same name, different table object + +Old table: orders + +New table: orders + + + +Existing stream + + + + + +The old stream does not follow the new table. + + +Illustrations · not retention defaults diff --git a/static/images/rdi/cdc-retention/spanner.svg b/static/images/rdi/cdc-retention/spanner.svg index b1ed8427cf..b3245bb5c2 100644 --- a/static/images/rdi/cdc-retention/spanner.svg +++ b/static/images/rdi/cdc-retention/spanner.svg @@ -1,80 +1,98 @@ - -Spanner: change-stream history -Illustrative Friday-to-Sunday interruption. A red bookmark marks the saved capture position. Solid dark blocks show retained history; hatched blocks show removed history. The first timeline can resume. The second cannot resume because required history is missing. + +Spanner: Two histories, two independent windows +Three cases compare database row versions with change-stream records. Both histories can remain available. A fixed snapshot timestamp can become too old while change records remain. After a snapshot completes, streaming history can expire independently of database version history. - - + + - + RDI / SOURCE HISTORY Spanner -Positions in change-stream history +Two histories, two independent windows 01 -History retained -Resume succeeds -Resume point -Restart - - - - - - - - - - - - - - - - - -Change records - -Friday -Sunday -Source keeps writing - - -The resume point is still in range. -Streaming can continue. +Both windows cover the work +Snapshot reads and streaming need history +Row versions +Change records + + + + + + + + + + + + + + + +Saved point +Now + + +Required history remains in both windows. - - + + 02 -History removed -Cannot resume -Resume point -Restart - - - - - - - - - - -Removed -Retained - -Friday -Sunday -Source keeps writing - - -Required change records expired. -Database version retention -does not extend this stream window. +The snapshot takes too long +Change records do not replace old row versions +Row versions +Change records + + + + + + + + + + + + + + + + +Saved point +Now + + +Old snapshot reads fail; stream still exists. - -Illustrative times · not retention defaults + + + +03 +Streaming falls behind +A finished snapshot cannot replay the gap +Row versions +Change records +Snapshot finished + + + + + + + + + +Saved point +Now + + +The stream no longer covers missed changes. + + +Illustrations · not retention defaults diff --git a/static/images/rdi/cdc-retention/sql-server.svg b/static/images/rdi/cdc-retention/sql-server.svg index c8bb1e00d8..3736e892f0 100644 --- a/static/images/rdi/cdc-retention/sql-server.svg +++ b/static/images/rdi/cdc-retention/sql-server.svg @@ -1,80 +1,82 @@ - -SQL Server: CDC change tables -Illustrative Friday-to-Sunday interruption. A red bookmark marks the saved capture position. Solid dark blocks show retained history; hatched blocks show removed history. The first timeline can resume. The second cannot resume because required history is missing. + +SQL Server: Two readers, then a separate cleanup window +SQL Server capture copies the transaction log to CDC change tables; RDI reads those tables. Retained CDC rows permit resume. Cleanup can move the oldest available LSN past RDI. Disabling a capture instance deletes its old table independently of the retention window. - - + + - + RDI / SOURCE HISTORY SQL Server -Positions in CDC change tables +Two readers, then a separate cleanup window 01 -History retained -Resume succeeds -Saved LSN -Restart - - - - - - - - - - - - - - - - - -Captured change rows - -Friday -Sunday -Source keeps writing - - -Required change rows remain. -RDI continues reading CDC tables. +Two readers, two delays +SQL Server captures first; RDI reads second + +Transaction +log + + +SQL Server +capture + + +CDC +tables + + +RDI +reader + + +Saved LSN + + +Retained CDC rows allow continuation. - - + + 02 -History removed -Cannot resume -Saved LSN -Restart - - - - - - - - - - -Removed -Retained - -Friday -Sunday -Source keeps writing - - -Cleanup removed required rows. -Transaction-log backups alone -do not restore this CDC history. +Cleanup overtakes RDI +The available range moves forward + + +Saved LSN + + + +Oldest available row + +New changes + + +Required rows are outside the CDC range. - -Illustrative times · not retention defaults + + + +03 +Capture instance removed +Maintenance can remove history immediately + +Old CDC table +Deleted + +New CDC table +Starts later + + + +The retention window did not save these rows. + + +A replacement does not restore old rows. + + +Illustrations · not retention defaults From 9ebbfb7afd728b52c742cd3830dc90cf0c4c32d7 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Sun, 18 Oct 2026 02:44:31 +0300 Subject: [PATCH 4/8] Clarify MongoDB change streams and pre-image requirements --- .../data-pipelines/prepare-dbs/mongodb.md | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md index 02d2308a5c..746924996d 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md @@ -33,7 +33,7 @@ The following table summarizes the considerations to prepare a MongoDB database | MongoDB Topology | Replica Set, Sharded Cluster, or MongoDB Atlas | | User Roles | readAnyDatabase, clusterMonitor | | Oplog | Sufficient size for snapshot and streaming | -| Pre/Post Images | Enable on collections **only if using a custom key** | +| Pre/Post Images | Enable when the pipeline needs old document values, including custom keys and set output types | | Connection String | Must include all hosts, replicaSet (if applicable), authSource, credentials | | MongoDB Atlas | **[SSL required](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-property-mongodb-ssl-enabled)**, provide root CA as the source's `CACERT` secret in RDI | | MongoDB mTLS | X.509 authentication requires source TLS secrets and MongoDB SSL properties | @@ -48,7 +48,7 @@ complete each step. - [ ] [Configure oplog size](#1-configure-oplog-size) - [ ] [Create a MongoDB user for RDI](#2-create-a-mongodb-user-for-rdi) - [ ] [Connection string format](#3-connection-string-format) -- [ ] [Enable change streams and pre/post images (only if using a custom key)](#4-enable-change-streams-and-prepost-images-only-if-using-a-custom-key) +- [ ] [Enable pre-images when required](#4-enable-pre-images-when-required) - [ ] [MongoDB Atlas specific requirements](#5-mongodb-atlas-specific-requirements) - [ ] [Self-hosted MongoDB mTLS and X.509 authentication](#6-self-hosted-mongodb-mtls-and-x509-authentication) - [ ] [Network and security](#7-network-and-security) @@ -107,13 +107,15 @@ mongodb://${MONGODB_DB_USERNAME}:${MONGODB_DB_PASSWORD}@host:30000 - For Atlas, adjust the connection string accordingly (see example below). - Set `replicaSet` and `authSource` as appropriate for your deployment. -## 4. Enable change streams and pre/post images (only if using a custom key) + -Change Streams are required only if you are using a custom key in your RDI pipeline. Change streams are available by default on replica sets, sharded clusters, and MongoDB Atlas. +## 4. Enable pre-images when required -If your RDI pipeline uses a custom key, you must enable pre- and post-images on the relevant collections to capture the document state before and after updates or deletes. This allows RDI to access both the previous and updated versions of documents during change events, ensuring accurate synchronization. +RDI uses MongoDB change streams for ongoing replication, with both default and custom Redis keys. Change streams are available on replica sets, sharded clusters, and MongoDB Atlas. -Use the command below to enable change streams and pre/post images: +A **pre-image** contains a document's values before an update or deletion. For example, if a custom Redis key uses an email address, changing that address requires RDI to remove the old Redis key and write the new one. The pre-image supplies the old address. A **post-image** contains the document after an insert or update; a deleted document has no post-image. + +Jobs that use custom output keys or set output types require pre-images. Enable document pre- and post-images on the relevant collections with this command. It enables the extra document images, not change streams themselves: ```javascript db.runCommand({ @@ -122,6 +124,10 @@ db.runCommand({ }); ``` +Also set `capture.mode` in the source's advanced settings to a mode ending in `_with_pre_image`, such as `change_streams_update_full_with_pre_image`. Enabling images on the collection alone does not make the collector request them. See the [source configuration reference](/content/integrate/redis-data-integration/reference/config-yaml-reference.md#sourcesadvancedsource-advanced-source-settings). + +Keep required pre-images available for the lag and interruptions you need to tolerate. A change event can remain available after its pre-image is gone. See [MongoDB history and old values](/content/integrate/redis-data-integration/cdc-offsets-and-retention.md#mongodb-account-for-write-volume-and-old-values) and [MongoDB pre- and post-images](https://www.mongodb.com/docs/manual/changestreams/#change-streams-with-document-pre--and-post-images). + ## 5. MongoDB Atlas specific requirements MongoDB Atlas only supports secure connections via SSL. From 8458e460c0e218c56243f96b1747ff69eb020066 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Sun, 18 Oct 2026 02:55:49 +0300 Subject: [PATCH 5/8] Identify Oracle as the example source in CDC introduction --- .../redis-data-integration/cdc-offsets-and-retention.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/cdc-offsets-and-retention.md b/content/integrate/redis-data-integration/cdc-offsets-and-retention.md index 5af9fcf005..343783d349 100644 --- a/content/integrate/redis-data-integration/cdc-offsets-and-retention.md +++ b/content/integrate/redis-data-integration/cdc-offsets-and-retention.md @@ -17,7 +17,7 @@ weight: 31 Redis Data Integration (RDI) can resume after an interruption only if the source still provides the change history and capture state it needs. Saving a position does not preserve that history. -Imagine that RDI stops on Friday and starts again on Sunday. Your application keeps writing to Oracle during those two days. RDI remembers where it stopped, but Oracle has removed some of the redo logs needed to continue. The collector reports that its saved system change number (SCN) is no longer available. +Imagine that RDI stops on Friday and starts again on Sunday. In this example, Oracle is the source database. Your application keeps writing to it during those two days. RDI remembers where it stopped, but Oracle has removed some of the redo logs needed to continue. The collector reports that its saved system change number (SCN) is no longer available. The source tables still exist. RDI has not forgotten its position. What is missing is the history between that position and the present. From 9e5d6201cb62804da2e4ea739aef1b12ad05dbc9 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Sun, 18 Oct 2026 03:04:05 +0300 Subject: [PATCH 6/8] Introduce processing before linking to checkpointing --- .../redis-data-integration/cdc-offsets-and-retention.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/integrate/redis-data-integration/cdc-offsets-and-retention.md b/content/integrate/redis-data-integration/cdc-offsets-and-retention.md index 343783d349..c204291f73 100644 --- a/content/integrate/redis-data-integration/cdc-offsets-and-retention.md +++ b/content/integrate/redis-data-integration/cdc-offsets-and-retention.md @@ -39,7 +39,7 @@ The source continues maintaining its history while RDI is disconnected. If the r *These timelines are illustrations, not database defaults. The important boundary is the oldest history still available when capture resumes.* -The collector's offset is also different from the processor's **checkpoint**. The offset tracks capture from the source. The checkpoint tracks processing of events already captured into RDI. A processor can be behind even while capture is current. If the backlog eventually stops capture through [backpressure](/content/integrate/redis-data-integration/architecture/_index.md#backpressure-mechanism), source retention becomes important again. +RDI handles changes in two steps: the collector reads them from the source, then the processor transforms them and writes them to the target. This page focuses on the collector's bookmark. The processor separately tracks its progress so it can recover after a restart. See [Checkpointing](/content/integrate/redis-data-integration/architecture/_index.md#checkpointing) for that part of the pipeline. ## See how each source keeps history @@ -145,7 +145,7 @@ Snowflake support is in preview. See [Prepare Snowflake](/content/integrate/redi ## Plan for the whole interruption -RDI does not have to be completely down for history to become unavailable. A network failure, lost source access, or sustained capture lag can leave its position behind. A slow target can build a backlog until backpressure stops capture. Source cleanup continues independently. +RDI does not have to be completely down for history to become unavailable. A network failure, lost source access, or sustained capture lag can leave its position behind. A slow target can build a backlog until [backpressure](/content/integrate/redis-data-integration/architecture/_index.md#backpressure-mechanism) stops capture. Source cleanup continues independently. Use a conservative planning budget: From 83cd92a00906c4031a9ed72c1ebd027f907b0365 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Sun, 18 Oct 2026 03:14:15 +0300 Subject: [PATCH 7/8] Clarify source retention controls and replay limits --- .../cdc-offsets-and-retention.md | 34 +++++++++++++++++-- .../data-pipelines/prepare-dbs/postgresql.md | 7 ++++ 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/content/integrate/redis-data-integration/cdc-offsets-and-retention.md b/content/integrate/redis-data-integration/cdc-offsets-and-retention.md index c204291f73..b360e9a811 100644 --- a/content/integrate/redis-data-integration/cdc-offsets-and-retention.md +++ b/content/integrate/redis-data-integration/cdc-offsets-and-retention.md @@ -43,10 +43,12 @@ RDI handles changes in two steps: the collector reads them from the source, then ## See how each source keeps history -The bookmark idea applies across sources, but the way each database retains history differs. The examples assume capture was working before the interruption. A source restart, replacement, or failover can introduce separate capture-state requirements. +Each source has its own retention rules. These are source-database settings, not an RDI memory limit. A size limit usually means space on disk. The examples assume capture was working before the interruption. ### Oracle: keep a complete, readable redo chain +**Retention controls: cleanup rules and disk space.** There is no single "keep CDC for this many hours" setting. Backup and cleanup jobs can remove archived logs based on their age or backup status. Oracle can also free disk space in its fast recovery area by removing logs that its deletion rules allow. Check both the [archive deletion policy](https://docs.oracle.com/en/database/oracle/oracle-database/19/bradv/configuring-rman-client-basic.html) and any scheduled cleanup jobs. + Oracle records changes in redo logs. With LogMiner, RDI reads online and archived redo and tracks system change numbers (SCNs). An SCN does not expire as a number. The problem is whether Oracle can provide the history needed from that position. Cleanup can remove an old archive, but a gap in the middle also matters. The file containing the saved SCN and the newest files can all exist while a required file between them is missing. With Oracle Real Application Clusters (RAC), mining needs the required redo from all relevant threads. @@ -56,6 +58,8 @@ Cleanup can remove an old archive, but a gap in the middle also matters. The fil Unavailable does not always mean deleted. Archiving might be delayed, or the archive destination might be inaccessible. Check availability before choosing a fresh snapshot. See [Oracle LogMiner requirements](https://docs.oracle.com/en/database/oracle/oracle-database/19/sutil/oracle-logminer-utility.html). +The logs also need matching table definitions. For example, an old change might refer to a column that has since been renamed or removed. LogMiner can fail to read that change if it uses today's table definition. Keeping more logs does not fix this. See the [LogMiner strategy guidance](https://debezium.io/documentation/reference/3.5/connectors/oracle.html#oracle-logminer-mining-strategies). + An outage is not the only risk. A long transaction can hold the restart position in older redo while other transactions continue. Quiet captured activity can also leave the saved offset behind. Monitor the saved position, not just recent events or process uptime. Use the [connector's offset and heartbeat guidance](https://debezium.io/documentation/reference/3.5/connectors/oracle.html#low-change-frequency-offset-management) for your Oracle configuration. ![A long-running Oracle transaction holds an older restart position while other transactions commit. Removing that older redo makes a later restart unsafe.](/images/rdi/cdc-retention/oracle-transactions.svg) @@ -65,6 +69,8 @@ Two other cases need a different diagnosis. During a snapshot, a “snapshot too ### MySQL and MariaDB: preserve files and source continuity +**Retention controls: an age limit, automatic cleanup, and manual or scheduled purges.** [MySQL](https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_expire_logs_seconds) and MariaDB can expire old log files. Cleanup happens at events such as server startup or log rotation, not necessarily at the exact expiry time. [MariaDB 11.4 and later](https://mariadb.com/docs/server/server-management/server-monitoring-logs/binary-log/using-and-maintaining-the-binary-log#limiting-the-binlog-size) can also limit the total log size on disk. A limit on the size of each individual file only controls rotation; it does not set the total retention window. + MySQL and MariaDB record changes in binary logs, often called *binlogs*. A bookmark can name a file and a position inside it. Rotating to a new file is normal. Older files must remain available until capture has read them. Expiration or manual purging can remove required events. Resetting the binary logs can remove history immediately, regardless of the planned retention window. The source tables can remain intact throughout. @@ -74,10 +80,14 @@ Expiration or manual purging can remove required events. Resetting the binary lo A source switch is a separate risk. File positions belong to a particular server's history. Global transaction identifiers (GTIDs) help identify transactions across servers; they do not recreate deleted events. Before a migration or failover, check that the replacement source has the required history and that your RDI configuration can resume from it. +Reading older changes also needs the right **schema history**: the record of how tables and columns changed. Rewinding across a column addition, rename, or removal can fail or produce wrong fields if that history does not match. Having the log files is not enough. See [how the connector uses schema history](https://debezium.io/documentation/reference/3.5/connectors/mysql.html#mysql-schema-history-topic). + Size retention and storage for peak writes. Coordinate purges and log resets with capture. See [Prepare MySQL and MariaDB](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md), [MySQL purging](https://dev.mysql.com/doc/refman/8.0/en/purge-binary-logs.html), [MySQL log resets](https://dev.mysql.com/doc/refman/8.4/en/reset-binary-logs-and-gtids.html), and [MariaDB transaction positioning](https://mariadb.com/docs/server/ha-and-performance/standard-replication/gtid). ### PostgreSQL: preserve WAL and slot state +**Retention controls: slot progress, an optional disk-size limit, and an optional idle-time limit.** A slot normally keeps the history its reader still needs, rather than keeping a fixed number of days. Size limits and idle timeouts depend on the PostgreSQL version. They take effect at database checkpoints, when PostgreSQL saves recovery state. See [version-specific retention controls](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#manage-wal-retention). + PostgreSQL records changes in its write-ahead log (WAL). RDI uses a logical replication slot and tracks a log sequence number (LSN). A valid slot can retain needed WAL while RDI is stopped. Disk usage can therefore grow throughout an outage. A full disk is a storage failure, not an expired LSN. Two configured limits can break continuity differently. A finite slot size limit can allow required WAL to be removed at a checkpoint. PostgreSQL 18 can also invalidate an inactive slot at a checkpoint after an enabled idle timeout, even if WAL files remain. That timeout is disabled by default. @@ -87,6 +97,8 @@ Two configured limits can break continuity differently. A finite slot size limit Keeping WAL is only part of recovery. Dropping and recreating a slot does not restore its old decoding state. After failover, the new primary needs usable slot state and the required history. A routine restart of the same database does not itself remove a persistent slot. +Resuming is also different from rewinding. The slot remembers which changes the collector has confirmed as received. Moving RDI's bookmark backward does not make that same slot send changes from before its confirmed position, even if the old WAL files still exist. See [the slot's confirmed position](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#manage-wal-retention). + ![An original PostgreSQL primary has saved slot state, while a replacement primary has WAL but lacks that slot state. Reusing the slot name does not restore it.](/images/rdi/cdc-retention/postgresql-slot-state.svg) {width="560"} @@ -94,6 +106,8 @@ Monitor slot health, retained WAL, and free disk together. See [Manage WAL reten ### SQL Server: watch both capture stages +**Retention controls: a time window and a scheduled cleanup job.** The window is configured in minutes. The cleanup job removes old rows from the CDC change tables when it runs. This is not a memory-size limit; the tables also need enough disk space. See [CDC cleanup jobs](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/administer-and-monitor-change-data-capture-sql-server#cleanup-job). + SQL Server first captures transaction-log changes into CDC change tables. RDI then reads those tables using an LSN-based position. If source capture stops, newer changes cannot reach RDI yet and the transaction log can grow. If RDI falls behind while capture continues, CDC cleanup can remove rows it still needs. ![SQL Server copies the transaction log into CDC tables before RDI reads them. Cleanup can overtake RDI's saved LSN, or maintenance can delete the old capture instance and its table.](/images/rdi/cdc-retention/sql-server.svg) @@ -105,6 +119,8 @@ Monitor source capture and RDI reading separately. Keeping transaction-log backu ### MongoDB: account for write volume and old values +**Retention controls: log size, with an optional minimum time in hours.** MongoDB normally removes the oldest entries automatically as its operation log fills. A minimum retention time can make the log grow beyond its configured size. It is a minimum age to preserve, not a deadline to delete. Check the [oplog settings](https://www.mongodb.com/docs/manual/core/replica-set-oplog/) and your hosting service's limits. + MongoDB change streams provide a **resume token**. Resuming requires enough history in the replica set's operation log, or *oplog*. For a size-limited oplog, a busy day covers fewer hours than a quiet day. The same weekend interruption can therefore succeed once and fail after a large write burst. ![Two equal-size MongoDB oplogs cover different periods. Quiet writes preserve Friday's token through Sunday; busy writes leave only Sunday's history.](/images/rdi/cdc-retention/mongodb.svg) @@ -112,13 +128,19 @@ MongoDB change streams provide a **resume token**. Resuming requires enough hist Some pipelines also need a document's old values, called a **pre-image**. Custom Redis keys can depend on fields that changed or were deleted. An event can remain in the oplog after its pre-image is gone. A usable resume token does not guarantee that all required old values remain. +Pre-images have separate cleanup rules: they can expire by age, or disappear when the matching event leaves the oplog. Background cleanup removes them. + ![MongoDB retains a change event in the oplog while its document pre-image is unavailable. This can affect pipelines using custom Redis keys without making the resume token invalid.](/images/rdi/cdc-retention/mongodb-pre-images.svg) {width="560"} -Monitor the actual oplog time window and any required pre-images. A minimum oplog retention period can require extra storage. Dropping or renaming watched objects can also [invalidate a stream, depending on its watch scope](https://www.mongodb.com/docs/manual/reference/change-events/invalidate/); this is separate from history expiry. See [Prepare MongoDB](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md), [oplog retention](https://www.mongodb.com/docs/manual/core/replica-set-oplog/), and [pre-image retention](https://www.mongodb.com/docs/manual/changestreams/#change-streams-with-document-pre--and-post-images). +Reading an old event does not always return the document as it looked then. When the collector uses a [full-document lookup](https://www.mongodb.com/docs/manual/changestreams/#lookup-full-document-for-update-operations), it gets the document's values at lookup time. For example, an older update can return Sunday's values instead of Friday's. + +Monitor the actual oplog time window and any required pre-images. Dropping or renaming watched objects can also [invalidate a stream, depending on its watch scope](https://www.mongodb.com/docs/manual/reference/change-events/invalidate/); this is separate from history expiry. See [Prepare MongoDB](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md) and [pre-image retention](https://www.mongodb.com/docs/manual/changestreams/#change-streams-with-document-pre--and-post-images). ### Spanner: distinguish two retention windows +**Retention controls: two time windows, with automatic cleanup.** You set how long to keep change-stream records and, separately, old database versions. These are durations, not memory or log-size limits. Spanner manages cleanup; you do not schedule a cleanup job. See [change-stream retention](https://docs.cloud.google.com/spanner/docs/change-streams/manage) and [database version retention](https://docs.cloud.google.com/spanner/docs/timestamp-bounds#maximum_timestamp_staleness). + Spanner has two independent histories. **Database version retention** keeps older row versions for RDI's fixed-timestamp snapshot. A slow or interrupted snapshot can lose access to that timestamp. **Change-stream retention** keeps change records for streaming. Increasing one window does not extend the other. ![Spanner compares two separate histories: both can cover the work, snapshot row versions can expire while change records remain, or streaming history can expire after a completed snapshot.](/images/rdi/cdc-retention/spanner.svg) @@ -132,6 +154,8 @@ Plan version history for snapshot duration and change-stream history for lag, in ### Snowflake: preserve table history and identity +**Retention controls: time in days, with automatic management.** Table-history retention and the allowed extension for streams with unread changes determine the window. You do not set a stream memory limit or schedule its cleanup. Extra retained history uses storage. See [stream retention and staleness](https://docs.snowflake.com/en/user-guide/streams-intro#data-retention-period-and-staleness). + Snowflake streams track an offset against source-table history. They do not hold a permanent copy of every change. If the required history is no longer retained, the stream becomes **stale**. Replacing the source table can also break continuity immediately: an old stream does not follow a new table that reuses the same name. ![Snowflake retains table versions for a stream offset, loses required versions through retention, or replaces the source table with a new object that the old stream does not follow.](/images/rdi/cdc-retention/snowflake.svg) @@ -163,6 +187,10 @@ First identify whether required history is missing or temporarily inaccessible. If the database can make the original history available again through a supported procedure, resuming may still be possible. Otherwise, rebuilding from a fresh snapshot establishes a new starting point. Increasing retention protects future history; it cannot recreate removed records. +Changing an offset does not bring back missing history either. An offset can record both a log position and progress within a group of changes. Editing only part of it can skip or repeat changes. Rewinding can send events again; moving forward skips events. Restoring an old offset does not undo changes already written to Redis. + +A schema-history rebuild is different from a data snapshot. It reads current table definitions, not the rows. It cannot recreate older definitions needed to read old changes, or repair data already written with wrong fields. See the [connector's schema-recovery restrictions](https://debezium.io/documentation/reference/3.5/connectors/mysql.html#mysql-property-snapshot-mode). + A snapshot reads what exists now. It does not recover every intermediate event. Consider a row that existed in Redis on Friday but was deleted from the source on Saturday: Sunday's snapshot contains no row to overwrite that old Redis key. Recovery must account for such stale target data as well as current source rows. -Use the [pipeline reset guidance](/content/integrate/redis-data-integration/data-pipelines/deploy.md#reset-a-pipeline) with your source's snapshot configuration and target reconciliation plan. Do not manually move an offset forward to silence an error: that can skip changes. Verify recovered inserts, updates, and deletes before relying on the target again. +Use the [pipeline reset guidance](/content/integrate/redis-data-integration/data-pipelines/deploy.md#reset-a-pipeline) with your source's snapshot configuration and a plan to remove stale target data. Do not move an offset forward just to silence an error. Verify actual field values, inserts, updates, and deletes before relying on the target again; matching row counts alone is not enough. diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md index 171aa1b28c..8adde9c1f9 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md @@ -211,6 +211,13 @@ slot storage. A finite slot limit can protect disk space at the cost of losing required history. Increasing the limit after that history is removed does not restore it. Do not drop or recreate an RDI slot as a routine disk-cleanup step. +The slot's `confirmed_flush_lsn` is the position the collector has acknowledged. +PostgreSQL starts logical streaming at the requested position or +`confirmed_flush_lsn`, whichever is later. Moving only RDI's saved offset backward +does not rewind the slot, even if older WAL remains on disk. Keeping unread history +for normal resume is different from replaying already acknowledged changes. +See the [PostgreSQL replication protocol](https://www.postgresql.org/docs/17/protocol-replication.html). + ## Set up permissions The Debezium connector needs a database user that has the REPLICATION and LOGIN roles so that it From e3bb66e5111044388259155e2eb0713dc1fae824 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Sun, 18 Oct 2026 04:42:56 +0300 Subject: [PATCH 8/8] Correct Snowflake timestamp recovery and retention guidance --- .../cdc-offsets-and-retention.md | 18 ++-- static/images/rdi/cdc-retention/snowflake.svg | 95 +++++++++---------- 2 files changed, 57 insertions(+), 56 deletions(-) diff --git a/content/integrate/redis-data-integration/cdc-offsets-and-retention.md b/content/integrate/redis-data-integration/cdc-offsets-and-retention.md index b360e9a811..ea8f671660 100644 --- a/content/integrate/redis-data-integration/cdc-offsets-and-retention.md +++ b/content/integrate/redis-data-integration/cdc-offsets-and-retention.md @@ -47,7 +47,7 @@ Each source has its own retention rules. These are source-database settings, not ### Oracle: keep a complete, readable redo chain -**Retention controls: cleanup rules and disk space.** There is no single "keep CDC for this many hours" setting. Backup and cleanup jobs can remove archived logs based on their age or backup status. Oracle can also free disk space in its fast recovery area by removing logs that its deletion rules allow. Check both the [archive deletion policy](https://docs.oracle.com/en/database/oracle/oracle-database/19/bradv/configuring-rman-client-basic.html) and any scheduled cleanup jobs. +**Retention controls: cleanup rules and disk space.** There is no single "keep CDC for this many hours" setting. Backup and cleanup jobs can remove archived logs based on their age or backup status. Oracle can also free disk space in its fast recovery area by removing logs that its deletion rules allow. Check both the [archive deletion policy](https://docs.oracle.com/en/database/oracle/oracle-database/19/bradv/configuring-rman-client-basic.html#GUID-FA142760-8ACB-43DA-939C-8407618D2D0F) and any scheduled cleanup jobs. Oracle records changes in redo logs. With LogMiner, RDI reads online and archived redo and tracks system change numbers (SCNs). An SCN does not expire as a number. The problem is whether Oracle can provide the history needed from that position. @@ -69,7 +69,7 @@ Two other cases need a different diagnosis. During a snapshot, a “snapshot too ### MySQL and MariaDB: preserve files and source continuity -**Retention controls: an age limit, automatic cleanup, and manual or scheduled purges.** [MySQL](https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_expire_logs_seconds) and MariaDB can expire old log files. Cleanup happens at events such as server startup or log rotation, not necessarily at the exact expiry time. [MariaDB 11.4 and later](https://mariadb.com/docs/server/server-management/server-monitoring-logs/binary-log/using-and-maintaining-the-binary-log#limiting-the-binlog-size) can also limit the total log size on disk. A limit on the size of each individual file only controls rotation; it does not set the total retention window. +**Retention controls: an age limit, automatic cleanup, and manual or scheduled purges.** [MySQL](https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_expire_logs_seconds) and MariaDB can expire old log files. Cleanup can happen at server startup or when the binary log is flushed, including during rotation. It does not necessarily happen at the exact expiry time. In MySQL 8.0.29 and later, `binlog_expire_logs_auto_purge` can disable automatic cleanup even when an age limit is set. [MariaDB 11.4 and later](https://mariadb.com/docs/server/server-management/server-monitoring-logs/binary-log/using-and-maintaining-the-binary-log#limiting-the-binlog-size) can also limit the total log size on disk. A limit on each individual file's size only controls rotation; it does not set the total retention window. MySQL and MariaDB record changes in binary logs, often called *binlogs*. A bookmark can name a file and a position inside it. Rotating to a new file is normal. Older files must remain available until capture has read them. @@ -143,6 +143,8 @@ Monitor the actual oplog time window and any required pre-images. Dropping or re Spanner has two independent histories. **Database version retention** keeps older row versions for RDI's fixed-timestamp snapshot. A slow or interrupted snapshot can lose access to that timestamp. **Change-stream retention** keeps change records for streaming. Increasing one window does not extend the other. +Keep both retention durations in [RDI's source configuration](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md#5-configure-rdi-for-spanner) in sync with the database settings. RDI uses its configured durations for retention checks; changing only the database settings does not update RDI. + ![Spanner compares two separate histories: both can cover the work, snapshot row versions can expire while change records remain, or streaming history can expire after a completed snapshot.](/images/rdi/cdc-retention/spanner.svg) {width="560"} @@ -154,18 +156,20 @@ Plan version history for snapshot duration and change-stream history for lag, in ### Snowflake: preserve table history and identity -**Retention controls: time in days, with automatic management.** Table-history retention and the allowed extension for streams with unread changes determine the window. You do not set a stream memory limit or schedule its cleanup. Extra retained history uses storage. See [stream retention and staleness](https://docs.snowflake.com/en/user-guide/streams-intro#data-retention-period-and-staleness). +**Retention controls: time in days, with automatic management.** On the source table, `DATA_RETENTION_TIME_IN_DAYS` sets how long Snowflake keeps old table versions through its **Time Travel** feature. `MAX_DATA_EXTENSION_TIME_IN_DAYS` limits how far Snowflake can extend retention to protect streams; its default is 14 days. Check both settings and any extension held by RDI's existing stream. Do not assume that every saved timestamp has 14 days of protection. These controls use storage, not a stream memory limit. See [Snowflake retention rules](https://docs.snowflake.com/en/user-guide/streams-intro#data-retention-period-and-staleness). -Snowflake streams track an offset against source-table history. They do not hold a permanent copy of every change. If the required history is no longer retained, the stream becomes **stale**. Replacing the source table can also break continuity immediately: an old stream does not follow a new table that reuses the same name. +RDI's bookmark is a saved table timestamp. When RDI resumes, it creates a stream at that timestamp to read the changes since then. It also recreates the stream when fetching later batches. If the required history is gone, creating the stream at the saved timestamp fails. Creating a new stream does not bring back old history. -![Snowflake retains table versions for a stream offset, loses required versions through retention, or replaces the source table with a new object that the old stream does not follow.](/images/rdi/cdc-retention/snowflake.svg) +![Snowflake lets RDI recreate a stream when its saved timestamp is still available. Creation fails when that history has expired or when the timestamp predates a replacement table's history.](/images/rdi/cdc-retention/snowflake.svg) {width="560"} -Monitor staleness and the predicted `STALE_AFTER` time. A plain query does not advance a stream's offset. RDI manages its streams; do not manually consume or replace them to clear an error. +Replacing a table is another risk. RDI creates streams using the configured table name, which can now refer to the replacement. The name alone does not prove that the original history is still available. If the saved timestamp predates the replacement table's history, RDI cannot create the stream at that time. + +Monitor how old RDI's saved timestamp is against the history still available for that table. The existing stream's `STALE_AFTER` value is a useful warning, but it does not prove that RDI can recreate a stream from its saved timestamp. RDI manages its streams; do not manually consume or replace them to clear an error. RDI also checks whether streams have data. Snowflake can protect an empty stream from staleness when that check returns false. This does not protect pending changes during a collector outage. -Snowflake support is in preview. See [Prepare Snowflake](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md), [stream staleness](https://docs.snowflake.com/en/user-guide/streams-intro#data-retention-period-and-staleness), and [empty-stream checks](https://docs.snowflake.com/en/sql-reference/functions/system_stream_has_data#usage-notes). +Snowflake support is in private preview. See [Prepare Snowflake](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md), [creating a stream at a past timestamp](https://docs.snowflake.com/en/sql-reference/sql/create-stream#usage-notes), and [empty-stream checks](https://docs.snowflake.com/en/sql-reference/functions/system_stream_has_data#usage-notes). ## Plan for the whole interruption diff --git a/static/images/rdi/cdc-retention/snowflake.svg b/static/images/rdi/cdc-retention/snowflake.svg index f3c50c58a2..b0f1f3ebc7 100644 --- a/static/images/rdi/cdc-retention/snowflake.svg +++ b/static/images/rdi/cdc-retention/snowflake.svg @@ -1,79 +1,76 @@ - -Snowflake: Table history and identity · Preview -A stream offset references retained versions of its source table. Removing required versions makes the stream stale. Replacing the source table is a separate failure: the old stream does not attach itself to a new table with the same name. + +Snowflake: Saved timestamps · Private preview +RDI saves a table timestamp and recreates a stream from it when fetching changes. Available table history allows resume. Expired history or a saved timestamp before a replacement table existed prevents stream creation. - - + + RDI / SOURCE HISTORY Snowflake -Table history and identity · Preview +Saved timestamps · Private preview 01 -Table history available -A stream refers to history; it does not copy it +Saved time is available +Resume needs history from the saved time -Version 1 +Friday -Version 2 +Saturday -Version 3 -Stream offset - - - - -Retained versions can supply changes. +Sunday +Saved time + + +RDI creates a stream at Friday’s timestamp. + + +The new stream can read missed changes. - + 02 -Required table history lost -A stream refers to history; it does not copy it +Saved time is too old +Resume needs history from the saved time -Version 1 +Friday -Version 2 +Saturday -Version 3 -Stream offset - - - - -The stream becomes stale. -Increasing retention cannot restore versions. +Sunday +Saved time + + +Friday’s table history is no longer available. + + +Stream creation at the saved time fails. - + 03 -The source table is replaced -Same name, different table object - -Old table: orders - -New table: orders - - - -Existing stream - - - - - -The old stream does not follow the new table. +The table is replaced +Same name does not preserve old history + +Old table: orders + +New table: orders + +RDI uses the table now at that name. +If its history starts after the saved time: + + +Stream creation at the saved time fails. - -Illustrations · not retention defaults + +Illustrations · not retention defaults