diff --git a/content/embeds/rdi-snapshot-gap.md b/content/embeds/rdi-snapshot-gap.md new file mode 100644 index 0000000000..65bbcbdf35 --- /dev/null +++ b/content/embeds/rdi-snapshot-gap.md @@ -0,0 +1,30 @@ +A new snapshot copies existing source rows. It does not replay missed deletes or remove +target records for rows absent from the snapshot. + +The timeline shows one row whose delete event is lost before collection resumes. +Time progresses from top to bottom. + +```mermaid +%%{init: {"sequence": {"actorMargin": 15, "diagramMarginX": 5, "width": 90, "mirrorActors": false}}}%% +sequenceDiagram + participant Source as Source
database + participant RDI as RDI
collector + participant Redis as Target
Redis + + Source->>RDI: Capture
row 42 + RDI->>Redis: Write
row 42 + Note over RDI: Collection stops + Note over Source: Delete row 42 + Note over Source,RDI: Required log
history expires + RDI->>Source: Restart with
when_needed + Note over Source,RDI: Saved position unavailable
New snapshot + Source->>RDI: Current rows
(without row 42) + RDI->>Redis: Write snapshot
rows + Note over Redis: Row 42 is stale
Delete missed +``` + +An interruption alone does not cause this gap. If the saved position and the required +log history are still available, the collector can resume and capture the delete. +In this example, the history expires before the collector resumes, and the new snapshot +contains no row 42 to overwrite or delete its target record. Streaming resumes after +the snapshot, but the stale record can remain in Redis. diff --git a/content/embeds/rdi-snapshot-modes.md b/content/embeds/rdi-snapshot-modes.md new file mode 100644 index 0000000000..e7a42995c1 --- /dev/null +++ b/content/embeds/rdi-snapshot-modes.md @@ -0,0 +1,71 @@ +The collector saves an *offset*, which is its position in the source's change history. +A missing offset means there is no saved position, as on first deployment or after a +reset. An unavailable position means an offset exists, but the source no longer has +the log history needed to resume from it. + +With the default `initial` mode, the collector snapshots when there is no saved +position, then streams changes. A restart with a completed snapshot and a valid saved +position resumes streaming. Deployment or restart alone does not require a new snapshot. +If the saved position is no longer available, `initial` does not automatically +re-snapshot to recover, and the collector stops and reports an error. + +With `when_needed` mode, the collector also takes a new snapshot when it detects that the +saved position is no longer available. This can reload all selected data after an +outage without a manual reset. The snapshot captures current rows, rather than +reconstructing every missed change. + +If the collector restarts during a snapshot, the snapshot starts again. Rows deleted +while the snapshot is interrupted can remain as stale records in the target because +the new snapshot does not replay their delete events. + +Use `when_needed` with monitoring and alerting. Recovery can start a long-running +snapshot, which can increase latency. Missed deletes can leave stale target records +even after the snapshot finishes. + +#### Missed deletes can leave stale target records {#missed-deletes} + +{{< embed-md "rdi-snapshot-gap.md" >}} + +#### Compare snapshot modes + +The available modes and their exact behavior depend on the source connector and its +version. The following table describes the six values listed in the RDI configuration +reference. A listed value is not necessarily supported by every connector. + +| Mode | Existing source data | Restart with a saved position | After a reset clears the position | +| :-- | :-- | :-- | :-- | +| `initial` (default) | Snapshot when no offset exists, then stream changes. | Resume after a completed snapshot if the position is valid. No automatic snapshot recovery for unavailable log history. | Snapshot, then stream changes. | +| `when_needed` | Snapshot when no offset exists, then stream changes. | Resume if the position is valid. Snapshot again if the connector detects an unavailable position. | Snapshot, then stream changes. | +| `always` | Snapshot on every collector start, then stream changes. | Take a new snapshot even if the position is valid. | Snapshot, then stream changes. | +| `initial_only` | Take an initial snapshot without streaming subsequent changes. | Restart an interrupted initial snapshot; do not capture subsequent changes after it completes. | Snapshot without ongoing change capture. | +| `no_data` | Skip existing rows. Initialize any required schema or source position, then stream changes. | Resume streaming if the position is valid. No data snapshot to recover unavailable history. | Skip existing rows and start streaming according to the connector's rules. | +| `never` | Skip the data snapshot where this mode is supported. | Resume according to the connector's rules; no data snapshot recovery. | Connector-specific startup behavior; existing rows are not reloaded. | + +{{< note >}} +On Kubernetes deployments, the collector exits after completing an `initial_only` +snapshot. Kubernetes restarts it, but the collector sees that the snapshot is complete +and exits again. This restart loop can make the source appear disconnected in +monitoring. Check snapshot completion separately from collector connectivity; +`initial_only` mode does not capture subsequent changes. +{{< /note >}} + +An interrupted initial snapshot can restart when the collector starts again, including +with `initial` and `initial_only`. + +For Debezium 3.5, note these connector differences: + +- PostgreSQL `no_data` performs no snapshot. It resumes from a saved log sequence + number (LSN), or from the replication slot's creation position when no LSN is saved. + This requires the relevant write-ahead log (WAL) history to remain available. +- Other connectors can capture schema or initialize a position without copying + existing rows in `no_data` mode. +- `never` is a deprecated alias for `no_data` in PostgreSQL and MongoDB. MySQL and + MariaDB retain `never` with connector-specific prerequisites. Oracle and SQL Server + do not offer `never`; use `no_data` to skip existing rows. + +See the Debezium 3.5 connector references for [MariaDB](https://debezium.io/documentation/reference/3.5/connectors/mariadb.html), +[MongoDB](https://debezium.io/documentation/reference/3.5/connectors/mongodb.html), +[MySQL](https://debezium.io/documentation/reference/3.5/connectors/mysql.html), +[Oracle](https://debezium.io/documentation/reference/3.5/connectors/oracle.html), +[PostgreSQL](https://debezium.io/documentation/reference/3.5/connectors/postgresql.html), +and [SQL Server](https://debezium.io/documentation/reference/3.5/connectors/sqlserver.html). diff --git a/content/integrate/redis-data-integration/1.19.1/architecture/_index.md b/content/integrate/redis-data-integration/1.19.1/architecture/_index.md index 0f0a2b3ec7..b6c8f5918a 100644 --- a/content/integrate/redis-data-integration/1.19.1/architecture/_index.md +++ b/content/integrate/redis-data-integration/1.19.1/architecture/_index.md @@ -80,6 +80,15 @@ are [*idempotent*](https://en.wikipedia.org/wiki/Idempotence) (that is to say that all writes after the first one make no change to the overall state). +{{< note >}} +The delivery guarantee applies to change events captured by RDI. If a Debezium +collector's saved source position becomes unavailable, a new snapshot cannot replay +changes whose source log history has expired. Missed deletes can leave stale records +in the target. See the +[missed-delete timeline](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#missed-deletes) +and [recovery guidance](/content/integrate/redis-data-integration/1.19.1/troubleshooting.md#unavailable-source-history). +{{< /note >}} + ## Checkpointing RDI uses Redis streams to store the sequence of change events diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/deploy.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/deploy.md index 841b6ec3b3..cb1bcd3db3 100644 --- a/content/integrate/redis-data-integration/1.19.1/data-pipelines/deploy.md +++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/deploy.md @@ -324,7 +324,10 @@ Use [`redis-di stop`](/content/integrate/redis-data-integration/1.19.1/reference to pause a running pipeline and [`redis-di start`](/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-start.md) to resume it. Stopping a pipeline halts data processing without deleting the pipeline or its -configuration, so you can start it again later from where it left off. +configuration. A Debezium source restarts according to its +[snapshot mode](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) +and the availability of its saved source position. +If you keep the source stopped for long periods, the stored offset may be removed, making it impossible to resume streaming. ```bash redis-di stop @@ -334,10 +337,15 @@ redis-di start ## Reset a pipeline Use [`redis-di reset`](/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset.md) -to return a pipeline to initial full-sync mode. This reloads a fresh -[snapshot](/content/integrate/redis-data-integration/1.19.1/architecture/_index.md#overview) of the source -data and then resumes change data capture (CDC), which is useful when the source and target have -drifted out of sync. +to clear the pipeline's saved source positions. With the default `initial` snapshot +mode, a Debezium collector reloads selected source data and then resumes change data capture +(CDC). Reset does not change the configured +[snapshot mode](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#choose-a-snapshot-mode). +Modes that skip existing rows do not reload them after a reset, and `initial_only` +does not capture subsequent changes. + +Reset does not flush the target Redis database. A new snapshot can overwrite target +records, but [missed deletes can leave stale records](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#missed-deletes). ```bash redis-di reset @@ -381,4 +389,4 @@ redis-di set-secret SOURCE_DB_USERNAME newUsername --wait=false redis-di set-secret SOURCE_DB_PASSWORD newPassword ``` -The same applies to any set of changes that are only valid together. \ No newline at end of file +The same applies to any set of changes that are only valid together. diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md index 7e1eb7876c..fb962db493 100644 --- a/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md +++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md @@ -160,6 +160,51 @@ configuration contains the following data: - `java_options`: controls the JAVA_OPTS environment variable (for RDI 1.15.1 and above). Use it to modify the default values for Java heap size and other Java options for the Debezium server. For example, set it to `"-Xmx2g -Xms512m"` to set the maximum heap size to 2 GB and the initial heap size to 512 MB. +### Choose a snapshot mode {#choose-a-snapshot-mode} + +Use `sources..advanced.source.snapshot.mode` to control when a +Debezium collector loads existing source data before streaming changes. This property +applies to sources with `type: cdc`, including MariaDB, MongoDB, MySQL, Oracle, +PostgreSQL, and SQL Server. It does not configure Spanner, Snowflake, or external +collectors. + +Add the property under the source you want to configure. For example, this file +uses `initial` for a MySQL source: + +```yaml +sources: + mysql: + type: cdc + connection: + type: mysql + host: + port: 3306 + user: ${SOURCE_DB_USERNAME} + password: ${SOURCE_DB_PASSWORD} + advanced: + source: + snapshot.mode: initial + +targets: + target: + connection: + type: redis + host: + port: 6379 + password: ${TARGET_DB_PASSWORD} +``` + +RDI 1.19.1 bundles Debezium 3.5. The comparison and connector notes in this +section apply to that release. + +{{< embed-md "rdi-snapshot-modes.md" >}} + +A reset clears saved positions without changing `snapshot.mode` or flushing the target +Redis database. If you need to reload existing rows, select a mode that takes a data +snapshot. A snapshot can overwrite target records, but it does not remove stale records +for rows that are absent from the source. See +[Recover from unavailable source log history](/content/integrate/redis-data-integration/1.19.1/troubleshooting.md#unavailable-source-history). + ### Targets Use this section to provide the connection details for the target Redis diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset.md index 917ce3ac92..8c18f57588 100644 --- a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset.md +++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset.md @@ -8,9 +8,13 @@ categories: ["redis-di"] url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset/' --- -Resets a pipeline into initial full-sync mode, so it reloads a snapshot of the source data before -resuming change data capture. By default, the command waits for the pipeline to reach a terminal -state before returning. +Clears a pipeline's saved source positions without changing its configured snapshot mode +or flushing the target Redis database. With the default `initial` mode, the collector +takes a new data snapshot and then resumes change data capture (CDC). Other modes can +skip existing rows or omit ongoing change capture. See +[Choose a snapshot mode](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#choose-a-snapshot-mode). + +By default, the command waits for the pipeline to reach a terminal state before returning. ## Usage diff --git a/content/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference.md b/content/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference.md index e2190ab93b..8ef36e6fd8 100644 --- a/content/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference.md +++ b/content/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference.md @@ -459,7 +459,7 @@ redis.wait.retry.delay.ms: 1000 #### sources\.advanced\.source: Advanced source settings -Advanced configuration properties for the source database connection and CDC behavior. **Applies to the `cdc` and `flink` collector types.**

For the `cdc` collector type, available properties depend on the source database — refer to the relevant Debezium connector documentation: [MySQL](https://debezium.io/documentation/reference/stable/connectors/mysql.html), [MariaDB](https://debezium.io/documentation/reference/stable/connectors/mariadb.html), [PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html), [Oracle](https://debezium.io/documentation/reference/stable/connectors/oracle.html), [SQL Server](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html), [Db2](https://debezium.io/documentation/reference/stable/connectors/db2.html), [MongoDB](https://debezium.io/documentation/reference/stable/connectors/mongodb.html). When using a property from those pages, omit the `debezium.source.` prefix.

**The named properties below cover the most commonly tuned settings: `spanner.*` properties apply to the `flink` collector type, all others apply to the `cdc` collector type. Any other property from the Debezium documentation can still be set as a free-form key-value pair.** +Advanced configuration properties for the source database connection and CDC behavior. **Applies to the `cdc` and `flink` collector types.**

For the `cdc` collector type, available properties depend on the source database — refer to the relevant Debezium connector documentation: [MySQL](https://debezium.io/documentation/reference/stable/connectors/mysql.html), [MariaDB](https://debezium.io/documentation/reference/stable/connectors/mariadb.html), [PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html), [Oracle](https://debezium.io/documentation/reference/stable/connectors/oracle.html), [SQL Server](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html), [MongoDB](https://debezium.io/documentation/reference/stable/connectors/mongodb.html). When using a property from those pages, omit the `debezium.source.` prefix.

Set `snapshot.mode` as a free-form source property to control data snapshots. The default is `initial`; available modes and restart behavior depend on the connector. A reset clears the saved position without changing the mode. See [Choose a snapshot mode](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#choose-a-snapshot-mode), including the [missed-delete timeline](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#missed-deletes).

**The named properties below cover the most commonly tuned settings: `spanner.*` properties apply to the `flink` collector type, all others apply to the `cdc` collector type. Any other property from the Debezium documentation can still be set as a free-form key-value pair.** **Properties** diff --git a/content/integrate/redis-data-integration/1.19.1/troubleshooting.md b/content/integrate/redis-data-integration/1.19.1/troubleshooting.md index 7a967150ee..545793a9ba 100644 --- a/content/integrate/redis-data-integration/1.19.1/troubleshooting.md +++ b/content/integrate/redis-data-integration/1.19.1/troubleshooting.md @@ -57,6 +57,40 @@ log analysis tools can use. > [Backpressure mechanism](/content/integrate/redis-data-integration/1.19.1/architecture/_index.md#backpressure-mechanism) > in the Architecture guide for more information. +## Recover from unavailable source log history {#unavailable-source-history} + +If the collector cannot resume from its saved position, check the collector source log +for unavailable or expired source log history. This differs from a missing offset, +which is expected on first deployment or after a reset. + +With `initial` mode, the collector can fail rather than start a new snapshot when the saved +position is unavailable. With `when_needed` mode, it can automatically snapshot the current +selected data and resume streaming. A new snapshot after a restart can therefore +indicate that the source history was no longer available. See +[Choose a snapshot mode](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) +for the connector-specific behavior. + +Possible causes include: + +- The collector was stopped long enough for the source change log to expire. +- A configuration change made the collector connect to another server with the same + data but different offsets. +- A cleanup process removed the change log while RDI was running. +- A crash and restore left the source without the saved offset. + +Before you use a new snapshot to recover, check for +[missed deletes and stale target records](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#missed-deletes). +A snapshot does not replay lost history or delete target records that are absent from +the source. Resetting the pipeline also leaves target records in place. + +If the saved position is unavailable, work with Redis support to plan a target cleanup +or rebuild before relying on the target as a complete copy. Check whether the target +contains data from other sources or applications before deleting any records. To reload +existing source rows, configure a mode that takes a data snapshot, then +[reset the pipeline](/content/integrate/redis-data-integration/1.19.1/data-pipelines/deploy.md#reset-a-pipeline). +Review the source's log retention so the required history remains available during +future interruptions. + ## Dump support package If you need to send a comprehensive set of forensics data to Redis support, diff --git a/content/integrate/redis-data-integration/architecture/_index.md b/content/integrate/redis-data-integration/architecture/_index.md index c55e9db9f5..8053c05634 100644 --- a/content/integrate/redis-data-integration/architecture/_index.md +++ b/content/integrate/redis-data-integration/architecture/_index.md @@ -80,6 +80,15 @@ are [*idempotent*](https://en.wikipedia.org/wiki/Idempotence) (that is to say that all writes after the first one make no change to the overall state). +{{< note >}} +The delivery guarantee applies to change events captured by RDI. If a Debezium +collector's saved source position becomes unavailable, a new snapshot cannot replay +changes whose source log history has expired. Missed deletes can leave stale records +in the target. See the +[missed-delete timeline](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#missed-deletes) +and [recovery guidance](/content/integrate/redis-data-integration/troubleshooting.md#unavailable-source-history). +{{< /note >}} + ## Checkpointing RDI uses Redis streams to store the sequence of change events diff --git a/content/integrate/redis-data-integration/data-pipelines/deploy.md b/content/integrate/redis-data-integration/data-pipelines/deploy.md index 438d886de6..02615595f9 100644 --- a/content/integrate/redis-data-integration/data-pipelines/deploy.md +++ b/content/integrate/redis-data-integration/data-pipelines/deploy.md @@ -340,7 +340,9 @@ Use [`redis-di stop`](/content/integrate/redis-data-integration/reference/cli/re to pause a running pipeline and [`redis-di start`](/content/integrate/redis-data-integration/reference/cli/redis-di-start.md) to resume it. Stopping a pipeline halts data processing without deleting the pipeline or its -configuration, so you can start it again later from where it left off. +configuration. A Debezium source restarts according to its +[snapshot mode](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) +and the availability of its saved source position. ```bash redis-di stop @@ -360,10 +362,15 @@ Note that a source can only run if its parent pipeline is running. See ## Reset a pipeline Use [`redis-di reset`](/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md) -to return a pipeline to initial full-sync mode. This reloads a fresh -[snapshot](/content/integrate/redis-data-integration/architecture/_index.md#overview) of the source -data and then resumes change data capture (CDC), which is useful when the source and target have -drifted out of sync. +to clear the pipeline's saved source positions. With the default `initial` snapshot +mode, a Debezium collector reloads selected source data and then resumes change data capture +(CDC). Reset does not change the configured +[snapshot mode](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode). +Modes that skip existing rows do not reload them after a reset, and `initial_only` mode +does not capture subsequent changes. + +Reset does not flush the target Redis database. A new snapshot can overwrite target +records, but [missed deletes can leave stale records](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#missed-deletes). ```bash redis-di reset diff --git a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md index 92bf0f4542..5d8a132ee7 100644 --- a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md +++ b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md @@ -207,10 +207,9 @@ collector and deletes that source's data from the RDI database, including its ch Debezium offsets, schema history, dead-letter queue entries, statistics, deduplication state, and record counters. The other sources keep their data, and RDI stops the whole pipeline while the deletion runs and starts it again afterwards. No further action is -needed for this cleanup, but it means that a source you add later under the same name starts -from a new -[initial snapshot](/content/integrate/redis-data-integration/architecture/_index.md) -rather than from the position it had reached. +needed for this cleanup, but a source you add later under the same name has no saved +position. With the default `initial` mode, it takes a new snapshot. Other modes follow +[their configured snapshot behavior](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode). The source's secrets are not deleted, so remove them yourself with [`redis-di delete-secret`](/content/integrate/redis-data-integration/reference/cli/redis-di-delete-secret.md) @@ -241,8 +240,9 @@ to removing the source and adding a new source with the new name. This implies i its `connection` section. - You must update `server_name` for every job that reads from the source. - The data present in the RDI database under the old name is deleted, as it is for any removed source. -- The source starts with a new - [initial snapshot](/content/integrate/redis-data-integration/architecture/_index.md). +- The new source has no saved position. With the default `initial` mode, it takes a + new snapshot. Other modes follow + [their configured snapshot behavior](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode). ## Start, stop, and reset a single source @@ -259,14 +259,23 @@ pipeline. Generally, stopping one source leaves the others running, and when one `external`. RDI creates no collector for this, so you cannot start or stop it. Stopping a source scales its collector down to zero replicas and leaves the rest of the -source's resources in place. RDI records a captured position for each source, so when you restart a collector, it resumes from where it stopped. +source's resources in place. When you restart a collector, its +[snapshot mode](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) +and the availability of its saved position determine whether it resumes streaming or +takes a new snapshot. Resetting a single source deletes that source's data from the RDI database, including its change data streams, Debezium offsets, schema history, dead-letter queue entries, statistics, deduplication state, and record counters. -A new [initial snapshot](/content/integrate/redis-data-integration/architecture/_index.md) is then -taken for that source, while every other source keeps its data. RDI stops the whole pipeline while -the reset runs and starts it again afterwards, exactly as it does for a reset of the whole -pipeline. +The reset source then follows its configured +[snapshot mode](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) +with no saved position. With the default `initial` mode, it takes a new snapshot and +resumes streaming. Every other source keeps its saved position, but all collectors +restart and follow their configured modes. For example, a source using `always` mode takes +another snapshot on restart even though its position was preserved. RDI stops the whole +pipeline while the reset runs and starts it again afterwards. + +Reset does not delete records from the target Redis database. A new snapshot can +[leave stale target records after missed deletes](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#missed-deletes). ## Monitor each source @@ -384,7 +393,8 @@ redis-di set-secret USERNAME --db mysql redis-di set-secret PASSWORD --db mysql ``` -The source then takes a fresh -[initial snapshot](/content/integrate/redis-data-integration/architecture/_index.md), because the position it had -reached was deleted along with the rest of its data. Records the pipeline already wrote to the -target database are not deleted, so the snapshot overwrites them. +The source has no saved position because its internal data was deleted. With the +default `initial` mode, it takes a fresh snapshot. Other modes follow +[their configured snapshot behavior](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode). +Records already written to the target remain. A new snapshot can overwrite them, but +it does not remove records for rows that are absent from the source. diff --git a/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md b/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md index 97fe54715c..36acf0d4d0 100644 --- a/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md +++ b/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md @@ -170,6 +170,52 @@ The example configuration contains the following data: - `java_options`: controls the JAVA_OPTS environment variable (for RDI 1.15.1 and above). Use it to modify the default values for Java heap size and other Java options for the Debezium server. For example, set it to `"-Xmx2g -Xms512m"` to set the maximum heap size to 2 GB and the initial heap size to 512 MB. +### Choose a snapshot mode {#choose-a-snapshot-mode} + +Use `sources..advanced.source.snapshot.mode` to control when a +Debezium collector loads existing source data before streaming changes. This property +applies to sources with `type: cdc`, including MariaDB, MongoDB, MySQL, Oracle, +PostgreSQL, and SQL Server. It does not configure Spanner, Snowflake, or external +collectors. + +Add the property under the source you want to configure. For example, this file +uses `initial` mode for a MySQL source: + +```yaml +sources: + mysql: + type: cdc + connection: + type: mysql + host: + port: 3306 + user: ${MYSQL_DB_USERNAME} + password: ${MYSQL_DB_PASSWORD} + advanced: + source: + snapshot.mode: initial + +targets: + target: + connection: + type: redis + host: + port: 6379 + password: ${TARGET_DB_PASSWORD} +``` + +RDI 1.19.1 and RDI 2.0.0 bundle Debezium 3.5. The comparison and connector +notes in this section apply to those releases. RDI 2.2.0 is planned to use +Debezium 3.7. + +{{< embed-md "rdi-snapshot-modes.md" >}} + +A reset clears saved positions without changing `snapshot.mode` or flushing the target +Redis database. If you need to reload existing rows, select a mode that takes a data +snapshot. A snapshot can overwrite target records, but it does not remove stale records +for rows that are absent from the source. See +[Recover from unavailable source log history](/content/integrate/redis-data-integration/troubleshooting.md#unavailable-source-history). + ### Targets Use this section to provide the connection details for the target Redis diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md b/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md index 050fdcc4f5..4fe8da3624 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md @@ -9,9 +9,13 @@ aliases: - /integrate/redis-data-integration/ingest/reference/cli/redis-di-reset/ --- -Resets a pipeline into initial full-sync mode, so it reloads a snapshot of the source data before -resuming change data capture. By default, the command waits for the pipeline to reach a terminal -state before returning. +Clears a pipeline's saved source positions without changing its configured snapshot mode +or flushing the target Redis database. With the default `initial` mode, the collector +takes a new data snapshot and then resumes change data capture (CDC). Other modes can +skip existing rows or omit ongoing change capture. See +[Choose a snapshot mode](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode). + +By default, the command waits for the pipeline to reach a terminal state before returning. With `--source`, only the specified source is reset. See [Start, stop, and reset a single source](/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md#start-stop-and-reset-a-single-source) for more information. diff --git a/content/integrate/redis-data-integration/reference/config-yaml-reference.md b/content/integrate/redis-data-integration/reference/config-yaml-reference.md index d63df53a7e..1f90237053 100644 --- a/content/integrate/redis-data-integration/reference/config-yaml-reference.md +++ b/content/integrate/redis-data-integration/reference/config-yaml-reference.md @@ -490,7 +490,7 @@ Advanced configuration properties for the source database connection and CDC beh |Name|Type|Description|Required| |----|----|-----------|--------| |**record\.processing\.threads**
(Record processing threads)|`integer`|Controls how many worker threads process captured records before they are written downstream.
Minimum: `1`
|| -|**snapshot\.mode**
(Snapshot mode)|`string`|Determines when the collector snapshots the source database. A pipeline reset clears the recorded offset, so this also decides whether the collector snapshots again after a reset.
Default: `"initial"`
Enum: `"always"`, `"initial"`, `"initial_only"`, `"never"`, `"no_data"`, `"when_needed"`
|| +|**snapshot\.mode**
(Snapshot mode)|`string`|Determines when a Debezium collector snapshots selected source data. Available modes and restart behavior depend on the source connector. A reset clears the saved offset without changing this mode or flushing the target. A new snapshot can leave stale target records after missed deletes. See [Choose a snapshot mode](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) for the mode comparison.
Default: `"initial"`
Enum: `"always"`, `"initial"`, `"initial_only"`, `"never"`, `"no_data"`, `"when_needed"`
|| |**snapshot\.max\.threads**
(Snapshot max threads)|`integer`|Sets the maximum number of threads used while taking the initial snapshot.
Default: `1`
Minimum: `1`
|| |**poll\.interval\.ms**
(Poll interval ms)|`integer`|Defines how often the collector polls the source for new changes.
Default: `500`
Minimum: `1`
|| |**snapshot\.fetch\.size**
(Snapshot fetch size)|`integer`|Defines how many rows are fetched per batch during the initial snapshot.
Minimum: `0`
|| diff --git a/content/integrate/redis-data-integration/troubleshooting.md b/content/integrate/redis-data-integration/troubleshooting.md index 4886cc33f9..5c254ef13c 100644 --- a/content/integrate/redis-data-integration/troubleshooting.md +++ b/content/integrate/redis-data-integration/troubleshooting.md @@ -57,6 +57,40 @@ log analysis tools can use. > [Backpressure mechanism](/content/integrate/redis-data-integration/architecture/_index.md#backpressure-mechanism) > in the Architecture guide for more information. +## Recover from unavailable source log history {#unavailable-source-history} + +If the collector cannot resume from its saved position, check the collector source log +for unavailable or expired source log history. This differs from a missing offset, +which is expected on first deployment or after a reset. + +With `initial` mode, the collector can fail rather than start a new snapshot when the saved +position is unavailable. With `when_needed` mode, it will automatically snapshot the current +selected data and resume streaming. A new snapshot after a restart can therefore +indicate that the source history was no longer available. See +[Choose a snapshot mode](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) +for the connector-specific behavior. + +Possible causes include: + +- The collector was stopped long enough for the source change log to expire. +- A configuration change made the collector connect to another server with the same + data but different offsets. +- A cleanup process removed the change log while RDI was running. +- A crash and restore left the source without the saved offset. + +Before you use a new snapshot to recover, check for +[missed deletes and stale target records](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#missed-deletes). +A snapshot does not replay lost history or delete target records that are absent from +the source. Resetting the pipeline also leaves target records in place. + +If the saved position is unavailable, work with Redis support to plan a target cleanup +or rebuild before relying on the target as a complete copy. Check whether the target +contains data from other sources or applications before deleting any records. To reload +existing source rows, configure a mode that takes a data snapshot, then +[reset the pipeline](/content/integrate/redis-data-integration/data-pipelines/deploy.md#reset-a-pipeline). +Review the source's log retention so the required history remains available during +future interruptions. + ## Dump support package If you need to send a comprehensive set of forensics data to Redis support, diff --git a/content/operate/rc/rdi/define.md b/content/operate/rc/rdi/define.md index 692bd553f7..bef1028d51 100644 --- a/content/operate/rc/rdi/define.md +++ b/content/operate/rc/rdi/define.md @@ -124,6 +124,13 @@ Enter the selected source's database settings. The fields depend on the database Use **Collector properties** for additional source and sink settings. These settings apply to the selected source. See the [collector source properties](/content/integrate/redis-data-integration/reference/config-yaml-reference.md#sourcesadvancedsource-advanced-source-settings) and [collector sink properties](/content/integrate/redis-data-integration/reference/config-yaml-reference.md#sourcesadvancedsink-rdi-collector-stream-writer-configuration). +For Debezium sources, set `snapshot.mode` in **Collector source properties** to +control data snapshots. The default mode is `initial`. Set it for each source separately; +available modes depend on the source database. See [Choose a snapshot mode](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode). +With `when_needed` mode, an unavailable saved position can trigger a new snapshot, but +[missed deletes can leave stale target records](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#missed-deletes). +This property does not configure the Snowflake collector. + ![The Edit advanced properties control under Collector properties.](/images/rc/rdi/rdi-source-configuration-collector-properties.png) {width="500px"} @@ -179,7 +186,7 @@ For the Flink processor, source matchers can use lists or entries prefixed with Review every source's connection, selected data, and transformation jobs, together with the shared target and settings. Use the source actions to return to a section that needs changes. -Select **Deploy pipeline** to start the pipeline. Each source performs its initial snapshot and then captures ongoing changes. Snapshot duration depends on the amount of selected data and source performance. +Select **Deploy pipeline** to start the pipeline. With the default `initial` mode, each Debezium source without a saved position takes a snapshot and then captures ongoing changes. Other [snapshot modes](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) can skip existing rows or omit ongoing change capture. Snapshot duration depends on the amount of selected data and source performance. ![The Deploy pipeline button.](/images/rc/rdi/rdi-confirm-deploy.png) {width="175px"} diff --git a/content/operate/rc/rdi/faq.md b/content/operate/rc/rdi/faq.md index 55c03028f3..617b8e13e8 100644 --- a/content/operate/rc/rdi/faq.md +++ b/content/operate/rc/rdi/faq.md @@ -14,22 +14,47 @@ weight: 6 ### What happens when I reset the pipeline? {#reset-pipeline} -A pipeline reset clears the internal RDI state for all sources, including their saved positions. When the pipeline runs, RDI takes a new snapshot of the selected data from every source, applies the current transformations, and then resumes streaming changes. If you reset a stopped pipeline, it remains stopped until you start it. +A pipeline reset clears the internal RDI state for all sources, including their saved positions. When the pipeline runs, each source follows its configured initial data load settings. With the default `initial` mode, Debezium sources take new snapshots, apply the current transformations, and resume streaming. Other [snapshot modes](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) can skip existing rows or omit ongoing change capture. If you reset a stopped pipeline, it remains stopped until you start it. A reset does **not** flush the target Redis database. Records already in the target remain until RDI overwrites or deletes them through normal processing. Keys that are no longer produced by the current dataset or transformations can remain in the target after a reset. For example, changing a transformation's key prefix and resetting creates keys with the new prefix without deleting keys with the old prefix. See [Reset data pipeline](/content/operate/rc/rdi/view-edit.md#reset-data-pipeline) for the steps. +### Why can records remain after a recovery snapshot? {#missed-deletes} + +A new snapshot can leave stale records in Redis. If a source row was deleted while +collection was interrupted and the required log history expired, the snapshot cannot +replay its delete event. This applies to Debezium sources with `type: cdc`, including +automatic snapshots with `when_needed` and snapshots after a manual reset. + +The sequence is: + +1. A source row is copied to Redis. +1. Collection stops, and the collector saves its source position. +1. The source deletes the row while Redis still has it. +1. The saved position becomes unavailable, so the delete cannot be replayed. +1. `when_needed` mode takes a new snapshot. The deleted row is absent. +1. Streaming resumes, but Redis still has the stale row because its delete was missed. + +An interruption alone does not cause this gap. The saved position must become +unavailable. The snapshot does not clean up the target, and a reset does not flush it. +See the [detailed missed-delete sequence](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#missed-deletes) +and [recovery guidance](/content/integrate/redis-data-integration/troubleshooting.md#unavailable-source-history). + ### What happens when I flush the target database? **Flush target database** permanently deletes all data from the shared target, including data from every source and data written outside RDI. Stop the pipeline before flushing. The source databases are not changed. -Flushing does not clear RDI's saved source positions. If you only start the pipeline afterwards, it resumes from those positions. It does not automatically reload records that have not changed in the source. +Flushing does not clear RDI's saved source positions. Starting a Debezium source follows its [snapshot mode](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode). With the default `initial` mode and a valid saved position, it resumes streaming without reloading unchanged source rows. See [Flush the target database](/content/operate/rc/rdi/view-edit.md#flush-the-target-database). ### How do I reload data after a flush? {#reload-after-flush} +Before reloading, check each source's initial data load settings. For Debezium sources, +select a [snapshot mode](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) that loads existing rows, such as +`initial`. Modes that skip data snapshots cannot refill the target's existing rows. + 1. Wait for the flush to finish. 1. [Reset the pipeline](/content/operate/rc/rdi/view-edit.md#reset-data-pipeline) while it is stopped, and wait for the reset to finish. 1. [Start the pipeline](/content/operate/rc/rdi/view-edit.md#stop-and-restart-data-pipeline) to take new snapshots of the selected data from all sources. @@ -41,7 +66,7 @@ RDI reloads data available in the source databases using the current dataset and ### What happens when I reset one source? {#reset-one-source} -RDI clears the selected source's internal state and takes a new snapshot of its selected data. The whole pipeline, including the other sources and the processor, restarts during this operation. Other sources keep their saved positions and resume streaming without a new snapshot. +RDI clears the selected source's internal state, including its saved position. With the default `initial` mode, a Debezium source takes a new data snapshot. Other [snapshot modes](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) can skip existing rows or omit ongoing change capture. The whole pipeline, including the other sources and the processor, restarts. Other sources keep their saved positions but follow their configured modes on restart; `always` mode takes another snapshot even with a preserved position. All records already in the target remain, including records from the reset source. The new snapshot can overwrite that source's records. Resetting a source does not selectively delete its target data. @@ -51,7 +76,7 @@ See [Reset a source](/content/operate/rc/rdi/view-edit.md#reset-source). Currently, RDI cannot flush target data for a single source. Connect to the target Redis database and selectively delete the records you want to remove. Identify them from your key naming and transformation rules, and check that other sources do not write to the same keys. Do not use **Flush target database** for this purpose: it deletes data for all sources. -Stop the affected source before deleting its records and allow its pending records to finish processing. Starting the source resumes ingestion, and later changes can recreate records. To reload all its selected data, [reset that source](/content/operate/rc/rdi/view-edit.md#reset-source). +Stop the affected source before deleting its records and allow its pending records to finish processing. Starting the source resumes ingestion, and later changes can recreate records. To reload all its selected data, select a mode that takes a data snapshot, then [reset that source](/content/operate/rc/rdi/view-edit.md#reset-source). ### Does deleting a source delete its target data? @@ -63,7 +88,7 @@ No. Deleting a source removes its pipeline configuration and internal RDI state. Redis manages RDI upgrades in Redis Cloud. Maintenance follows your Redis Cloud Pro subscription's [maintenance window](/content/operate/rc/rdi/_index.md#maintenance-windows). -During an upgrade, monitoring may be temporarily unavailable, and ingestion pauses while the pipeline components restart. A streaming pipeline then resumes from its saved state and processes the changes accumulated during the interruption. +During an upgrade, monitoring may be temporarily unavailable, and ingestion pauses while the pipeline components restart. With the default `initial` mode and valid saved positions, Debezium sources resume streaming and process the accumulated changes. Other [snapshot modes](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) or unavailable source history can change this behavior. A routine upgrade does not flush the target database or require you to reset the pipeline. Existing target records remain available, but they may temporarily lag behind the source data. diff --git a/content/operate/rc/rdi/view-edit.md b/content/operate/rc/rdi/view-edit.md index b438c306d8..ba669f9700 100644 --- a/content/operate/rc/rdi/view-edit.md +++ b/content/operate/rc/rdi/view-edit.md @@ -62,11 +62,29 @@ When extending an older single-source pipeline, preserve the original source nam 1. On **Dashboard**, open the source card's **More actions** menu. 1. Select **Stop source**, then confirm with **Stop source**. -Stopping one source pauses data capture from that source without stopping the other sources. Starting it again resumes from its saved position. To resume a source, open its **More actions** menu and select **Start source**, then confirm with **Start source**. A source processes data only while the pipeline is running. See [How does stopping sources affect billing?](/content/operate/rc/rdi/faq.md#stopping-and-billing). +Stopping one source pauses data capture from that source without stopping the other sources. When you start a source again, its behavior depends on the source type: + +- Debezium sources follow their [snapshot mode](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) and the availability of their saved position. +- Snowflake sources use RIOT-X and Snowflake Streams. They resume from the saved stream offset when the stream retains the required changes. If the retention window expires while the source is stopped, changes can be lost. + +To recover a Snowflake source with an unusable saved position, set `clearOffset` to +`true` in that source's **Collector source properties**, then apply the changes. In the +pipeline configuration, the property is +`sources..advanced.riotx.clearOffset`. The Snowflake `snapshot` property +controls whether the collector reloads existing rows: + +- `INITIAL` (default): Load current source rows, then capture new changes. +- `NEVER`: Skip existing rows and capture only new changes. + +Clearing the offset does not replay missed events or remove stale target records. +After recovery, set `clearOffset` back to `false` so later collector starts preserve +the saved position. See [Edit collector properties](#edit-collector-properties). + +To resume a source, open its **More actions** menu and select **Start source**, then confirm with **Start source**. A source processes data only while the pipeline is running. See [How does stopping sources affect billing?](/content/operate/rc/rdi/faq.md#stopping-and-billing). ### Reset a source {#reset-source} -Resetting one source starts a new snapshot and reprocesses its selected data. +Resetting one source clears its saved position. With the default `initial` mode, a Debezium source takes a new snapshot and reprocesses its selected data. Other [snapshot modes](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) can skip existing rows or omit ongoing change capture. 1. On **Dashboard**, open the source card's **More actions** menu. 1. Select **Reset source**. @@ -75,7 +93,7 @@ Resetting one source starts a new snapshot and reprocesses its selected data. ![Reset source confirmation explaining that other sources retain their data and the pipeline temporarily stops.](/images/rc/rdi/rdi-2-reset-source.png) {width="600px"} -The reset clears that source's internal RDI streams, offsets, schema history, rejected records, and processing counters. The whole pipeline and all its sources restart during the reset. Other sources keep their saved positions and resume streaming. All records already in the shared target database remain, including those from the reset source. The new snapshot can overwrite records for that source. See [What happens when I reset one source?](/content/operate/rc/rdi/faq.md#reset-one-source). +The reset clears that source's internal RDI streams, offsets, schema history, rejected records, and processing counters. The whole pipeline and all its sources restart during the reset. Other sources keep their saved positions. Debezium sources restart according to their snapshot modes; for example, `always` mode takes another snapshot even when its saved position is preserved. All records already in the shared target database remain, including those from the reset source. The new snapshot can overwrite records for that source. See [What happens when I reset one source?](/content/operate/rc/rdi/faq.md#reset-one-source). ### Remove a source {#remove-source} @@ -100,7 +118,7 @@ You can change the target to another database in the same subscription. 1. Select the new target database. 1. Confirm with **Change target**. -All sources share the new target. Changing the target restarts the pipeline but does not automatically re-ingest existing records. [Reset the pipeline](#reset-data-pipeline) if you need to copy the existing source data to the new target. +All sources share the new target. Changing the target restarts the pipeline. Whether a Debezium source reloads existing rows depends on its [snapshot mode](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode). To copy existing source data to the new target, select modes that take data snapshots, then [reset the pipeline](#reset-data-pipeline). ## Metrics @@ -210,7 +228,7 @@ Select a source in the **Sources** list to view the schemas, tables, columns, an 1. Select **Save changes**. 1. Review the restart warning and select **Apply and restart**. -Dataset changes belong to the selected source. Applying changes restarts the pipeline and can temporarily interrupt processing. Use [Reset a source](#reset-source) when you need a new snapshot of that source. +Dataset changes belong to the selected source. Applying changes restarts the pipeline and can temporarily interrupt processing. To reload existing rows, select a snapshot mode that takes a data snapshot, then use [Reset a source](#reset-source). ## Transformations @@ -231,12 +249,12 @@ The Flink processor accepts lists and `regex:` entries in source selection patte ## Reset data pipeline -Resetting the whole pipeline clears its internal RDI state for all sources. A running pipeline restarts and takes a new snapshot for every source. A stopped pipeline remains stopped until you [start it](#stop-and-restart-data-pipeline). RDI reprocesses the selected data using the current transformations. Reset alone does not delete records from the target database. See [What happens when I reset the pipeline?](/content/operate/rc/rdi/faq.md#reset-pipeline). +Resetting the whole pipeline clears its internal RDI state for all sources. A running pipeline restarts each source according to its configured initial data load settings. With the default `initial` mode, Debezium sources take new snapshots. Other [snapshot modes](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) can skip existing rows or omit ongoing change capture. A stopped pipeline remains stopped until you [start it](#stop-and-restart-data-pipeline). When a source takes a data snapshot, RDI reprocesses the selected data using the current transformations. Reset alone does not delete target records, and a new snapshot can [leave stale records after missed deletes](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#missed-deletes). See [What happens when I reset the pipeline?](/content/operate/rc/rdi/faq.md#reset-pipeline). 1. Open **Pipeline actions** and select **Reset pipeline**. 1. Review the confirmation and select **Reset data pipeline**. -To re-run the snapshot for just one source while preserving other sources' internal data, see [Reset a source](#reset-source). +To clear the saved position of one source while preserving other sources' internal data, see [Reset a source](#reset-source). ## Stop and restart data pipeline diff --git a/layouts/shortcodes/embed-md.html b/layouts/shortcodes/embed-md.html index be2deec933..7d79f4d15d 100644 --- a/layouts/shortcodes/embed-md.html +++ b/layouts/shortcodes/embed-md.html @@ -2,4 +2,7 @@ {{ $page := .Site.GetPage $file }} {{ with $page }} {{ .Content }} -{{ end }} \ No newline at end of file + {{ if .Store.Get "hasMermaid" }} + {{ $.Page.Store.Set "hasMermaid" true }} + {{ end }} +{{ end }}