Clarify RDI snapshot modes and missed deletes

content/embeds/rdi-snapshot-gap.md

@@ -0,0 +1,30 @@
+A new snapshot copies existing source rows. It does not replay missed deletes or remove
+target records for rows absent from the snapshot.
+
+The timeline shows one row whose delete event is lost before collection resumes.
+Time progresses from top to bottom.
+
+```mermaid
+%%{init: {"sequence": {"actorMargin": 15, "diagramMarginX": 5, "width": 90, "mirrorActors": false}}}%%
+sequenceDiagram
+    participant Source as Source<br/>database
+    participant RDI as RDI<br/>collector
+    participant Redis as Target<br/>Redis
+
+    Source->>RDI: Capture<br/>row 42
+    RDI->>Redis: Write<br/>row 42
+    Note over RDI: Collection stops
+    Note over Source: Delete row 42
+    Note over Source,RDI: Required log<br/>history expires
+    RDI->>Source: Restart with<br/>when_needed
+    Note over Source,RDI: Saved position unavailable<br/>New snapshot
+    Source->>RDI: Current rows<br/>(without row 42)
+    RDI->>Redis: Write snapshot<br/>rows
+    Note over Redis: Row 42 is stale<br/>Delete missed
+```
+
+An interruption alone does not cause this gap. If the saved position and the required
+log history are still available, the collector can resume and capture the delete.
+In this example, the history expires before the collector resumes, and the new snapshot
+contains no row 42 to overwrite or delete its target record. Streaming resumes after
+the snapshot, but the stale record can remain in Redis.

content/embeds/rdi-snapshot-modes.md

@@ -0,0 +1,63 @@
+The collector saves an *offset*, which is its position in the source's change history.
+A missing offset means there is no saved position, as on first deployment or after a
+reset. An unavailable position means an offset exists, but the source no longer has
+the log history needed to resume from it.
+
+With the default `initial` mode, the collector snapshots when there is no saved
+position, then streams changes. A restart with a completed snapshot and a valid saved
+position resumes streaming. Deployment or restart alone does not require a new snapshot.
+If the saved position is no longer available, `initial` does not automatically
+re-snapshot to recover, and the collector stops and reports an error.
+
+With `when_needed` mode, the collector also takes a new snapshot when it detects that the
+saved position is no longer available. This can reload all selected data after an
+outage without a manual reset. The snapshot captures current rows, rather than
+reconstructing every missed change.
+
+If the collector restarts during a snapshot, the snapshot starts again. Rows deleted
+while the snapshot is interrupted can remain as stale records in the target because
+the new snapshot does not replay their delete events.
+
+Use `when_needed` with monitoring and alerting. Recovery can start a long-running
+snapshot, which can increase latency and leave stale target records until recovery
+finishes.
+
+#### Missed deletes can leave stale target records {#missed-deletes}
+
+{{< embed-md "rdi-snapshot-gap.md" >}}
+
+#### Compare snapshot modes
+
+The available modes and their exact behavior depend on the source connector and its
+version. The following table describes the six values listed in the RDI configuration
+reference. A listed value is not necessarily supported by every connector.
+
+| Mode | Existing source data | Restart with a saved position | After a reset clears the position |
+| :-- | :-- | :-- | :-- |
+| `initial` (default) | Snapshot when no offset exists, then stream changes. | Resume after a completed snapshot if the position is valid. No automatic snapshot recovery for unavailable log history. | Snapshot, then stream changes. |
+| `when_needed` | Snapshot when no offset exists, then stream changes. | Resume if the position is valid. Snapshot again if the connector detects an unavailable position. | Snapshot, then stream changes. |
+| `always` | Snapshot on every collector start, then stream changes. | Take a new snapshot even if the position is valid. | Snapshot, then stream changes. |
+| `initial_only` | Take an initial snapshot without streaming subsequent changes. | Restart an interrupted initial snapshot; do not capture subsequent changes after it completes. | Snapshot without ongoing change capture. |
+| `no_data` | Skip existing rows. Initialize any required schema or source position, then stream changes. | Resume streaming if the position is valid. No data snapshot to recover unavailable history. | Skip existing rows and start streaming according to the connector's rules. |
+| `never` | Skip the data snapshot where this mode is supported. | Resume according to the connector's rules; no data snapshot recovery. | Connector-specific startup behavior; existing rows are not reloaded. |
+
+An interrupted initial snapshot can restart when the collector starts again, including
+with `initial` and `initial_only`.
+
+For Debezium 3.5, note these connector differences:
+
+- PostgreSQL `no_data` performs no snapshot. It resumes from a saved log sequence
+  number (LSN), or from the replication slot's creation position when no LSN is saved.
+  This requires the relevant write-ahead log (WAL) history to remain available.
+- Other connectors can capture schema or initialize a position without copying
+  existing rows in `no_data` mode.
+- `never` is a deprecated alias for `no_data` in PostgreSQL and MongoDB. MySQL and
+  MariaDB retain `never` with connector-specific prerequisites. Oracle and SQL Server
+  do not offer `never`; use `no_data` to skip existing rows.
+
+See the Debezium 3.5 connector references for [MariaDB](https://debezium.io/documentation/reference/3.5/connectors/mariadb.html),
+[MongoDB](https://debezium.io/documentation/reference/3.5/connectors/mongodb.html),
+[MySQL](https://debezium.io/documentation/reference/3.5/connectors/mysql.html),
+[Oracle](https://debezium.io/documentation/reference/3.5/connectors/oracle.html),
+[PostgreSQL](https://debezium.io/documentation/reference/3.5/connectors/postgresql.html),
+and [SQL Server](https://debezium.io/documentation/reference/3.5/connectors/sqlserver.html).

content/integrate/redis-data-integration/1.19.1/architecture/_index.md

@@ -80,6 +80,15 @@ are [*idempotent*](https://en.wikipedia.org/wiki/Idempotence) (that is
 to say that all writes after the first one make no change to the
 overall state).
 
+{{< note >}}
+The delivery guarantee applies to change events captured by RDI. If a Debezium
+collector's saved source position becomes unavailable, a new snapshot cannot replay
+changes whose source log history has expired. Missed deletes can leave stale records
+in the target. See the
+[missed-delete timeline](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#missed-deletes)
+and [recovery guidance](/content/integrate/redis-data-integration/1.19.1/troubleshooting.md#unavailable-source-history).
+{{< /note >}}
+
 ## Checkpointing
 
 RDI uses Redis streams to store the sequence of change events

content/integrate/redis-data-integration/1.19.1/data-pipelines/deploy.md

@@ -324,7 +324,10 @@ Use [`redis-di stop`](/content/integrate/redis-data-integration/1.19.1/reference
 to pause a running pipeline and
 [`redis-di start`](/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-start.md)
 to resume it. Stopping a pipeline halts data processing without deleting the pipeline or its
-configuration, so you can start it again later from where it left off.
+configuration. A Debezium source restarts according to its
+[snapshot mode](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#choose-a-snapshot-mode)
+and the availability of its saved source position.
+If you keep the source stopped for long periods, the stored offset may be removed, making it impossible to resume streaming.
 
 ```bash
 redis-di stop
@@ -334,10 +337,15 @@ redis-di start
 ## Reset a pipeline
 
 Use [`redis-di reset`](/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset.md)
-to return a pipeline to initial full-sync mode. This reloads a fresh
-[snapshot](/content/integrate/redis-data-integration/1.19.1/architecture/_index.md#overview) of the source
-data and then resumes change data capture (CDC), which is useful when the source and target have
-drifted out of sync.
+to clear the pipeline's saved source positions. With the default `initial` snapshot
+mode, a Debezium collector reloads selected source data and then resumes change data capture
+(CDC). Reset does not change the configured
+[snapshot mode](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#choose-a-snapshot-mode).
+Modes that skip existing rows do not reload them after a reset, and `initial_only`
+does not capture subsequent changes.
+
+Reset does not flush the target Redis database. A new snapshot can overwrite target
+records, but [missed deletes can leave stale records](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#missed-deletes).
 
 ```bash
 redis-di reset
@@ -381,4 +389,4 @@ redis-di set-secret SOURCE_DB_USERNAME newUsername --wait=false
 redis-di set-secret SOURCE_DB_PASSWORD newPassword
 ```
 
-The same applies to any set of changes that are only valid together.
+The same applies to any set of changes that are only valid together.

content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md

@@ -160,6 +160,51 @@ configuration contains the following data:
   - `java_options`: controls the JAVA_OPTS environment variable (for RDI 1.15.1 and above). Use it to modify the default values for Java heap size and other Java options for the Debezium server.
     For example, set it to `"-Xmx2g -Xms512m"` to set the maximum heap size to 2 GB and the initial heap size to 512 MB.
 
+### Choose a snapshot mode {#choose-a-snapshot-mode}
+
+Use `sources.<source-name>.advanced.source.snapshot.mode` to control when a
+Debezium collector loads existing source data before streaming changes. This property
+applies to sources with `type: cdc`, including MariaDB, MongoDB, MySQL, Oracle,
+PostgreSQL, and SQL Server. It does not configure Spanner, Snowflake, or external
+collectors.
+
+Add the property under the source you want to configure. For example, this file
+uses `initial` for a MySQL source:
+
+```yaml
+sources:
+  mysql:
+    type: cdc
+    connection:
+      type: mysql
+      host: <your-source-host>
+      port: 3306
+      user: ${SOURCE_DB_USERNAME}
+      password: ${SOURCE_DB_PASSWORD}
+    advanced:
+      source:
+        snapshot.mode: initial
+
+targets:
+  target:
+    connection:
+      type: redis
+      host: <your-target-host>
+      port: 6379
+      password: ${TARGET_DB_PASSWORD}
+```
+
+RDI 1.19.1 bundles Debezium 3.5. The comparison and connector notes in this
+section apply to that release.
+
+{{< embed-md "rdi-snapshot-modes.md" >}}
+
+A reset clears saved positions without changing `snapshot.mode` or flushing the target
+Redis database. If you need to reload existing rows, select a mode that takes a data
+snapshot. A snapshot can overwrite target records, but it does not remove stale records
+for rows that are absent from the source. See
+[Recover from unavailable source log history](/content/integrate/redis-data-integration/1.19.1/troubleshooting.md#unavailable-source-history).
+
 ### Targets
 
 Use this section to provide the connection details for the target Redis

content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset.md

@@ -8,9 +8,13 @@ categories: ["redis-di"]
 url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset/'
 ---
 
-Resets a pipeline into initial full-sync mode, so it reloads a snapshot of the source data before
-resuming change data capture. By default, the command waits for the pipeline to reach a terminal
-state before returning.
+Clears a pipeline's saved source positions without changing its configured snapshot mode
+or flushing the target Redis database. With the default `initial` mode, the collector
+takes a new data snapshot and then resumes change data capture (CDC). Other modes can
+skip existing rows or omit ongoing change capture. See
+[Choose a snapshot mode](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#choose-a-snapshot-mode).
+
+By default, the command waits for the pipeline to reach a terminal state before returning.
 
 ## Usage
 

content/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference.md

@@ -459,7 +459,7 @@ redis.wait.retry.delay.ms: 1000
 <a name="sourcesadvancedsource"></a>
 #### sources\.advanced\.source: Advanced source settings
 
-Advanced configuration properties for the source database connection and CDC behavior. **Applies to the `cdc` and `flink` collector types.**<br/><br/>For the `cdc` collector type, available properties depend on the source database — refer to the relevant Debezium connector documentation: [MySQL](https://debezium.io/documentation/reference/stable/connectors/mysql.html), [MariaDB](https://debezium.io/documentation/reference/stable/connectors/mariadb.html), [PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html), [Oracle](https://debezium.io/documentation/reference/stable/connectors/oracle.html), [SQL Server](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html), [Db2](https://debezium.io/documentation/reference/stable/connectors/db2.html), [MongoDB](https://debezium.io/documentation/reference/stable/connectors/mongodb.html). When using a property from those pages, omit the `debezium.source.` prefix.<br/><br/>**The named properties below cover the most commonly tuned settings: `spanner.*` properties apply to the `flink` collector type, all others apply to the `cdc` collector type. Any other property from the Debezium documentation can still be set as a free-form key-value pair.**
+Advanced configuration properties for the source database connection and CDC behavior. **Applies to the `cdc` and `flink` collector types.**<br/><br/>For the `cdc` collector type, available properties depend on the source database — refer to the relevant Debezium connector documentation: [MySQL](https://debezium.io/documentation/reference/stable/connectors/mysql.html), [MariaDB](https://debezium.io/documentation/reference/stable/connectors/mariadb.html), [PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html), [Oracle](https://debezium.io/documentation/reference/stable/connectors/oracle.html), [SQL Server](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html), [MongoDB](https://debezium.io/documentation/reference/stable/connectors/mongodb.html). When using a property from those pages, omit the `debezium.source.` prefix.<br/><br/>Set `snapshot.mode` as a free-form source property to control data snapshots. The default is `initial`; available modes and restart behavior depend on the connector. A reset clears the saved position without changing the mode. See [Choose a snapshot mode](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#choose-a-snapshot-mode), including the [missed-delete timeline](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#missed-deletes).<br/><br/>**The named properties below cover the most commonly tuned settings: `spanner.*` properties apply to the `flink` collector type, all others apply to the `cdc` collector type. Any other property from the Debezium documentation can still be set as a free-form key-value pair.**
 
 
 **Properties**

content/integrate/redis-data-integration/1.19.1/troubleshooting.md

@@ -57,6 +57,40 @@ log analysis tools can use.
 > [Backpressure mechanism](/content/integrate/redis-data-integration/1.19.1/architecture/_index.md#backpressure-mechanism)
 > in the Architecture guide for more information.
 
+## Recover from unavailable source log history {#unavailable-source-history}
+
+If the collector cannot resume from its saved position, check the collector source log
+for unavailable or expired source log history. This differs from a missing offset,
+which is expected on first deployment or after a reset.
+
+With `initial` mode, the collector can fail rather than start a new snapshot when the saved
+position is unavailable. With `when_needed` mode, it can automatically snapshot the current
+selected data and resume streaming. A new snapshot after a restart can therefore
+indicate that the source history was no longer available. See
+[Choose a snapshot mode](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#choose-a-snapshot-mode)
+for the connector-specific behavior.
+
+Possible causes include:
+
+- The collector was stopped long enough for the source change log to expire.
+- A configuration change made the collector connect to another server with the same
+  data but different offsets.
+- A cleanup process removed the change log while RDI was running.
+- A crash and restore left the source without the saved offset.
+
+Before you use a new snapshot to recover, check for
+[missed deletes and stale target records](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#missed-deletes).
+A snapshot does not replay lost history or delete target records that are absent from
+the source. Resetting the pipeline also leaves target records in place.
+
+If the saved position is unavailable, work with Redis support to plan a target cleanup
+or rebuild before relying on the target as a complete copy. Check whether the target
+contains data from other sources or applications before deleting any records. To reload
+existing source rows, configure a mode that takes a data snapshot, then
+[reset the pipeline](/content/integrate/redis-data-integration/1.19.1/data-pipelines/deploy.md#reset-a-pipeline).
+Review the source's log retention so the required history remains available during
+future interruptions.
+
 ## Dump support package
 
 If you need to send a comprehensive set of forensics data to Redis support,

content/integrate/redis-data-integration/architecture/_index.md

@@ -80,6 +80,15 @@ are [*idempotent*](https://en.wikipedia.org/wiki/Idempotence) (that is
 to say that all writes after the first one make no change to the
 overall state).
 
+{{< note >}}
+The delivery guarantee applies to change events captured by RDI. If a Debezium
+collector's saved source position becomes unavailable, a new snapshot cannot replay
+changes whose source log history has expired. Missed deletes can leave stale records
+in the target. See the
+[missed-delete timeline](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#missed-deletes)
+and [recovery guidance](/content/integrate/redis-data-integration/troubleshooting.md#unavailable-source-history).
+{{< /note >}}
+
 ## Checkpointing
 
 RDI uses Redis streams to store the sequence of change events

content/integrate/redis-data-integration/data-pipelines/deploy.md

@@ -340,7 +340,9 @@ Use [`redis-di stop`](/content/integrate/redis-data-integration/reference/cli/re
 to pause a running pipeline and
 [`redis-di start`](/content/integrate/redis-data-integration/reference/cli/redis-di-start.md)
 to resume it. Stopping a pipeline halts data processing without deleting the pipeline or its
-configuration, so you can start it again later from where it left off.
+configuration. A Debezium source restarts according to its
+[snapshot mode](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode)
+and the availability of its saved source position.
 
 ```bash
 redis-di stop
@@ -360,10 +362,15 @@ Note that a source can only run if its parent pipeline is running. See
 ## Reset a pipeline
 
 Use [`redis-di reset`](/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md)
-to return a pipeline to initial full-sync mode. This reloads a fresh
-[snapshot](/content/integrate/redis-data-integration/architecture/_index.md#overview) of the source
-data and then resumes change data capture (CDC), which is useful when the source and target have
-drifted out of sync.
+to clear the pipeline's saved source positions. With the default `initial` snapshot
+mode, a Debezium collector reloads selected source data and then resumes change data capture
+(CDC). Reset does not change the configured
+[snapshot mode](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode).
+Modes that skip existing rows do not reload them after a reset, and `initial_only` mode
+does not capture subsequent changes.
+
+Reset does not flush the target Redis database. A new snapshot can overwrite target
+records, but [missed deletes can leave stale records](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#missed-deletes).
 
 ```bash
 redis-di reset