Repository navigation
Clarify RDI snapshot modes and missed deletes - #4238
ZdravkoDonev-redis wants to merge 6 commits into
Conversation
🧠 Redis MemoryFound 9 related items from repository history:
Memory updated at d87ebb8 |
andy-stark-redis
left a comment
andy-stark-redis
left a comment
There was a problem hiding this comment.
Some suggestions (mainly adding the word "mode" after always, initial, etc) and I'd prefer not to use that new shortcode unless you really feel it's necessary. Otherwise looks good, so I'll approve to avoid holding you up.
| | `initial` (default) | Snapshot when no offset exists, then stream changes. | Resume after a completed snapshot if the position is valid. No automatic snapshot recovery for unavailable log history. | Snapshot, then stream changes. | | ||
| | `when_needed` | Snapshot when no offset exists, then stream changes. | Resume if the position is valid. Snapshot again if the connector detects an unavailable position. | Snapshot, then stream changes. | | ||
| | `always` | Snapshot on every collector start, then stream changes. | Take a new snapshot even if the position is valid. | Snapshot, then stream changes. | | ||
| | `initial_only` | Take an initial snapshot without streaming subsequent changes. | Restart an interrupted initial snapshot; do not capture subsequent changes after it completes. | Snapshot without ongoing change capture. | |
There was a problem hiding this comment.
It may be worth noting that using initial_only will cause the collector to enter a crash loop upon completion, causing the source to appear as "not connected" in monitoring.
There was a problem hiding this comment.
Why would initial_only cause a crash loop on its own?
There was a problem hiding this comment.
With initial_only Debezium performs the snapshot, stores the offset at which the snapshot is made, and then shuts down. K8s will restart the pods after it exits, causing it to start again, see that snapshot has been completed and shutdown again, this way entering a loop starting, checking and shutting down.
| Stopping one source pauses data capture from that source without stopping the other sources. When you start a source again, its behavior depends on the source type: | ||
|
|
||
| - Debezium sources follow their [snapshot mode](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) and the availability of their saved position. | ||
| - Snowflake sources use RIOT-X and Snowflake Streams. They resume from the saved stream offset when the stream retains the required changes. If the retention window expires while the source is stopped, changes can be lost. Set `clearOffset: true` to restart from the current state. |
There was a problem hiding this comment.
I am not familiar with this clearOffset: true. Where and how do you set it? Does restart from the current state mean "snapshot and start" or "continue CDC from now and ignore missed events"
| | `initial` (default) | Snapshot when no offset exists, then stream changes. | Resume after a completed snapshot if the position is valid. No automatic snapshot recovery for unavailable log history. | Snapshot, then stream changes. | | ||
| | `when_needed` | Snapshot when no offset exists, then stream changes. | Resume if the position is valid. Snapshot again if the connector detects an unavailable position. | Snapshot, then stream changes. | | ||
| | `always` | Snapshot on every collector start, then stream changes. | Take a new snapshot even if the position is valid. | Snapshot, then stream changes. | | ||
| | `initial_only` | Take an initial snapshot without streaming subsequent changes. | Restart an interrupted initial snapshot; do not capture subsequent changes after it completes. | Snapshot without ongoing change capture. | |
There was a problem hiding this comment.
With initial_only Debezium performs the snapshot, stores the offset at which the snapshot is made, and then shuts down. K8s will restart the pods after it exits, causing it to start again, see that snapshot has been completed and shutdown again, this way entering a loop starting, checking and shutting down.
3 participants
When a saved source log position expires,
when_neededcan take a new snapshot and resume collection while leaving deleted rows in Redis. The current docs do not explain this gap or the connector differences between snapshot modes.This change compares all six listed modes for Debezium sources and adds a shared Mermaid timeline to the latest and v1.19.1 configuration guides. It updates the configuration references, deployment/reset guidance, troubleshooting, multiple-source guidance, and Cloud pages. Reset and target cleanup remain separate. The embed shortcode now enables Mermaid rendering in the parent page.
The Cloud FAQ includes a six-event timeline with colored markers and alternating labels on wide screens. It switches to a vertical timeline when the content area is narrow. The diagram uses semantic HTML and CSS, with no JavaScript dependency.
The Architecture guarantee remains unchanged, with an open product-review note about expired history and missed deletes.
Validation:
no_data.Companion schema-description PR: https://github.com/RedisLabs/redis-data-integration/pull/2886
Context: Slack discussion.
Note
Low Risk
Documentation-only changes that clarify existing CDC behavior; no runtime or configuration schema changes in this repo.
Overview
Documents Debezium
snapshot.modefor RDI CDC sources and the stale-target risk when source log history expires before the collector resumes.Adds shared embeds (
rdi-snapshot-modes.md,rdi-snapshot-gap.md) with a mode comparison table, Debezium 3.5 connector notes, and a Mermaid missed-delete timeline. Pipeline config gains a Choose a snapshot mode section; architecture adds a note that at-least-once delivery does not cover replay of expired history.Reset / start / stop wording is corrected:
redis-di resetclears saved positions without changingsnapshot.modeor flushing Redis; recovery behavior depends on the configured mode. New troubleshooting covers unavailable source log history; multiple-source and Redis Cloud (define, FAQ, view-edit) pages are aligned.The
embed-mdshortcode now propagateshasMermaidso embedded diagrams render on parent pages. Config reference text forsnapshot.modeis expanded and linked to the new guide.Reviewed by Cursor Bugbot for commit d87ebb8. Bugbot is set up for automated code reviews on this repo. Configure here.