RDSC-6045: Document draining streams before Flink processor migration by ZdravkoDonev-redis · Pull Request #4017 · redis/docs

ZdravkoDonev-redis · 2026-09-16T07:37:55Z

Switching an existing pipeline from the classic processor to the Flink processor while input streams still contain records can leave records unprocessed. Consumer-group pending counts and lag can be zero while physical stream entries still remain.

This PR documents a processor migration that can be completed on RDI 1.19.0:

Disable source collection and explicitly keep the classic processor running.
Confirm that every input stream has length zero in three complete checks, five seconds apart.
Set processors.type: flink, resume collection, and deploy the complete configuration.
Verify that Flink is healthy and source changes made during the pause reach the target.

The RDI version upgrade is a separate procedure. The upgrade page links to this guide and requires an explicit processor type before upgrading to 2.0.0.

Tracking: https://redislabs.atlassian.net/browse/RDSC-6045

Validation

Passed: Hugo 0.143.1 build and rendered-page inspection. Both changed pages render, all seven step anchors exist, YAML examples are valid, and git diff --check passes. Existing unrelated repository warnings remain.
Passed: 14 local qualification-runner tests. The runner is outside this docs PR.
Passed: three independent RDI 1.19.0 cases: explicit classic with 520 queued records, omitted processor type with 665 queued records, and already-empty streams. The cases recorded 154, 154, and 153 passing checks, with no failed checks.
Passed in all three cases: collection stopped, the classic processor drained all eight input streams, three zero-length checks passed, the combined Flink/source-resume deployment succeeded, and writes made during the pause reached the target.
Passed in all three cases: exact target values, inserts, updates, deletes, ledger IDs, source-position continuity, no unexpected snapshot, completed Flink checkpoints, and DLQ preservation.
Passed after upgrading each case to candidate 0.0.202609101145: exact target comparison, empty input streams, completed checkpoints, and ten further minutes of writes.
Passed: direct SCAN/XLEN, packaged redis-di describe default, and raw classic-processor metrics reported the same eight-stream inventory and compatible physical lengths. The guide explains that the CLI Pending field is a stream length, not an XPENDING count.
Passed: Redis Insight Browser returned all eight input streams. Stream Data showed Entries: 0 after three manual refreshes five seconds apart. Its built-in CLI returned the same eight keys with SCAN, and XLEN returned 0.
Passed: the combined deployment showed no classic-processor and resumed-collector overlap, so a separate switch-then-resume sequence was not needed.

The tested upgrade artifact is 0.0.202609101145, the agreed 2.0.0 candidate. This PR does not claim validation of the final 2.0.0 release.

Related work

PR #3916 changes some of the same processor-default and rollback text. Keep explicit processor selection and the stream-drain procedure when resolving conflicts.

* Update content/operate/rc/api/api-reference/openapi.json * Update content/operate/rc/api/api-reference/openapi.json --------- Co-authored-by: redisdocsapp[bot] <177626021+redisdocsapp[bot]@users.noreply.github.com>

* initial draft * crop image * edits and questions * add in missing info * edits * review feedback changes * 2026.9.5 RN

* Let scrollbar clicks pass through the Qualified widget strip On production the Qualified chat widget (injected via GTM, not part of this repo) restyles <html> to overflow:hidden so <body> becomes the page's scroll container, and parks an invisible ~20px-wide, full-height, max-z-index iframe (#q-messenger-frame) against the right edge of the viewport — exactly where body's scrollbar sits. Chrome hit-tests its scrollbars before page content, but Firefox hit-tests content first, so the invisible strip swallowed every click and hover on the scrollbar: it never reacted and, with overlay scrollbars, never faded in. Reported as 'the scrollbar doesn't show up on Firefox'; never reproduces on hugo server because GTM only runs on production. Disable pointer events on the strip while the chat is closed. The widget sets the q-docked attribute on <html> whenever the panel is open (on mobile too, where the docked width is 0), so scoping the override to html:not([q-docked]) hands pointer events back to the open panel. Verified by hit-testing (elementFromPoint) a repro page built from the widget's captured stylesheet plus a simulated messenger iframe: without the fix the strip captures the point next to the scrollbar; with the fix clicks pass through; with q-docked set the frame captures again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Keep the page scrollbar visible under the Qualified widget Reviewers on DOC-7044 still saw the scrollbar flash at load and vanish. The pointer-events fix in the previous commit was real but secondary; the disappearance itself has a different mechanism. The Qualified widget replaces root scrolling with a <body> scroller. At init it measures the root scrollbar width N, then applies html { width: calc(100dvw - N); margin-right: N; overflow: hidden } and body { width: calc(100% + N) }, so that body's scrollbar lands exactly where the root scrollbar was. With overlay scrollbars N = 0 and nothing breaks, which is why this never reproduced on trackpad Macs. With classic always-visible scrollbars (Windows, or macOS with a mouse) N is ~15-17px: body overshoots html's box by N, and html's own overflow: hidden clips body's scrollbar out of view. The root scrollbar dies when the widget loads and its replacement is never visible. While the chat is closed (html[q-docked-target]:not([q-docked])), neutralize the compensation: html back to 100dvw with no margin, body back to 100%. Body then fills the window and its scrollbar sits at the window edge, unclipped. While the panel is docked open the widget's own geometry applies untouched — verified html still shrinks by the panel width. Also scope our html { scrollbar-gutter: stable } to auto while the widget is active: a reserved gutter inside the widget's overflow-hidden html box would push body's scrollbar inboard again. Verified against the live staging build in Firefox by injecting these rules at document start: body flush with the window, a real mouse drag on the thumb at page top scrolls (previously dead), wheel scrolling unaffected, and docked mode still lays out correctly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Co-authored-by: redisdocsapp[bot] <177626021+redisdocsapp[bot]@users.noreply.github.com>

* make OLM distinction * clarify versions

* helm chart link fix * postgres 16 change, radar over mcm * remove checksums from download center steps * bugbot fixes

* RS: fix incorrect alerts email link (DOC-6925) The "database and cluster alerts" link in the send-alerts-by-email procedure pointed at security/access-control/manage-users, which documents password complexity, expiration, lockouts, and account inactivity timeout. It says nothing about alerts. Point it at create-users#add-users instead, the section that documents selecting which database and cluster alerts each user receives by email. Fixed in current content only: references/alerts/_index.md (the page on the ticket) and monitoring/v1_monitoring.md, which carries the same procedure. The versioned snapshots under 7.4/, 7.8/, 7.22/, and 8.0/ have the same wrong link and are left as-is. * RS: fix alerts email link in 8.0 snapshot (DOC-6925) Applies the same create-users#add-users correction to the 8.0 snapshot, which is the current snapshot readers are most likely on. The 7.4, 7.8, and 7.22 snapshots carry the same wrong link and are left as-is, per the versioned-content convention in content/operate/rs/AGENTS.md.

Add AISIX to Redis AI ecosystem integrations

github-actions · 2026-09-16T07:38:13Z

Staging links:
https://redis.io/docs/staging/RDSC-6045-flink-migration-drain/integrate/redis-data-integration/installation/migration-classic-to-flink/
https://redis.io/docs/staging/RDSC-6045-flink-migration-drain/integrate/redis-data-integration/installation/upgrade/

github-actions · 2026-09-16T07:38:14Z

🧠 Redis Memory

Found 8 related items from repository history (1 new this commit):

PR — RDSC-4603 Update RDI public documentation

See https://redislabs.atlassian.net/browse/RDSC-4603 and https://github.com/RedisLabs/redis-data-integration/pull/2565
PR — RDSC-5692 Update public docs that Flink processor is GA

See https://redislabs.atlassian.net/browse/RDSC-5692
PR — RDSC-6038: Add RDI 2.0.0 release notes

Adds the September 2026 RDI 2.0.0 release notes and changes rdi_current_version from 1.19.1 to 2.0.0. The site currently has no 2.0.0
PR — RDSC-6038: Update Redis Cloud RDI documentation for 2.0.0

Updates the Cloud RDI guides for the RDI 2.0.0 console. The current guides describe one source and the older five-step setup. The updated gu
PR — RDSC-5718 Update RDI Flink VM and WAIT documentation

What changed
Commit — RDSC-6045: Document draining streams before Flink processor migration

RDSC-6045: Document draining streams before Flink processor migration
Commit — RDSC-6045: Separate processor migration from version upgrade

RDSC-6045: Separate processor migration from version upgrade
PR — RDI 1.19.0 release notes 🆕

Summary

Memory updated at c9e8a95

stoyanr · 2026-09-16T11:23:54Z

+Run:
+
+```bash
+redis-di describe default


Suggested change

redis-di describe default

redis-di describe

stoyanr · 2026-09-16T11:25:03Z

+
+Wait for the deployment to finish and the source collector to stop. Keep
+the pipeline active so the classic processor can process the remaining input records.
+Do not use `redis-di stop` for this step, because it also stops the processor.


Suggested change

Do not use `redis-di stop` for this step, because it also stops the processor.

stoyanr · 2026-09-16T11:25:29Z

+Applications can continue writing to the source database while collection
+is disabled. Make sure the database change log retains the whole paused
+interval. When the collector restarts, it resumes from the saved source
+position and processes those changes.


Suggested change

Applications can continue writing to the source database while collection

is disabled. Make sure the database change log retains the whole paused

interval. When the collector restarts, it resumes from the saved source

position and processes those changes.

Applications can continue writing to the source database while collection

is disabled. When the collector restarts, it resumes from the saved source

position and processes those changes.

stoyanr · 2026-09-16T11:27:06Z

+### Check with Redis Insight
+
+1.  Connect Redis Insight to the RDI database and open **Browse**.
+1.  Filter by the pipeline's input stream pattern. For the default pipeline,
+    use `data:{rdi}:*`. Confirm that all input streams are listed.
+1.  Open each stream, select **Stream Data**, and use the refresh button.
+    Confirm that **Entries** is `0`.
+1.  Repeat the complete inventory and entry check three times, five seconds
+    apart.
+
+You can also open the built-in **CLI** and run the `SCAN` and `XLEN` commands
+shown above. The Browser and CLI results must contain the same streams and
+lengths.
+
+If records remain, keep the classic processor running and resolve its
+processing errors before continuing. Do not delete stream entries, reset the
+pipeline, or move consumer-group positions to make the count reach `0`.


Suggested change

### Check with Redis Insight

1. Connect Redis Insight to the RDI database and open **Browse**.

1. Filter by the pipeline's input stream pattern. For the default pipeline,

use `data:{rdi}:*`. Confirm that all input streams are listed.

1. Open each stream, select **Stream Data**, and use the refresh button.

Confirm that **Entries** is `0`.

1. Repeat the complete inventory and entry check three times, five seconds

apart.

You can also open the built-in **CLI** and run the `SCAN` and `XLEN` commands

shown above. The Browser and CLI results must contain the same streams and

lengths.

If records remain, keep the classic processor running and resolve its

processing errors before continuing. Do not delete stream entries, reset the

pipeline, or move consumer-group positions to make the count reach `0`.

stoyanr · 2026-09-16T11:28:04Z

+In the **Statistics** table, the **Pending** value for each classic processor
+stream is its current length. Confirm that every input stream is listed and
+that the values agree with the Redis command checks below. This **Pending**
+value is different from consumer-group pending entries. `XPENDING` or group
+lag of `0` alone does not prove that a stream is empty.


Suggested change

In the **Statistics** table, the **Pending** value for each classic processor

stream is its current length. Confirm that every input stream is listed and

that the values agree with the Redis command checks below. This **Pending**

value is different from consumer-group pending entries. `XPENDING` or group

lag of `0` alone does not prove that a stream is empty.

In the **Statistics** table, the **Pending** value for each classic processor

stream is its current length. Confirm that every input stream is listed.

stoyanr · 2026-09-16T11:29:31Z


-Then redeploy the pipeline. The operator stops the classic processor pods
-and starts the Flink JobManager and TaskManager workloads for the pipeline.
+RDI 1.19.0 requires this setting because its default processor is `classic`.


Suggested change

RDI 1.19.0 requires this setting because its default processor is `classic`.

RDI 1.19.x requires this setting because its default processor is `classic`. On RDI 2.0.0 the default processor is `flink`, so this setting is no longer required.

github-actions Bot and others added 13 commits September 14, 2026 06:25

update rc openapi spec (#3995)

492e76d

* Update content/operate/rc/api/api-reference/openapi.json * Update content/operate/rc/api/api-reference/openapi.json --------- Co-authored-by: redisdocsapp[bot] <177626021+redisdocsapp[bot]@users.noreply.github.com>

Radar: connect with agents (#3955)

cbdb948

* initial draft * crop image * edits and questions * add in missing info * edits * review feedback changes * 2026.9.5 RN

OP: add sundry release notes (#3999)

72c6155

RS: add -65 build bug fixes to -78 RN (#4000)

2346af8

Add AISIX to Redis AI ecosystem integrations

37ed611

Update content/operate/rc/api/api-reference/openapi.json (#4005)

35d1366

Co-authored-by: redisdocsapp[bot] <177626021+redisdocsapp[bot]@users.noreply.github.com>

K8s: make OLM distinction in limitation (#3975)

57c2c73

* make OLM distinction * clarify versions

Radar: install/agent page fixes (#4010)

a0c45aa

* helm chart link fix * postgres 16 change, radar over mcm * remove checksums from download center steps * bugbot fixes

Merge pull request #4007 from Yilialinn/docs/add-aisix-redis-integration

d1b43b9

Add AISIX to Redis AI ecosystem integrations

OP: delete FT.DROPINDEX from A-A unsupported cmds page (#4013)

059c44c

RDSC-6045: Document draining streams before Flink processor migration

93e57c8

ZdravkoDonev-redis self-assigned this Sep 16, 2026

RDSC-6045: Separate processor migration from version upgrade

bb55b9e