Skip to content

RDSC-5610: Multi-source documentation - #3916

Open
stoyanr wants to merge 23 commits into
redis:mainfrom
stoyanr:RDSC-5610-multi-source-docs
Open

stoyanr wants to merge 23 commits into
redis:mainfrom
stoyanr:RDSC-5610-multi-source-docs

Conversation

@stoyanr

@stoyanr stoyanr commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

See https://redislabs.atlassian.net/browse/RDSC-5610

Enhances #3841 with numerous changes and additions. I couldn't update the original PR since I don't have write permissions to this repository.

Note that it now documents 2 PRs that are not yet merged, but should be before the release:


Note

Low Risk
Documentation-only changes; operational risk is misconfiguration if published before the referenced RDI product changes ship.

Overview
Documents multi-source RDI pipelines: one pipeline can ingest from several source databases (each with its own collector, secrets, streams, and metrics), with a new Multiple sources in one pipeline guide covering naming, config.yaml, jobs (server_name), add/remove sources, per-source stop/start/reset, monitoring, and legacy naming after upgrade.

Secrets and deploy are rewritten around per-database keys (USERNAME, PASSWORD, TLS keys) and redis-di set-secret … --db <source|target>, with env vars like ${MYSQL_DB_PASSWORD}; K8s examples move from source-db to <source>-db. Legacy SOURCE_DB_* / TARGET_DB_* remain for single-source or pre-upgrade sources.

Product defaults and fit: architecture, FAQ, pipeline config, and upgrade pages state processors.type defaults to Flink (set classic explicitly to opt out), including a 2.0.0 upgrade warning that undeployed type pipelines move to Flink on next deploy. The “when to use” embed and decision tree drop the single-source-only gate and allow one or more sources.

Cross-cutting updates cover API v2 requirement for multi-source, DLQ/stream naming with source segments, Flink job matcher lists/regex:, observability per-collector URLs, HA/standby VM behavior, and refreshed database prep guides (MongoDB keystore, MySQL TLS, Snowflake/Spanner secrets, etc.).

Reviewed by Cursor Bugbot for commit 4c412ec. Bugbot is set up for automated code reviews on this repo. Configure here.

@andy-stark-redis andy-stark-redis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite a few suggestions but they are easily applied, so I'll approve to avoid delays.

Comment thread content/integrate/redis-data-integration/architecture/classic-vs-flink.md Outdated
Comment thread content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md Outdated
Comment thread content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md Outdated
Comment thread content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md Outdated
Comment thread content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md Outdated
Comment thread content/integrate/redis-data-integration/reference/cli/_index.md Outdated
Comment thread content/integrate/redis-data-integration/reference/cli/redis-di-describe.md Outdated
Comment thread content/integrate/redis-data-integration/reference/api-migration.md Outdated
Comment thread content/integrate/redis-data-integration/faq.md Outdated
Comment thread content/integrate/redis-data-integration/quick-start-guide.md Outdated
stoyanr and others added 15 commits September 8, 2026 14:16
…vs-flink.md

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
…e-dbs/_index.md

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
…e-dbs/snowflake.md

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
…e-dbs/snowflake.md

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
…e-dbs/spanner.md

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
…e-dbs/spanner.md

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Comment thread content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md Outdated
Comment thread content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md Outdated

@zhivko-kirishev-redis zhivko-kirishev-redis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything about the automatic population of the ssl properties as well as the default Flink processor looks good

@nicoweidner nicoweidner left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a few comments. We can tackle them in a followup so as not to block the release, none of them are critical

Comment thread content/integrate/redis-data-integration/data-pipelines/rejected-records.md Outdated
Comment thread content/integrate/redis-data-integration/data-pipelines/deploy.md Outdated
Comment thread content/integrate/redis-data-integration/data-pipelines/multiple-sources.md Outdated
Comment thread content/integrate/redis-data-integration/data-pipelines/multiple-sources.md Outdated
Comment thread content/integrate/redis-data-integration/data-pipelines/multiple-sources.md Outdated
Comment thread content/integrate/redis-data-integration/data-pipelines/multiple-sources.md Outdated
Comment thread content/integrate/redis-data-integration/data-pipelines/multiple-sources.md Outdated
Comment thread content/integrate/redis-data-integration/data-pipelines/multiple-sources.md Outdated

@nicoweidner nicoweidner left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing all comments, looks good to me now!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants