Skip to content

RDSC-5610: Multi-source documentation - #4028

Closed
andy-stark-redis wants to merge 38 commits into
redis:mainfrom
stoyanr:RDSC-5610-multi-source-docs
Closed

andy-stark-redis wants to merge 38 commits into
redis:mainfrom
stoyanr:RDSC-5610-multi-source-docs

Conversation

@andy-stark-redis

@andy-stark-redis andy-stark-redis commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Recreated from #3916 after its base branch (RDSC-6038-rdi-2.0.0-release-notes) was deleted following #3996's squash-merge, which auto-closed the original PR and left it unable to be reopened via the API (GitHub disallows retargeting a closed PR's base). Same head commit, same diff — see #3916 for the original review history.


See https://redislabs.atlassian.net/browse/RDSC-5610

Enhances #3841 with numerous changes and additions. I couldn't update the original PR since I don't have write permissions to this repository.

Note that it now documents 2 PRs that are not yet merged, but should be before the release:

Release-note additions

The release notes add the Collector API startup fix for OpenShift, identifying it as a regression introduced in RDI 1.19.1. The Flink processor and collector security entry adds 15 CVE identifiers from the final 2.0.0 changelog, preserving all existing identifiers.

Validation for this update: confirmed that the diff contains only these two changes, checked the security identifiers against the 2.0.0 changelog, and ran git diff --check. The Hugo build was not rerun for this text-only update.

Generated documentation consolidation

The OpenAPI version is set to 2.0.0 in commit dcc84dbfb, taking the only new API/schema content from generated PR #4026. The full parsed OpenAPI document now matches #4026. The configuration references are equivalent apart from formatting, and the reviewed API migration and CLI documentation in this PR is retained. #4026 is closed as superseded.

Validation: parsed JSON comparison with #4026, confirmation that only info.version changed in this commit, and git diff --check passed.


Note

Low Risk
Documentation-only changes in the docs repo; no application code is modified, though readers must follow the new upgrade and secret guidance when moving to RDI 2.0.0.

Overview
Updates Redis Data Integration docs for RDI 2.0.0 (rdi_current_version in config.toml) and documents multi-source pipelines, per-database secrets, and several 2.0 breaking operational changes.

Multi-source: New guide Multiple sources in one pipeline covers naming rules, config.yaml with several sources, job server_name (and Flink list/regex matchers), per-source collectors/metrics/DLQ keys, and add/remove/start/stop/reset with --source. Architecture, pipelines, FAQ, and API migration now state that v2 is required for multiple sources and that legacy upgraded sources keep SOURCE_DB_*, server_name: rdi, and old stream names until removed or the pipeline is cleared.

Secrets & deploy: Secret management is rewritten around generic keys (USERNAME, PASSWORD, CACERT, …) plus redis-di set-secret --db <source|target>, derived env vars like ${MYSQL_DB_PASSWORD}, and K8s secrets named <source>-db / <source>-db-ssl. Prepare-database pages and examples are aligned to that model.

Processor & upgrade: Default stream processor is documented as Flink (explicit classic to opt out). Upgrade and classic→Flink migration guides add drain-before-switch steps, 2.0.0-specific warnings, and redis-di deploy --empty (and API v2 empty PUT) replacing pipeline delete/undeploy. Observability and HA examples use per-source collector names (e.g. collector-mysql). Minor image pin updates appear in the K8s install doc (Debezium, RIOT-X).

Reviewed by Cursor Bugbot for commit dcc84db. Bugbot is set up for automated code reviews on this repo. Configure here.

ZdravkoDonev-redis and others added 30 commits September 14, 2026 02:21
Co-authored-by: Stoyan Rachev <stoyanr@gmail.com>
Co-authored-by: Stoyan Rachev <stoyanr@gmail.com>
…vs-flink.md

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
…e-dbs/_index.md

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
…e-dbs/snowflake.md

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
…e-dbs/snowflake.md

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
…e-dbs/spanner.md

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
…e-dbs/spanner.md

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>
…e-dbs/mongodb.md

Co-authored-by: Zdravko Donev <zdravko.donev@redis.com>
@andy-stark-redis

Copy link
Copy Markdown
Contributor Author

Superseded by #4029, which resolves the merge conflicts against main (this PR's branch lives on a fork I can't push a resolution to).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants