From 85f467527d9e4aeaf177b91da4157161b6d5bd67 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Thu, 3 Sep 2026 16:15:05 +0300 Subject: [PATCH 01/29] RDSC-5610: Multi-source documentation --- content/embeds/rdi-tls-secrets.md | 4 +- content/embeds/rdi-when-to-use-dec-tree.md | 15 - content/embeds/rdi-when-to-use.md | 2 +- .../architecture/_index.md | 28 +- .../architecture/classic-vs-flink.md | 6 + .../data-pipelines/_index.md | 10 +- .../data-pipelines/deploy.md | 183 +- .../data-pipelines/multiple-sources.md | 279 ++ .../data-pipelines/pipeline-config.md | 319 ++- .../data-pipelines/prepare-dbs/_index.md | 7 +- .../aws-aurora-rds/aws-aur-mysql.md | 1 - .../data-pipelines/prepare-dbs/mongodb.md | 29 +- .../data-pipelines/prepare-dbs/oracle.md | 10 +- .../data-pipelines/prepare-dbs/snowflake.md | 36 +- .../data-pipelines/prepare-dbs/spanner.md | 31 +- .../data-pipelines/prepare-dbs/sql-server.md | 27 +- .../data-pipelines/prepare-dbs/supabase.md | 13 +- .../data-pipelines/rejected-records.md | 20 +- .../transform-examples/_index.md | 20 +- .../transform-examples/redis-json-example.md | 2 +- .../integrate/redis-data-integration/faq.md | 6 + .../installation/_index.md | 4 +- .../installation/ha-test.md | 27 +- .../installation/install-k8s.md | 10 +- .../installation/install-vm.md | 26 +- .../installation/upgrade.md | 13 +- .../redis-data-integration/observability.md | 23 +- .../quick-start-guide.md | 4 + .../reference/api-migration.md | 18 +- .../reference/api-reference/openapi.json | 2335 +++++++++++------ .../reference/cli/_index.md | 5 + .../reference/cli/redis-di-delete-secret.md | 3 +- .../reference/cli/redis-di-deploy.md | 2 +- .../reference/cli/redis-di-describe-secret.md | 9 +- .../reference/cli/redis-di-describe.md | 52 +- .../reference/cli/redis-di-get-dlq.md | 6 +- .../cli/redis-di-get-metric-collection.md | 3 +- .../reference/cli/redis-di-get-secret.md | 11 +- .../cli/redis-di-list-dlq-records.md | 8 +- .../reference/cli/redis-di-list-dlqs.md | 3 + .../reference/cli/redis-di-list-secrets.md | 10 +- .../reference/cli/redis-di-reset.md | 4 + .../reference/cli/redis-di-scaffold.md | 21 +- .../reference/cli/redis-di-set-secret.md | 17 +- .../reference/cli/redis-di-start.md | 4 + .../reference/cli/redis-di-stop.md | 4 + .../reference/config-yaml-reference.md | 89 +- content/operate/rc/rdi/define.md | 2 +- content/operate/rc/rdi/quick-start.md | 2 +- content/operate/rc/rdi/setup.md | 2 +- 50 files changed, 2631 insertions(+), 1134 deletions(-) create mode 100644 content/integrate/redis-data-integration/data-pipelines/multiple-sources.md diff --git a/content/embeds/rdi-tls-secrets.md b/content/embeds/rdi-tls-secrets.md index 147e76b516..4b6c74c70f 100644 --- a/content/embeds/rdi-tls-secrets.md +++ b/content/embeds/rdi-tls-secrets.md @@ -1,4 +1,4 @@ -When creating secrets for TLS or mTLS, ensure that all certificates and keys are in `PEM` format. The only exception to this is that for PostgreSQL, the private key `SOURCE_DB_KEY` secret must be in `DER` format. If you have a key in `PEM` format, you must convert it to `DER` before creating the `SOURCE_DB_KEY` secret using the command: +When creating secrets for TLS or mTLS, ensure that all certificates and keys are in `PEM` format. The only exception to this is that for PostgreSQL, the source's private key secret must be in `DER` format. If you have a key in `PEM` format, you must convert it to `DER` before creating the secret using the command: ```bash openssl pkcs8 -topk8 -inform PEM -outform DER \ @@ -6,4 +6,4 @@ openssl pkcs8 -topk8 -inform PEM -outform DER \ -out /path/to/myclient.pk8 -nocrypt ``` -This command assumes that the private key is not encrypted. See the [`openssl` documentation](https://docs.openssl.org/master/) to learn how to convert an encrypted private key. \ No newline at end of file +This command assumes that the private key is not encrypted. See the [`openssl` documentation](https://docs.openssl.org/master/) to learn how to convert an encrypted private key. diff --git a/content/embeds/rdi-when-to-use-dec-tree.md b/content/embeds/rdi-when-to-use-dec-tree.md index f5219bef37..10d4f558ae 100644 --- a/content/embeds/rdi-when-to-use-dec-tree.md +++ b/content/embeds/rdi-when-to-use-dec-tree.md @@ -32,21 +32,6 @@ questions: label: "⚠️ Check deployment options to see if RDI is suitable for your needs before proceeding" id: deploymentMismatch sentiment: "indeterminate" - yes: - value: "Yes" - nextQuestion: singleSource - singleSource: - text: | - Are you transferring data from a single source database? - whyAsk: | - RDI is designed to work with a single source database. Multiple sources create conflicting change events. - answers: - no: - value: "No" - outcome: - label: "❌ RDI won't work with multiple source databases" - id: multipleSourcesOrActiveActive - sentiment: "negative" yes: value: "Yes" nextQuestion: consistency diff --git a/content/embeds/rdi-when-to-use.md b/content/embeds/rdi-when-to-use.md index 3251321552..7ef4734d35 100644 --- a/content/embeds/rdi-when-to-use.md +++ b/content/embeds/rdi-when-to-use.md @@ -3,7 +3,7 @@ RDI is a good fit when: - You want your app/micro-services to read from Redis to scale reads at speed. -- You want to transfer data to Redis from a *single* source database. +- You want to transfer data to Redis from one or more source databases. - You must use a slow database as the system of record for the app. - The app must always *write* its data to the slow database. - Your app can tolerate *eventual* consistency of data in the Redis cache. diff --git a/content/integrate/redis-data-integration/architecture/_index.md b/content/integrate/redis-data-integration/architecture/_index.md index f9d09f0453..323a81950a 100644 --- a/content/integrate/redis-data-integration/architecture/_index.md +++ b/content/integrate/redis-data-integration/architecture/_index.md @@ -20,13 +20,13 @@ weight: 30 ## Overview -RDI implements a [change data capture](https://en.wikipedia.org/wiki/Change_data_capture) (CDC) pattern that tracks changes to the data in a -non-Redis *source* database and makes corresponding changes to a Redis +RDI implements a [change data capture](https://en.wikipedia.org/wiki/Change_data_capture) (CDC) pattern that tracks changes to the data in one or +more non-Redis *source* databases and makes corresponding changes to a Redis *target* database. You can use the target as a cache to improve performance because it will typically handle read queries much faster than the source. To use RDI, you define a *dataset* that specifies which data items -you want to capture from the source and how you want to +you want to capture from each source and how you want to represent them in the target. For example, if the source is a relational database then you specify which table columns you want to capture but you don't need to store them in an equivalent table @@ -35,15 +35,15 @@ representation is most suitable for your app. To convert from the source to the target representation, RDI applies *transformations* to the data after capture. -RDI synchronizes the dataset between the source and target using +RDI synchronizes the dataset between the sources and the target using a *data pipeline* that implements several processing steps in sequence: -1. A *CDC collector* captures changes to the source database. RDI - currently uses an open source collector called +1. A *CDC collector* per source captures changes to the source + databases. RDI currently uses an open source collector called [Debezium](https://debezium.io/) for this step. -1. The collector records the captured changes using +1. Each collector records the captured changes using [Redis streams]({{< relref "/develop/data-types/streams" >}}) in the RDI database. @@ -60,12 +60,12 @@ its state and configuration data and also the change data streams in a Redis dat {{< image filename="images/rdi/ingest/ingest-dataflow.webp" >}} When you first start RDI, the target database is empty and so all -of the data in the source database is essentially "change" data. +of the data in the source databases is essentially "change" data. RDI collects this data in a phase called *initial cache loading*, which can take minutes or hours to finish, depending on the size of the source data. Once the initial cache loading is complete, there is a *snapshot* dataset in the target that will gradually -change when new data gets captured from the source. At this point, +change when new data gets captured from the sources. At this point, RDI automatically enters a second phase called *change streaming*, where changes in the data are captured as they happen. Changes are usually added to the target within a few seconds after capture. @@ -83,7 +83,7 @@ overall state). ## Checkpointing RDI uses Redis streams to store the sequence of change events -captured from the source. The events are then retrieved in order +captured from the sources. The events are then retrieved in order from the streams, processed, and written to the target. The stream processor uses a *checkpoint* mechanism to keep track of the last event in the sequence that it has successfully processed and stored. If the processor fails @@ -96,11 +96,11 @@ face of failures. Sometimes, data records can get added to the streams faster than RDI can process them. This can happen if the target is slowed or disconnected -or simply if the source quickly generates a lot of change data. +or simply if a source quickly generates a lot of change data. If this continues, then the streams will eventually occupy all the available memory. When RDI detects this situation, it applies a *backpressure* mechanism to slow or stop the flow of incoming data. -Change data is held at the source until RDI clears the backlog and has +Change data is held at the sources until RDI clears the backlog and has enough free memory to resume streaming. {{}}The Debezium log sometimes reports that RDI has run out @@ -128,7 +128,7 @@ It includes: and exports them as [Prometheus](https://prometheus.io/) metrics. The *data plane* contains the processes that actually move the data. -It includes the *CDC collector* and the *stream processor* that implement +It includes a *CDC collector* for each source and the *stream processor*, which implement the two phases of the pipeline lifecycle (initial cache loading and change streaming). The *management plane* provides tools that let you interact @@ -164,7 +164,7 @@ deploy RDI. ### RDI on your own VMs -For this deployment, you must provide two VMs. The collector and stream processor +For this deployment, you must provide two VMs. The collectors and stream processor are active on one VM, while on the other they are in standby to provide high availability. The two operators running on both VMs use a leader election algorithm to decide which VM is the active one (the "leader"). diff --git a/content/integrate/redis-data-integration/architecture/classic-vs-flink.md b/content/integrate/redis-data-integration/architecture/classic-vs-flink.md index 3818dfe91b..76f4942bcb 100644 --- a/content/integrate/redis-data-integration/architecture/classic-vs-flink.md +++ b/content/integrate/redis-data-integration/architecture/classic-vs-flink.md @@ -102,6 +102,12 @@ for examples and [`redis.lookup`]({{< relref "/integrate/redis-data-integration/reference/data-transformation/lookup" >}}) for the full property list. +The Flink processor also accepts an advanced matcher syntax in a job's `source` section, where +`server_name`, `db`, `schema`, and `table` each accept a list of names, and an entry prefixed +with `regex:` selects all names that match the regular expression. One job can then process +multiple tables, potentially from different sources, databases, or schemas. See +[Job files]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}). + ## Metrics The two processors expose different Prometheus metric sets and use diff --git a/content/integrate/redis-data-integration/data-pipelines/_index.md b/content/integrate/redis-data-integration/data-pipelines/_index.md index 62f7f255b6..0f364c4e61 100644 --- a/content/integrate/redis-data-integration/data-pipelines/_index.md +++ b/content/integrate/redis-data-integration/data-pipelines/_index.md @@ -28,9 +28,9 @@ deploy them. ## How a pipeline works -An RDI pipeline captures change data records from the source database, and transforms them -into Redis data structures. It writes each of these new structures to a Redis target -database under its own key. +An RDI pipeline captures change data records from one or more source databases, and +transforms them into Redis data structures. It writes each of these new structures to a +Redis target database under its own key. By default, RDI transforms the source data into [hashes]({{< relref "/develop/data-types/hashes" >}}) or @@ -94,6 +94,10 @@ to the snapshot phase. When this is complete, the pipeline continues with CDC as Follow the steps described in the sections below to prepare and run an RDI pipeline. +The following example uses a single-source pipeline. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) +for details on how to capture multiple sources in one pipeline. + ### 1. Prepare the source database Before using the pipeline you must first prepare your source database to use diff --git a/content/integrate/redis-data-integration/data-pipelines/deploy.md b/content/integrate/redis-data-integration/data-pipelines/deploy.md index d126291b45..1cb84b4439 100644 --- a/content/integrate/redis-data-integration/data-pipelines/deploy.md +++ b/content/integrate/redis-data-integration/data-pipelines/deploy.md @@ -21,33 +21,39 @@ The sections below explain how to deploy a pipeline after you have created the r ## Set secrets -Before you deploy your pipeline, you must set the authentication secrets for the -source and target databases. Each secret has a name that you pass to the +Before you deploy your pipeline, you must set the authentication secrets for the source +and target databases. Every secret belongs to one database: a source, identified by its +name in `config.yaml`, or the target. You name that database with the `--db` option of the [`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}}) -command to set the secret value. -You can then refer to these secrets in the `config.yaml` file using the syntax "`${SECRET_NAME}`" -(the sample -[config.yaml file]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#example" >}}) -shows these secrets in use). +command. -The table below lists all valid secret names. Note that the -username and password are required for the source and target, but the other -secrets are only relevant for TLS/mTLS connections. +The table below lists the available secret keys. The username and password are required, +while the other keys are only relevant for TLS/mTLS connections. -| Secret name | Description | +| Secret key | Description | | :-- | :-- | -| `SOURCE_DB_USERNAME` | Username for the source database | -| `SOURCE_DB_PASSWORD` | Password for the source database | -| `SOURCE_DB_CACERT` | (For TLS only) Source database CA certificate | -| `SOURCE_DB_CERT` | (For mTLS only) Source database client certificate | -| `SOURCE_DB_KEY` | (For mTLS only) Source database private key | -| `SOURCE_DB_KEY_PASSWORD` | (For mTLS only) Source database private key password | -| `TARGET_DB_USERNAME` | Username for the target database | -| `TARGET_DB_PASSWORD` | Password for the target database | -| `TARGET_DB_CACERT` | (For TLS only) Target database CA certificate | -| `TARGET_DB_CERT` | (For mTLS only) Target database client certificate | -| `TARGET_DB_KEY` | (For mTLS only) Target database private key | -| `TARGET_DB_KEY_PASSWORD` | (For mTLS only) Target database private key password | +| `USERNAME` | Username for the database | +| `PASSWORD` | Password for the database | +| `CACERT` | (For TLS only) CA certificate | +| `CERT` | (For mTLS only) Client certificate | +| `KEY` | (For mTLS only) Private key | +| `KEY_PASSWORD` | (For mTLS only) Private key password | + +You can reference a secret in `config.yaml` using an environment variable that is derived from +the secret key and the database name. The variable name consists of the database name in +uppercase (with each dash replaced by an underscore), followed by `_DB_`, followed by the key. +For example, if you set `PASSWORD` with `--db mysql` the corresponding environment variable +is `MYSQL_DB_PASSWORD`, which the source references as +`${MYSQL_DB_PASSWORD}`. If you set `PASSWORD` with `--db target`, the environment variable is `TARGET_DB_PASSWORD`. The sample +[config.yaml file]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#example" >}}) +shows these references in use, and +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) +covers the source naming rules. + +{{< note >}}The scope-prefixed keys `SOURCE_DB_*` and `TARGET_DB_*` are also accepted, and can be +used without specifying `--db`. RDI accepts a `SOURCE_DB_*` key only for a pipeline with +exactly one source that predates per-source naming. See +[Existing names are kept after an upgrade]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}).{{< /note >}} {{< note >}} {{< embed-md "rdi-tls-secrets.md" >}} @@ -58,38 +64,38 @@ secrets are only relevant for TLS/mTLS connections. Use [`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}}) to set secrets for any installation type (VM, Kubernetes, or Redis Cloud). -The specific command lines for source secrets are as follows: +The command lines for a source named `mysql` are shown below. If your pipeline has multiple sources, you should run the command once for each source, using the appropriate database name. ```bash # For username and password -redis-di set-secret SOURCE_DB_USERNAME yourUsername -redis-di set-secret SOURCE_DB_PASSWORD yourPassword +redis-di set-secret USERNAME --db mysql yourUsername +redis-di set-secret PASSWORD --db mysql yourPassword # With source TLS, in addition to the above -redis-di set-secret SOURCE_DB_CACERT /path/to/myca.crt +redis-di set-secret CACERT --db mysql /path/to/myca.crt # With source mTLS, in addition to the above -redis-di set-secret SOURCE_DB_CERT /path/to/myclient.crt -redis-di set-secret SOURCE_DB_KEY /path/to/myclient.key -# Use this only if SOURCE_DB_KEY is password-protected -redis-di set-secret SOURCE_DB_KEY_PASSWORD yourKeyPassword +redis-di set-secret CERT --db mysql /path/to/myclient.crt +redis-di set-secret KEY --db mysql /path/to/myclient.key +# Use this only if the private key is password-protected +redis-di set-secret KEY_PASSWORD --db mysql yourKeyPassword ``` The corresponding command lines for target secrets are: ```bash # For username and password -redis-di set-secret TARGET_DB_USERNAME yourUsername -redis-di set-secret TARGET_DB_PASSWORD yourPassword +redis-di set-secret USERNAME --db target yourUsername +redis-di set-secret PASSWORD --db target yourPassword # With target TLS, in addition to the above -redis-di set-secret TARGET_DB_CACERT /path/to/myca.crt +redis-di set-secret CACERT --db target /path/to/myca.crt # With target mTLS, in addition to the above -redis-di set-secret TARGET_DB_CERT /path/to/myclient.crt -redis-di set-secret TARGET_DB_KEY /path/to/myclient.key -# Use this only if TARGET_DB_KEY is password-protected -redis-di set-secret TARGET_DB_KEY_PASSWORD yourKeyPassword +redis-di set-secret CERT --db target /path/to/myclient.crt +redis-di set-secret KEY --db target /path/to/myclient.key +# Use this only if the private key is password-protected +redis-di set-secret KEY_PASSWORD --db target yourKeyPassword ``` By default, `set-secret` waits for the pipeline to apply the change before returning. When you set @@ -103,14 +109,17 @@ never returns secret values, these commands show only the secret keys and whethe the stored values. ```bash -# List all the secrets of a pipeline and whether each one is set +# List all the secrets of a pipeline, with the database each one belongs to redis-di list-secrets +# List only the secrets of one database +redis-di list-secrets --db mysql + # Show a single secret and whether it is set -redis-di describe-secret SOURCE_DB_PASSWORD +redis-di describe-secret PASSWORD --db mysql # Delete a secret (prompts for confirmation unless you add --force) -redis-di delete-secret SOURCE_DB_CACERT +redis-di delete-secret CACERT --db mysql ``` See the reference pages for @@ -135,9 +144,12 @@ kubectl create secret generic \ --from-literal== ``` -Where `` is either `source-db` for source secrets or `target-db` for target secrets. +Where `` is `-db` for the secrets of a source, or `target-db` for target secrets. +The examples below use a source named `mysql`, so its secret is `mysql-db`. -If you use TLS or mTLS for either the source or target databases, you also need to create the `source-db-ssl` and/or `target-db-ssl` K8s secrets that contain the certificates used to establish secure connections. The general pattern of the commands is: +If you use TLS or mTLS for either the source or target databases, you also need to create the +`-db-ssl` and/or `target-db-ssl` K8s secrets that contain the certificates used +to establish secure connections. The general pattern of the commands is: ```bash kubectl create secret generic -ssl \ @@ -149,36 +161,37 @@ The specific command lines for source secrets are as follows: ```bash # Without source TLS -# Create or update source-db secret -kubectl create secret generic source-db --namespace=rdi \ ---from-literal=SOURCE_DB_USERNAME=yourUsername \ ---from-literal=SOURCE_DB_PASSWORD=yourPassword \ +# Create or update mysql-db secret +kubectl create secret generic mysql-db --namespace=rdi \ +--from-literal=MYSQL_DB_USERNAME=yourUsername \ +--from-literal=MYSQL_DB_PASSWORD=yourPassword \ --save-config --dry-run=client -o yaml | kubectl apply -f - # With source TLS -# Create of update source-db secret -kubectl create secret generic source-db --namespace=rdi \ ---from-literal=SOURCE_DB_USERNAME=yourUsername \ ---from-literal=SOURCE_DB_PASSWORD=yourPassword \ ---from-literal=SOURCE_DB_CACERT=/etc/certificates/source_db/ca.crt \ +# Create or update mysql-db secret +kubectl create secret generic mysql-db --namespace=rdi \ +--from-literal=MYSQL_DB_USERNAME=yourUsername \ +--from-literal=MYSQL_DB_PASSWORD=yourPassword \ +--from-literal=MYSQL_DB_CACERT=/etc/certificates/mysql_db/ca.crt \ --save-config --dry-run=client -o yaml | kubectl apply -f - -# Create or update source-db-ssl secret -kubectl create secret generic source-db-ssl --namespace=rdi \ +# Create or update mysql-db-ssl secret +kubectl create secret generic mysql-db-ssl --namespace=rdi \ --from-file=ca.crt=/path/to/myca.crt \ --save-config --dry-run=client -o yaml | kubectl apply -f - # With source mTLS -# Create or update source-db secret -kubectl create secret generic source-db --namespace=rdi \ ---from-literal=SOURCE_DB_USERNAME=yourUsername \ ---from-literal=SOURCE_DB_PASSWORD=yourPassword \ ---from-literal=SOURCE_DB_CACERT=/etc/certificates/source_db/ca.crt \ ---from-literal=SOURCE_DB_CERT=/etc/certificates/source_db/client.crt \ ---from-literal=SOURCE_DB_KEY=/etc/certificates/source_db/client.key \ ---from-literal=SOURCE_DB_KEY_PASSWORD=yourKeyPassword \ # add this only if SOURCE_DB_KEY is password-protected +# Create or update mysql-db secret. Include the MYSQL_DB_KEY_PASSWORD line +# only if the private key is password-protected. +kubectl create secret generic mysql-db --namespace=rdi \ +--from-literal=MYSQL_DB_USERNAME=yourUsername \ +--from-literal=MYSQL_DB_PASSWORD=yourPassword \ +--from-literal=MYSQL_DB_CACERT=/etc/certificates/mysql_db/ca.crt \ +--from-literal=MYSQL_DB_CERT=/etc/certificates/mysql_db/client.crt \ +--from-literal=MYSQL_DB_KEY=/etc/certificates/mysql_db/client.key \ +--from-literal=MYSQL_DB_KEY_PASSWORD=yourKeyPassword \ --save-config --dry-run=client -o yaml | kubectl apply -f - -# Create or update source-db-ssl secret -kubectl create secret generic source-db-ssl --namespace=rdi \ +# Create or update mysql-db-ssl secret +kubectl create secret generic mysql-db-ssl --namespace=rdi \ --from-file=ca.crt=/path/to/myca.crt \ --from-file=client.crt=/path/to/myclient.crt \ --from-file=client.key=/path/to/myclient.key \ @@ -196,7 +209,7 @@ kubectl create secret generic target-db --namespace=rdi \ --save-config --dry-run=client -o yaml | kubectl apply -f - # With target TLS -# Create of update target-db secret +# Create or update target-db secret kubectl create secret generic target-db --namespace=rdi \ --from-literal=TARGET_DB_USERNAME=yourUsername \ --from-literal=TARGET_DB_PASSWORD=yourPassword \ @@ -208,14 +221,15 @@ kubectl create secret generic target-db-ssl --namespace=rdi \ --save-config --dry-run=client -o yaml | kubectl apply -f - # With target mTLS -# Create or update target-db secret +# Create or update target-db secret. Include the TARGET_DB_KEY_PASSWORD line +# only if the private key is password-protected. kubectl create secret generic target-db --namespace=rdi \ --from-literal=TARGET_DB_USERNAME=yourUsername \ --from-literal=TARGET_DB_PASSWORD=yourPassword \ --from-literal=TARGET_DB_CACERT=/etc/certificates/target_db/ca.crt \ --from-literal=TARGET_DB_CERT=/etc/certificates/target_db/client.crt \ --from-literal=TARGET_DB_KEY=/etc/certificates/target_db/client.key \ ---from-literal=TARGET_DB_KEY_PASSWORD=yourKeyPassword \ # add this only if TARGET_DB_KEY is password-protected +--from-literal=TARGET_DB_KEY_PASSWORD=yourKeyPassword \ --save-config --dry-run=client -o yaml | kubectl apply -f - # Create or update target-db-ssl secret kubectl create secret generic target-db-ssl --namespace=rdi \ @@ -225,11 +239,12 @@ kubectl create secret generic target-db-ssl --namespace=rdi \ --save-config --dry-run=client -o yaml | kubectl apply -f - ``` -Note that the certificate paths contained in the secrets `SOURCE_DB_CACERT`, `SOURCE_DB_CERT`, and `SOURCE_DB_KEY` (for the source database) and `TARGET_DB_CACERT`, `TARGET_DB_CERT`, and `TARGET_DB_KEY` (for the target database) are internal to RDI, so you *must* use the values shown in the example above. You should only change the certificate paths when you create the `source-db-ssl` and `target-db-ssl` secrets. +Note that the certificate paths contained in the `CACERT`, `CERT`, and `KEY` secrets are internal to RDI, so you *must* use the values shown in the example above. Each source has its own certificate directory, named after the source (for example, `/etc/certificates/mysql_db/` for a source named `mysql`). You should only change the certificate paths when you create the `-db-ssl` and `target-db-ssl` secrets. -Secrets that you create directly with `kubectl` must also be labeled so that the RDI operator -discovers them as pipeline secrets. Each secret needs the following labels, where the -`app.kubernetes.io/instance` label is the pipeline name (`default` for the default pipeline): +You must also label any secrets that you create directly with `kubectl` so that the RDI operator +discovers them as pipeline secrets. Give each secret the following labels, where the +`app.kubernetes.io/instance` label corresponds to the pipeline name (the name is just +`default` for the default pipeline): | Label | Value | | :-- | :-- | @@ -240,7 +255,7 @@ discovers them as pipeline secrets. Each secret needs the following labels, wher Apply the labels to each secret with [`kubectl label`](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_label/): ```bash -kubectl label secret source-db --namespace=rdi --overwrite \ +kubectl label secret mysql-db --namespace=rdi --overwrite \ app.kubernetes.io/name=pipeline \ app.kubernetes.io/instance=default \ product=rdi @@ -250,7 +265,7 @@ kubectl label secret target-db --namespace=rdi --overwrite \ product=rdi # With source TLS or mTLS -kubectl label secret source-db-ssl --namespace=rdi --overwrite \ +kubectl label secret mysql-db-ssl --namespace=rdi --overwrite \ app.kubernetes.io/name=pipeline \ app.kubernetes.io/instance=default \ product=rdi @@ -331,6 +346,16 @@ redis-di stop redis-di start ``` +To act on a single source instead of the whole pipeline, add `--source`: + +```bash +redis-di stop --source mysql +redis-di start --source mysql +``` + +Note that a source can only run if its parent pipeline is running. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for more information. + ## Reset a pipeline Use [`redis-di reset`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-reset" >}}) @@ -343,6 +368,12 @@ drifted out of sync. redis-di reset ``` +Add `--source` to reset a single source and leave the others untouched: + +```bash +redis-di reset --source mysql +``` + ## Undeploy a pipeline To remove a pipeline, use the @@ -377,8 +408,8 @@ To avoid this, set all the related secrets, or at least all of them except the l become healthy: ```bash -redis-di set-secret SOURCE_DB_USERNAME newUsername --wait=false -redis-di set-secret SOURCE_DB_PASSWORD newPassword +redis-di set-secret USERNAME --db mysql newUsername --wait=false +redis-di set-secret PASSWORD --db mysql newPassword ``` -The same applies to any set of changes that are only valid together. \ No newline at end of file +The same applies to any set of changes that are only valid together. diff --git a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md new file mode 100644 index 0000000000..ff3b4ed71e --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md @@ -0,0 +1,279 @@ +--- +Title: Multiple sources in one pipeline +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: Learn how to ingest from several source databases into one Redis target. +group: di +linkTitle: Multiple sources +summary: Redis Data Integration keeps Redis in sync with the primary database in near + real time. +type: integration +weight: 4 +--- + +One RDI pipeline can capture changes from several source databases and write them all to the +same Redis target. The sources can be of different database types and each has its +own collector, its own credentials, and its own set of Redis streams to ensure it +is independent of the other sources. + +{{< note >}}Use RDI API v2 to manage a pipeline with several sources. RDI API v1 supports only +single-source pipelines. See the +[RDI API migration guide]({{< relref "/integrate/redis-data-integration/reference/api-migration" >}}).{{< /note >}} + +## Name your sources + +Each source is an entry in the `sources` section of +[`config.yaml`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config" >}}), +with the source name as key: + +```yaml +sources: + mysql: # this source is named 'mysql' + type: cdc +``` + +A source name must: + +- Start with a lowercase letter. +- Contain only lowercase letters, digits, and dashes. +- End with a letter or a digit. +- Be at most 22 characters long. + +The names `rdi` and `target` are reserved and cannot be used for sources. + +RDI derives the environment variables that contain the source's credentials from the source +name, so the `connection` section of a source named `mysql` references `${MYSQL_DB_USERNAME}` +and `${MYSQL_DB_PASSWORD}`. See +[Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) +for details of how RDI derives those names and for the full list of secret keys. + +The source name also appears in the resources RDI creates for the source. The table below +lists the names derived from a source named `mysql`. + +| Resource | Name | +| :-- | :-- | +| Credentials secret | `mysql-db` | +| TLS secret | `mysql-db-ssl` | +| Environment variable prefix | `MYSQL_DB_` | +| Certificate directory | `/etc/certificates/mysql_db/` | +| Collector deployment | `collector-mysql` | +| Change data streams | `data:{rdi}:mysql..` | +| Dead-letter queue streams | `dlq:data:{rdi}:mysql..
` | +| Metric collection | `collector-mysql_metrics` | +| Metrics endpoint path on a VM installation | `/collector-mysql/metrics` | + +Each source also accepts an optional `name` property, which is a display name +of up to 100 characters. Unlike the source name, it is not used as an identifier, +so there is no restriction on the characters you can use. + +## Configure several sources + +Add one entry per source in the `config.yaml` file (see +[Pipeline configuration file]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config" >}}) +for a full description of this file). +[`redis-di scaffold`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-scaffold" >}}) +generates a configuration with one source, named by its `--source-name` option, so add any +further sources by editing `config.yaml`. + +Sources of different types can be mixed freely, but a source's collector `type` and its +`connection` type have to match. Use `cdc`, the default, for the relational databases and +MongoDB, `flink` for a Spanner connection, and `riotx` for a Snowflake connection. RDI +rejects any other combination when you deploy the pipeline. See +[Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) +for how to prepare each source database. + +The following example captures from a MySQL database and a PostgreSQL database, each with its +own credentials: + +```yaml +sources: + mysql: + type: cdc + connection: + type: mysql + host: + port: 3306 + user: ${MYSQL_DB_USERNAME} + password: ${MYSQL_DB_PASSWORD} + databases: + - inventory + tables: + inventory.customers: {} + inventory.orders: {} + postgresql: + type: cdc + connection: + type: postgresql + host: + port: 5432 + database: billing + user: ${POSTGRESQL_DB_USERNAME} + password: ${POSTGRESQL_DB_PASSWORD} + schemas: + - public + tables: + public.customers: {} +targets: + target: + connection: + type: redis + host: + port: 6379 + password: ${TARGET_DB_PASSWORD} +``` + +## Set secrets for each source + +Set a source's credentials with the source name in the `--db` option: + +```bash +redis-di set-secret USERNAME --db mysql +redis-di set-secret PASSWORD --db mysql +redis-di set-secret USERNAME --db postgresql +redis-di set-secret PASSWORD --db postgresql +``` + +The secret keys used as CLI arguments are the same for every source: `USERNAME`, `PASSWORD`, and, for +[Transport Layer Security (TLS)]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) +connections, `CACERT`, `CERT`, `KEY`, and `KEY_PASSWORD`. Use `--db target` for the +target database: `redis-di set-secret PASSWORD --db target `. + +Each source's `connection` section then references its own secrets: `${MYSQL_DB_USERNAME}` and +`${MYSQL_DB_PASSWORD}` for `mysql`, `${POSTGRESQL_DB_USERNAME}` and `${POSTGRESQL_DB_PASSWORD}` +for `postgresql`, and `${TARGET_DB_PASSWORD}` for the target. + +See [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) +for the full secret reference. + +## Select sources in jobs + +A [job]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) +selects the source it processes by setting `server_name` to the source name: + +```yaml +source: + server_name: mysql + db: inventory + table: customers +``` + +When a pipeline has more than one source, every job must set `server_name`, and the value has +to match one of the sources in `config.yaml`. RDI rejects the pipeline when a job has no +`server_name`, or when its `server_name` matches no source. + +In a pipeline with a single source, `server_name` is optional. If you omit it, the +job does not filter by source. + +No two jobs may select the same records, so make sure the source selectors of your jobs do +not overlap. RDI rejects the pipeline when it finds two jobs that intersect. + +With the [Flink processor]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}}), +`server_name` also accepts a list of source names, and an entry prefixed with `regex:` selects +all sources that match the regular expression, so one job can process multiple tables, +potentially from different sources, databases, or schemas. See +[Job files]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) +for details. + +## Add or remove a source + +To add a source, set its secrets first, then add it to `config.yaml` and deploy. Adding a +source does not interrupt other sources that are already running. + +To remove a source, delete its entry from `config.yaml` and deploy. RDI removes the source's +collector and deletes the source's keys from the RDI database. No further action is +needed for this cleanup, but it means that a source you add later under the same name starts +from a new +[initial snapshot]({{< relref "/integrate/redis-data-integration/architecture" >}}) +rather than resuming from the position it had reached. + +The source's secrets are not deleted, so remove them yourself with +[`redis-di delete-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-delete-secret" >}}) +if you no longer need them. + +Note that renaming a source is not supported. Renaming a source in `config.yaml` is equivalent +to removing the source and adding a new source with the new name. This implies in particular: + +- The source's secrets have to be created under the new name and `${...}` references in + its `connection` section updated. +- `server_name` has to be updated for every job that reads from the source. +- The source starts with a new + [initial snapshot]({{< relref "/integrate/redis-data-integration/architecture" >}}). + +## Start, stop, and reset a single source + +Pass `--source` to act on a single source instead of the whole pipeline: + +```bash +redis-di stop --source mysql +redis-di start --source mysql +redis-di reset --source mysql +``` + +A source runs only while its pipeline runs, so starting one source does not start a stopped +pipeline. Generally, stopping one source leaves the others running, and when one source fails, the other sources keep capturing changes. The only exception to this is a source of type +`external`. RDI creates no collector for this, so you cannot start or stop it. + +Stopping a source scales its collector down to zero replicas and leaves the rest of the +source's resources in place. RDI records a captured position per source, so a collector you +start again resumes from where it stopped. + +Resetting a single source deletes only that source's keys, so a new +[initial snapshot]({{< relref "/integrate/redis-data-integration/architecture" >}}) is taken for that source, +while the stream processor and the other sources keep running. + +## Monitor each source + +Use [`redis-di describe`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-describe" >}}) +to see the state of every source at once. + +In its output, the `Sources` section lists each source with its sync mode and +whether it is connected. +The `Components` section lists one collector per source. Errors are reported against the +component they came from. See the +[`redis-di describe`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-describe" >}}) +reference page for more details. + +Note that while the sources are independent of each other in the data they capture, the +pipeline status is not per source. RDI reports the whole pipeline in an error state when a +single source fails, so use the `Components` section to find out which one it is. + +Each source's collector has its own metric collection, named after the collector, such as +`collector-mysql_metrics`. In Prometheus, the stream processor's `rdi_incoming_entries` and +`rdi_stream_event_latency_ms` metrics contain a `data_source` label that identifies the stream +the value belongs to, including the source name, so you can break both of them down per +source. See +[Stream processor metrics]({{< relref "/integrate/redis-data-integration/observability#stream-processor-metrics" >}}) +and, for the per-source collector endpoints, +[Accessing the metrics]({{< relref "/integrate/redis-data-integration/observability#accessing-the-metrics" >}}). + +Dead-letter queue streams have Redis keys containing a +`..
` section. +This makes it easy to attribute rejected records to their source. See +[Rejected records]({{< relref "/integrate/redis-data-integration/data-pipelines/rejected-records" >}}) for more information. + +## Existing names are kept after an upgrade + +Before RDI supported multiple sources per pipeline, every source-scoped resource had +a name including the word `source` instead of the actual source name in `config.yaml`. +For a source that existed before you upgraded to a version that supports multiple sources, +those names are kept unchanged, whatever the source is called in `config.yaml`. +In particular, for such a source: + +- Its secret environment variables are still named `SOURCE_DB_*`, so its `connection` section + can keep referencing these secrets. +- Its Kubernetes secrets are still named `source-db` and `source-db-ssl`. +- Its Kubernetes deployment and other resources are still named `collector-source`. +- Its data streams are still named `data:{rdi}:.
`, and its offset + and schema history keys are still `metadata:debezium:offsets` and + `metadata:debezium:schema_history`. +- Its `server_name` is still `rdi`. + +For a source you add after the upgrade, RDI derives all of these names from the source name, +as described on this page. + +See [Upgrading RDI]({{< relref "/integrate/redis-data-integration/installation/upgrade" >}}) +for more information. diff --git a/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md b/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md index 93b73120c0..1348255a7b 100644 --- a/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md +++ b/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md @@ -54,15 +54,15 @@ sources: type: mysql host: port: 3306 - user: ${SOURCE_DB_USERNAME} - password: ${SOURCE_DB_PASSWORD} + user: ${MYSQL_DB_USERNAME} + password: ${MYSQL_DB_PASSWORD} targets: target: connection: type: redis - host: - port: + host: + port: 6379 password: ${TARGET_DB_PASSWORD} processors: @@ -111,11 +111,17 @@ and [`processors`](#processors). ### Sources -The `sources` section has a subsection for the source that -you need to configure. The source section starts with a unique name -to identify the source (in the example, there is a source -called `mysql` but you can choose any name you like). The example -configuration contains the following data: +The `sources` section has one subsection per source database. Each subsection starts +with the source name, which identifies the source and must be unique (in the example, +the source is called `mysql`). + +RDI also derives the environment variables that contain the source's credentials from the +source name, for example `${MYSQL_DB_USERNAME}` and `${MYSQL_DB_PASSWORD}` for a source +named `mysql`. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) +for the source naming rules and for capturing from more than one source database. + +The example configuration contains the following data: - `type`: The collector to use for the pipeline. Use `cdc` for MariaDB, MySQL, MongoDB, Oracle, PostgreSQL, or SQL Server. Use `flink` for Google Cloud @@ -284,141 +290,222 @@ for full details of the other available properties. ## Extended configuration example -This example combines the commonly used options from this page. Remove properties -that you don't need. See the +The example below is the configuration that +[`redis-di scaffold`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-scaffold" >}}) +generates for a MySQL source named `mysql`, with every property documented inline. Uncomment what +you need and delete the rest. See the [configuration file reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference" >}}) for every supported property. ```yaml +# Configuration of the RDI sources, targets, and processor. Commented-out properties are optional, and where a default +# exists the value shown is that default. +# For an introduction to the pipeline configuration, see +# https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config/ +# For a reference of every configuration property, see +# https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference/ + +# Source databases that are used to capture changes from. Each key is a unique source name. sources: mysql: + # Type of the source collector. Use `cdc` for Debezium, `flink` for Spanner, or `riotx` for Snowflake. type: cdc + + # Log verbosity of the source collector, one of `trace`, `debug`, `info`, `warn`, or `error`. logging: level: info + + # Connection to the source database. connection: type: mysql - host: # e.g. localhost + # Hostname or IP address of the source database server, for example `localhost`. + host: + # Port that the source database server listens on. port: 3306 - # User and password are injected from the secrets. - user: ${SOURCE_DB_USERNAME} - password: ${SOURCE_DB_PASSWORD} - # Additional properties for the source collector: - # List of databases to include (optional). + # User and password are resolved from the source database secret. + user: ${MYSQL_DB_USERNAME} + password: ${MYSQL_DB_PASSWORD} + + # Databases to capture from the source database. When omitted, all databases are captured. # databases: - # - database1 - # - database2 + # - - # List of tables to be synced (optional). - # tables: - # If only one database is specified in the databases property above, - # then tables can be defined without the database prefix. - # .: - # List of columns to be synced (optional). - # columns: - # - - # - - # List of columns to be used as keys (optional). - # keys: - # - - - # Example: Sync specific tables. + # Tables to capture from the source database, keyed by table name. A table with no properties, or with an empty + # mapping, captures all of its columns. If only one database is listed above, table names can omit the + # database prefix. # tables: - # Sync a specific table with all its columns: - # redislabscdc.account: {} - # Sync a specific table with selected columns: - # redislabscdc.emp: - # columns: - # - empno - # - fname - # - lname - - # Advanced collector properties (optional): + # .: {} + # .: + # Columns to capture. When omitted, all columns are captured. + # columns: + # - + # Columns that form a unique identifier. Only needed when the table has no primary key or unique constraint. + # keys: + # - + + # Advanced properties that override the collector defaults. Only needed for non-standard tuning. # advanced: - # Sink collector properties - see the full list at - # https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream - # sink: - # Optional hard limits on memory usage of RDI streams. - # redis.memory.limit.mb: 300 - # redis.memory.threshold.percentage: 85 - - # Uncomment for production so RDI Collector will wait on replica - # when writing entries. - # redis.wait.enabled: true - # redis.wait.timeout.ms: 1000 - # redis.wait.retry.enabled: true - # redis.wait.retry.delay.ms: 1000 - - # Source specific properties - see the full list at - # https://debezium.io/documentation/reference/stable/connectors/ - # source: - # snapshot.mode: initial - # Uncomment if you want a snapshot to include only a subset of the rows - # in a table. This property affects snapshots only. - # snapshot.select.statement.overrides: . - # The specified SELECT statement determines the subset of table rows to - # include in the snapshot. - # snapshot.select.statement.overrides..: - - # Example: Snapshot filtering by order status. - # To include only orders with non-pending status from customers.orders - # table: - # snapshot.select.statement.overrides: customer.orders - # snapshot.select.statement.overrides.customer.orders: SELECT * FROM customers.orders WHERE status != 'pending' ORDER BY order_id DESC - - # Quarkus framework properties - see the full list at - # https://quarkus.io/guides/all-config - # quarkus: - # banner.enabled: "false" - - # `java_options` (for RDI 1.15.1 and above) controls the JAVA_OPTS environment variable. Use it to modify the default values for - # Java heap size and other Java options for the Debezium server. - # java_options: "-Xmx2g -Xms512m" - + # Properties of the RDI Collector stream writer. See the full list in + # https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream + # sink: + # Number of records that the sink writes to Redis in a single batch. Raise it, together with the source + # `max.batch.size` and `max.queue.size`, for higher snapshot throughput. + # redis.batch.size: 2048 + + # Back pressure on the RDI streams. The sink stops writing once the used memory of the RDI database reaches + # `redis.memory.threshold.percentage` of its `maxmemory`, or of `redis.memory.limit.mb` when that is set. + # redis.memory.limit.mb: 0 + # redis.memory.threshold.percentage: 85 + + # Whether every write to the RDI streams is verified to have reached a replica shard, which prevents losing + # captured changes when the primary shard fails over. It costs write latency, and it has no effect unless the + # RDI database is replicated. `redis.wait.retry.enabled` keeps retrying when the verification times out. + # redis.wait.enabled: false + # redis.wait.timeout.ms: 1000 + # redis.wait.retry.enabled: false + # redis.wait.retry.delay.ms: 1000 + + # Properties of the source database connection and of Debezium. See the full list for your source database in + # https://debezium.io/documentation/reference/stable/connectors/ + # source: + # When and whether the collector takes an initial snapshot of the source database. + # snapshot.mode: initial + + # Performance tuning of the Debezium collector. Rows read from the source database are buffered in a queue of + # `max.queue.size` records, serialized by `record.processing.threads` worker threads, and passed on in batches + # of `max.batch.size` records, which the sink then writes to Redis in batches of `sink.redis.batch.size`. + # Lower this to 2 or 1 when fewer CPUs are available to the collector. + # record.processing.threads: 4 + + # Interval between two polls of the source database for new changes in milliseconds. Lower it to 100 or below + # for lower CDC latency. + # poll.interval.ms: 500 + + # Raise both sizes, together with `sink.redis.batch.size`, for higher snapshot throughput, and keep the queue + # about four times the batch. A larger queue needs a larger heap, so raise `advanced.resources.memory` and set + # `-Xmx` in `advanced.java_options` accordingly. + # max.batch.size: 2048 + # max.queue.size: 8192 + + # Restrict the snapshot of a table to the rows that the given statement selects. This affects the snapshot only, + # for example to skip orders that are still pending. + # snapshot.select.statement.overrides: . + # snapshot.select.statement.overrides..: + + # Let the client retrieve the public key of the server for the TLS handshake. Uncomment when the source + # database user authenticates with `caching_sha2_password` over a connection that is not encrypted. + # database.allowPublicKeyRetrieval: true + + # Properties of the Quarkus runtime that hosts Debezium Server. See the full list in + # https://quarkus.io/guides/all-config + # quarkus: + # banner.enabled: "false" + +# Target Redis databases that the processed records are written to. RDI currently only supports a single target that +# must be named `target`. targets: - # Redis target database connection. - # RDI supports one target database. Name it 'target'. target: + # Connection to the target Redis database. connection: type: redis - # Host of the Redis database to which RDI will - # write the processed data. - host: # e.g. localhost - # Port for the Redis database to which RDI will - # write the processed data. - port: # e.g. 12000 - # User of the Redis database to which RDI will write the processed data. - # Uncomment if you are not using the default user. + # Hostname or IP address of the target Redis database, for example `localhost`. + host: + # Port that the target Redis database listens on. + port: 6379 + # User of the target Redis database, uncomment when not using the default user. # user: ${TARGET_DB_USERNAME} - # Password for Redis target database. + # Password of the target Redis database. password: ${TARGET_DB_PASSWORD} - # SSL/TLS configuration: Uncomment to enable secure connections. + # TLS configuration, uncomment to connect securely. The key and the certificate must be set together, and the key + # password only when the key is protected by one. + # cacert: ${TARGET_DB_CACERT} + # cert: ${TARGET_DB_CERT} # key: ${TARGET_DB_KEY} # key_password: ${TARGET_DB_KEY_PASSWORD} - # cert: ${TARGET_DB_CERT} - # cacert: ${TARGET_DB_CACERT} + +# Settings that control how the processor writes the captured records to the targets. processors: + # Processor implementation to run, one of `classic` or `flink`. + # The default is `classic` for backward compatibility, while `flink` is strongly recommended for new pipelines. + # See https://redis.io/docs/latest/integrate/redis-data-integration/faq/#which-processor-should-i-use type: flink - # Target data type: hash or json. + + # Maximum number of records read from the source streams in a single batch. + # read_batch_size: 2000 + + # Maximum time in milliseconds to wait for a read batch to fill before processing it. + # read_batch_timeout_ms: 100 + + # Maximum number of records written to a target database in a single batch. Must not exceed read_batch_size. + # write_batch_size: 200 + + # Whether batches are processed asynchronously, which improves throughput and reduces latency. Applies to the classic + # processor only. + # enable_async_processing: true + + # Maximum number of batches queued for processing. Applies to the classic processor only. + # batch_queue_size: 3 + + # Maximum number of batches queued for asynchronous acknowledgement. Applies to the classic processor only. + # ack_queue_size: 10 + + # Whether incoming records are deduplicated. Applies to the classic processor only. + # dedup: false + + # Maximum number of entries kept in the deduplication set. Applies to the classic processor only. + # dedup_max_size: 1024 + + # How failed records are handled. `ignore` drops them, and `dlq` writes them to the dead letter queue. + # error_handling: dlq + + # Maximum number of messages stored per dead letter queue stream. + # dlq_max_messages: 1000 + + # Data type that the records are stored as. `hash` writes a Redis hash, and `json` writes a RedisJSON document, which + # requires the RedisJSON module in the target database. # target_data_type: hash - # Enable merge as the default strategy for writing JSON documents. - # json_update_strategy: merge - # Confirm that writes reached a target database replica. + + # How existing JSON documents are updated. `replace` overwrites the whole document, and `merge` merges the incoming + # fields into it. + # json_update_strategy: replace + + # Whether JSON documents are merged with the native JSON.MERGE command rather than with Lua scripts. Applies to the + # classic processor only. + # use_native_json_merge: true + + # Number of parallel processes that perform the initial synchronization. Applies to the classic processor only. + # initial_sync_processes: 4 + + # Time in milliseconds to sleep between batches while idle. Applies to the classic processor only. + # idle_sleep_time_ms: 200 + + # Time in milliseconds between two checks for new streams while idle. Applies to the classic processor only. + # idle_streams_check_interval_ms: 1000 + + # Time in milliseconds between two checks for new streams while busy. Applies to the classic processor only. + # busy_streams_check_interval_ms: 5000 + + # Maximum number of attempts for a failed write to a target database before giving up. + # retry_max_attempts: 5 + + # Initial delay in milliseconds before the first retry of a failed write. + # retry_initial_delay_ms: 1000 + + # Maximum delay in milliseconds between two retries of a failed write. + # retry_max_delay_ms: 10000 + + # Whether every write is verified to have reached the replica shards of the target database. Enable this only when the + # target database is replicated and a healthy replica is available. # wait_enabled: false + + # Maximum time in milliseconds to wait for the replica write verification. # wait_timeout: 1000 + + # Whether a write is retried until the replica verification succeeds, rather than given up on after the first failure. # retry_on_replica_failure: true - # Flink processor performance settings. - # advanced: - # source: - # batch.size: 2000 - # batch.timeout.ms: 100 - # discovery.interval.ms: 1000 - # target: - # batch.size: 200 - # flush.interval.ms: 100 - # flink: - # taskmanager.numberOfTaskSlots: 1 - # taskmanager.memory.process.size: 2048m - # resources: - # taskManager: - # replicas: 2 + + # Logging settings of the processor. Applies to the Flink processor only. + # logging: + # Log verbosity of the processor, one of `trace`, `debug`, `info`, `warn`, or `error`. + # level: info ``` diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md index e1cef5a262..127b363ab5 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md @@ -20,7 +20,12 @@ weight: 1 Each database uses a different mechanism to track changes to its data and generally, these mechanisms are not switched on by default. RDI's Debezium collector uses these mechanisms for change data capture (CDC), -so you must prepare your source database before you can use it with RDI. +so you must prepare each source database before you can use it with RDI. + +A pipeline can capture from more than one source database, and you must prepare each of them +separately. Follow the page for each source's database type. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) +for how to configure several sources in one pipeline. RDI supports the following source databases: diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md index 8299b1948b..42e03772a5 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md @@ -17,7 +17,6 @@ weight: 2 --- Follow the steps in the sections below to prepare an [AWS Aurora MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.Aurora.html) or [AWS RDS MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.MySQL.html) database. -database to work with RDI. Select the steps for your database type. diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md index a49be68c82..17c678c255 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md @@ -34,7 +34,7 @@ The following table summarizes the considerations to prepare a MongoDB database | Oplog | Sufficient size for snapshot and streaming | | Pre/Post Images | Enable on collections **only if using a custom key** | | Connection String | Must include all hosts, replicaSet (if applicable), authSource, credentials | -| MongoDB Atlas | **[SSL required](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-property-mongodb-ssl-enabled)**, provide root CA as `SOURCE_DB_CACERT` secret in RDI | +| MongoDB Atlas | **[SSL required](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-property-mongodb-ssl-enabled)**, provide root CA as the source's `CACERT` secret in RDI | | MongoDB mTLS | X.509 authentication requires source TLS secrets and MongoDB SSL properties | | Network | RDI Collector must reach all MongoDB nodes on required ports | @@ -90,13 +90,18 @@ db.createUser({ The RDI Collector requires a MongoDB connection string that includes all relevant hosts and authentication details. +The credential references come from the source name, which is `mongodb` in the examples on +this page. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) +for the source naming rules. + Example (Replica Set): ``` -mongodb://${SOURCE_DB_USERNAME}:${SOURCE_DB_PASSWORD}@host1:27017,host2:27017,host3:27017/?replicaSet=rs0&authSource=admin +mongodb://${MONGODB_DB_USERNAME}:${MONGODB_DB_PASSWORD}@host1:27017,host2:27017,host3:27017/?replicaSet=rs0&authSource=admin ``` Example (Sharded Cluster): ``` -mongodb://${SOURCE_DB_USERNAME}:${SOURCE_DB_PASSWORD}@host:30000 +mongodb://${MONGODB_DB_USERNAME}:${MONGODB_DB_PASSWORD}@host:30000 ``` - For Atlas, adjust the connection string accordingly (see example below). - Set `replicaSet` and `authSource` as appropriate for your deployment. @@ -119,22 +124,23 @@ db.runCommand({ ## 5. MongoDB Atlas specific requirements MongoDB Atlas only supports secure connections via SSL. -The root CA certificate for MongoDB Atlas must be added as a SOURCE_DB_CACERT secret in RDI. +The root CA certificate for MongoDB Atlas must be added as the source's `CACERT` secret in RDI. - Download the MongoDB Atlas root CA certificate. -- In RDI, add this certificate as a secret named SOURCE_DB_CACERT. +- In RDI, add this certificate with + `redis-di set-secret CACERT --db mongodb /path/to/atlas-ca.crt`. - Ensure that the `mongodb.ssl.enabled: true` setting is present in your RDI configuration. Example connection string for Atlas: ``` -mongodb+srv://${SOURCE_DB_USERNAME}:${SOURCE_DB_PASSWORD}@cluster0.mongodb.net/?authSource=admin +mongodb+srv://${MONGODB_DB_USERNAME}:${MONGODB_DB_PASSWORD}@cluster0.mongodb.net/?authSource=admin ``` ## 6. Self-hosted MongoDB mTLS and X.509 authentication -For self-hosted MongoDB deployments that require TLS, set the source CA certificate -as the `SOURCE_DB_CACERT` secret. For X.509 client certificate authentication, also -set the `SOURCE_DB_CERT` and `SOURCE_DB_KEY` secrets. See +For self-hosted MongoDB deployments that require TLS, set the source's `CACERT` secret to +the source CA certificate. For X.509 client certificate authentication, also set the +source's `CERT` and `KEY` secrets. See [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) for the full list of source database TLS and mTLS secrets. @@ -145,11 +151,12 @@ properties in the source `advanced.source` section: advanced: source: mongodb.ssl.enabled: true - mongodb.ssl.keystore: /debezium/certs/source_db_keystore + mongodb.ssl.keystore: /debezium/certs/mongodb_db_keystore mongodb.ssl.keystore.password: debezium ``` -The RDI Collector builds `/debezium/certs/source_db_keystore` from the source +The keystore is named after the source, so a source named `mongodb` uses +`/debezium/certs/mongodb_db_keystore`. The RDI Collector builds it from the source database client certificate and private key secrets. Debezium requires the `mongodb.ssl.keystore` and `mongodb.ssl.keystore.password` properties to present the client certificate to MongoDB. diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md index 5d8d2b09f7..7cea77ec11 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md @@ -1034,8 +1034,8 @@ sources: type: oracle host: host.docker.internal port: 1521 - user: ${SOURCE_DB_USERNAME} - password: ${SOURCE_DB_PASSWORD} + user: ${ORACLE_DB_USERNAME} + password: ${ORACLE_DB_PASSWORD} advanced: source: database.dbname: ORCLCDB @@ -1118,8 +1118,8 @@ sources: type: oracle host: oracle port: 1521 - user: ${SOURCE_DB_USERNAME} - password: ${SOURCE_DB_PASSWORD} + user: ${ORACLE_DB_USERNAME} + password: ${ORACLE_DB_PASSWORD} database: ORCLCDB advanced: source: @@ -1411,4 +1411,4 @@ With an XStream outbound server: - Debezium/RDI uses Oracle’s supported XStream API. - Transactions are preserved and grouped correctly. - Offsets/positions allow clean resume after restarts. -- Multiple consumers can be attached to the same outbound server if needed. \ No newline at end of file +- Multiple consumers can be attached to the same outbound server if needed. diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md index 4b4550fc80..58d63f4530 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md @@ -38,7 +38,7 @@ Snowflake is only supported with RDI deployed on Kubernetes/Helm. RDI VM mode do ```checklist {id="snowflakelist"} - [ ] [Set up Snowflake permissions](#1-set-up-snowflake-permissions) - [ ] [Configure authentication](#2-configure-authentication) -- [ ] [Set up secrets for Kubernetes deployment](#3-set-up-secrets-for-kubernetes-deployment) +- [ ] [Set up secrets](#3-set-up-secrets) - [ ] [Configure RDI for Snowflake](#4-configure-rdi-for-snowflake) ``` @@ -140,37 +140,39 @@ For enhanced security, use key-pair authentication: ALTER USER rdi_user SET RSA_PUBLIC_KEY=''; ``` -## 3. Set up secrets for Kubernetes deployment +## 3. Set up secrets -Before deploying the RDI pipeline, configure the necessary secrets. +Before deploying the RDI pipeline, configure the necessary secrets with +[`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}}). +Pass the source name with `--db`; the source configured in the next step is named `snowflake`. ### Password authentication ```bash -kubectl create secret generic source-db \ - --namespace=rdi \ - --from-literal=SOURCE_DB_USERNAME=your_username \ - --from-literal=SOURCE_DB_PASSWORD=your_password +redis-di set-secret USERNAME --db snowflake your_username +redis-di set-secret PASSWORD --db snowflake your_password ``` ### Private key authentication -Create a secret with the private key file: +Omit the password and set the private key from its file instead: ```bash -kubectl create secret generic source-db-ssl \ - --namespace=rdi \ - --from-file=client.key=/path/to/rsa_key.p8 +redis-di set-secret USERNAME --db snowflake your_username +redis-di set-secret KEY --db snowflake --file /path/to/rsa_key.p8 ``` -Also create the source-db secret with the username: +If the `.p8` file is protected by a passphrase, also set that passphrase: ```bash -kubectl create secret generic source-db \ - --namespace=rdi \ - --from-literal=SOURCE_DB_USERNAME=your_username +redis-di set-secret KEY_PASSWORD --db snowflake your_passphrase ``` +RDI stores the private key in the source's TLS secret, `snowflake-db-ssl`, and the RIOTX +collector reads it from there. See +[Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) +for the full secret reference, including how to create these secrets with `kubectl` instead. + ## 4. Configure RDI for Snowflake Use the following example configuration in your `config.yaml` file: @@ -182,8 +184,8 @@ sources: connection: type: snowflake url: "jdbc:snowflake://myaccount.snowflakecomputing.com/" - user: "${SOURCE_DB_USERNAME}" - password: "${SOURCE_DB_PASSWORD}" # Omit for key-pair auth + user: "${SNOWFLAKE_DB_USERNAME}" + password: "${SNOWFLAKE_DB_PASSWORD}" # Omit for key-pair auth database: "MYDB" warehouse: "COMPUTE_WH" # role: "RDI_ROLE" # Optional: Snowflake role diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md index 1e5b16966d..9a8d0a8029 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md @@ -35,7 +35,7 @@ checklist as you complete each step. - [ ] [Prepare for snapshot](#1-prepare-for-snapshot) - [ ] [Prepare for streaming](#2-prepare-for-streaming) - [ ] [Create a service account](#3-create-a-service-account) -- [ ] [Set up secrets for Kubernetes deployment (optional)](#4-set-up-secrets-for-kubernetes-deployment-optional) +- [ ] [Set up secrets (optional)](#4-set-up-secrets-optional) - [ ] [Configure RDI for Spanner](#5-configure-rdi-for-spanner) - [ ] [Additional Kubernetes configuration](#6-additional-kubernetes-configuration) ``` @@ -143,19 +143,29 @@ RDI supports two authentication methods for accessing Spanner: 2. **Service account credentials file**: You provide the service account key file as a Kubernetes secret. This method requires setting `use_credentials_file: true` in your RDI configuration. -## 4. Set up secrets for Kubernetes deployment (optional) +## 4. Set up secrets (optional) Before deploying the RDI pipeline, you need to configure the necessary secrets for the target -database. Instructions for setting up the target database secrets are available in the -[RDI deployment guide]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets-for-k8shelm-deployment-using-kubectl-command" >}}). +database. Set them with +[`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}}), +as described in +[Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}). +A Spanner source needs no credentials secret of its own, because it authenticates through +Workload Identity or a service account credentials file. **Optional**: If you prefer to use a service account credentials file instead of Workload Identity -authentication, you'll need to create a Spanner-specific secret named `source-db-credentials`. +authentication, you'll need to create a Spanner-specific secret named +`-db-credentials`, where `` is the name of the source in `config.yaml`. +The source configured in the next step is named `spanner`, so its secret is +`spanner-db-credentials`. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) +for the source naming rules. + This secret should contain the service account key file generated during the Spanner setup phase. Use the command below to create it: ```bash -kubectl create secret generic source-db-credentials --namespace=rdi \ +kubectl create secret generic spanner-db-credentials --namespace=rdi \ --from-file=gcp-service-account.json=~/spanner-reader-account.json \ --save-config --dry-run=client -o yaml | kubectl apply -f - ``` @@ -163,8 +173,13 @@ kubectl create secret generic source-db-credentials --namespace=rdi \ Be sure to adjust the file path (`~/spanner-reader-account.json`) if your service account key is stored elsewhere. +{{< note >}}`redis-di set-secret` has no key for this secret, so you must +create it with `kubectl`. A secret created that way also has to be labeled so that the RDI operator +discovers it as a pipeline secret. See +[Set secrets for K8s/Helm deployment using Kubectl command]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets-for-k8shelm-deployment-using-kubectl-command" >}}).{{< /note >}} + {{< note >}} -If you create the `source-db-credentials` secret, you must also set `use_credentials_file: true` +If you create the `spanner-db-credentials` secret, you must also set `use_credentials_file: true` in your RDI configuration to use the credentials file instead of Workload Identity authentication. {{< /note >}} @@ -175,7 +190,7 @@ When configuring your RDI pipeline for Spanner, use the following example config ```yaml sources: - source: + spanner: type: flink connection: type: spanner diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md index 19f91ca52f..8201a91cae 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md @@ -610,8 +610,8 @@ sources: host: .database.windows.net port: 1433 database: - user: ${SOURCE_DB_USERNAME} - password: ${SOURCE_DB_PASSWORD} + user: ${SQLSERVER_DB_USERNAME} + password: ${SQLSERVER_DB_PASSWORD} logging: level: info schemas: @@ -646,20 +646,27 @@ Debezium SQL Server connector and JDBC driver. The Azure-specific values are: | `snapshot.mode` | The Debezium snapshot strategy. | `initial`. Captures a snapshot of the existing rows, then streams subsequent changes from the CDC tables. | For SQL authentication, omit the `driver.authentication` line and set -`${SOURCE_DB_USERNAME}` and `${SOURCE_DB_PASSWORD}` to the SQL user's credentials. +`${SQLSERVER_DB_USERNAME}` and `${SQLSERVER_DB_PASSWORD}` to the SQL user's credentials. Keep the other Azure-specific properties. #### Secret mapping -For Microsoft Entra service-principal authentication, the RDI source secret must +For Microsoft Entra service-principal authentication, the source's credentials must provide: -| Secret key | Value | +| Config reference | Value | | --- | --- | -| `SOURCE_DB_USERNAME` | The service principal's **Application (client) ID** (a GUID). | -| `SOURCE_DB_PASSWORD` | The service principal's **client secret**. | +| `${SQLSERVER_DB_USERNAME}` | The service principal's **Application (client) ID** (a GUID). | +| `${SQLSERVER_DB_PASSWORD}` | The service principal's **client secret**. | -{{< warning >}}The `SOURCE_DB_USERNAME` value is the client ID (a GUID), but the contained +Set them with the source name, which for the source in this example is `sqlserver`: + +```bash +redis-di set-secret USERNAME --db sqlserver +redis-di set-secret PASSWORD --db sqlserver +``` + +{{< warning >}}The username value is the client ID (a GUID), but the contained database user created in the previous section uses the service principal's **display name**. These are two different identifiers for the same principal — mixing them up is the most common cause of `Login failed for user ''` errors @@ -711,7 +718,7 @@ GO - **`Login failed for user ''`** — the contained database user was not created for this service principal, or it was created with the wrong identifier. Verify that the `CREATE USER ... FROM EXTERNAL PROVIDER` statement used - the service principal's display name, and that `SOURCE_DB_USERNAME` contains its + the service principal's display name, and that `${SQLSERVER_DB_USERNAME}` contains its client ID. Query `sys.database_principals` on the source database to see which principals exist. - **`SSL Server certificate validation failed` or hostname mismatch** — @@ -781,4 +788,4 @@ documentation for further details. {{< note >}}RDI will *not* correctly capture changes that happen in the time gap between changing the source schema (step 1 above) and updating the value of `@capture_instance` (step 2). Try to keep the gap as short as possible or perform the update at a time when you expect -few changes to the data.{{< /note >}} \ No newline at end of file +few changes to the data.{{< /note >}} diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md index bcc2e31454..1da4762516 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md @@ -163,12 +163,13 @@ In the Supabase dashboard, go to 1. Enable **Enforce SSL on incoming connections**. 1. Download the Supabase CA certificate. -Store the database username, password, and CA certificate as RDI secrets: +Store the database username, password, and CA certificate as RDI secrets. Pass the +source name with `--db`; the source configured in the next step is named `supabase`: ```bash -redis-di set-secret SOURCE_DB_USERNAME rdi_replication -redis-di set-secret SOURCE_DB_PASSWORD '' -redis-di set-secret SOURCE_DB_CACERT /path/to/prod-ca-2021.crt +redis-di set-secret USERNAME --db supabase rdi_replication +redis-di set-secret PASSWORD --db supabase '' +redis-di set-secret CACERT --db supabase /path/to/prod-ca-2021.crt ``` RDI verifies that the direct endpoint hostname matches the certificate. @@ -187,8 +188,8 @@ sources: host: db..supabase.co port: 5432 database: postgres - user: ${SOURCE_DB_USERNAME} - password: ${SOURCE_DB_PASSWORD} + user: ${SUPABASE_DB_USERNAME} + password: ${SUPABASE_DB_PASSWORD} schemas: - public tables: diff --git a/content/integrate/redis-data-integration/data-pipelines/rejected-records.md b/content/integrate/redis-data-integration/data-pipelines/rejected-records.md index 73a8c50b92..6e490252f8 100644 --- a/content/integrate/redis-data-integration/data-pipelines/rejected-records.md +++ b/content/integrate/redis-data-integration/data-pipelines/rejected-records.md @@ -41,25 +41,23 @@ RDI stores rejected records in the RDI database as capped Redis streams. Each DL stream corresponds to a source table and tracks the records rejected for that table. -DLQ stream names use the `dlq:` prefix followed by the source data stream name. -In current RDI versions, the stream name is typically: +A DLQ stream is named after the source data stream it corresponds to, with a `dlq:` prefix: ```text -dlq:data:{rdi}:.
+dlq:data:{rdi}:..
``` -For example, rejected records for the `public.users` table are stored in: +For example, rejected records for the `public.users` table of a source named `postgresql` +are stored in: ```text -dlq:data:{rdi}:public.users +dlq:data:{rdi}:postgresql.public.users ``` -For sources that include the source name in the stream qualifier, the final part -can contain three components: - -```text -dlq:data:{rdi}:..
-``` +The dead-letter queue endpoints and the CLI report table names in that same qualified form, +so a rejected record is always attributed to the source it came from. Sources that existed +before the upgrade to a version that supports multiple sources do not have the source segment +in the key. Some RDI versions or configurations can use a hash-tagged variant such as `dlq:{data:rdi:.
}`. To find all DLQ streams in the diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md index 3b0b2cfa35..e1a9c13a6a 100644 --- a/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md @@ -58,7 +58,11 @@ The main sections of these files are: - `source`: This is a mandatory section that specifies the data items that you want to use. You can add the following properties here: - - `server_name`: Logical server name (optional). + - `server_name`: The name of the source whose records this job processes, as it appears + in the `sources` section of + [config.yaml]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config" >}}). + This is required when the pipeline has more than one source but is optional in a pipeline with a single source. See + [Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for more information. - `db`: Database name (optional). This refers to a database name you supplied in [config.yaml]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config" >}}). - `schema`: Database schema (optional). This refers to a schema name you supplied in @@ -72,6 +76,18 @@ The main sections of these files are: and is set to `true` by default. Set it to `false` if you need to use case-sensitive values for these properties. + With the + [Flink processor]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}}), + `server_name`, `db`, `schema`, and `table` each accept either a single name or a list of names. + An entry prefixed with `regex:` selects all names that match the regular expression, which has + to match the whole name. For example, `server_name: [mysql, postgresql]` selects two sources, + and `table: "regex:orders_[0-9]+"` selects every numbered `orders` table. A job processes the + records that match all of the properties it sets, so a single job can process multiple tables, + potentially from different sources, databases, or schemas. + + The classic processor takes a single name in each of these properties. It rejects a list or a + `regex:` entry when you deploy the pipeline, so write one job per table instead. + - `transform`: This is an optional section describing the transformation that the pipeline applies to the data before writing it to the target. The `uses` property specifies a *transformation block* that will use the parameters supplied in the `with` section. See the @@ -126,7 +142,7 @@ the default logic. ```yaml name: Rename field example source: - server_name: redislabs + server_name: sqlserver schema: dbo table: emp transform: diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-json-example.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-json-example.md index d2b0380d81..668176b41b 100644 --- a/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-json-example.md +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-json-example.md @@ -26,7 +26,7 @@ database to use this feature. In the example below, the data is captured from the source table named `invoice` and is written to the Redis database as a JSON document. The `connection` is an optional parameter that refers to the corresponding connection name defined in `config.yaml`. When you specify the `data_type` parameter for the job, it overrides the system-wide setting `target_data_type` defined in `config.yaml`. -Another optional parameter, `on_update`, specifies the writing strategy. You can set this to either `replace` (the default) or `merge`. This affects the way the document is written to the target. Replacing the document will overwrite it completely, while merging will update it with the fields captured in the source, keeping the rest of the document intact. The `replace` option is usually more performant, while `merge` allows other jobs and applications to set extra fields in the same JSON documents. +Another optional parameter, `on_update`, specifies the writing strategy. You can set this to either `replace` (the default) or `merge`. This affects the way the document is written to the target. Replacing the document will overwrite it completely, while merging will update it with the fields captured in the source, keeping the rest of the document intact. The `replace` option is usually more performant, while with `merge` other jobs and applications can set extra fields in the same JSON documents. In this case, the result will be Redis JSON documents with key names based on the key expression (for example, `invoice_id:1`) and with an expiration of 100 seconds. If you don't supply an `expire` parameter, the keys will never expire. diff --git a/content/integrate/redis-data-integration/faq.md b/content/integrate/redis-data-integration/faq.md index 5c86fc184c..3c3e43eab5 100644 --- a/content/integrate/redis-data-integration/faq.md +++ b/content/integrate/redis-data-integration/faq.md @@ -86,6 +86,12 @@ deployment, use separate Kubernetes clusters. See [Install on Kubernetes]({{< relref "/integrate/redis-data-integration/installation/install-k8s" >}}) for installation details. +## Can one pipeline capture from several source databases? + +Yes. Add one entry per source to the `sources` section of `config.yaml`. Each source has its +own collector, which captures change records independently of the other sources. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}). + ## Can RDI automatically track changes to the source database schema? If you don't configure RDI to capture a specific set of tables in the schema then it will diff --git a/content/integrate/redis-data-integration/installation/_index.md b/content/integrate/redis-data-integration/installation/_index.md index c5e7bdee75..5dac61f0bb 100644 --- a/content/integrate/redis-data-integration/installation/_index.md +++ b/content/integrate/redis-data-integration/installation/_index.md @@ -19,6 +19,6 @@ weight: 20 The guides in this section explain the options you have for installing and upgrading RDI on your own servers. See the [Redis Cloud RDI guide]({{< relref "/operate/rc/rdi" >}}) to learn how to set up RDI for a cloud database. -Before you use RDI, you must also configure your source database to enable CDC. See the +Before you use RDI, you must also configure each of your source databases to enable CDC. See the [Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) -section to learn how to do this. \ No newline at end of file +section to learn how to do this. diff --git a/content/integrate/redis-data-integration/installation/ha-test.md b/content/integrate/redis-data-integration/installation/ha-test.md index 7f89c08514..910248528d 100644 --- a/content/integrate/redis-data-integration/installation/ha-test.md +++ b/content/integrate/redis-data-integration/installation/ha-test.md @@ -24,7 +24,7 @@ kubectl -n rdi get pods # Example output: NAME READY STATUS RESTARTS AGE collector-api-577d95bfd8-5wbg6 1/1 Running 0 12m -collector-source-95f45bcf7-vwn5l 1/1 Running 0 12m +collector-mysql-95f45bcf7-vwn5l 1/1 Running 0 12m fluentd-zq2lc 1/1 Running 0 72m logrotate-29530445-j729x 0/1 Completed 0 14m logrotate-29530450-dprr2 0/1 Completed 0 9m40s @@ -36,7 +36,27 @@ rdi-operator-7f7f6c7dfd-5qmjd 1/1 Running 0 71m rdi-reloader-77df5f7854-lwmvz 1/1 Running 0 71m ``` -2. Identify the leader node - this is the one that has a running `collector-source` pod. +2. Identify the leader node (this is the one that has a running collector pod). A + collector is named after its source, so the example above shows `collector-mysql` for a + source named `mysql`. + + You can also ask each node about the pipeline directly. On the follower, the pipeline + stands by: + + ``` + kubectl -n rdi get pipeline default -o jsonpath='{.status.phase}' + + # Example output on the follower: + Standby + + # Example output on the leader: + Active + ``` + + [`redis-di describe`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-describe" >}}) + shows `Status: standby` on the follower, and + [`redis-di info`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-info" >}}) + shows `Leader Election Mode: follower`. ## Performing the HA Failover Testing @@ -73,6 +93,9 @@ kubectl -n rdi logs rdi-operator-7f7f6c7dfd-5qmjd -f In about 10 seconds you will start seeing log entries from the leader saying that it could not acquire the leadership. When the leader lock expires, the second node will acquire the leadership and you will see log entries from the second node indicating that it has become the leader. +Once the failover completes, the two nodes swap their pipeline phases: the pipeline becomes +`Active` on the new leader and `Standby` on the old one. + ## Cleanup To clean up after the test, remove the `iptables` rule that you added to block the traffic: diff --git a/content/integrate/redis-data-integration/installation/install-k8s.md b/content/integrate/redis-data-integration/installation/install-k8s.md index 55b0ebfa6f..47ef52b70c 100644 --- a/content/integrate/redis-data-integration/installation/install-k8s.md +++ b/content/integrate/redis-data-integration/installation/install-k8s.md @@ -423,15 +423,17 @@ Specifically, ensure that one or both of the following Helm chart values is set: - `controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz` - `controller.service.externalTrafficPolicy=Local` -## Prepare your source database +## Prepare your source databases -Before deploying a pipeline, you must configure your source database to enable CDC. See the +Before deploying a pipeline, you must configure each source database to enable CDC. See the [Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) -section to learn how to do this. +section to learn how to do this. A pipeline can capture from more than one source database, +and each one has to be prepared. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for details. ## Deploy a pipeline -When the Helm installation is complete and you have prepared the source database for CDC, +When the Helm installation is complete and you have prepared your source databases for CDC, you are ready to start using RDI. See the guides on how to [configure]({{< relref "/integrate/redis-data-integration/data-pipelines" >}}) and [deploy]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}}) diff --git a/content/integrate/redis-data-integration/installation/install-vm.md b/content/integrate/redis-data-integration/installation/install-vm.md index 63700922fa..37580f71e9 100644 --- a/content/integrate/redis-data-integration/installation/install-vm.md +++ b/content/integrate/redis-data-integration/installation/install-vm.md @@ -17,7 +17,7 @@ weight: 10 --- This guide explains how to install Redis Data Integration (RDI) on one or more VMs and integrate it with -your source database. You can also +your source databases. You can also [Install RDI on Kubernetes]({{< relref "/integrate/redis-data-integration/installation/install-k8s" >}}). {{< note >}}We recommend you always use the latest version, which is RDI v{{< rdi-version >}}. @@ -269,17 +269,31 @@ to renew the lease in the RDI database, it will lose the leadership and a failov will take place. After the failover, the secondary instance will become the primary one, and the RDI pipeline will be active on that VM. +The secondary instance keeps the pipeline and its configuration, and reports it with the +`standby` status, but runs none of its collectors or its stream processor, and it does not +touch the pipeline's keys in the RDI database. A standby pipeline is read-only, so +running [`redis-di describe`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-describe" >}}) +against the secondary instance works and shows `Status: standby`. Attempting to change a standby pipeline +fails with `423 Locked` and a message saying that the pipeline is standing by because this instance is +currently a leader election follower. Operations such as deploy, start, stop, reset, and delete +can only be performed against the primary instance. + +Setting pipeline secrets works on a standby instance. Secrets are maintained per +instance and are not replicated between the two, because they can legitimately differ in some cases. + You may find it useful to trigger a failover deliberately to check that RDI is correctly configured to handle it. See [Test HA failover]({{< relref "/integrate/redis-data-integration/installation/ha-test" >}}) to learn how to do this. -## Prepare your source database +## Prepare your source databases -Before deploying a pipeline, you must configure your source database to enable CDC. See the +Before deploying a pipeline, you must configure each source database to enable CDC. See the [Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) -section to learn how to do this. +section to learn how to do this. A pipeline can capture from more than one source database, +and you must prepare each one separately. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for more information. ## Deploy a pipeline -When the installation is complete, and you have prepared the source database for CDC, +When the installation is complete, and you have prepared your source databases for CDC, you are ready to start using RDI. See the guides on how to [configure]({{< relref "/integrate/redis-data-integration/data-pipelines" >}}) and [deploy]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}}) @@ -331,4 +345,4 @@ This will uninstall RDI and its dependencies, are you sure? [y, N] ``` If you type anything other than "y" here, the script will abort without making any changes -to RDI or your source database. +to RDI or your source databases. diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index 964881e79d..bbc454af6b 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -96,7 +96,7 @@ run the CLI binary of the previous RDI version, which still provided the `redis- sudo redis-di upgrade --rdi-host --rdi-port ``` -{{< note >}}If the `collector-source` or the `processor` pods are not in the `Running` state after +{{< note >}}If the `collector` or `processor` pods are not in the `Running` state after the upgrade, you must run `redis-di deploy` and check again that they are both in the `Running` state. {{< /note >}} @@ -196,6 +196,15 @@ described in [Uninstall RDI]({{< relref "/integrate/redis-data-integration/insta and then install the old version. {{< /note >}} +## Source names after the upgrade + +For a source that existed before upgrading to a version that supports multiple sources, the +resource names from before the upgrade are kept unchanged, so it will keep running without +changes. + +For more details on how to handle sources created after upgrading, see +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}). + ## Enabling the Flink processor The @@ -225,7 +234,7 @@ The upgrade process replaces the current RDI components with their new versions: - Firstly, the control plane components are replaced. At this point, the pipeline is still active but monitoring will be disconnected. - Secondly, the pipeline data plane components are replaced. - If a pipeline is active while upgrading, the `collector-source` and `processor` + If a pipeline is active while upgrading, the `collector` and `processor` pods will be restarted. The pipeline will pause for up to two minutes but it will catch up very quickly after restarting. The pipeline data and state are both stored in Redis, so data will not diff --git a/content/integrate/redis-data-integration/observability.md b/content/integrate/redis-data-integration/observability.md index eabcb2ab0d..219b380626 100644 --- a/content/integrate/redis-data-integration/observability.md +++ b/content/integrate/redis-data-integration/observability.md @@ -50,10 +50,25 @@ The way you access the metrics endpoints depends on whether you are using a VM i ### VM Installation +You can use the CLI to determine the metrics endpoints instead of assembling them yourself. +The `Components` section of +[`redis-di describe`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-describe" >}}) +has a `Metrics Endpoints` column listing each component's externally-reachable metrics URLs, +so a pipeline with several sources shows the URL of every collector. + +RDI discovers these endpoints from each component's `Ingress` resources, so a component +without an ingress has none, and the list is empty until the ingress controller assigns an +address to the ingress. + For VM installations, the metrics are available by default on the following endpoints: -- Collector metrics: `https:///collector-source/metrics` -- Stream processor metrics: `https:///processor/metrics` -- Operator metrics: `https:///operator/metrics` +- Collector metrics: `https://:/collector-/metrics`, where `` is + the source name from `config.yaml`. Each source has its own collector, so a pipeline with + several sources has one endpoint per source. +- Stream processor metrics: `https://:/processor/metrics` (Flink processor) + or `https://:/metrics` (classic processor). +- Operator metrics: `https://:/operator/metrics` + +Above, `` is the HTTPS port you specified when installing RDI, or 443 if you kept the default. Please note that for RDI versions prior to 1.16.0 the collector metrics are not accessible. @@ -223,7 +238,7 @@ RDI reports with their descriptions. - Where the metric name has the `rdi_` prefix, this will be replaced by the Kubernetes namespace name if you supplied a custom name during installation. The prefix is always `rdi_` for VM installations. - Metrics with the `_created` suffix are automatically generated by Prometheus for counters and gauges to track when they were first created. -- The `rdi_incoming_entries` metric provides a detailed breakdown for each data source by operation type. +- The `rdi_incoming_entries` metric provides a detailed breakdown for each data source by operation type. The `data_source` label identifies the data stream the value belongs to, as the stream's key without the `data:` prefix (for example, `{rdi}:mysql.inventory.customers`). The key contains the source name, so you can break the metric down per source. Sources that existed before the upgrade to a version that supports multiple sources do not have the source segment in the key. - The `rdi_stream_event_latency_ms` metric helps monitor data freshness and processing delays. - The processor performance metrics are divided into two categories: - **Total metrics**: Accumulate values across all processed batches for historical analysis diff --git a/content/integrate/redis-data-integration/quick-start-guide.md b/content/integrate/redis-data-integration/quick-start-guide.md index 44134d605f..bedf666619 100644 --- a/content/integrate/redis-data-integration/quick-start-guide.md +++ b/content/integrate/redis-data-integration/quick-start-guide.md @@ -71,6 +71,10 @@ the `config.yaml` file and then edit the following settings: At this point, the pipeline is ready to deploy. +This tutorial uses one source database, but a pipeline can capture from several source databases +into the same Redis target. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}). + ### Create a context (optional) {#create-context} To manage and inspect RDI, you can use the diff --git a/content/integrate/redis-data-integration/reference/api-migration.md b/content/integrate/redis-data-integration/reference/api-migration.md index eed803b967..cd4dddbe79 100644 --- a/content/integrate/redis-data-integration/reference/api-migration.md +++ b/content/integrate/redis-data-integration/reference/api-migration.md @@ -27,6 +27,16 @@ RDI 1.19.0 supports only one pipeline, which must be named `default`. Support fo The API version is part of the URL. Update `/api/v1` requests to use `/api/v2` where a corresponding v2 endpoint is available. You should also review the request and response models, because they can differ between versions. +## API v1 and multiple sources + +API v1 handles single-source pipelines only, so +[a pipeline with several sources]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) +has to be managed through API v2. In particular: + +- `POST /api/v1/pipelines`, `PATCH /api/v1/pipelines`, and `POST /api/v1/pipelines/sources/dry-run` reject a configuration with more than one source with `422 Unprocessable Content` and `Only a single source per pipeline is supported`. +- `PUT /api/v1/secrets`, `PUT /api/v1/secrets/{secret_name}`, and the v1 source management endpoints only handle a pipeline whose single source still uses the legacy `source` names, which covers a pipeline that predates per-source naming and one that you create through v1. They reject any other pipeline with `Use API v2 to manage sources and secrets of this pipeline.` +- `GET /api/v1/status` and `GET /api/v1/monitoring/statistics` report the first source of the pipeline only. + ## Endpoint mapping | API v1 | API v2 | @@ -45,7 +55,8 @@ The API version is part of the URL. Update `/api/v1` requests to use `/api/v2` w | `PUT /api/v1/pipelines/sources` and source subresources | `PATCH /api/v2/pipelines/{name}` with `sources` in the payload | | `PUT /api/v1/pipelines/targets` and target subresources | `PATCH /api/v2/pipelines/{name}` with `targets` in the payload | | `PUT /api/v1/pipelines/processors` and `PUT /api/v1/pipelines/processors/{prop}` | `PATCH /api/v2/pipelines/{name}` with `processors` in the payload | -| Secret provider endpoints | `POST`, `PUT`, or `DELETE /api/v2/pipelines/{name}/secrets[/{key}]` | +| `PUT /api/v1/secrets` and `PUT /api/v1/secrets/{secret_name}` | `POST`, `PUT`, or `DELETE /api/v2/pipelines/{name}/secrets[/{key}]` with the `db` query parameter | +| `POST`, `PUT`, or `DELETE /api/v1/pipelines/secret-providers/{name}` | `PATCH /api/v2/pipelines/{name}` with `secret-providers` in the payload | | Source metadata, schemas, databases, tables, and columns endpoints | `GET /api/v2/pipelines/{name}/source-schemas/{source_name}` with the appropriate filters | | `POST /api/v1/pipelines/sources/dry-run` | `POST /api/v2/pipelines?dry_run=true` | | `POST /api/v1/pipelines/targets/dry-run` | `POST /api/v2/pipelines?dry_run=true` | @@ -118,7 +129,8 @@ Note: 2. Add the pipeline name to each v2 request. The only pipeline in 1.19.0 is always named `default`. 3. Check the pipeline response, or call `GET /api/v2/pipelines/{name}/status`, instead of polling an action ID. 4. Use `POST /api/v2/pipelines`, `PUT /api/v2/pipelines/{name}`, or `PATCH /api/v2/pipelines/{name}` to update source, target, processor, and secret-provider settings as needed. When using `PATCH`, omit the configuration sections that you do not want to change. -5. Use `GET /api/v2/pipelines/{name}/metric-collections/{collection_name}` for monitoring and `GET /api/v2/pipelines/{name}/source-schemas/{source_name}` for source metadata. -6. Test creating, updating, validating, starting, stopping, resetting, and deleting a pipeline on a non-production RDI 1.19.0 or later installation before updating production applications. +5. Change secret requests to address a secret by a database-independent key and a `db` parameter. Where a v1 request set `SOURCE_DB_PASSWORD`, a v2 request sets the `PASSWORD` key with `db` naming the source, and `TARGET_DB_PASSWORD` becomes the `PASSWORD` key with `db=target`. See [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}). +6. Use `GET /api/v2/pipelines/{name}/metric-collections/{collection_name}` for monitoring and `GET /api/v2/pipelines/{name}/source-schemas/{source_name}` for source metadata. +7. Test creating, updating, validating, starting, stopping, resetting, and deleting a pipeline on a non-production RDI 1.19.0 or later installation before updating production applications. Authentication and the API base URL do not change. The migration requires updates to the endpoint paths, pipeline scoping, request models, and operation status handling. diff --git a/content/integrate/redis-data-integration/reference/api-reference/openapi.json b/content/integrate/redis-data-integration/reference/api-reference/openapi.json index 8c3c37d013..5e4d90d779 100644 --- a/content/integrate/redis-data-integration/reference/api-reference/openapi.json +++ b/content/integrate/redis-data-integration/reference/api-reference/openapi.json @@ -50,6 +50,16 @@ } } } + }, + "401": { + "description": "Unauthorized", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } } }, "security": [] @@ -77,7 +87,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -87,7 +97,27 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Not Found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "423": { + "description": "Locked", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -122,7 +152,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -132,7 +162,27 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Not Found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "423": { + "description": "Locked", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -149,7 +199,7 @@ "post": { "tags": ["secure"], "summary": "Reset Pipeline", - "description": "Reset pipeline.", + "description": "Resets a pipeline.", "operationId": "reset_pipeline_api_v1_pipelines_reset_post", "responses": { "200": { @@ -167,7 +217,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -177,7 +227,27 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Not Found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "423": { + "description": "Locked", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -228,7 +298,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -238,7 +308,17 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Action not found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ActionResponse" } } } @@ -270,7 +350,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -280,16 +360,13 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } } }, "security": [ - { - "JWTBearer": [] - }, { "JWTBearer": [] } @@ -338,15 +415,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Update Sources Api V1 Pipelines Sources Put" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -356,7 +425,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -366,7 +435,37 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Not Found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Unprocessable Entity", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + } + }, + "423": { + "description": "Locked", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -435,15 +534,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Upsert Source By Name Api V1 Pipelines Sources Name Put" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -453,7 +544,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -463,10 +554,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } }, @@ -519,15 +640,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Partial Update Source By Name Api V1 Pipelines Sources Name Patch" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -537,7 +650,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -547,10 +660,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } }, @@ -581,15 +724,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Delete Source By Name Api V1 Pipelines Sources Name Delete" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -599,7 +734,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -609,10 +744,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } } @@ -652,15 +817,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Update Targets Api V1 Pipelines Targets Put" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -670,7 +827,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -680,7 +837,37 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Not Found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Unprocessable Entity", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + } + }, + "423": { + "description": "Locked", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -742,15 +929,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Upsert Target By Name Api V1 Pipelines Targets Name Put" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -760,7 +939,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -770,10 +949,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } }, @@ -823,15 +1032,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Partial Update Target By Name Api V1 Pipelines Targets Name Patch" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -841,7 +1042,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -851,16 +1052,46 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } - } - } - }, - "delete": { - "tags": ["secure", "targets"], - "summary": "Delete Target By Name", + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" + } + } + }, + "delete": { + "tags": ["secure", "targets"], + "summary": "Delete Target By Name", "description": "Deletes a target from RDI settings by its name.", "operationId": "delete_target_by_name_api_v1_pipelines_targets__name__delete", "security": [ @@ -885,15 +1116,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Delete Target By Name Api V1 Pipelines Targets Name Delete" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -903,7 +1126,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -913,10 +1136,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } } @@ -978,15 +1231,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Upsert Secret Provider By Name Api V1 Pipelines Secret Providers Name Put" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -996,7 +1241,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1006,10 +1251,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } }, @@ -1057,15 +1332,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Partial Update Secret Provider By Name Api V1 Pipelines Secret Providers Name Patch" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -1075,7 +1342,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1085,10 +1352,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } }, @@ -1119,15 +1416,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Delete Secret Provider By Name Api V1 Pipelines Secret Providers Name Delete" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -1137,7 +1426,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1147,10 +1436,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } } @@ -1159,7 +1478,7 @@ "post": { "tags": ["secure", "connection"], "summary": "Sources Dry Run", - "description": "Tests the connection to the source database.\n\nArgs:\n body: The connection details according to the OpenAPI Scheme.\n\nReturns:\n Union[ConnectionStatusSingleConnection, Error]: The connection status.", + "description": "Tests the connection to the source database.", "operationId": "sources_dry_run_api_v1_pipelines_sources_dry_run_post", "requestBody": { "content": { @@ -1195,15 +1514,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ConnectionStatusSingleConnection" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Sources Dry Run Api V1 Pipelines Sources Dry Run Post" + "$ref": "#/components/schemas/ConnectionStatusSingleConnection" } } } @@ -1213,7 +1524,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1223,7 +1534,47 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Unprocessable Entity", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "500": { + "description": "Internal Server Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "503": { + "description": "Service Unavailable", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "504": { + "description": "Gateway Timeout", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1287,7 +1638,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1297,7 +1648,27 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "400": { + "description": "Bad Request", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Unprocessable Entity", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1351,7 +1722,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1361,7 +1732,17 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Unprocessable Entity", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1441,7 +1822,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1451,7 +1832,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1461,20 +1842,60 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, "423": { "description": "Collector API is not available for the specified source or is disabled", "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, + "503": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } } @@ -1517,7 +1938,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1527,7 +1948,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1537,20 +1958,60 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, "423": { "description": "Collector API is not available for the specified source or is disabled", "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, + "503": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } } @@ -1593,7 +2054,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1603,7 +2064,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1613,20 +2074,60 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, "423": { "description": "Collector API is not available for the specified source or is disabled", "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, + "503": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } } @@ -1705,7 +2206,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1715,30 +2216,70 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "The specified source is not found in the current pipeline", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "description": "Collector API is not available for the specified source or is disabled", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } }, - "404": { - "description": "The specified source is not found in the current pipeline", + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, + "503": { "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } - } + }, + "description": "Service Unavailable" }, - "423": { - "description": "Collector API is not available for the specified source or is disabled", + "504": { "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } - } + }, + "description": "Gateway Timeout" } } } @@ -1792,7 +2333,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1802,7 +2343,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1812,20 +2353,60 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, "423": { "description": "Collector API is not available for the specified source or is disabled", "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, + "503": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } } @@ -1833,9 +2414,9 @@ "/api/v1/pipelines": { "get": { "tags": ["secure", "deploy"], - "summary": "Retrieve Pipelines", + "summary": "Get Pipelines", "description": "Returns the current pipeline.", - "operationId": "retrieve_pipelines_api_v1_pipelines_get", + "operationId": "get_pipelines_api_v1_pipelines_get", "security": [ { "JWTBearer": [] @@ -1849,7 +2430,7 @@ "schema": { "type": "object", "additionalProperties": true, - "title": "Response Retrieve Pipelines Api V1 Pipelines Get" + "title": "Response Get Pipelines Api V1 Pipelines Get" } } } @@ -1859,7 +2440,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1869,18 +2450,28 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" } } }, "post": { "tags": ["secure", "deploy"], - "summary": "Pipelines", + "summary": "Post Pipelines", "description": "Updates the current pipeline completely, or creates it if it doesn't exist, using the provided configuration.\nBefore creating or updating, validates the provided configuration for schema compliance and integrity.", - "operationId": "pipelines_api_v1_pipelines_post", + "operationId": "post_pipelines_api_v1_pipelines_post", "security": [ { "JWTBearer": [] @@ -1892,14 +2483,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "type": "boolean", "description": "Only validate the provided configuration without making changes", "default": false, "title": "Dry Run" @@ -2102,7 +2686,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2112,7 +2696,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2164,6 +2748,46 @@ } } } + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, + "503": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } }, @@ -2183,14 +2807,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "type": "boolean", "description": "Only validate the provided configuration without making changes", "default": false, "title": "Dry Run" @@ -2406,7 +3023,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2416,7 +3033,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2468,6 +3085,46 @@ } } } + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, + "503": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } } @@ -2484,15 +3141,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Undeploy Pipelines Api V1 Pipelines Undeploy Post" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -2502,7 +3151,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2512,7 +3161,37 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Not Found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Unprocessable Entity", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + } + }, + "423": { + "description": "Locked", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2547,7 +3226,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2557,7 +3236,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2607,6 +3286,24 @@ "title": "Db Flavor" }, "description": "Optional database flavor" + }, + { + "name": "source_name", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "Optional source name; defaults to the database type", + "title": "Source Name" + }, + "description": "Optional source name; defaults to the database type" } ], "responses": { @@ -2615,15 +3312,6 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/TemplateResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" - } - ], - "title": "Response Pipelines Config Templates Api V1 Pipelines Config Templates Ingest Db Type Get", "$ref": "#/components/schemas/TemplateResponse" } } @@ -2634,7 +3322,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2644,7 +3332,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2653,7 +3341,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } }, @@ -2663,7 +3351,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } }, @@ -2678,22 +3366,38 @@ "summary": "Pipelines Jobs Templates", "description": "Returns the YAML templates describing how to create a job", "operationId": "pipelines_jobs_templates_api_v1_pipelines_jobs_templates_ingest_get", + "security": [ + { + "JWTBearer": [] + } + ], + "parameters": [ + { + "name": "source_name", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "Optional source name; defaults to a placeholder", + "title": "Source Name" + }, + "description": "Optional source name; defaults to a placeholder" + } + ], "responses": { "200": { "description": "Successful Response", "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/TemplateResponse", - "anyOf": [ - { - "$ref": "#/components/schemas/TemplateResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" - } - ], - "title": "Response Pipelines Jobs Templates Api V1 Pipelines Jobs Templates Ingest Get" + "$ref": "#/components/schemas/TemplateResponse" } } } @@ -2703,7 +3407,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2713,27 +3417,22 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } }, - "501": { - "description": "Not Implemented", + "422": { "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } - } - } - }, - "security": [ - { - "JWTBearer": [] + }, + "description": "Unprocessable Entity" } - ] + } } }, "/api/v1/pipelines/jobs/dry-run": { @@ -2758,15 +3457,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/JobDryRunSuccessResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Job Dry Run Api V1 Pipelines Jobs Dry Run Post" + "$ref": "#/components/schemas/JobDryRunSuccessResponse" } } } @@ -2776,7 +3467,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2786,7 +3477,17 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "400": { + "description": "Bad Request", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2796,7 +3497,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" + "$ref": "#/components/schemas/ExtendedErrorResponse" } } } @@ -2831,7 +3532,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2841,7 +3542,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2878,7 +3579,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2888,7 +3589,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2925,7 +3626,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2935,7 +3636,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2983,35 +3684,57 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Update Processors Api V1 Pipelines Processors Put" + "$ref": "#/components/schemas/ActionIdResponse" + } + } + } + }, + "401": { + "description": "Unauthorized", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "403": { + "description": "Forbidden", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Not Found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } }, - "401": { - "description": "Unauthorized", + "422": { + "description": "Unprocessable Entity", "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/ExtendedErrorResponse" } } } }, - "403": { - "description": "Forbidden", + "423": { + "description": "Locked", "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3084,15 +3807,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Upsert Processors Property Api V1 Pipelines Processors Prop Put" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -3102,7 +3817,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3112,10 +3827,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } } @@ -3168,7 +3913,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3178,7 +3923,17 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Use API v2 to manage sources and secrets of this pipeline.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3218,7 +3973,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3228,7 +3983,17 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Use API v2 to manage sources and secrets of this pipeline.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3263,7 +4028,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3273,7 +4038,17 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Not Found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3308,7 +4083,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3318,7 +4093,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3335,7 +4110,7 @@ "post": { "tags": ["secure"], "summary": "Start Trace", - "description": "Starts a trace session for troubleshooting data transformation.", + "description": "Starts a trace session for troubleshooting data transformation.\n\nTracing is no longer supported; the operator does not process trace tasks anymore.", "operationId": "start_trace_api_v1_trace_start_post", "requestBody": { "content": { @@ -3363,7 +4138,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3373,7 +4148,17 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "501": { + "description": "Not Implemented", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3450,14 +4235,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "type": "boolean", "description": "Only validate the provided configuration without making changes", "default": false, "title": "Dry Run" @@ -3551,6 +4329,26 @@ }, "description": "Unprocessable Entity" }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Locked" + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, "503": { "content": { "application/json": { @@ -3560,6 +4358,16 @@ } }, "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } } @@ -3634,7 +4442,7 @@ "put": { "tags": ["v2", "pipelines"], "summary": "Update Pipeline", - "description": "Updates a pipeline completely, or creates it if it doesn't exist.\nRetries 409 conflicts received from the K8s API server.", + "description": "Updates a pipeline completely, or creates it if it doesn't exist.", "operationId": "update_pipeline_api_v2_pipelines__name__put", "security": [ { @@ -3658,14 +4466,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "type": "boolean", "description": "Only validate the provided configuration without making changes", "default": false, "title": "Dry Run" @@ -3749,6 +4550,26 @@ }, "description": "Unprocessable Entity" }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Locked" + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, "503": { "content": { "application/json": { @@ -3758,13 +4579,23 @@ } }, "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } }, "patch": { "tags": ["v2", "pipelines"], "summary": "Patch Pipeline", - "description": "Updates a pipeline partially (only updates specified fields), with retries on 409 conflicts received from the K8s API server.", + "description": "Updates a pipeline partially (only updates specified fields).", "operationId": "patch_pipeline_api_v2_pipelines__name__patch", "security": [ { @@ -3788,14 +4619,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "type": "boolean", "description": "Only validate the provided configuration without making changes", "default": false, "title": "Dry Run" @@ -3889,6 +4713,26 @@ }, "description": "Unprocessable Entity" }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Locked" + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, "503": { "content": { "application/json": { @@ -3898,13 +4742,23 @@ } }, "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } }, "delete": { "tags": ["v2", "pipelines"], "summary": "Delete Pipeline", - "description": "Replaces a pipeline with an empty inactive pipeline and resets it.\n\nThis is done to ensure that no artifacts (offsets, schema, data streams etc.) are left behind,\nas this may cause errors and unexpected behavior when trying to re-create the pipeline.", + "description": "Replaces a pipeline with an empty inactive pipeline.", "operationId": "delete_pipeline_api_v2_pipelines__name__delete", "security": [ { @@ -3957,6 +4811,16 @@ } }, "description": "Not Found" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Locked" } } } @@ -4033,7 +4897,7 @@ "post": { "tags": ["v2", "pipelines"], "summary": "Start Pipeline", - "description": "Starts a pipeline, retrying on 409 conflicts received from the K8s API server.", + "description": "Starts a pipeline, or a single source when one is given.", "operationId": "start_pipeline_api_v2_pipelines__name__start_post", "security": [ { @@ -4051,6 +4915,24 @@ "title": "Name" }, "description": "Pipeline name" + }, + { + "name": "source", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "Start only this source; it runs only while the pipeline itself is running. Must be an existing source that is not of type 'external'.", + "title": "Source" + }, + "description": "Start only this source; it runs only while the pipeline itself is running. Must be an existing source that is not of type 'external'." } ], "responses": { @@ -4084,6 +4966,16 @@ }, "description": "Forbidden" }, + "400": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Bad Request" + }, "404": { "content": { "application/json": { @@ -4093,6 +4985,16 @@ } }, "description": "Not Found" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Locked" } } } @@ -4101,7 +5003,7 @@ "post": { "tags": ["v2", "pipelines"], "summary": "Stop Pipeline", - "description": "Stops a pipeline, retrying on 409 conflicts received from the K8s API server.", + "description": "Stops a pipeline, or a single source when one is given.", "operationId": "stop_pipeline_api_v2_pipelines__name__stop_post", "security": [ { @@ -4119,6 +5021,24 @@ "title": "Name" }, "description": "Pipeline name" + }, + { + "name": "source", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "Stop only this source; the pipeline and its other sources keep running. Must be an existing source that is not of type 'external'.", + "title": "Source" + }, + "description": "Stop only this source; the pipeline and its other sources keep running. Must be an existing source that is not of type 'external'." } ], "responses": { @@ -4130,9 +5050,29 @@ "$ref": "#/components/schemas/PipelineResponse" } } - } + } + }, + "401": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Unauthorized" + }, + "403": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Forbidden" }, - "401": { + "400": { "content": { "application/json": { "schema": { @@ -4140,9 +5080,9 @@ } } }, - "description": "Unauthorized" + "description": "Bad Request" }, - "403": { + "404": { "content": { "application/json": { "schema": { @@ -4150,9 +5090,9 @@ } } }, - "description": "Forbidden" + "description": "Not Found" }, - "404": { + "423": { "content": { "application/json": { "schema": { @@ -4160,7 +5100,7 @@ } } }, - "description": "Not Found" + "description": "Locked" } } } @@ -4169,7 +5109,7 @@ "post": { "tags": ["v2", "pipelines"], "summary": "Reset Pipeline", - "description": "Resets a pipeline, retrying on 409 conflicts received from the K8s API server.", + "description": "Resets a pipeline.", "operationId": "reset_pipeline_api_v2_pipelines__name__reset_post", "security": [ { @@ -4229,6 +5169,16 @@ } }, "description": "Not Found" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Locked" } } } @@ -4380,6 +5330,16 @@ } }, "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" } } } @@ -4710,7 +5670,7 @@ "get": { "tags": ["v2", "pipelines", "secrets"], "summary": "Get Pipeline Secrets", - "description": "Lists all pipeline secrets.", + "description": "Lists pipeline secrets, optionally filtered to a single database.", "operationId": "get_pipeline_secrets_api_v2_pipelines__name__secrets_get", "security": [ { @@ -4728,6 +5688,24 @@ "title": "Name" }, "description": "Pipeline name" + }, + { + "name": "db", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "If set, list only the secrets of this database.", + "title": "Db" + }, + "description": "If set, list only the secrets of this database." } ], "responses": { @@ -4798,6 +5776,36 @@ "title": "Name" }, "description": "Pipeline name" + }, + { + "name": "db", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "The database a secret belongs to: a source name, or 'target' for the target database. Omit for legacy SOURCE_DB_* and TARGET_DB_* keys.", + "title": "Db" + }, + "description": "The database a secret belongs to: a source name, or 'target' for the target database. Omit for legacy SOURCE_DB_* and TARGET_DB_* keys." + }, + { + "name": "validate_source", + "in": "query", + "required": false, + "schema": { + "type": "boolean", + "description": "When true, require the source to already exist in the pipeline.", + "default": false, + "title": "Validate Source" + }, + "description": "When true, require the source to already exist in the pipeline." } ], "requestBody": { @@ -4861,6 +5869,16 @@ } }, "description": "Conflict" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" } } } @@ -4898,6 +5916,24 @@ "title": "Key" }, "description": "Pipeline secret key" + }, + { + "name": "db", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "The database a secret belongs to: a source name, or 'target' for the target database. Omit for legacy SOURCE_DB_* and TARGET_DB_* keys.", + "title": "Db" + }, + "description": "The database a secret belongs to: a source name, or 'target' for the target database. Omit for legacy SOURCE_DB_* and TARGET_DB_* keys." } ], "responses": { @@ -4940,6 +5976,16 @@ } }, "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" } } }, @@ -4975,6 +6021,24 @@ "title": "Key" }, "description": "Pipeline secret key" + }, + { + "name": "db", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "The database a secret belongs to: a source name, or 'target' for the target database. Omit for legacy SOURCE_DB_* and TARGET_DB_* keys.", + "title": "Db" + }, + "description": "The database a secret belongs to: a source name, or 'target' for the target database. Omit for legacy SOURCE_DB_* and TARGET_DB_* keys." } ], "requestBody": { @@ -5073,6 +6137,24 @@ "title": "Key" }, "description": "Pipeline secret key" + }, + { + "name": "db", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "The database a secret belongs to: a source name, or 'target' for the target database. Omit for legacy SOURCE_DB_* and TARGET_DB_* keys.", + "title": "Db" + }, + "description": "The database a secret belongs to: a source name, or 'target' for the target database. Omit for legacy SOURCE_DB_* and TARGET_DB_* keys." } ], "responses": { @@ -5108,6 +6190,16 @@ } }, "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" } } } @@ -5244,6 +6336,46 @@ } }, "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Locked" + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, + "503": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } } @@ -5334,6 +6466,16 @@ }, "description": "Not Found" }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, "502": { "content": { "application/json": { @@ -5353,6 +6495,16 @@ } }, "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } } @@ -5373,6 +6525,16 @@ } } } + }, + "500": { + "description": "Internal Server Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + } } }, "security": [] @@ -5426,7 +6588,7 @@ "error": { "anyOf": [ { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__sources__Error" + "$ref": "#/components/schemas/redis_di_api__v1__models__sources__Error" }, { "type": "null" @@ -5499,8 +6661,7 @@ "collector-api", "stream-processor", "metrics-exporter", - "flink-processor-jobmanager", - "flink-processor-taskmanager", + "flink-collector", "flink-processor", "riotx-collector", "unknown" @@ -5517,7 +6678,7 @@ "error": { "anyOf": [ { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__sources__Error" + "$ref": "#/components/schemas/redis_di_api__v1__models__sources__Error" }, { "type": "null" @@ -5884,7 +7045,6 @@ "DbType": { "type": "string", "enum": [ - "cassandra", "mariadb", "mongodb", "mysql", @@ -6089,54 +7249,29 @@ "title": "InfoResponse", "description": "Response model containing information about the API and operator." }, - "Job": { + "JobDryRunBody": { "properties": { - "source": { - "$ref": "#/components/schemas/Source" - }, - "transform": { - "anyOf": [ + "job": { + "additionalProperties": true, + "type": "object", + "title": "Job settings to dry run with.", + "examples": [ { - "items": { - "$ref": "#/components/schemas/Transform" + "source": { + "table": "employee" }, - "type": "array" - }, - { - "type": "null" - } - ], - "title": "Transform" - }, - "output": { - "items": { - "$ref": "#/components/schemas/Output" - }, - "type": "array", - "title": "Output" - }, - "name": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" + "transform": [ + { + "uses": "add_field", + "with": { + "expression": "FNAME || ' ' || LAST_NAME", + "field": "FullName", + "language": "sql" + } + } + ] } - ], - "title": "Name", - "examples": ["Alex"] - } - }, - "type": "object", - "required": ["source", "output"], - "title": "Job" - }, - "JobDryRunBody": { - "properties": { - "job": { - "$ref": "#/components/schemas/Job", - "title": "Job settings to dry run with." + ] }, "input_data": { "items": { @@ -6200,6 +7335,19 @@ "title": "LeaderElectionMode", "description": "Operator leader-election mode." }, + "LegacySecretName": { + "type": "string", + "enum": [ + "source-db", + "target-db", + "rdi-sys-config", + "rdi-db-ssl", + "source-db-ssl", + "target-db-ssl" + ], + "title": "LegacySecretName", + "description": "The secret names API v1 accepts as the request key. They are kept for backward compatibility\nonly, since the K8s secret is derived from each key instead." + }, "MetadataResponse": { "properties": { "tables": { @@ -6240,35 +7388,6 @@ "title": "OperationResponse", "description": "Represents the response of an operation." }, - "Output": { - "properties": { - "uses": { - "type": "string", - "title": "Uses", - "examples": ["redis.write"] - }, - "with": { - "anyOf": [ - { - "$ref": "#/components/schemas/WithInput" - }, - { - "$ref": "#/components/schemas/WithFields" - }, - { - "$ref": "#/components/schemas/WithNest" - }, - { - "$ref": "#/components/schemas/WithConnection" - } - ], - "title": "With" - } - }, - "type": "object", - "required": ["uses", "with"], - "title": "Output" - }, "Pipeline": { "properties": { "status": { @@ -6312,6 +7431,17 @@ "type": { "$ref": "#/components/schemas/ComponentType" }, + "source": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Source" + }, "version": { "type": "string", "title": "Version" @@ -6352,6 +7482,14 @@ } ], "title": "Metric Collections" + }, + "external_endpoints": { + "items": { + "type": "string" + }, + "type": "array", + "title": "External Endpoints", + "default": [] } }, "additionalProperties": false, @@ -6583,6 +7721,10 @@ "type": "string", "title": "Key" }, + "db": { + "type": "string", + "title": "Db" + }, "type": { "type": "string", "enum": ["simple", "file", "binary_file"], @@ -6591,7 +7733,7 @@ }, "additionalProperties": false, "type": "object", - "required": ["key", "type"], + "required": ["key", "db", "type"], "title": "PipelineSecretResponse", "description": "Response model for a pipeline secret." }, @@ -6917,25 +8059,12 @@ } ] }, - "SecretName": { - "type": "string", - "enum": [ - "source-db", - "target-db", - "rdi-sys-config", - "rdi-db-ssl", - "source-db-ssl", - "target-db-ssl" - ], - "title": "SecretName", - "description": "Defines supported secret names." - }, "SecretsModel": { "additionalProperties": { "$ref": "#/components/schemas/SecretKeyModel" }, "propertyNames": { - "$ref": "#/components/schemas/SecretName" + "$ref": "#/components/schemas/LegacySecretName" }, "type": "object", "title": "SecretsModel", @@ -6959,50 +8088,6 @@ "title": "SortOrder", "description": "Sort order for DLQ records." }, - "Source": { - "properties": { - "schema": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Schema", - "examples": ["public"] - }, - "table": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Table", - "examples": ["employee"] - }, - "row_format": { - "anyOf": [ - { - "type": "string", - "enum": ["full", "partial"] - }, - { - "type": "null" - } - ], - "title": "Row Format", - "examples": ["full"] - } - }, - "additionalProperties": true, - "type": "object", - "title": "Source" - }, "SourceColumnResponse": { "properties": { "type": { @@ -7312,6 +8397,7 @@ "stopping", "resetting", "pending", + "standby", "unknown" ], "title": "Status", @@ -7501,35 +8587,6 @@ "title": "TraceRequest", "description": "Request model for the trace endpoint." }, - "Transform": { - "properties": { - "uses": { - "type": "string", - "title": "Uses", - "examples": ["add_field"] - }, - "with": { - "anyOf": [ - { - "$ref": "#/components/schemas/WithInput" - }, - { - "$ref": "#/components/schemas/WithFields" - }, - { - "$ref": "#/components/schemas/WithNest" - }, - { - "$ref": "#/components/schemas/WithConnection" - } - ], - "title": "With" - } - }, - "type": "object", - "required": ["uses", "with"], - "title": "Transform" - }, "ValidationError": { "properties": { "loc": { @@ -7553,226 +8610,18 @@ "type": { "type": "string", "title": "Error Type" - } - }, - "type": "object", - "required": ["loc", "msg", "type"], - "title": "ValidationError" - }, - "WithConnection": { - "properties": { - "connection": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Connection", - "examples": ["target"] - }, - "data_type": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Data Type", - "examples": ["hash"] - }, - "args": { - "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "items": {}, - "type": "array" - }, - { - "type": "null" - } - ], - "title": "Args", - "examples": [ - { - "member": "value" - }, - ["expr1"] - ] - }, - "key": { - "anyOf": [ - { - "$ref": "#/components/schemas/WithInput" - }, - { - "type": "null" - } - ] - }, - "cmd": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Cmd", - "description": "Redis command to execute, e.g. HGETALL.", - "examples": ["HGETALL"] - }, - "language": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Language", - "examples": ["jmespath", "sql"] - }, - "field": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Field", - "description": "Target field name for the lookup result." - } - }, - "type": "object", - "title": "WithConnection" - }, - "WithFieldRef": { - "properties": { - "field": { - "type": "string", - "title": "Field", - "description": "Field name to reference or remove.", - "examples": ["branch"] - } - }, - "type": "object", - "required": ["field"], - "title": "WithFieldRef" - }, - "WithFields": { - "properties": { - "fields": { - "items": { - "anyOf": [ - { - "$ref": "#/components/schemas/WithInput" - }, - { - "$ref": "#/components/schemas/WithFieldRef" - } - ] - }, - "type": "array", - "title": "Fields" - } - }, - "type": "object", - "required": ["fields"], - "title": "WithFields" - }, - "WithInput": { - "properties": { - "expression": { - "type": "string", - "title": "Expression", - "examples": [ - "FNAME || ' ' || LAST_NAME", - "concat([fname, ' ' , last_name])" - ] }, - "field": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Field", - "description": "This field it available only for a subset of with expressions e.g. `add_field`, `remove_field`.", - "examples": ["FullName"] + "input": { + "title": "Input" }, - "language": { - "type": "string", - "title": "Language", - "examples": ["sql", "jmespath"] - } - }, - "type": "object", - "required": ["expression", "language"], - "title": "WithInput" - }, - "WithNest": { - "properties": { - "nest": { - "additionalProperties": true, + "ctx": { "type": "object", - "title": "Nest", - "description": "Nesting configuration for hierarchical data structures." - }, - "on_update": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "On Update", - "examples": ["merge", "replace"] - }, - "data_type": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Data Type", - "examples": ["json", "hash"] - }, - "connection": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Connection", - "examples": ["target"] + "title": "Context" } }, "type": "object", - "required": ["nest"], - "title": "WithNest" + "required": ["loc", "msg", "type"], + "title": "ValidationError" }, "redis_di_api__models__errors__Error": { "properties": { @@ -7832,27 +8681,7 @@ "title": "ErrorResponse", "description": "Generic error response model" }, - "redis_di_api__v1__modules__shared__response__Error": { - "properties": { - "status": { - "type": "string", - "const": "error", - "title": "The status of the run.", - "default": "error", - "examples": ["error"] - }, - "error": { - "type": "string", - "title": "The error message.", - "examples": ["Job is malformed"] - } - }, - "type": "object", - "required": ["error"], - "title": "Error", - "description": "Model representing an error status." - }, - "redis_di_api__v1__modules__shared__response__ErrorResponse": { + "redis_di_api__v1__models__responses__ErrorResponse": { "properties": { "detail": { "type": "string", @@ -7865,7 +8694,7 @@ "title": "ErrorResponse", "description": "Model representing an API error response." }, - "redis_di_api__v1__modules__shared__sources__Error": { + "redis_di_api__v1__models__sources__Error": { "properties": { "message": { "type": "string", diff --git a/content/integrate/redis-data-integration/reference/cli/_index.md b/content/integrate/redis-data-integration/reference/cli/_index.md index c584fc120e..34f8887417 100644 --- a/content/integrate/redis-data-integration/reference/cli/_index.md +++ b/content/integrate/redis-data-integration/reference/cli/_index.md @@ -86,6 +86,11 @@ Pipeline-scoped commands take the pipeline name as an optional positional argume take their own key or name as the positional argument and target the pipeline with the `-p` / `--pipeline` option, which also defaults to `default`. +Two options narrow a command to one database of the pipeline. The secret commands take `--db` with +a source name or `target` to address that database's secrets, and `start`, `stop`, and `reset` take +`--source` with a source name to act on that source alone. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}). + The commands group as follows: - **Information**: [`info`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-info" >}}). diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-delete-secret.md b/content/integrate/redis-data-integration/reference/cli/redis-di-delete-secret.md index d55224832c..e82e81f785 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-delete-secret.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-delete-secret.md @@ -22,6 +22,7 @@ redis-di delete-secret [flags] | Option | Description | | :----------------- | :-------------------------------------------------------------------------------- | | `-p`, `--pipeline` | Pipeline to target (default `default`). | +| `--db` | Database the secret belongs to: a source name, or `target`. | | `--force` | Skip the confirmation prompt. | | `--wait` | Wait for the pipeline to reach the expected state (default `true`). | | `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). | @@ -32,5 +33,5 @@ This command also accepts the ## Example ```bash -redis-di delete-secret SOURCE_DB_CACERT --force +redis-di delete-secret CACERT --db mysql --force ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-deploy.md b/content/integrate/redis-data-integration/reference/cli/redis-di-deploy.md index 9e66e27268..c21fec3834 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-deploy.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-deploy.md @@ -29,7 +29,7 @@ The pipeline name is an optional argument that defaults to `default`. | `--dir` | Directory containing the pipeline configuration (default `.`). | | `--dry-run` | Validate the configuration without deploying. | | `--validate-tables` | Validate the configuration against the source and target databases (default `true`). | -| `--validate-cdc` | Validate the source database CDC configuration. | +| `--validate-cdc` | Validate the CDC configuration of the source databases. | | `--start` | Start the pipeline after deploying (default `true`). | | `--wait` | Wait for the pipeline to reach the expected state (default `true`). | | `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). | diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-describe-secret.md b/content/integrate/redis-data-integration/reference/cli/redis-di-describe-secret.md index 3ca2e2c7a2..e84fdd362f 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-describe-secret.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-describe-secret.md @@ -19,9 +19,10 @@ redis-di describe-secret [flags] ## Options -| Option | Description | -| :----------------- | :-------------------------------------- | -| `-p`, `--pipeline` | Pipeline to target (default `default`). | +| Option | Description | +| :----------------- | :---------------------------------------------------------- | +| `-p`, `--pipeline` | Pipeline to target (default `default`). | +| `--db` | Database the secret belongs to: a source name, or `target`. | This command also accepts the [global options]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di#global-options" >}}). @@ -29,5 +30,5 @@ This command also accepts the ## Example ```bash -redis-di describe-secret TARGET_DB_PASSWORD +redis-di describe-secret PASSWORD --db target ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-describe.md b/content/integrate/redis-data-integration/reference/cli/redis-di-describe.md index 0ff5a63243..c0cbda106f 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-describe.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-describe.md @@ -42,7 +42,8 @@ To watch the status update live, pair the command with `watch`: watch -n 1 redis-di describe ``` -The output has a section for each part of the pipeline, for example: +The output has a section for each part of the pipeline. The example below shows a pipeline with +two sources, `mysql` and `postgresql`: ``` Name: default @@ -51,30 +52,43 @@ Status: started Current: yes Sources: - Name Type Db Type Connection Sync Mode Connected - ---- ---- ------- ---------- --------- --------- - mysql cdc mysql ${HOST_IP}:13000 streaming yes + Name Type Db Type Connection Sync Mode Connected + ---- ---- ------- ---------- --------- --------- + mysql cdc mysql :3306 streaming yes + postgresql cdc postgresql :5432 streaming yes Targets: - Name Db Type Connection Connected - ---- ------- ---------- --------- - target redis ${HOST_IP}:12000 yes + Name Db Type Connection Connected + ---- ------- ---------- --------- + target redis :12000 yes + +Processor: + Type: flink Jobs: - Name Source Transformations Outputs Connections - ---- ------ --------------- ------- ----------- - address_job inventory.addresses 1 1 target - customers_hash_job inventory.customers 0 1 target + Name Server Name Db / Schema Table Transformations Outputs Connections + ---- ----------- ----------- ----- --------------- ------- ----------- + billing_job postgresql public customers 1 1 target + customers_hash_job mysql inventory customers 0 1 target + orders_job mysql inventory orders 1 1 target Components: - Name Type Version Status - ---- ---- ------- ------ - collector-api collector-api 0.0.0 started - collector-source debezium-collector ... started - processor processor 0.0.0 started + Name Type Version Status Metrics Endpoints + ---- ---- ------- ------ ----------------- + collector-api collector-api started + collector-mysql debezium-collector started https://:443/collector-mysql/metrics + collector-postgresql debezium-collector started https://:443/collector-postgresql/metrics + flink-processor flink-processor started https://:443/processor/metrics Statistics: - Name Total Pending Inserted Updated Deleted Filtered Rejected Deduplicated Last Arrival - ---- ----- ------- -------- ------- ------- -------- -------- ------------ ------------ - {rdi}:inventory.customers 4 0 4 0 0 0 0 0 2026-06-18T13:42:44Z + Name Incoming Pending Inserted Updated Deleted Filtered Rejected Last Arrival + ---- -------- ------- -------- ------- ------- -------- -------- ------------ + mysql.inventory.customers 4 0 4 0 0 0 0 2026-06-18T13:42:44Z + mysql.inventory.orders 12 0 12 0 0 0 0 2026-06-18T13:42:51Z + postgresql.public.customers 7 0 7 0 0 0 0 2026-06-18T13:42:49Z ``` + +The `Metrics Endpoints` column shows the externally-reachable metrics URLs of each component, +which RDI discovers from the component's `Ingress` resources. It is empty for a component that +has no ingress, or while the ingress controller has not assigned an address yet. See +[Accessing the metrics]({{< relref "/integrate/redis-data-integration/observability#accessing-the-metrics" >}}). diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-get-dlq.md b/content/integrate/redis-data-integration/reference/cli/redis-di-get-dlq.md index 994ac42c9d..f782d2b593 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-get-dlq.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-get-dlq.md @@ -11,6 +11,10 @@ aliases: Gets a single dead-letter queue (DLQ) of a pipeline and prints it in the compact `list-dlqs` table format. +The name is the DLQ's source-qualified table name, `..
`, as +[`list-dlqs`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs" >}}) +reports it. + ## Usage ``` @@ -30,5 +34,5 @@ This command also accepts the ## Example ```bash -redis-di get-dlq inventory.customers +redis-di get-dlq mysql.inventory.customers ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-get-metric-collection.md b/content/integrate/redis-data-integration/reference/cli/redis-di-get-metric-collection.md index 7a427c22d0..74716a4e6f 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-get-metric-collection.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-get-metric-collection.md @@ -32,5 +32,6 @@ This command also accepts the ## Example ```bash -redis-di get-metric-collection processor -o json +redis-di get-metric-collection processor_metrics -o json +redis-di get-metric-collection collector-mysql_metrics -o json ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-get-secret.md b/content/integrate/redis-data-integration/reference/cli/redis-di-get-secret.md index 0b94f16073..4b710d3e9b 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-get-secret.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-get-secret.md @@ -20,10 +20,11 @@ redis-di get-secret [flags] ## Options -| Option | Description | -| :----------------- | :--------------------------------------------------- | -| `-p`, `--pipeline` | Pipeline to target (default `default`). | -| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. | +| Option | Description | +| :----------------- | :---------------------------------------------------------- | +| `-p`, `--pipeline` | Pipeline to target (default `default`). | +| `--db` | Database the secret belongs to: a source name, or `target`. | +| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. | This command also accepts the [global options]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di#global-options" >}}). @@ -31,5 +32,5 @@ This command also accepts the ## Example ```bash -redis-di get-secret SOURCE_DB_USERNAME +redis-di get-secret USERNAME --db mysql ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records.md b/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records.md index 49b1744364..23d1668be3 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records.md @@ -16,6 +16,10 @@ and paging with `--limit`, `--offset`, and `--sort-order`. The operation code is Use [`list-dlqs`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs" >}}) to see all the pipeline's dead-letter queues and their record counts. +The DLQ name is its source-qualified table name, `..
`, as +[`list-dlqs`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs" >}}) +reports it. + ## Usage ``` @@ -48,8 +52,8 @@ This command also accepts the ```bash # Newest 20 rejected records of a queue -redis-di list-dlq-records inventory.customers +redis-di list-dlq-records mysql.inventory.customers # Oldest 100 records, as JSON -redis-di list-dlq-records inventory.customers --limit 100 --sort-order asc -o json +redis-di list-dlq-records mysql.inventory.customers --limit 100 --sort-order asc -o json ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs.md b/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs.md index 13b96386a7..2f67107b68 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs.md @@ -13,6 +13,9 @@ that RDI rejected. Use [`list-dlq-records`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records" >}}) to read the records of a single queue. +There is one queue per source table, named `..
`, where +`` is the source name from `config.yaml`. + ## Usage ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-list-secrets.md b/content/integrate/redis-data-integration/reference/cli/redis-di-list-secrets.md index 14c64f63c5..aea6a1bb9c 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-list-secrets.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-list-secrets.md @@ -19,10 +19,11 @@ redis-di list-secrets [flags] ## Options -| Option | Description | -| :----------------- | :--------------------------------------------------- | -| `-p`, `--pipeline` | Pipeline to target (default `default`). | -| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. | +| Option | Description | +| :----------------- | :------------------------------------------------------------------ | +| `-p`, `--pipeline` | Pipeline to target (default `default`). | +| `--db` | List only the secrets of this database: a source name, or `target`. | +| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. | This command also accepts the [global options]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di#global-options" >}}). @@ -31,5 +32,6 @@ This command also accepts the ```bash redis-di list-secrets +redis-di list-secrets --db mysql redis-di list-secrets -p my-pipeline ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md b/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md index 14d627c0a4..e5e7728f3d 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md @@ -25,6 +25,7 @@ The pipeline name is an optional argument that defaults to `default`. | Option | Description | | :---------- | :-------------------------------------------------------------------------------- | +| `--source` | Target only this source instead of the whole pipeline. | | `--wait` | Wait for the pipeline to reach the expected state (default `true`). | | `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). | @@ -35,4 +36,7 @@ This command also accepts the ```bash redis-di reset + +# Reset only source mysql +redis-di reset --source mysql ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-scaffold.md b/content/integrate/redis-data-integration/reference/cli/redis-di-scaffold.md index 927acb678d..20aef77298 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-scaffold.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-scaffold.md @@ -10,8 +10,9 @@ aliases: --- Generates a starter pipeline configuration for the given source database type. With `--dir`, the -command writes a `config.yaml` file into that directory, prompting before overwriting an existing -file unless `--force` is set. Without `--dir`, it prints the configuration to standard output. +command writes a `config.yaml` file and a `jobs/job.yaml` file into that directory, prompting +before overwriting existing files unless `--force` is set. Without `--dir`, it prints the +configuration to standard output. ## Usage @@ -21,12 +22,13 @@ redis-di scaffold [flags] ## Options -| Option | Description | -| :------------ | :------------------------------------------------------------------------------------------------------------------------------ | -| `--db-type` | Source database type (required): `mariadb`, `mongodb`, `mysql`, `oracle`, `postgresql`, `snowflake`, `sqlserver`, or `spanner`. | -| `--db-flavor` | Source database flavor: `mongodb-atlas`, `mongodb-replica-set`, or `mongodb-sharded-cluster`. | -| `--dir` | Directory to write `config.yaml` to; prints to standard output when omitted. | -| `--force` | Skip the confirmation prompt when overwriting an existing file. | +| Option | Description | +| :-------------- | :------------------------------------------------------------------------------------------------------------------------------ | +| `--db-type` | Source database type (required): `mariadb`, `mongodb`, `mysql`, `oracle`, `postgresql`, `snowflake`, `sqlserver`, or `spanner`. | +| `--db-flavor` | Source database flavor: `mongodb-atlas`, `mongodb-replica-set`, or `mongodb-sharded-cluster`. | +| `--source-name` | Name of the generated source; defaults to the database type. | +| `--dir` | Directory to write the configuration and job files to; prints the configuration to standard output when omitted. | +| `--force` | Skip the confirmation prompt when overwriting an existing file. | This command also accepts the [global options]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di#global-options" >}}). @@ -39,4 +41,7 @@ redis-di scaffold --db-type postgresql # Write a MySQL configuration into a directory redis-di scaffold --db-type mysql --dir /opt/rdi/config + +# Name the generated source 'inventory' instead of 'mysql' +redis-di scaffold --db-type mysql --source-name inventory --dir /opt/rdi/config ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-set-secret.md b/content/integrate/redis-data-integration/reference/cli/redis-di-set-secret.md index f82e5978db..1bb2e01988 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-set-secret.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-set-secret.md @@ -10,10 +10,11 @@ aliases: --- Creates or updates a secret of a pipeline. Secrets hold the credentials and certificates that the -pipeline uses to connect to the source and target databases (see +pipeline uses to connect to its source and target databases (see [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) -for the list of secret names). You can then refer to a secret in the `config.yaml` file with the -syntax `${SECRET_NAME}`. +for the secret keys). Every secret belongs to one database, which you name with `--db`: a source, or +`target`. RDI combines the key and the database into the environment variable that `config.yaml` +references, so `PASSWORD` with `--db mysql` becomes `${MYSQL_DB_PASSWORD}`. The secret value comes from the `[value]` argument, the `--file` option, or the `--literal` option. If you provide none of these on an interactive terminal, the command prompts for the value without @@ -30,6 +31,7 @@ redis-di set-secret [value] [flags] | Option | Description | | :----------------- | :-------------------------------------------------------------------------------- | | `-p`, `--pipeline` | Pipeline to target (default `default`). | +| `--db` | Database the secret belongs to: a source name, or `target`. | | `--file` | Read the secret value from the file at this path. | | `--literal` | Use this literal string as the secret value. | | `--wait` | Wait for the pipeline to reach the expected state (default `true`). | @@ -42,11 +44,14 @@ This command also accepts the ```bash # Value from an argument -redis-di set-secret SOURCE_DB_USERNAME myuser +redis-di set-secret USERNAME --db mysql myuser # Value from a file (for example, a certificate) -redis-di set-secret SOURCE_DB_CACERT --file /path/to/myca.crt +redis-di set-secret CACERT --db mysql --file /path/to/myca.crt # Value read from an interactive prompt -redis-di set-secret SOURCE_DB_PASSWORD +redis-di set-secret PASSWORD --db mysql + +# Target database credentials +redis-di set-secret PASSWORD --db target mypassword ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-start.md b/content/integrate/redis-data-integration/reference/cli/redis-di-start.md index 23a5b9b124..53e61df7c5 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-start.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-start.md @@ -24,6 +24,7 @@ The pipeline name is an optional argument that defaults to `default`. | Option | Description | | :---------- | :-------------------------------------------------------------------------------- | +| `--source` | Target only this source instead of the whole pipeline. | | `--wait` | Wait for the pipeline to reach the expected state (default `true`). | | `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). | @@ -35,4 +36,7 @@ This command also accepts the ```bash redis-di start redis-di start my-pipeline --wait=false + +# Start only source mysql. Does not start the pipeline if it's currently stopped. +redis-di start --source mysql ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-stop.md b/content/integrate/redis-data-integration/reference/cli/redis-di-stop.md index 17b8dd5877..f2849cfe0f 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-stop.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-stop.md @@ -24,6 +24,7 @@ The pipeline name is an optional argument that defaults to `default`. | Option | Description | | :---------- | :-------------------------------------------------------------------------------- | +| `--source` | Target only this source instead of the whole pipeline. | | `--wait` | Wait for the pipeline to reach the expected state (default `true`). | | `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). | @@ -34,4 +35,7 @@ This command also accepts the ```bash redis-di stop + +# Stop only source mysql +redis-di stop --source mysql ``` diff --git a/content/integrate/redis-data-integration/reference/config-yaml-reference.md b/content/integrate/redis-data-integration/reference/config-yaml-reference.md index a7a94416b1..63921de22a 100644 --- a/content/integrate/redis-data-integration/reference/config-yaml-reference.md +++ b/content/integrate/redis-data-integration/reference/config-yaml-reference.md @@ -33,7 +33,7 @@ Source collectors that capture changes from upstream databases. Each key is a un |Name|Type|Description|Required| |----|----|-----------|--------| -|[**connection**](#sourcesconnection)
(Source database connection)|`object`|Connection configuration for a non-Redis source database. The exact set of properties depends on the database type.
|yes| +|[**connection**](#sourcesconnection)
(Source database connection)|`object`|Connection configuration for a non-Redis source database. The exact set of properties depends on the database type.
|no| |**name**
(Source name)|`string`|Human-readable name for the source collector. Maximum 100 characters.
Maximal Length: `100`
|no| |**type**
(Collector type)|`string`|Type of the source collector. Use `cdc` (default) for change data capture using [Debezium](https://debezium.io/). Use `flink` for Spanner change streams using the Apache Flink-based collector. Use `riotx` for Snowflake CDC using [RIOT-X](https://redis.github.io/riotx/).
Default: `"cdc"`
Enum: `"cdc"`, `"flink"`, `"riotx"`
|yes| |**active**
(Collector enabled)|`boolean`|When `true`, the collector runs; when `false`, the collector is disabled and produces no events.
Default: `true`
|no| @@ -44,6 +44,7 @@ Source collectors that capture changes from upstream databases. Each key is a un |[**advanced**](#sourcesadvanced)
(Advanced configuration)|`object`|Advanced configuration that overrides the underlying engine's defaults. Only required for non-standard tuning.
|no| +**Property Name Pattern:** `^[a-z]([a-z0-9-]*[a-z0-9])?$` ### sources\.connection: Source database connection @@ -238,7 +239,7 @@ Connection configuration for a Snowflake database. |**type**
(Database type)|`string`|Database type identifier. Always `snowflake` for this connection.
Constant Value: `"snowflake"`
|yes| |**url**
(JDBC URL)|`string`|Snowflake JDBC connection URL, for example `jdbc:snowflake://account.snowflakecomputing.com/`.
|yes| |**user**
(Snowflake user)|`string`|Username for authentication to Snowflake.
|yes| -|**password**
(Snowflake password)|`string`|Password for authentication to Snowflake. For key-pair authentication, omit this field and provide the private key via the `source-db-ssl` secret (`client.key` field).
|no| +|**password**
(Snowflake password)|`string`|Password for authentication to Snowflake. For key-pair authentication, omit this field and provide the private key via the `-db-ssl` secret (`client.key` field).
|no| |**database**
(Snowflake database)|`string`|Name of the Snowflake database to connect to.
|yes| |**warehouse**
(Snowflake warehouse)|`string`|Name of the Snowflake warehouse used for compute.
|yes| |**role**
(Snowflake role)|`string`|Snowflake role used for the connection.
|no| @@ -352,6 +353,7 @@ Advanced configuration that overrides the underlying engine's defaults. Only req |[**resources**](#sourcesadvancedresources)
(Collector resource settings)|`object`|Compute resources allocated to the collector. **Only applies to the `cdc` collector type.**
|| |[**riotx**](#sourcesadvancedriotx)
(Advanced RIOT\-X settings)|`object`|Advanced configuration properties for the RIOT-X Snowflake collector. **Only applies to the `riotx` collector type.**
|| |**java\_options**
(Advanced Java options)|`string`|These Java options will be passed to the command line command when launching the source collector. **Only applies to the `cdc` collector type.**
|| +|[**diagnostics**](#sourcesadvanceddiagnostics)
(Collector diagnostics)|`object`|Opt-in JVM diagnostics for the source collector. Each artifact is written to the `/opt/rdi/diagnostics` directory on the host and included in the `dump-support-package` bundle. **Only applies to the `cdc` collector type on VM installations, and is ignored otherwise.**
|| **Additional Properties:** not allowed **Minimal Properties:** 1 @@ -381,6 +383,8 @@ source: max.batch.size: 2048 max.queue.size: 8192 heartbeat.interval.ms: 0 + database.trustServerCertificate: false + driver.applicationIntent: ReadWrite lob.enabled: false publication.autocreate.mode: all_tables publication.name: dbz_publication @@ -400,6 +404,11 @@ riotx: streamPrefix: 'data:' clearOffset: false count: 0 +diagnostics: + heap_dump: + enabled: false + java_flight_recorder: + enabled: false ``` @@ -460,7 +469,7 @@ redis.wait.retry.delay.ms: 1000 #### sources\.advanced\.source: Advanced source settings -Advanced configuration properties for the source database connection and CDC behavior. **Applies to the `cdc` and `flink` collector types.**

For the `cdc` collector type, available properties depend on the source database — refer to the relevant Debezium connector documentation: [MySQL](https://debezium.io/documentation/reference/stable/connectors/mysql.html), [MariaDB](https://debezium.io/documentation/reference/stable/connectors/mariadb.html), [PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html), [Oracle](https://debezium.io/documentation/reference/stable/connectors/oracle.html), [SQL Server](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html), [Db2](https://debezium.io/documentation/reference/stable/connectors/db2.html), [MongoDB](https://debezium.io/documentation/reference/stable/connectors/mongodb.html). When using a property from those pages, omit the `debezium.source.` prefix.

**The named properties below cover the most commonly tuned settings: `spanner.*` properties apply to the `flink` collector type, all others apply to the `cdc` collector type. Any other property from the Debezium documentation can still be set as a free-form key-value pair.** +Advanced configuration properties for the source database connection and CDC behavior. **Applies to the `cdc` and `flink` collector types.**

For the `cdc` collector type, available properties depend on the source database — refer to the relevant Debezium connector documentation: [MySQL](https://debezium.io/documentation/reference/stable/connectors/mysql.html), [MariaDB](https://debezium.io/documentation/reference/stable/connectors/mariadb.html), [PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html), [Oracle](https://debezium.io/documentation/reference/stable/connectors/oracle.html), [SQL Server](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html), [Db2](https://debezium.io/documentation/reference/stable/connectors/db2.html), [MongoDB](https://debezium.io/documentation/reference/stable/connectors/mongodb.html). When using a property from those pages, omit the `debezium.source.` prefix.

**The named properties below cover the most commonly tuned settings: `spanner.*` properties apply to the `flink` collector type, all others apply to the `cdc` collector type. Any other property from the Debezium documentation can still be set as a free-form key-value pair, except for `topic.prefix`, which RDI derives from the source name.** **Properties** @@ -475,6 +484,8 @@ Advanced configuration properties for the source database connection and CDC beh |**max\.queue\.size**
(Max queue size)|`integer`|Limits how many records can be buffered in memory before processing catches up.
Default: `8192`
Minimum: `1`
|| |**heartbeat\.interval\.ms**
(Heartbeat interval ms)|`integer`|Sets how often heartbeat events are emitted to keep change tracking active. Use 0 to disable them.
Default: `0`
Minimum: `0`
|| |**heartbeat\.action\.query**
(Heartbeat action query)|`string`|SQL query executed on the source whenever a heartbeat is emitted.
|| +|**database\.trustServerCertificate**
(Trust SQL Server certificate)|`boolean`|Controls whether the RDI collector skips validation of the SQL Server TLS certificate. Enable this only when the server certificate cannot be validated through a trusted certificate authority.
Default: `false`
|| +|**driver\.applicationIntent**
(SQL Server application intent)|`string`|Declares the SQL Server workload type for the RDI collector. Set to `ReadOnly` when connecting to an Always On read-only replica. CDC must be enabled on the primary SQL Server node.
Default: `"ReadWrite"`
Enum: `"ReadOnly"`, `"ReadWrite"`
|| |**lob\.enabled**
(Lob enabled)|`boolean`|Determines whether large object columns are included in change capture.
Default: `false`
|| |**gtid\.source\.includes**
(GTID source includes)|`string`|Restricts MySQL GTID processing to the listed source UUIDs.
|| |**publication\.autocreate\.mode**
(Publication autocreate mode)|`string`|Controls whether and how the PostgreSQL publication is created or updated automatically.
Default: `"all_tables"`
Enum: `"all_tables"`, `"filtered"`, `"disabled"`
|| @@ -493,6 +504,8 @@ Advanced configuration properties for the source database connection and CDC beh |**Additional Properties**|`string`, `number`, `boolean`||| **Minimal Properties:** 1 + + **Example** ```yaml @@ -502,6 +515,8 @@ snapshot.fetch.size: 10000 max.batch.size: 2048 max.queue.size: 8192 heartbeat.interval.ms: 0 +database.trustServerCertificate: false +driver.applicationIntent: ReadWrite lob.enabled: false publication.autocreate.mode: all_tables publication.name: dbz_publication @@ -607,6 +622,74 @@ count: 0 Deprecated RIOTX global fallback list of columns to use as message keys for every captured table. Prefer `tables..keys` + +#### sources\.advanced\.diagnostics: Collector diagnostics + +Opt-in JVM diagnostics for the source collector. Each artifact is written to the `/opt/rdi/diagnostics` directory on the host and included in the `dump-support-package` bundle. **Only applies to the `cdc` collector type on VM installations, and is ignored otherwise.** + + +**Properties** + +|Name|Type|Description|Required| +|----|----|-----------|--------| +|[**heap\_dump**](#sourcesadvanceddiagnosticsheap_dump)
(Heap dump on out\-of\-memory)|`object`|Writes a heap dump if the JVM runs out of memory, then exits so the collector restarts cleanly.
|| +|[**java\_flight\_recorder**](#sourcesadvanceddiagnosticsjava_flight_recorder)
(Java Flight Recorder recording)|`object`|Records a Java Flight Recorder profile of the collector at startup. RDI controls `filename` and `dumponexit`, and defaults `settings=profile`, `duration=5m`, and `maxsize=128m` unless set under `options`.
|| + +**Additional Properties:** not allowed +**Minimal Properties:** 1 +**Example** + +```yaml +heap_dump: + enabled: false +java_flight_recorder: + enabled: false + +``` + + +##### sources\.advanced\.diagnostics\.heap\_dump: Heap dump on out\-of\-memory + +Writes a heap dump if the JVM runs out of memory, then exits so the collector restarts cleanly. + + +**Properties** + +|Name|Type|Description|Required| +|----|----|-----------|--------| +|**enabled**
(Enable heap dump)|`boolean`|When `true`, the collector writes a heap dump if the JVM runs out of memory, then exits so it restarts cleanly.
Default: `false`
|| + +**Additional Properties:** not allowed +**Minimal Properties:** 1 +**Example** + +```yaml +enabled: false + +``` + + +##### sources\.advanced\.diagnostics\.java\_flight\_recorder: Java Flight Recorder recording + +Records a Java Flight Recorder profile of the collector at startup. RDI controls `filename` and `dumponexit`, and defaults `settings=profile`, `duration=5m`, and `maxsize=128m` unless set under `options`. + + +**Properties** + +|Name|Type|Description|Required| +|----|----|-----------|--------| +|**enabled**
(Enable flight recording)|`boolean`|When `true`, starts a Java Flight Recorder recording when the collector starts.
Default: `false`
|| +|**options**
(Flight Recorder options)|`string`|Comma-separated `-XX:StartFlightRecording` options, for example `duration=10m,maxsize=256m,path-to-gc-roots=true`.
|| + +**Additional Properties:** not allowed +**Minimal Properties:** 1 +**Example** + +```yaml +enabled: false + +``` + ## targets: Target connections diff --git a/content/operate/rc/rdi/define.md b/content/operate/rc/rdi/define.md index bef1b571fa..102d2714dd 100644 --- a/content/operate/rc/rdi/define.md +++ b/content/operate/rc/rdi/define.md @@ -46,7 +46,7 @@ In the **Pipeline setup** step: - Snowflake _(Preview)_ {{}} -1. Enter a name for your source database in the **Source name** field. This is a name for the source database that will appear on Redis Cloud. +1. Enter a name for your source database in the **Source name** field. This is a display label for the source database on Redis Cloud. It does not affect the pipeline's configuration. 1. Select the target Redis Cloud database from the **Target database** list. {{}} diff --git a/content/operate/rc/rdi/quick-start.md b/content/operate/rc/rdi/quick-start.md index 298ae00591..944f26b0fb 100644 --- a/content/operate/rc/rdi/quick-start.md +++ b/content/operate/rc/rdi/quick-start.md @@ -45,7 +45,7 @@ Before you can create your first Data Integration pipeline for a Redis Cloud sub {{}} 1. Select **PostgreSQL** as the source database type. {{}} -1. Enter a name for your source database in the **Source name** field. This is a name for the source database that will appear on Redis Cloud. +1. Enter a name for your source database in the **Source name** field. This is a display label for the source database on Redis Cloud. It does not affect the pipeline's configuration. 1. Select **Continue to source** to move to the **Source configuration** step. {{}} diff --git a/content/operate/rc/rdi/setup.md b/content/operate/rc/rdi/setup.md index 64856ce15a..a743d92276 100644 --- a/content/operate/rc/rdi/setup.md +++ b/content/operate/rc/rdi/setup.md @@ -49,7 +49,7 @@ Before you can set up your source connectivity and secrets, you need the AWS Acc - MongoDB - Snowflake {{}} -1. Enter a name for your source database in the **Source name** field. This is a name for the source database that will appear on Redis Cloud. +1. Enter a name for your source database in the **Source name** field. This is a display label for the source database on Redis Cloud. It does not affect the pipeline's configuration. 1. Select **Continue to source** to move to the **Source configuration** step. {{}} From 70d4f9b90db50aaa7a064b7195cc7da30b1d85c9 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 14:16:44 +0300 Subject: [PATCH 02/29] Update content/integrate/redis-data-integration/architecture/classic-vs-flink.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../redis-data-integration/architecture/classic-vs-flink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/architecture/classic-vs-flink.md b/content/integrate/redis-data-integration/architecture/classic-vs-flink.md index 76f4942bcb..6c164a21b8 100644 --- a/content/integrate/redis-data-integration/architecture/classic-vs-flink.md +++ b/content/integrate/redis-data-integration/architecture/classic-vs-flink.md @@ -106,7 +106,7 @@ The Flink processor also accepts an advanced matcher syntax in a job's `source` `server_name`, `db`, `schema`, and `table` each accept a list of names, and an entry prefixed with `regex:` selects all names that match the regular expression. One job can then process multiple tables, potentially from different sources, databases, or schemas. See -[Job files]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}). +[Job files]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) for more information. ## Metrics From 126bf7a33e69cf6cc8c442ea6107d148d418d298 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 14:17:05 +0300 Subject: [PATCH 03/29] Update content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../redis-data-integration/data-pipelines/prepare-dbs/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md index 127b363ab5..0027587e5b 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md @@ -25,7 +25,7 @@ so you must prepare each source database before you can use it with RDI. A pipeline can capture from more than one source database, and you must prepare each of them separately. Follow the page for each source's database type. See [Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) -for how to configure several sources in one pipeline. +to learn how to configure several sources in one pipeline. RDI supports the following source databases: From 43a503985b49a04883e6f69886f527e2b4e85fc2 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 14:17:28 +0300 Subject: [PATCH 04/29] Update content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../data-pipelines/prepare-dbs/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md index 58d63f4530..39bb91f039 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md @@ -144,7 +144,7 @@ For enhanced security, use key-pair authentication: Before deploying the RDI pipeline, configure the necessary secrets with [`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}}). -Pass the source name with `--db`; the source configured in the next step is named `snowflake`. +Pass the source name with `--db` (the source configured in the following example is named `snowflake` for demonstration purposes). ### Password authentication From 335a76be9c9174f446baa9196c412b3f2cb457ac Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 14:17:56 +0300 Subject: [PATCH 05/29] Update content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../data-pipelines/prepare-dbs/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md index 39bb91f039..8fafda6515 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md @@ -162,7 +162,7 @@ redis-di set-secret USERNAME --db snowflake your_username redis-di set-secret KEY --db snowflake --file /path/to/rsa_key.p8 ``` -If the `.p8` file is protected by a passphrase, also set that passphrase: +If the `.p8` file is protected by a passphrase, you must also set that passphrase: ```bash redis-di set-secret KEY_PASSWORD --db snowflake your_passphrase From 6fb3d361e072a6fc6fbfa7dc35c25a788733b675 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 14:18:37 +0300 Subject: [PATCH 06/29] Update content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../data-pipelines/prepare-dbs/spanner.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md index 9a8d0a8029..6f80559774 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md @@ -156,7 +156,7 @@ Workload Identity or a service account credentials file. **Optional**: If you prefer to use a service account credentials file instead of Workload Identity authentication, you'll need to create a Spanner-specific secret named `-db-credentials`, where `` is the name of the source in `config.yaml`. -The source configured in the next step is named `spanner`, so its secret is +The example below assumes a source named `spanner`, so its corresponding secret is `spanner-db-credentials`. See [Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for the source naming rules. From 255116739cafc1d92f2385406888014a83c4799c Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 14:19:13 +0300 Subject: [PATCH 07/29] Update content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../data-pipelines/prepare-dbs/spanner.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md index 6f80559774..ca145615dd 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md @@ -174,9 +174,9 @@ Be sure to adjust the file path (`~/spanner-reader-account.json`) if your servic stored elsewhere. {{< note >}}`redis-di set-secret` has no key for this secret, so you must -create it with `kubectl`. A secret created that way also has to be labeled so that the RDI operator +create it with `kubectl`. A secret created that way must also be labeled so that the RDI operator discovers it as a pipeline secret. See -[Set secrets for K8s/Helm deployment using Kubectl command]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets-for-k8shelm-deployment-using-kubectl-command" >}}).{{< /note >}} +[Set secrets for K8s/Helm deployment using Kubectl command]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets-for-k8shelm-deployment-using-kubectl-command" >}}) for more information.{{< /note >}} {{< note >}} If you create the `spanner-db-credentials` secret, you must also set `use_credentials_file: true` From de7ee59859819d0837d6dd0ea06d734a7e781a51 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 14:30:56 +0300 Subject: [PATCH 08/29] Apply batched suggestions from code review Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- content/integrate/redis-data-integration/faq.md | 2 +- content/integrate/redis-data-integration/quick-start-guide.md | 2 +- .../redis-data-integration/reference/api-migration.md | 3 +-- 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/content/integrate/redis-data-integration/faq.md b/content/integrate/redis-data-integration/faq.md index 3c3e43eab5..0d2a941dc6 100644 --- a/content/integrate/redis-data-integration/faq.md +++ b/content/integrate/redis-data-integration/faq.md @@ -90,7 +90,7 @@ for installation details. Yes. Add one entry per source to the `sources` section of `config.yaml`. Each source has its own collector, which captures change records independently of the other sources. See -[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}). +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for more information. ## Can RDI automatically track changes to the source database schema? diff --git a/content/integrate/redis-data-integration/quick-start-guide.md b/content/integrate/redis-data-integration/quick-start-guide.md index bedf666619..6418287ff6 100644 --- a/content/integrate/redis-data-integration/quick-start-guide.md +++ b/content/integrate/redis-data-integration/quick-start-guide.md @@ -73,7 +73,7 @@ At this point, the pipeline is ready to deploy. This tutorial uses one source database, but a pipeline can capture from several source databases into the same Redis target. See -[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}). +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for more information. ### Create a context (optional) {#create-context} diff --git a/content/integrate/redis-data-integration/reference/api-migration.md b/content/integrate/redis-data-integration/reference/api-migration.md index cd4dddbe79..4385b79cf6 100644 --- a/content/integrate/redis-data-integration/reference/api-migration.md +++ b/content/integrate/redis-data-integration/reference/api-migration.md @@ -30,8 +30,7 @@ The API version is part of the URL. Update `/api/v1` requests to use `/api/v2` w ## API v1 and multiple sources API v1 handles single-source pipelines only, so -[a pipeline with several sources]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) -has to be managed through API v2. In particular: +you must use API v2 to manage [a pipeline with several sources]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}). In particular: - `POST /api/v1/pipelines`, `PATCH /api/v1/pipelines`, and `POST /api/v1/pipelines/sources/dry-run` reject a configuration with more than one source with `422 Unprocessable Content` and `Only a single source per pipeline is supported`. - `PUT /api/v1/secrets`, `PUT /api/v1/secrets/{secret_name}`, and the v1 source management endpoints only handle a pipeline whose single source still uses the legacy `source` names, which covers a pipeline that predates per-source naming and one that you create through v1. They reject any other pipeline with `Use API v2 to manage sources and secrets of this pipeline.` From 70c558b975f20668117b526f784715954720db42 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 14:32:19 +0300 Subject: [PATCH 09/29] Apply batched suggestions from code review Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../redis-data-integration/data-pipelines/multiple-sources.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md index ff3b4ed71e..39877e4f1f 100644 --- a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md +++ b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md @@ -260,7 +260,7 @@ This makes it easy to attribute rejected records to their source. See Before RDI supported multiple sources per pipeline, every source-scoped resource had a name including the word `source` instead of the actual source name in `config.yaml`. For a source that existed before you upgraded to a version that supports multiple sources, -those names are kept unchanged, whatever the source is called in `config.yaml`. +those names are kept unchanged, regardless of what the source is called in `config.yaml`. In particular, for such a source: - Its secret environment variables are still named `SOURCE_DB_*`, so its `connection` section From 07363904dee31d4237179c3b6cec9d59fd46a2b6 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 14:32:41 +0300 Subject: [PATCH 10/29] Apply batched suggestions from code review Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../data-pipelines/prepare-dbs/sql-server.md | 2 +- .../data-pipelines/prepare-dbs/supabase.md | 2 +- .../data-pipelines/transform-examples/_index.md | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md index 8201a91cae..4cc700c09c 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md @@ -659,7 +659,7 @@ provide: | `${SQLSERVER_DB_USERNAME}` | The service principal's **Application (client) ID** (a GUID). | | `${SQLSERVER_DB_PASSWORD}` | The service principal's **client secret**. | -Set them with the source name, which for the source in this example is `sqlserver`: +Set them with the source name, which is `sqlserver` for the source in this example: ```bash redis-di set-secret USERNAME --db sqlserver diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md index 1da4762516..98e71bd453 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md @@ -164,7 +164,7 @@ In the Supabase dashboard, go to 1. Download the Supabase CA certificate. Store the database username, password, and CA certificate as RDI secrets. Pass the -source name with `--db`; the source configured in the next step is named `supabase`: +source name with `--db` (the source configured in the example below is named `supabase`): ```bash redis-di set-secret USERNAME --db supabase rdi_replication diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md index e1a9c13a6a..ca0965596d 100644 --- a/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md @@ -79,14 +79,14 @@ The main sections of these files are: With the [Flink processor]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}}), `server_name`, `db`, `schema`, and `table` each accept either a single name or a list of names. - An entry prefixed with `regex:` selects all names that match the regular expression, which has - to match the whole name. For example, `server_name: [mysql, postgresql]` selects two sources, + An entry prefixed with `regex:` selects all names that match the regular expression (note that this + must match the whole name, not just a substring). For example, `server_name: [mysql, postgresql]` selects two sources, and `table: "regex:orders_[0-9]+"` selects every numbered `orders` table. A job processes the records that match all of the properties it sets, so a single job can process multiple tables, potentially from different sources, databases, or schemas. The classic processor takes a single name in each of these properties. It rejects a list or a - `regex:` entry when you deploy the pipeline, so write one job per table instead. + `regex:` entry when you deploy the pipeline, so you should use one job per table instead. - `transform`: This is an optional section describing the transformation that the pipeline applies to the data before writing it to the target. The `uses` property specifies a From f6f14d55ab5ea4e1bef066a1b9ab5a248e3a4c26 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 14:33:24 +0300 Subject: [PATCH 11/29] Apply batched suggestions from code review Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../data-pipelines/multiple-sources.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md index 39877e4f1f..d497641369 100644 --- a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md +++ b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md @@ -20,9 +20,9 @@ same Redis target. The sources can be of different database types and each has i own collector, its own credentials, and its own set of Redis streams to ensure it is independent of the other sources. -{{< note >}}Use RDI API v2 to manage a pipeline with several sources. RDI API v1 supports only +{{< note >}}You must use RDI API v2 to manage a pipeline with several sources. RDI API v1 supports only single-source pipelines. See the -[RDI API migration guide]({{< relref "/integrate/redis-data-integration/reference/api-migration" >}}).{{< /note >}} +[RDI API migration guide]({{< relref "/integrate/redis-data-integration/reference/api-migration" >}}) for more information.{{< /note >}} ## Name your sources @@ -197,8 +197,8 @@ if you no longer need them. Note that renaming a source is not supported. Renaming a source in `config.yaml` is equivalent to removing the source and adding a new source with the new name. This implies in particular: -- The source's secrets have to be created under the new name and `${...}` references in - its `connection` section updated. +- You must create the source's secrets under the new name and update `${...}` references in + its `connection` section. - `server_name` has to be updated for every job that reads from the source. - The source starts with a new [initial snapshot]({{< relref "/integrate/redis-data-integration/architecture" >}}). From 93b64460c8b0b3e3515d4b6a1a249fecf22f3e2a Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 14:34:03 +0300 Subject: [PATCH 12/29] Apply batched suggestions from code review Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../redis-data-integration/data-pipelines/deploy.md | 2 +- .../data-pipelines/multiple-sources.md | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/deploy.md b/content/integrate/redis-data-integration/data-pipelines/deploy.md index 1cb84b4439..a87eb81c7c 100644 --- a/content/integrate/redis-data-integration/data-pipelines/deploy.md +++ b/content/integrate/redis-data-integration/data-pipelines/deploy.md @@ -53,7 +53,7 @@ covers the source naming rules. {{< note >}}The scope-prefixed keys `SOURCE_DB_*` and `TARGET_DB_*` are also accepted, and can be used without specifying `--db`. RDI accepts a `SOURCE_DB_*` key only for a pipeline with exactly one source that predates per-source naming. See -[Existing names are kept after an upgrade]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}).{{< /note >}} +[Existing names are kept after an upgrade]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}) for more information.{{< /note >}} {{< note >}} {{< embed-md "rdi-tls-secrets.md" >}} diff --git a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md index d497641369..9fab5018e6 100644 --- a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md +++ b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md @@ -46,7 +46,7 @@ A source name must: The names `rdi` and `target` are reserved and cannot be used for sources. RDI derives the environment variables that contain the source's credentials from the source -name, so the `connection` section of a source named `mysql` references `${MYSQL_DB_USERNAME}` +name. For example, the `connection` section of a source named `mysql` references `${MYSQL_DB_USERNAME}` and `${MYSQL_DB_PASSWORD}`. See [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) for details of how RDI derives those names and for the full list of secret keys. @@ -84,7 +84,7 @@ Sources of different types can be mixed freely, but a source's collector `type` MongoDB, `flink` for a Spanner connection, and `riotx` for a Snowflake connection. RDI rejects any other combination when you deploy the pipeline. See [Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) -for how to prepare each source database. +to learn how to prepare each source database. The following example captures from a MySQL database and a PostgreSQL database, each with its own credentials: @@ -161,8 +161,8 @@ source: table: customers ``` -When a pipeline has more than one source, every job must set `server_name`, and the value has -to match one of the sources in `config.yaml`. RDI rejects the pipeline when a job has no +When a pipeline has more than one source, every job must set `server_name`, and the value +must match one of the sources in `config.yaml`. RDI rejects the pipeline when a job has no `server_name`, or when its `server_name` matches no source. In a pipeline with a single source, `server_name` is optional. If you omit it, the From 959e0b9bf73fe3b03a9e75bb5295ac094968689e Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 14:34:29 +0300 Subject: [PATCH 13/29] Apply batched suggestions from code review Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../data-pipelines/multiple-sources.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md index 9fab5018e6..cdf2f797b9 100644 --- a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md +++ b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md @@ -199,7 +199,7 @@ to removing the source and adding a new source with the new name. This implies i - You must create the source's secrets under the new name and update `${...}` references in its `connection` section. -- `server_name` has to be updated for every job that reads from the source. +- You must update `server_name` for every job that reads from the source. - The source starts with a new [initial snapshot]({{< relref "/integrate/redis-data-integration/architecture" >}}). @@ -218,8 +218,7 @@ pipeline. Generally, stopping one source leaves the others running, and when one `external`. RDI creates no collector for this, so you cannot start or stop it. Stopping a source scales its collector down to zero replicas and leaves the rest of the -source's resources in place. RDI records a captured position per source, so a collector you -start again resumes from where it stopped. +source's resources in place. RDI records a captured position for each source, so when you restart a collector, it resumes from where it stopped. Resetting a single source deletes only that source's keys, so a new [initial snapshot]({{< relref "/integrate/redis-data-integration/architecture" >}}) is taken for that source, @@ -238,8 +237,8 @@ component they came from. See the reference page for more details. Note that while the sources are independent of each other in the data they capture, the -pipeline status is not per source. RDI reports the whole pipeline in an error state when a -single source fails, so use the `Components` section to find out which one it is. +pipeline status is not broken down per source. RDI reports the whole pipeline in an error state when a +single source fails, so you should use the `Components` section to find out which one has failed. Each source's collector has its own metric collection, named after the collector, such as `collector-mysql_metrics`. In Prometheus, the stream processor's `rdi_incoming_entries` and From 2c67588f5f46bd04601ad606a7850a58b83e2d67 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 14:34:46 +0300 Subject: [PATCH 14/29] Apply batched suggestions from code review Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../redis-data-integration/installation/install-vm.md | 2 +- .../redis-data-integration/installation/upgrade.md | 7 +++---- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/content/integrate/redis-data-integration/installation/install-vm.md b/content/integrate/redis-data-integration/installation/install-vm.md index 37580f71e9..dd6b34b4e7 100644 --- a/content/integrate/redis-data-integration/installation/install-vm.md +++ b/content/integrate/redis-data-integration/installation/install-vm.md @@ -278,7 +278,7 @@ fails with `423 Locked` and a message saying that the pipeline is standing by be currently a leader election follower. Operations such as deploy, start, stop, reset, and delete can only be performed against the primary instance. -Setting pipeline secrets works on a standby instance. Secrets are maintained per +You can still set pipeline secrets for an instance when it is on standby. Secrets are maintained per instance and are not replicated between the two, because they can legitimately differ in some cases. You may find it useful to trigger a failover deliberately to check that RDI is correctly configured to handle it. See [Test HA failover]({{< relref "/integrate/redis-data-integration/installation/ha-test" >}}) to learn how to do this. diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index bbc454af6b..5be1adcc5c 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -196,11 +196,10 @@ described in [Uninstall RDI]({{< relref "/integrate/redis-data-integration/insta and then install the old version. {{< /note >}} -## Source names after the upgrade +## Source names before and after the upgrade -For a source that existed before upgrading to a version that supports multiple sources, the -resource names from before the upgrade are kept unchanged, so it will keep running without -changes. +For a source that existed before upgrading to an RDI version that supports multiple sources, the +resource names from before the upgrade are kept unchanged. The source will keep running correctly without changes. For more details on how to handle sources created after upgrading, see [Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}). From 27858bb63020ea9fc64f3cae0161e6b2aa3a4f7c Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 14:35:16 +0300 Subject: [PATCH 15/29] Apply batched suggestions from code review Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../data-pipelines/rejected-records.md | 8 ++++---- .../redis-data-integration/installation/ha-test.md | 4 ++-- .../redis-data-integration/installation/install-k8s.md | 2 +- .../redis-data-integration/reference/cli/_index.md | 6 +++--- 4 files changed, 10 insertions(+), 10 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/rejected-records.md b/content/integrate/redis-data-integration/data-pipelines/rejected-records.md index 6e490252f8..cb48afa079 100644 --- a/content/integrate/redis-data-integration/data-pipelines/rejected-records.md +++ b/content/integrate/redis-data-integration/data-pipelines/rejected-records.md @@ -54,10 +54,10 @@ are stored in: dlq:data:{rdi}:postgresql.public.users ``` -The dead-letter queue endpoints and the CLI report table names in that same qualified form, -so a rejected record is always attributed to the source it came from. Sources that existed -before the upgrade to a version that supports multiple sources do not have the source segment -in the key. +The dead-letter queue endpoints and the CLI report table names are in the same form, +so you can easily attribute a rejected record to the source it came from. Sources that existed +before the upgrade to RDI v2.0.0 or above do not have the source segment +in the key because these earlier RDI versions did not support multiple sources. Some RDI versions or configurations can use a hash-tagged variant such as `dlq:{data:rdi:.
}`. To find all DLQ streams in the diff --git a/content/integrate/redis-data-integration/installation/ha-test.md b/content/integrate/redis-data-integration/installation/ha-test.md index 910248528d..9309bdde15 100644 --- a/content/integrate/redis-data-integration/installation/ha-test.md +++ b/content/integrate/redis-data-integration/installation/ha-test.md @@ -93,8 +93,8 @@ kubectl -n rdi logs rdi-operator-7f7f6c7dfd-5qmjd -f In about 10 seconds you will start seeing log entries from the leader saying that it could not acquire the leadership. When the leader lock expires, the second node will acquire the leadership and you will see log entries from the second node indicating that it has become the leader. -Once the failover completes, the two nodes swap their pipeline phases: the pipeline becomes -`Active` on the new leader and `Standby` on the old one. +Once the failover completes, the two nodes swap their pipeline phases: the pipeline has +`Active` status on the new leader and `Standby` status on the old one. ## Cleanup diff --git a/content/integrate/redis-data-integration/installation/install-k8s.md b/content/integrate/redis-data-integration/installation/install-k8s.md index 47ef52b70c..58fa91b893 100644 --- a/content/integrate/redis-data-integration/installation/install-k8s.md +++ b/content/integrate/redis-data-integration/installation/install-k8s.md @@ -428,7 +428,7 @@ Specifically, ensure that one or both of the following Helm chart values is set: Before deploying a pipeline, you must configure each source database to enable CDC. See the [Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) section to learn how to do this. A pipeline can capture from more than one source database, -and each one has to be prepared. See +so you must prepare each one each source separately. See [Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for details. ## Deploy a pipeline diff --git a/content/integrate/redis-data-integration/reference/cli/_index.md b/content/integrate/redis-data-integration/reference/cli/_index.md index 34f8887417..a6a5e5e7c5 100644 --- a/content/integrate/redis-data-integration/reference/cli/_index.md +++ b/content/integrate/redis-data-integration/reference/cli/_index.md @@ -86,10 +86,10 @@ Pipeline-scoped commands take the pipeline name as an optional positional argume take their own key or name as the positional argument and target the pipeline with the `-p` / `--pipeline` option, which also defaults to `default`. -Two options narrow a command to one database of the pipeline. The secret commands take `--db` with -a source name or `target` to address that database's secrets, and `start`, `stop`, and `reset` take +There are two command options that select a particular database of the pipeline. The secret commands take `--db` with +a source name or `target` to address that database's secrets, and the `start`, `stop`, and `reset` commands take `--source` with a source name to act on that source alone. See -[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}). +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for more information. The commands group as follows: From 96460fc83110b5a817a342a51526d1cda7d64262 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 14:35:30 +0300 Subject: [PATCH 16/29] Apply batched suggestions from code review Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../redis-data-integration/reference/cli/redis-di-describe.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-describe.md b/content/integrate/redis-data-integration/reference/cli/redis-di-describe.md index c0cbda106f..f71421c27a 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-describe.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-describe.md @@ -91,4 +91,4 @@ Statistics: The `Metrics Endpoints` column shows the externally-reachable metrics URLs of each component, which RDI discovers from the component's `Ingress` resources. It is empty for a component that has no ingress, or while the ingress controller has not assigned an address yet. See -[Accessing the metrics]({{< relref "/integrate/redis-data-integration/observability#accessing-the-metrics" >}}). +[Accessing the metrics]({{< relref "/integrate/redis-data-integration/observability#accessing-the-metrics" >}}) for more information. From 9e8389489a294e29eb613b69569cc832c09d69de Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 16:11:10 +0300 Subject: [PATCH 17/29] Update content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md Co-authored-by: Zdravko Donev --- .../data-pipelines/prepare-dbs/mongodb.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md index 17c678c255..af1d469b22 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md @@ -129,7 +129,7 @@ The root CA certificate for MongoDB Atlas must be added as the source's `CACERT` - Download the MongoDB Atlas root CA certificate. - In RDI, add this certificate with `redis-di set-secret CACERT --db mongodb /path/to/atlas-ca.crt`. -- Ensure that the `mongodb.ssl.enabled: true` setting is present in your RDI configuration. +- Ensure that the `mongodb.ssl.enabled: true` setting is present in your RDI configuration for the corresponding source DB collector. Example connection string for Atlas: ``` From 043e3cfb9a31f998382fe3f228c2a936c3b56000 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 16:26:38 +0300 Subject: [PATCH 18/29] Update per-source reset and deletion --- .../data-pipelines/deploy.md | 5 ++-- .../data-pipelines/multiple-sources.md | 26 ++++++++++++---- .../data-pipelines/rejected-records.md | 6 +--- .../reference/api-migration.md | 2 +- .../reference/api-reference/openapi.json | 30 ++++++++++++++++++- .../reference/cli/redis-di-reset.md | 3 ++ 6 files changed, 58 insertions(+), 14 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/deploy.md b/content/integrate/redis-data-integration/data-pipelines/deploy.md index a87eb81c7c..5d86981788 100644 --- a/content/integrate/redis-data-integration/data-pipelines/deploy.md +++ b/content/integrate/redis-data-integration/data-pipelines/deploy.md @@ -52,7 +52,7 @@ covers the source naming rules. {{< note >}}The scope-prefixed keys `SOURCE_DB_*` and `TARGET_DB_*` are also accepted, and can be used without specifying `--db`. RDI accepts a `SOURCE_DB_*` key only for a pipeline with -exactly one source that predates per-source naming. See +exactly one source created before RDI supported multiple sources. See [Existing names are kept after an upgrade]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}) for more information.{{< /note >}} {{< note >}} @@ -368,7 +368,8 @@ drifted out of sync. redis-di reset ``` -Add `--source` to reset a single source and leave the others untouched: +Add `--source` to reset a single source. The other sources keep their data, but the whole +pipeline stops while the reset runs and starts again afterwards: ```bash redis-di reset --source mysql diff --git a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md index cdf2f797b9..d1962341c6 100644 --- a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md +++ b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md @@ -45,6 +45,13 @@ A source name must: The names `rdi` and `target` are reserved and cannot be used for sources. +{{< warning >}}If your pipeline has a source created before RDI supported multiple sources, do not name a new +source after any schema or database of that older source. The change data streams of the older +source do not contain a source name segment, so a new source named after one of its schemas would claim keys +that belong to the older source, and resetting or removing the new source would delete the older +source's data. See +[Existing names are kept after an upgrade](#existing-names-are-kept-after-an-upgrade).{{< /warning >}} + RDI derives the environment variables that contain the source's credentials from the source name. For example, the `connection` section of a source named `mysql` references `${MYSQL_DB_USERNAME}` and `${MYSQL_DB_PASSWORD}`. See @@ -184,11 +191,14 @@ To add a source, set its secrets first, then add it to `config.yaml` and deploy. source does not interrupt other sources that are already running. To remove a source, delete its entry from `config.yaml` and deploy. RDI removes the source's -collector and deletes the source's keys from the RDI database. No further action is +collector and deletes that source's data from the RDI database, including its change data streams, +Debezium offsets, schema history, dead-letter queue entries, statistics, deduplication state, +and record counters. The other sources keep their data, and RDI stops the whole pipeline +while the deletion runs and starts it again afterwards. No further action is needed for this cleanup, but it means that a source you add later under the same name starts from a new [initial snapshot]({{< relref "/integrate/redis-data-integration/architecture" >}}) -rather than resuming from the position it had reached. +rather than from the position it had reached. The source's secrets are not deleted, so remove them yourself with [`redis-di delete-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-delete-secret" >}}) @@ -200,6 +210,7 @@ to removing the source and adding a new source with the new name. This implies i - You must create the source's secrets under the new name and update `${...}` references in its `connection` section. - You must update `server_name` for every job that reads from the source. +- The data present in the RDI database under the old name is deleted, as it is for any removed source. - The source starts with a new [initial snapshot]({{< relref "/integrate/redis-data-integration/architecture" >}}). @@ -220,9 +231,14 @@ pipeline. Generally, stopping one source leaves the others running, and when one Stopping a source scales its collector down to zero replicas and leaves the rest of the source's resources in place. RDI records a captured position for each source, so when you restart a collector, it resumes from where it stopped. -Resetting a single source deletes only that source's keys, so a new -[initial snapshot]({{< relref "/integrate/redis-data-integration/architecture" >}}) is taken for that source, -while the stream processor and the other sources keep running. +Resetting a single source deletes that source's data from the RDI database, including its change data streams, Debezium +offsets, schema history, dead-letter queue entries, statistics, deduplication state, and record counters. +A new [initial snapshot]({{< relref "/integrate/redis-data-integration/architecture" >}}) is then +taken for that source, while every other source keeps its data. RDI stops the whole pipeline while +the reset runs and starts it again afterwards, exactly as it does for a reset of the whole +pipeline. + +Data that is not partitioned by source is never deleted by a per-source reset. ## Monitor each source diff --git a/content/integrate/redis-data-integration/data-pipelines/rejected-records.md b/content/integrate/redis-data-integration/data-pipelines/rejected-records.md index cb48afa079..70ee60c641 100644 --- a/content/integrate/redis-data-integration/data-pipelines/rejected-records.md +++ b/content/integrate/redis-data-integration/data-pipelines/rejected-records.md @@ -48,11 +48,7 @@ dlq:data:{rdi}:..
``` For example, rejected records for the `public.users` table of a source named `postgresql` -are stored in: - -```text -dlq:data:{rdi}:postgresql.public.users -``` +are stored in `dlq:data:{rdi}:postgresql.public.users`. The dead-letter queue endpoints and the CLI report table names are in the same form, so you can easily attribute a rejected record to the source it came from. Sources that existed diff --git a/content/integrate/redis-data-integration/reference/api-migration.md b/content/integrate/redis-data-integration/reference/api-migration.md index 4385b79cf6..33c20588b1 100644 --- a/content/integrate/redis-data-integration/reference/api-migration.md +++ b/content/integrate/redis-data-integration/reference/api-migration.md @@ -33,7 +33,7 @@ API v1 handles single-source pipelines only, so you must use API v2 to manage [a pipeline with several sources]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}). In particular: - `POST /api/v1/pipelines`, `PATCH /api/v1/pipelines`, and `POST /api/v1/pipelines/sources/dry-run` reject a configuration with more than one source with `422 Unprocessable Content` and `Only a single source per pipeline is supported`. -- `PUT /api/v1/secrets`, `PUT /api/v1/secrets/{secret_name}`, and the v1 source management endpoints only handle a pipeline whose single source still uses the legacy `source` names, which covers a pipeline that predates per-source naming and one that you create through v1. They reject any other pipeline with `Use API v2 to manage sources and secrets of this pipeline.` +- `PUT /api/v1/secrets`, `PUT /api/v1/secrets/{secret_name}`, and the v1 source management endpoints only handle a pipeline whose single source still uses the legacy `source` names, which covers a pipeline created before RDI supported multiple sources and one that you create through v1. They reject any other pipeline with `Use API v2 to manage sources and secrets of this pipeline.` - `GET /api/v1/status` and `GET /api/v1/monitoring/statistics` report the first source of the pipeline only. ## Endpoint mapping diff --git a/content/integrate/redis-data-integration/reference/api-reference/openapi.json b/content/integrate/redis-data-integration/reference/api-reference/openapi.json index 5e4d90d779..4a759fd690 100644 --- a/content/integrate/redis-data-integration/reference/api-reference/openapi.json +++ b/content/integrate/redis-data-integration/reference/api-reference/openapi.json @@ -5109,7 +5109,7 @@ "post": { "tags": ["v2", "pipelines"], "summary": "Reset Pipeline", - "description": "Resets a pipeline.", + "description": "Resets a pipeline, or a single source when one is given.", "operationId": "reset_pipeline_api_v2_pipelines__name__reset_post", "security": [ { @@ -5127,6 +5127,24 @@ "title": "Name" }, "description": "Pipeline name" + }, + { + "name": "source", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "Reset only this source; the other sources keep their data. The whole pipeline is stopped while the reset runs and starts again afterwards. Must be an existing source that is not of type 'external'.", + "title": "Source" + }, + "description": "Reset only this source; the other sources keep their data. The whole pipeline is stopped while the reset runs and starts again afterwards. Must be an existing source that is not of type 'external'." } ], "responses": { @@ -5160,6 +5178,16 @@ }, "description": "Forbidden" }, + "400": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Bad Request" + }, "404": { "content": { "application/json": { diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md b/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md index e5e7728f3d..2b96992e4f 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md @@ -13,6 +13,9 @@ Resets a pipeline into initial full-sync mode, so it reloads a snapshot of the s resuming change data capture. By default, the command waits for the pipeline to reach a terminal state before returning. +With `--source`, only the specified source is reset. See +[Start, stop, and reset a single source]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#start-stop-and-reset-a-single-source" >}}) for more information. + ## Usage ``` From ee65ac5c9e4b7eca421779ef61bf7a00fe1634a6 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 17:38:57 +0300 Subject: [PATCH 19/29] Update for changes introduced with RDSC-5882 --- .../data-pipelines/prepare-dbs/mongodb.md | 16 ++++++------ .../prepare-dbs/my-sql-mariadb.md | 25 ++++++++++++++++++- .../reference/api-reference/openapi.json | 2 +- .../reference/config-yaml-reference.md | 4 +-- 4 files changed, 33 insertions(+), 14 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md index af1d469b22..6a8d93cc04 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md @@ -144,22 +144,20 @@ source's `CERT` and `KEY` secrets. See [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) for the full list of source database TLS and mTLS secrets. -When you use MongoDB X.509 authentication, include all of the following -properties in the source `advanced.source` section: +When you use MongoDB X.509 authentication, enable TLS in the source `advanced.source` section: ```yaml advanced: source: mongodb.ssl.enabled: true - mongodb.ssl.keystore: /debezium/certs/mongodb_db_keystore - mongodb.ssl.keystore.password: debezium ``` -The keystore is named after the source, so a source named `mongodb` uses -`/debezium/certs/mongodb_db_keystore`. The RDI Collector builds it from the source -database client certificate and private key secrets. Debezium requires the -`mongodb.ssl.keystore` and `mongodb.ssl.keystore.password` properties to present -the client certificate to MongoDB. +RDI builds the keystore that presents the client certificate to MongoDB from the source's +`CERT` and `KEY` secrets, and configures the collector to use it. + +{{< note >}}Do not set `mongodb.ssl.keystore` or `mongodb.ssl.keystore.password` yourself. RDI +manages the keystore, so these properties are not only unnecessary, they are rejected when you +deploy the pipeline.{{< /note >}} For X.509 authentication, the MongoDB connection string must also include the required authentication options, such as `authMechanism=MONGODB-X509` and diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md index 85f3e3d9f8..d97491ec20 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md @@ -31,6 +31,7 @@ checklist as you complete each step. - [ ] [Configure session timeouts](#4-configure-session-timeouts) - [ ] [Enable query log events](#5-enable-query-log-events) - [ ] [Check binlog_row_value_options](#6-check-binlog_row_value_options) +- [ ] [Connect over TLS or mTLS](#7-connect-over-tls-or-mtls) ``` ## 1. Create a CDC user @@ -227,7 +228,29 @@ If the value is `PARTIAL_JSON` then you should unset the variable: mysql> set @@global.binlog_row_value_options="" ; ``` -## 7. Configuration is complete +## 7. Connect over TLS or mTLS + +To connect to the source database over TLS, set the source's `CACERT` secret to the source CA +certificate. For mutual TLS, also set the source's `CERT` and `KEY` secrets. See +[Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) +for the full list of source database TLS and mTLS secrets. + +Select the TLS mode in the source `advanced.source` section, for example: + +```yaml +advanced: + source: + database.ssl.mode: verify_ca +``` + +RDI builds the keystore that presents the client certificate to the source database from the +source's `CERT` and `KEY` secrets, and configures the collector to use it. + +{{< note >}}Do not set `database.ssl.keystore` or `database.ssl.keystore.password` yourself. RDI +manages the keystore, so these properties are not only unnecessary, they are rejected when you +deploy the pipeline.{{< /note >}} + +## 8. Configuration is complete After following the steps above, your MySQL/MariaDB database is ready for Debezium to use. diff --git a/content/integrate/redis-data-integration/reference/api-reference/openapi.json b/content/integrate/redis-data-integration/reference/api-reference/openapi.json index 4a759fd690..57241fb3a4 100644 --- a/content/integrate/redis-data-integration/reference/api-reference/openapi.json +++ b/content/integrate/redis-data-integration/reference/api-reference/openapi.json @@ -2,7 +2,7 @@ "openapi": "3.1.0", "info": { "title": "Redis Data Integration API", - "description": "> **NOTE:** RDI API v1 is deprecated as of RDI 1.19.0. Use RDI API v2 for new integrations and migrate existing clients. API v1 will not be extended with new RDI features and may be removed in a future RDI version. See the [RDI API migration guide](/integrate/redis-data-integration/reference/api-migration/).\n\nAPI for Redis Data Integration services", + "description": "> **NOTE:** RDI API v1 is deprecated as of RDI 1.19.0. Use RDI API v2 for new integrations and migrate existing clients. API v1 will not be extended with new RDI features and may be removed in a future RDI version. See the [RDI API migration guide](https://redis.io/docs/latest/integrate/redis-data-integration/reference/api-migration/).\n\nAPI for Redis Data Integration services", "version": "1.19.0" }, "paths": { diff --git a/content/integrate/redis-data-integration/reference/config-yaml-reference.md b/content/integrate/redis-data-integration/reference/config-yaml-reference.md index 63921de22a..cd352702ec 100644 --- a/content/integrate/redis-data-integration/reference/config-yaml-reference.md +++ b/content/integrate/redis-data-integration/reference/config-yaml-reference.md @@ -469,7 +469,7 @@ redis.wait.retry.delay.ms: 1000 #### sources\.advanced\.source: Advanced source settings -Advanced configuration properties for the source database connection and CDC behavior. **Applies to the `cdc` and `flink` collector types.**

For the `cdc` collector type, available properties depend on the source database — refer to the relevant Debezium connector documentation: [MySQL](https://debezium.io/documentation/reference/stable/connectors/mysql.html), [MariaDB](https://debezium.io/documentation/reference/stable/connectors/mariadb.html), [PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html), [Oracle](https://debezium.io/documentation/reference/stable/connectors/oracle.html), [SQL Server](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html), [Db2](https://debezium.io/documentation/reference/stable/connectors/db2.html), [MongoDB](https://debezium.io/documentation/reference/stable/connectors/mongodb.html). When using a property from those pages, omit the `debezium.source.` prefix.

**The named properties below cover the most commonly tuned settings: `spanner.*` properties apply to the `flink` collector type, all others apply to the `cdc` collector type. Any other property from the Debezium documentation can still be set as a free-form key-value pair, except for `topic.prefix`, which RDI derives from the source name.** +Advanced configuration properties for the source database connection and CDC behavior. **Applies to the `cdc` and `flink` collector types.**

For the `cdc` collector type, available properties depend on the source database — refer to the relevant Debezium connector documentation: [MySQL](https://debezium.io/documentation/reference/stable/connectors/mysql.html), [MariaDB](https://debezium.io/documentation/reference/stable/connectors/mariadb.html), [PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html), [Oracle](https://debezium.io/documentation/reference/stable/connectors/oracle.html), [SQL Server](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html), [Db2](https://debezium.io/documentation/reference/stable/connectors/db2.html), [MongoDB](https://debezium.io/documentation/reference/stable/connectors/mongodb.html). When using a property from those pages, omit the `debezium.source.` prefix.

**The named properties below cover the most commonly tuned settings: `spanner.*` properties apply to the `flink` collector type, all others apply to the `cdc` collector type. Any other property from the Debezium documentation can still be set as a free-form key-value pair, except for `topic.prefix`, which RDI derives from the source name, and the keystore properties, which RDI configures from a source's TLS secret.** **Properties** @@ -504,8 +504,6 @@ Advanced configuration properties for the source database connection and CDC beh |**Additional Properties**|`string`, `number`, `boolean`||| **Minimal Properties:** 1 - - **Example** ```yaml From 53183fceee6a12d536f76acce6b80ce5932db310 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Tue, 8 Sep 2026 18:21:55 +0300 Subject: [PATCH 20/29] Update for making the Flink processor the default --- .../architecture/_index.md | 4 ++-- .../architecture/classic-vs-flink.md | 2 +- .../data-pipelines/pipeline-config.md | 19 +++++++++---------- .../integrate/redis-data-integration/faq.md | 18 +++++++----------- .../installation/install-k8s.md | 10 ++++------ .../installation/install-vm.md | 10 ++++------ .../migration-classic-to-flink.md | 6 +++--- .../installation/upgrade.md | 9 +++++---- 8 files changed, 35 insertions(+), 43 deletions(-) diff --git a/content/integrate/redis-data-integration/architecture/_index.md b/content/integrate/redis-data-integration/architecture/_index.md index 323a81950a..c49e54701a 100644 --- a/content/integrate/redis-data-integration/architecture/_index.md +++ b/content/integrate/redis-data-integration/architecture/_index.md @@ -148,8 +148,8 @@ The diagram below shows all RDI components and the interactions between them: RDI provides two implementations of the stream processor, *classic* and *Flink*. You select the implementation per pipeline through the [`processors.type`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}}) -property in `config.yaml`. The default is `classic`, so existing pipelines -keep their behavior unchanged. +property in `config.yaml`. The default is `flink`, so set `classic` explicitly to run a +pipeline on the classic processor. See [Differences between the classic and Flink processors]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}}) diff --git a/content/integrate/redis-data-integration/architecture/classic-vs-flink.md b/content/integrate/redis-data-integration/architecture/classic-vs-flink.md index 6c164a21b8..40dc1fe45d 100644 --- a/content/integrate/redis-data-integration/architecture/classic-vs-flink.md +++ b/content/integrate/redis-data-integration/architecture/classic-vs-flink.md @@ -69,7 +69,7 @@ for VM installations. The two processors share the same `config.yaml` envelope and the same `connections`, `sources`, `targets`, and `jobs` sections. The only differences are inside the `processors:` block, which is selected via -`processors.type` (`classic` or `flink`, default `classic`). Properties +`processors.type` (`classic` or `flink`, default `flink`). Properties that apply to only one implementation are annotated with **Classic processor only.** or **Flink processor only.** in the [pipeline configuration reference]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}}), diff --git a/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md b/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md index 1348255a7b..10fb64008f 100644 --- a/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md +++ b/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md @@ -66,12 +66,12 @@ targets: password: ${TARGET_DB_PASSWORD} processors: - type: flink target_data_type: hash ``` -Keep `type: flink` for new pipelines. The other processor properties have defaults, -so add them only when you need to change the default behavior. +The processor properties all have defaults, so add them only when you need to change the +default behavior. In particular, the pipeline runs on the Flink processor unless you set +`type` to `classic`. ## Build the file with an AI assistant @@ -184,12 +184,13 @@ that you should set as described in [Set secrets]({{< relref "/integrate/redis-d ### Processors -The `processors` section selects the stream processor and configures its behavior. -Use the Flink processor for new pipelines: +The `processors` section selects the stream processor and configures its behavior. The default +is the Flink processor, which is what we recommend. To run a pipeline on the classic processor instead, +set `type` to `classic`: ```yaml processors: - type: flink + type: classic ``` See [Differences between the classic and Flink processors]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}}) @@ -261,7 +262,6 @@ setting for its output. ```yaml processors: - type: flink target_data_type: hash ``` @@ -275,7 +275,6 @@ healthy replica is available: ```yaml processors: - type: flink wait_enabled: true wait_timeout: 1000 retry_on_replica_failure: true @@ -426,9 +425,9 @@ targets: # Settings that control how the processor writes the captured records to the targets. processors: # Processor implementation to run, one of `classic` or `flink`. - # The default is `classic` for backward compatibility, while `flink` is strongly recommended for new pipelines. + # The default is `flink`, which is the recommended processor. # See https://redis.io/docs/latest/integrate/redis-data-integration/faq/#which-processor-should-i-use - type: flink + # type: flink # Maximum number of records read from the source streams in a single batch. # read_batch_size: 2000 diff --git a/content/integrate/redis-data-integration/faq.md b/content/integrate/redis-data-integration/faq.md index 0d2a941dc6..ef74473a61 100644 --- a/content/integrate/redis-data-integration/faq.md +++ b/content/integrate/redis-data-integration/faq.md @@ -167,18 +167,14 @@ processor. It also adds optional expression and `redis.lookup` result caching. **We strongly recommend using the Flink processor** for new pipelines and -migrating existing pipelines to it, to benefit from these improvements. The -*classic* processor is still the default, so pipelines keep using it until -you opt in, and it remains a fully supported choice — for example, when you -want to ensure your pipelines continue to work as before until you have -consciously migrated them. In a future release, however, the Flink processor -will become the default and the classic processor may be deprecated, so adopting -the Flink processor now avoids a later migration. - -Switch a pipeline to the Flink processor by setting +migrating existing pipelines to it, to benefit from these improvements. It is +the default, so a pipeline whose `config.yaml` does not set a processor type +uses it. The *classic* processor remains a fully supported choice for now. +It may be deprecated in a future release. + +To switch a pipeline to the classic processor, set [`processors.type`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}}) -to `flink` (`classic` is the default). You can adopt it per pipeline without -changing the others. +to `classic`. You can do that per pipeline without changing the others. See [Differences between the classic and Flink processors]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}}) diff --git a/content/integrate/redis-data-integration/installation/install-k8s.md b/content/integrate/redis-data-integration/installation/install-k8s.md index 58fa91b893..b0584a2ea7 100644 --- a/content/integrate/redis-data-integration/installation/install-k8s.md +++ b/content/integrate/redis-data-integration/installation/install-k8s.md @@ -322,9 +322,8 @@ oc get projects -o yaml | grep "openshift.io/sa.scc" ### Configure the Flink processor -RDI ships with two stream processor implementations: the default *classic* -processor and the -[Apache Flink](https://flink.apache.org/)-based *Flink* processor. +RDI ships with two stream processor implementations: the *classic* processor and the +default [Apache Flink](https://flink.apache.org/)-based *Flink* processor. See [Stream processor implementations]({{< relref "/integrate/redis-data-integration/architecture#stream-processor-implementations" >}}) for an overview of the differences and @@ -356,10 +355,9 @@ operator: Configuring the Flink processor at the Helm chart level only sets the values that the operator will use when deploying the JobManager and TaskManager workloads. -To run a specific pipeline on the Flink processor, set +A pipeline runs on the Flink processor unless its `config.yaml` sets [`processors.type`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}}) -to `flink` in that pipeline's `config.yaml`. Pipelines without this setting -continue to use the classic processor. Fine-tune the Flink runtime +to `classic`. Fine-tune the Flink runtime through the `processors.advanced` section of `config.yaml` (see the [configuration reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors" >}})). diff --git a/content/integrate/redis-data-integration/installation/install-vm.md b/content/integrate/redis-data-integration/installation/install-vm.md index dd6b34b4e7..b9ab70864a 100644 --- a/content/integrate/redis-data-integration/installation/install-vm.md +++ b/content/integrate/redis-data-integration/installation/install-vm.md @@ -310,19 +310,17 @@ for your platform.{{< /note >}} ## Configure the Flink processor -RDI ships with two stream processor implementations: the default *classic* -processor and the -[Apache Flink](https://flink.apache.org/)-based *Flink* processor. +RDI ships with two stream processor implementations: the *classic* processor and the +default [Apache Flink](https://flink.apache.org/)-based *Flink* processor. See [Stream processor implementations]({{< relref "/integrate/redis-data-integration/architecture#stream-processor-implementations" >}}) for an overview of the differences and [Differences between the classic and Flink processors]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}}) for a side-by-side comparison. -To run a specific pipeline on the Flink processor, set +A pipeline runs on the Flink processor unless its `config.yaml` sets [`processors.type`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}}) -to `flink` in that pipeline's `config.yaml` and redeploy it. Pipelines without -this setting continue to use the classic processor. Fine-tune the Flink runtime +to `classic`. Fine-tune the Flink runtime through the `processors.advanced` section of `config.yaml` (see the [configuration reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors" >}})). diff --git a/content/integrate/redis-data-integration/installation/migration-classic-to-flink.md b/content/integrate/redis-data-integration/installation/migration-classic-to-flink.md index 419c048101..82b695e30a 100644 --- a/content/integrate/redis-data-integration/installation/migration-classic-to-flink.md +++ b/content/integrate/redis-data-integration/installation/migration-classic-to-flink.md @@ -17,7 +17,7 @@ weight: 35 --- RDI ships with two stream processor implementations. The *classic* -processor is implemented in Python. The *Flink* processor is built on top of +processor is implemented in Python. The default *Flink* processor is built on top of [Apache Flink](https://flink.apache.org/). Both run on VM and Kubernetes installations. The Flink processor can achieve much higher throughput during snapshots, scales horizontally by changing the number of TaskManager replicas, @@ -179,7 +179,7 @@ pipeline, or move consumer-group positions to make the count reach `0`. After the drain check passes, remove the source's `active: false` setting from the existing `config.yaml` and set [`processors.type`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}}) -to `flink`: +to `flink` (or remove this property, since `flink` is the default): ```yaml processors: @@ -260,7 +260,7 @@ for the `ServiceMonitor` configuration and the available metrics. ## Rolling back -To revert a pipeline to the classic processor, set `processors.type` back to +To revert a pipeline to the classic processor, set `processors.type` to `classic` and redeploy the pipeline. This setting is required on RDI 2.0.0, where the default is `flink`. The classic processor silently ignores `processors.advanced`, so you don't need to remove it before switching back. diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index 5be1adcc5c..9dec5e73e1 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -211,11 +211,12 @@ The fully supported on both VM and Kubernetes installations after upgrading to RDI 1.19.0. Once the upgrade completes, it is always available — no opt-in is required, and the defaults are sized for typical workloads. -On RDI 1.19.0, existing classic pipelines keep using that processor until -you switch them by setting + +{{< warning >}}The Flink processor is the default as of RDI 2.0.0. +Upgrading to that release or later moves a pipeline whose `config.yaml` does not set [`processors.type`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}}) -to `flink` in their `config.yaml`. -RDI 2.0.0 changes this default; see [Upgrading to RDI 2.0.0](#upgrading-to-rdi-200). +onto the Flink processor when you next deploy it. To keep such a pipeline on the classic +processor, set `processors.type` to `classic` before you upgrade.{{< /warning >}} On Kubernetes, to override the Flink processor defaults, add an `operator.dataPlane.flinkProcessor` block to your `rdi-values.yaml` file as From aba68c0b1c3004815d5a9a74b95d2c0406a073ad Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Mon, 14 Sep 2026 14:05:31 +0300 Subject: [PATCH 21/29] Address comments --- .../data-pipelines/deploy.md | 4 +- .../data-pipelines/multiple-sources.md | 44 ++++++++++++------- .../data-pipelines/rejected-records.md | 12 +++-- .../reference/cli/redis-di-get-dlq.md | 5 +-- .../cli/redis-di-list-dlq-records.md | 5 +-- .../reference/cli/redis-di-list-dlqs.md | 7 ++- 6 files changed, 49 insertions(+), 28 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/deploy.md b/content/integrate/redis-data-integration/data-pipelines/deploy.md index 5d86981788..da1ed7eb93 100644 --- a/content/integrate/redis-data-integration/data-pipelines/deploy.md +++ b/content/integrate/redis-data-integration/data-pipelines/deploy.md @@ -51,8 +51,8 @@ shows these references in use, and covers the source naming rules. {{< note >}}The scope-prefixed keys `SOURCE_DB_*` and `TARGET_DB_*` are also accepted, and can be -used without specifying `--db`. RDI accepts a `SOURCE_DB_*` key only for a pipeline with -exactly one source created before RDI supported multiple sources. See +used without specifying `--db`. RDI accepts a `SOURCE_DB_*` key only for a pipeline with exactly +one source. Prefer the per-database keys with `--db`, which work for any pipeline. See [Existing names are kept after an upgrade]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}) for more information.{{< /note >}} {{< note >}} diff --git a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md index d1962341c6..7cc0d7d6f8 100644 --- a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md +++ b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md @@ -68,11 +68,16 @@ lists the names derived from a source named `mysql`. | Environment variable prefix | `MYSQL_DB_` | | Certificate directory | `/etc/certificates/mysql_db/` | | Collector deployment | `collector-mysql` | -| Change data streams | `data:{rdi}:mysql..
` | -| Dead-letter queue streams | `dlq:data:{rdi}:mysql..
` | +| Change data streams | `data:{rdi}:mysql.` | +| Dead-letter queue streams | `dlq:data:{rdi}:mysql.` | | Metric collection | `collector-mysql_metrics` | | Metrics endpoint path on a VM installation | `/collector-mysql/metrics` | +In the stream names, `` is the qualified table name: +`.
` for MySQL and MariaDB, `.` for MongoDB, +`.
` for Oracle, PostgreSQL, Snowflake, and Spanner, and +`..
` for SQL Server. + Each source also accepts an optional `name` property, which is a display name of up to 100 characters. Unlike the source name, it is not used as an identifier, so there is no restriction on the characters you can use. @@ -123,7 +128,7 @@ sources: schemas: - public tables: - public.customers: {} + public.clients: {} targets: target: connection: @@ -170,7 +175,12 @@ source: When a pipeline has more than one source, every job must set `server_name`, and the value must match one of the sources in `config.yaml`. RDI rejects the pipeline when a job has no -`server_name`, or when its `server_name` matches no source. +`server_name`, or when its `server_name` matches no source. The one exception is the default +job for `table: "*"`: it is a source-agnostic catch-all, so it needs no `server_name`. + +For a source that existed before RDI supported multiple sources, set `server_name` to `rdi` +rather than to the name the source has in `config.yaml`. See +[Existing names are kept after an upgrade](#existing-names-are-kept-after-an-upgrade). In a pipeline with a single source, `server_name` is optional. If you omit it, the job does not filter by source. @@ -238,8 +248,6 @@ taken for that source, while every other source keeps its data. RDI stops the wh the reset runs and starts it again afterwards, exactly as it does for a reset of the whole pipeline. -Data that is not partitioned by source is never deleted by a per-source reset. - ## Monitor each source Use [`redis-di describe`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-describe" >}}) @@ -256,17 +264,23 @@ Note that while the sources are independent of each other in the data they captu pipeline status is not broken down per source. RDI reports the whole pipeline in an error state when a single source fails, so you should use the `Components` section to find out which one has failed. -Each source's collector has its own metric collection, named after the collector, such as -`collector-mysql_metrics`. In Prometheus, the stream processor's `rdi_incoming_entries` and -`rdi_stream_event_latency_ms` metrics contain a `data_source` label that identifies the stream -the value belongs to, including the source name, so you can break both of them down per -source. See -[Stream processor metrics]({{< relref "/integrate/redis-data-integration/observability#stream-processor-metrics" >}}) +Each Debezium collector has its own metric collection, named after the collector, such as +`collector-mysql_metrics`. The Spanner and Snowflake collectors don't have metric collections. + +In Prometheus, you can break the per-stream record counters down per source, since the stream +name contains the source name. With the +[Flink processor]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}}) +the counters are reported by +`flink_jobmanager_job_operator_coordinator_stream_type_rdiRecords`, which carries a `stream` +label; with the classic processor they are reported by `rdi_incoming_entries`, which carries an +equivalent `data_source` label. See +[Flink processor metrics]({{< relref "/integrate/redis-data-integration/observability#flink-processor-metrics" >}}), +[Stream processor metrics]({{< relref "/integrate/redis-data-integration/observability#stream-processor-metrics" >}}), and, for the per-source collector endpoints, [Accessing the metrics]({{< relref "/integrate/redis-data-integration/observability#accessing-the-metrics" >}}). Dead-letter queue streams have Redis keys containing a -`..
` section. +`.` section. This makes it easy to attribute rejected records to their source. See [Rejected records]({{< relref "/integrate/redis-data-integration/data-pipelines/rejected-records" >}}) for more information. @@ -282,10 +296,10 @@ In particular, for such a source: can keep referencing these secrets. - Its Kubernetes secrets are still named `source-db` and `source-db-ssl`. - Its Kubernetes deployment and other resources are still named `collector-source`. -- Its data streams are still named `data:{rdi}:.
`, and its offset +- Its data streams are still named `data:{rdi}:`, and its offset and schema history keys are still `metadata:debezium:offsets` and `metadata:debezium:schema_history`. -- Its `server_name` is still `rdi`. +- Its `server_name` is still `rdi`, or, for a Spanner source, its instance ID. For a source you add after the upgrade, RDI derives all of these names from the source name, as described on this page. diff --git a/content/integrate/redis-data-integration/data-pipelines/rejected-records.md b/content/integrate/redis-data-integration/data-pipelines/rejected-records.md index 70ee60c641..65c00b26f1 100644 --- a/content/integrate/redis-data-integration/data-pipelines/rejected-records.md +++ b/content/integrate/redis-data-integration/data-pipelines/rejected-records.md @@ -44,11 +44,17 @@ table. A DLQ stream is named after the source data stream it corresponds to, with a `dlq:` prefix: ```text -dlq:data:{rdi}:..
+dlq:data:{rdi}:. ``` +Above, `` is the qualified table name: `.
` +for MySQL and MariaDB, `.` for MongoDB, `.
` for Oracle, +PostgreSQL, Snowflake, and Spanner, and `..
` for SQL Server. + For example, rejected records for the `public.users` table of a source named `postgresql` -are stored in `dlq:data:{rdi}:postgresql.public.users`. +are stored in `dlq:data:{rdi}:postgresql.public.users`, and those for the `dbo.users` table of +the `inventory` database of a source named `sqlserver` are stored in +`dlq:data:{rdi}:sqlserver.inventory.dbo.users`. The dead-letter queue endpoints and the CLI report table names are in the same form, so you can easily attribute a rejected record to the source it came from. Sources that existed @@ -56,7 +62,7 @@ before the upgrade to RDI v2.0.0 or above do not have the source segment in the key because these earlier RDI versions did not support multiple sources. Some RDI versions or configurations can use a hash-tagged variant such as -`dlq:{data:rdi:.
}`. To find all DLQ streams in the +`dlq:{data:rdi:}`. To find all DLQ streams in the RDI database, scan for stream keys that start with `dlq:`. The maximum number of records stored per DLQ stream is controlled by diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-get-dlq.md b/content/integrate/redis-data-integration/reference/cli/redis-di-get-dlq.md index f782d2b593..ae50b3a426 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-get-dlq.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-get-dlq.md @@ -11,9 +11,8 @@ aliases: Gets a single dead-letter queue (DLQ) of a pipeline and prints it in the compact `list-dlqs` table format. -The name is the DLQ's source-qualified table name, `..
`, as -[`list-dlqs`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs" >}}) -reports it. +The name is the DLQ's source-qualified table name, `.`, as +reported by [`list-dlqs`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs" >}}). ## Usage diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records.md b/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records.md index 23d1668be3..f36effacee 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records.md @@ -16,9 +16,8 @@ and paging with `--limit`, `--offset`, and `--sort-order`. The operation code is Use [`list-dlqs`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs" >}}) to see all the pipeline's dead-letter queues and their record counts. -The DLQ name is its source-qualified table name, `..
`, as -[`list-dlqs`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs" >}}) -reports it. +The DLQ name is its source-qualified table name, `.`, as +reported by [`list-dlqs`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs" >}}). ## Usage diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs.md b/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs.md index 2f67107b68..82e6203849 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs.md @@ -13,8 +13,11 @@ that RDI rejected. Use [`list-dlq-records`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records" >}}) to read the records of a single queue. -There is one queue per source table, named `..
`, where -`` is the source name from `config.yaml`. +There is one queue per source table, named `.`, where `` +is the source name from `config.yaml` and `` is the qualified table name, +such as `public.users` for PostgreSQL or `inventory.dbo.users` for SQL Server. See +[Rejected records]({{< relref "/integrate/redis-data-integration/data-pipelines/rejected-records#how-rdi-stores-rejected-records" >}}) +for the form each database uses. ## Usage From 430f5b912b9d8e97aa8af4bd49e12d17291a3f6f Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Mon, 14 Sep 2026 14:27:07 +0300 Subject: [PATCH 22/29] Update image versions, fix some Claude-isms --- .../data-pipelines/multiple-sources.md | 6 +++--- .../data-pipelines/supported-types.md | 4 ++-- .../redis-data-integration/installation/install-k8s.md | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md index 7cc0d7d6f8..daa1b3ab6a 100644 --- a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md +++ b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md @@ -265,14 +265,14 @@ pipeline status is not broken down per source. RDI reports the whole pipeline in single source fails, so you should use the `Components` section to find out which one has failed. Each Debezium collector has its own metric collection, named after the collector, such as -`collector-mysql_metrics`. The Spanner and Snowflake collectors don't have metric collections. +`collector-mysql_metrics`. The Flink and RIOT-X collectors don't have metric collections. In Prometheus, you can break the per-stream record counters down per source, since the stream name contains the source name. With the [Flink processor]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}}) the counters are reported by -`flink_jobmanager_job_operator_coordinator_stream_type_rdiRecords`, which carries a `stream` -label; with the classic processor they are reported by `rdi_incoming_entries`, which carries an +`flink_jobmanager_job_operator_coordinator_stream_type_rdiRecords`, which has a `stream` +label; with the classic processor they are reported by `rdi_incoming_entries`, which has an equivalent `data_source` label. See [Flink processor metrics]({{< relref "/integrate/redis-data-integration/observability#flink-processor-metrics" >}}), [Stream processor metrics]({{< relref "/integrate/redis-data-integration/observability#stream-processor-metrics" >}}), diff --git a/content/integrate/redis-data-integration/data-pipelines/supported-types.md b/content/integrate/redis-data-integration/data-pipelines/supported-types.md index 51d53e8e9c..ef22c25e49 100644 --- a/content/integrate/redis-data-integration/data-pipelines/supported-types.md +++ b/content/integrate/redis-data-integration/data-pipelines/supported-types.md @@ -195,7 +195,7 @@ choose. When a connector captures large objects (for example, Oracle `CLOB`/`BLOB`), an update event never contains the value of an *unchanged* LOB column. Instead, the -column carries a placeholder. The default placeholder is `__debezium_unavailable_value`, +column contains a placeholder. The default placeholder is `__debezium_unavailable_value`, which you can change with `advanced.source.unavailable.value.placeholder` on the source (the Helm chart exposes this as `processor.lob.placeholder`). RDI skips these placeholder values rather than writing them to Redis as user data. @@ -453,7 +453,7 @@ What is available for updates depends on `capture.mode`: - A *create* event always includes the full document. - An *update* event includes the full document only when `capture.mode` is - `change_streams_update_full`; otherwise it carries only the changed fields + `change_streams_update_full`; otherwise it includes only the changed fields (`updatedFields`/`removedFields`). A `*_with_pre_image` mode is required to include the prior document state. diff --git a/content/integrate/redis-data-integration/installation/install-k8s.md b/content/integrate/redis-data-integration/installation/install-k8s.md index b0584a2ea7..c0aa45a81f 100644 --- a/content/integrate/redis-data-integration/installation/install-k8s.md +++ b/content/integrate/redis-data-integration/installation/install-k8s.md @@ -101,12 +101,12 @@ If you plan to use Spanner as a source for your pipeline, you'll also need [redis/rdi-flink-collector](https://hub.docker.com/r/redis/rdi-flink-collector). If you plan to use Snowflake as a source for any of your pipelines, you'll also need -[riotx/riotx:v1.8.0](https://hub.docker.com/r/riotx/riotx): +[riotx/riotx:v1.12.0](https://hub.docker.com/r/riotx/riotx): [RIOT-X](https://redis.github.io/riotx/), a data ingestion and replication tool for Redis. In addition, the RDI Helm chart uses the following 3rd party images: -- [redislabs/debezium-server:3.5.0.Final-rdi.1](https://hub.docker.com/r/redislabs/debezium-server), +- [redislabs/debezium-server:3.5.0.Final-rdi.3](https://hub.docker.com/r/redislabs/debezium-server), based on `quay.io/debezium/server/3.5.0.Final` with minor modifications: [Debezium](https://debezium.io/), an open source distributed platform for change data capture. - [redis/reloader:v1.4.13](https://hub.docker.com/r/redis/reloader), originally `ghcr.io/stakater/reloader:v1.4.13`: From 9df6c0c8b619523d142eb988382ab203661a9f67 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Mon, 14 Sep 2026 16:44:42 +0300 Subject: [PATCH 23/29] Clarify naming of legacy sources --- .../data-pipelines/multiple-sources.md | 8 ++++++-- .../data-pipelines/pipeline-config.md | 5 ++++- .../data-pipelines/transform-examples/_index.md | 2 ++ .../redis-data-integration/reference/api-migration.md | 2 +- .../reference/cli/redis-di-set-secret.md | 4 +++- 5 files changed, 16 insertions(+), 5 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md index daa1b3ab6a..34254ca5a7 100644 --- a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md +++ b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md @@ -301,8 +301,12 @@ In particular, for such a source: `metadata:debezium:schema_history`. - Its `server_name` is still `rdi`, or, for a Spanner source, its instance ID. -For a source you add after the upgrade, RDI derives all of these names from the source name, -as described on this page. +These names are permanent. If you remove such a source from `config.yaml` and later add a source +with the same name, RDI applies the names from before the upgrade to it again rather than +deriving them from the source name. + +For a source you add after the upgrade under any other name, RDI derives all of these names +from the source name, as described on this page. See [Upgrading RDI]({{< relref "/integrate/redis-data-integration/installation/upgrade" >}}) for more information. diff --git a/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md b/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md index 10fb64008f..dae7f0077a 100644 --- a/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md +++ b/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md @@ -119,7 +119,10 @@ RDI also derives the environment variables that contain the source's credentials source name, for example `${MYSQL_DB_USERNAME}` and `${MYSQL_DB_PASSWORD}` for a source named `mysql`. See [Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) -for the source naming rules and for capturing from more than one source database. +for the source naming rules and for capturing from more than one source database. For a source +that existed before RDI supported multiple sources, these variables are still named +`SOURCE_DB_*`; see +[Existing names are kept after an upgrade]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}). The example configuration contains the following data: diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md index ca0965596d..99a498b31e 100644 --- a/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md @@ -63,6 +63,8 @@ The main sections of these files are: [config.yaml]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config" >}}). This is required when the pipeline has more than one source but is optional in a pipeline with a single source. See [Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for more information. + For a source that existed before RDI supported multiple sources, use `rdi` instead, as described in + [Existing names are kept after an upgrade]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}). - `db`: Database name (optional). This refers to a database name you supplied in [config.yaml]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config" >}}). - `schema`: Database schema (optional). This refers to a schema name you supplied in diff --git a/content/integrate/redis-data-integration/reference/api-migration.md b/content/integrate/redis-data-integration/reference/api-migration.md index 33c20588b1..58ebbfd22b 100644 --- a/content/integrate/redis-data-integration/reference/api-migration.md +++ b/content/integrate/redis-data-integration/reference/api-migration.md @@ -128,7 +128,7 @@ Note: 2. Add the pipeline name to each v2 request. The only pipeline in 1.19.0 is always named `default`. 3. Check the pipeline response, or call `GET /api/v2/pipelines/{name}/status`, instead of polling an action ID. 4. Use `POST /api/v2/pipelines`, `PUT /api/v2/pipelines/{name}`, or `PATCH /api/v2/pipelines/{name}` to update source, target, processor, and secret-provider settings as needed. When using `PATCH`, omit the configuration sections that you do not want to change. -5. Change secret requests to address a secret by a database-independent key and a `db` parameter. Where a v1 request set `SOURCE_DB_PASSWORD`, a v2 request sets the `PASSWORD` key with `db` naming the source, and `TARGET_DB_PASSWORD` becomes the `PASSWORD` key with `db=target`. See [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}). +5. Change secret requests to address a secret by a database-independent key and a `db` parameter. Where a v1 request set `SOURCE_DB_PASSWORD`, a v2 request sets the `PASSWORD` key with `db` naming the source, and `TARGET_DB_PASSWORD` becomes the `PASSWORD` key with `db=target`. See [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}). The environment variable that `config.yaml` references is unchanged for a source that existed before RDI supported multiple sources; see [Existing names are kept after an upgrade]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}). 6. Use `GET /api/v2/pipelines/{name}/metric-collections/{collection_name}` for monitoring and `GET /api/v2/pipelines/{name}/source-schemas/{source_name}` for source metadata. 7. Test creating, updating, validating, starting, stopping, resetting, and deleting a pipeline on a non-production RDI 1.19.0 or later installation before updating production applications. diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-set-secret.md b/content/integrate/redis-data-integration/reference/cli/redis-di-set-secret.md index 1bb2e01988..e8e490730a 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-set-secret.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-set-secret.md @@ -14,7 +14,9 @@ pipeline uses to connect to its source and target databases (see [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) for the secret keys). Every secret belongs to one database, which you name with `--db`: a source, or `target`. RDI combines the key and the database into the environment variable that `config.yaml` -references, so `PASSWORD` with `--db mysql` becomes `${MYSQL_DB_PASSWORD}`. +references, so `PASSWORD` with `--db mysql` becomes `${MYSQL_DB_PASSWORD}`. For a source that +existed before RDI supported multiple sources, the variable is still named `SOURCE_DB_PASSWORD`; see +[Existing names are kept after an upgrade]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}). The secret value comes from the `[value]` argument, the `--file` option, or the `--literal` option. If you provide none of these on an interactive terminal, the command prompts for the value without From 6876d93b4c9dba44986c719724f5e5a68c3082b9 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Wed, 16 Sep 2026 13:55:50 +0300 Subject: [PATCH 24/29] Adapt to latest changes --- .../data-pipelines/deploy.md | 30 +++++-- .../data-pipelines/multiple-sources.md | 84 ++++++++++++++++++- .../installation/upgrade.md | 46 ++++++++-- .../reference/api-migration.md | 2 +- .../reference/api-reference/openapi.json | 36 +++----- .../reference/cli/_index.md | 1 - .../reference/cli/previous-cli-comparison.md | 2 +- .../reference/cli/redis-di-delete.md | 36 +++----- .../reference/cli/redis-di-deploy.md | 10 ++- .../reference/cli/redis-di.md | 1 - 10 files changed, 175 insertions(+), 73 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/deploy.md b/content/integrate/redis-data-integration/data-pipelines/deploy.md index da1ed7eb93..7dc334f754 100644 --- a/content/integrate/redis-data-integration/data-pipelines/deploy.md +++ b/content/integrate/redis-data-integration/data-pipelines/deploy.md @@ -375,23 +375,35 @@ pipeline stops while the reset runs and starts again afterwards: redis-di reset --source mysql ``` -## Undeploy a pipeline +## Clear a pipeline {#clear-a-pipeline} -To remove a pipeline, use the -[`redis-di delete`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-delete" >}}) -command. This stops the pipeline and deletes it, along with its configuration and status, from RDI. -The secrets you set for the pipeline are not affected. +To stop a pipeline and discard its configuration, deploy an empty configuration with the +`--empty` option of +[`redis-di deploy`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-deploy" >}}): ```bash -redis-di delete +redis-di deploy --empty ``` -Because deleting a pipeline is destructive, the command asks for confirmation unless you add the -`--force` option. If you omit the pipeline name, the `default` pipeline is deleted. +RDI removes the pipeline's data plane components and deletes its data from the RDI database, +including the change data streams, offsets, schema history, dead-letter queue entries, +statistics, deduplication state, and record counters of every source. The pipeline itself +remains, with an empty configuration, so you can deploy a new configuration to it at any time. + +The pipeline secrets are not affected, so remove them yourself with +[`redis-di delete-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-delete-secret" >}}) +if you no longer need them. The records the pipeline wrote to the target database are not +deleted either. + +{{< note >}}Clearing a pipeline discards the source aliases of any source that predates RDI's +support for multiple sources, so if you deploy such a source again under the same name you +must set its secrets again, and also change its secret references and job `server_name` values. See +[Redeploying a configuration after clearing a pipeline]({{< relref "/integrate/redis-data-integration/installation/upgrade#redeploying-a-configuration-after-clearing-a-pipeline" >}}) +for a before and after example.{{< /note >}} ## Wait for changes to complete {#wait} -The commands that change a pipeline's state, namely `deploy`, `delete`, `start`, `stop`, `reset`, +The commands that change a pipeline's state, namely `deploy`, `start`, `stop`, `reset`, `set-secret`, and `delete-secret`, do not return as soon as the API accepts the request. By default, they wait for the pipeline to finish transitioning to the expected state, polling its status until it succeeds, reaches an error, or the `--timeout` (2 minutes by default) elapses. This is usually what diff --git a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md index 34254ca5a7..75697681d3 100644 --- a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md +++ b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md @@ -212,7 +212,25 @@ rather than from the position it had reached. The source's secrets are not deleted, so remove them yourself with [`redis-di delete-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-delete-secret" >}}) -if you no longer need them. +if you no longer need them. The records the pipeline wrote to the target database are not deleted +either. + +Adding the same source again is straightforward, unless the source you removed predates RDI's support +for multiple sources. For such a source the names from before the upgrade are kept only while it +exists, so any source you add under the same name is treated as a new source, for which RDI derives the +names instead. Adapt the configuration accordingly: + +- Set the source's secrets again, for example + `redis-di set-secret PASSWORD --db mysql `. +- Change the secret references in its `connection` section from `${SOURCE_DB_*}` to + `${MYSQL_DB_*}`, for a source named `mysql`. +- Change `server_name` from `rdi` to the source name in every job associated with it. + +See +[Redeploying a configuration after clearing a pipeline](#redeploying-a-configuration-after-clearing-a-pipeline) +for a before and after example, and +[Existing names are kept after an upgrade](#existing-names-are-kept-after-an-upgrade) +for the full list of names involved. Note that renaming a source is not supported. Renaming a source in `config.yaml` is equivalent to removing the source and adding a new source with the new name. This implies in particular: @@ -301,12 +319,70 @@ In particular, for such a source: `metadata:debezium:schema_history`. - Its `server_name` is still `rdi`, or, for a Spanner source, its instance ID. -These names are permanent. If you remove such a source from `config.yaml` and later add a source -with the same name, RDI applies the names from before the upgrade to it again rather than -deriving them from the source name. +RDI keeps these names in a mapping from the source name in `config.yaml` to +the internal name the source had before the upgrade. This mapping lasts only as long as the source +does: RDI discards it as soon as the source is removed from the configuration, whether you remove that one +source or +[clear the whole pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#clear-a-pipeline" >}}). +A source you add afterwards under the same name is treated as a new source, so +RDI derives its names from the source name. See +[Add or remove a source](#add-or-remove-a-source) for what you have to change in that case. For a source you add after the upgrade under any other name, RDI derives all of these names from the source name, as described on this page. See [Upgrading RDI]({{< relref "/integrate/redis-data-integration/installation/upgrade" >}}) for more information. + +## Redeploying a configuration after clearing a pipeline + +A configuration exported from an upgraded pipeline still references the names from before the +upgrade, so deploying it again after +[clearing the pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#clear-a-pipeline" >}}) +fails, because the mapping that made those names resolve is gone. + +A source and a job of such an upgraded pipeline: + +```yaml +sources: + mysql: + connection: + user: ${SOURCE_DB_USERNAME} + password: ${SOURCE_DB_PASSWORD} +``` + +```yaml +source: + server_name: rdi + db: inventory + table: customers +``` + +The same source and job, adapted to deploy as a new source named `mysql`: + +```yaml +sources: + mysql: + connection: + user: ${MYSQL_DB_USERNAME} + password: ${MYSQL_DB_PASSWORD} +``` + +```yaml +source: + server_name: mysql + db: inventory + table: customers +``` + +Set the source's secrets under its actual name before you deploy: + +```bash +redis-di set-secret USERNAME --db mysql +redis-di set-secret PASSWORD --db mysql +``` + +The source then takes a fresh +[initial snapshot]({{< relref "/integrate/redis-data-integration/architecture" >}}), because the position it had +reached was deleted along with the rest of its data. Records the pipeline already wrote to the +target database are not deleted, so the snapshot overwrites them. diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index 9dec5e73e1..01f75e588c 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -196,13 +196,49 @@ described in [Uninstall RDI]({{< relref "/integrate/redis-data-integration/insta and then install the old version. {{< /note >}} -## Source names before and after the upgrade +## Upgrading to RDI 2.0.0 {#rdi-200} -For a source that existed before upgrading to an RDI version that supports multiple sources, the -resource names from before the upgrade are kept unchanged. The source will keep running correctly without changes. +RDI 2.0.0 changes several behaviors that affect a pipeline upgraded from RDI 1.19.x or earlier. -For more details on how to handle sources created after upgrading, see -[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}). +### The Flink processor becomes the default + +A pipeline whose `config.yaml` does not set `processors.type` deploys the Flink processor instead +of the classic one from RDI 2.0.0 on. See +[Enabling the Flink processor](#enabling-the-flink-processor) below. + +If your pipeline uses the classic processor, we recommend migrating it to the Flink processor deliberately either +before or after the upgrade to 2.0.0, rather than letting this happen during the upgrade. See +[Migrate from the classic processor to the Flink processor]({{< relref "/integrate/redis-data-integration/installation/migration-classic-to-flink" >}}). + +### Clearing a pipeline replaces deleting it + +`DELETE /api/v2/pipelines/{name}` no longer empties a pipeline, and `redis-di delete` no longer +works. Clear the configuration with `redis-di deploy --empty`, or with +`PUT /api/v2/pipelines/{name}` and an empty configuration, instead. See +[Clear a pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#clear-a-pipeline" >}}). + +Clearing a pipeline is not the same as resetting it or flushing the target database. Clearing +removes the configuration and the pipeline's data from the RDI database, while resetting keeps the +configuration and re-snapshots the sources; neither one deletes any records from the target +database. + +### Redeploying a configuration after clearing a pipeline + +A configuration of an upgraded pipeline still references the names from before the upgrade, so +deploying it again after clearing the pipeline fails. See +[Redeploying a configuration after clearing a pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#redeploying-a-configuration-after-clearing-a-pipeline" >}}) +for what you have to change, with a before and after example. + +### API changes + +RDI 2.0.0 changes several API v2 query parameters and response fields. Most of them affect only +an API client, and are listed in the +[RDI 2.0.0 release notes]({{< relref "/integrate/redis-data-integration/release-notes/rdi-2-0-0" >}}). + +The one to check for is the metric collections endpoint, which now keys `data_streams.streams` +by the source-qualified table name, such as `mysql.inventory.addresses`, instead of by the Redis +stream name. Update any dashboard or script that reads the previous form. See +[Observability]({{< relref "/integrate/redis-data-integration/observability" >}}) for the metrics RDI reports. ## Enabling the Flink processor diff --git a/content/integrate/redis-data-integration/reference/api-migration.md b/content/integrate/redis-data-integration/reference/api-migration.md index 58ebbfd22b..b906cded37 100644 --- a/content/integrate/redis-data-integration/reference/api-migration.md +++ b/content/integrate/redis-data-integration/reference/api-migration.md @@ -59,7 +59,7 @@ you must use API v2 to manage [a pipeline with several sources]({{< relref "/int | Source metadata, schemas, databases, tables, and columns endpoints | `GET /api/v2/pipelines/{name}/source-schemas/{source_name}` with the appropriate filters | | `POST /api/v1/pipelines/sources/dry-run` | `POST /api/v2/pipelines?dry_run=true` | | `POST /api/v1/pipelines/targets/dry-run` | `POST /api/v2/pipelines?dry_run=true` | -| `POST /api/v1/pipelines/undeploy` | `DELETE /api/v2/pipelines/{name}` | +| `POST /api/v1/pipelines/undeploy` | `PUT /api/v2/pipelines/{name}` with an empty configuration | | `POST /api/v1/trace/start` | `POST /api/v2/pipelines/{name}/traces` | API v2 also adds endpoints for DLQ inspection, target flushing, metric collections, and API information. See the [API reference]({{< relref "/integrate/redis-data-integration/reference/api-reference" >}}) for the complete list. diff --git a/content/integrate/redis-data-integration/reference/api-reference/openapi.json b/content/integrate/redis-data-integration/reference/api-reference/openapi.json index 57241fb3a4..d7bff387c8 100644 --- a/content/integrate/redis-data-integration/reference/api-reference/openapi.json +++ b/content/integrate/redis-data-integration/reference/api-reference/openapi.json @@ -4241,30 +4241,6 @@ "title": "Dry Run" }, "description": "Only validate the provided configuration without making changes" - }, - { - "name": "validate_tables", - "in": "query", - "required": false, - "schema": { - "type": "boolean", - "description": "Validate against source and target databases via collector API", - "default": true, - "title": "Validate Tables" - }, - "description": "Validate against source and target databases via collector API" - }, - { - "name": "validate_cdc", - "in": "query", - "required": false, - "schema": { - "type": "boolean", - "description": "Validate CDC configuration via collector API", - "default": false, - "title": "Validate Cdc" - }, - "description": "Validate CDC configuration via collector API" } ], "requestBody": { @@ -4758,7 +4734,7 @@ "delete": { "tags": ["v2", "pipelines"], "summary": "Delete Pipeline", - "description": "Replaces a pipeline with an empty inactive pipeline.", + "description": "Deletes a pipeline by name.", "operationId": "delete_pipeline_api_v2_pipelines__name__delete", "security": [ { @@ -4812,6 +4788,16 @@ }, "description": "Not Found" }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, "423": { "content": { "application/json": { diff --git a/content/integrate/redis-data-integration/reference/cli/_index.md b/content/integrate/redis-data-integration/reference/cli/_index.md index a6a5e5e7c5..440f55a866 100644 --- a/content/integrate/redis-data-integration/reference/cli/_index.md +++ b/content/integrate/redis-data-integration/reference/cli/_index.md @@ -98,7 +98,6 @@ The commands group as follows: [`get`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-get" >}}), [`describe`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-describe" >}}) (alias `status`), [`deploy`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-deploy" >}}) (alias `set`), - [`delete`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-delete" >}}), [`start`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-start" >}}), [`stop`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-stop" >}}), and [`reset`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-reset" >}}). diff --git a/content/integrate/redis-data-integration/reference/cli/previous-cli-comparison.md b/content/integrate/redis-data-integration/reference/cli/previous-cli-comparison.md index 1bbaa51cf7..8aaaf1915a 100644 --- a/content/integrate/redis-data-integration/reference/cli/previous-cli-comparison.md +++ b/content/integrate/redis-data-integration/reference/cli/previous-cli-comparison.md @@ -119,7 +119,7 @@ Other common tasks changed as follows: | Continuously refresh status | `redis-di status --live` | `watch -n 1 redis-di describe [pipeline]` | | List pipelines | Not available | `redis-di list` | | Get a pipeline | Not available | `redis-di get [pipeline]` | -| Delete a pipeline | Not available | `redis-di delete [pipeline]` | +| Clear a pipeline | Not available | `redis-di deploy --empty` | | Inspect rejected records | `redis-di get-rejected [options]` | `redis-di list-dlqs`, then `redis-di list-dlq-records `; `get-rejected` remains an alias | | Inspect jobs | `redis-di list-jobs` and `redis-di describe-job ` | The same commands, with `--pipeline ` for a non-default pipeline | | Manage secrets | `redis-di set-secret ` or `rdi-secret.sh` | `list-secrets`, `get-secret`, `describe-secret`, `set-secret`, and `delete-secret` | diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-delete.md b/content/integrate/redis-data-integration/reference/cli/redis-di-delete.md index 1ea013d9ef..694b084803 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-delete.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-delete.md @@ -1,38 +1,24 @@ --- Title: redis-di delete linkTitle: redis-di delete -description: Deletes a pipeline +description: Deleting a pipeline is not supported weight: 10 alwaysopen: false categories: ["redis-di"] aliases: -- /integrate/redis-data-integration/ingest/reference/cli/redis-di-delete/ + - /integrate/redis-data-integration/ingest/reference/cli/redis-di-delete/ --- -Deletes a pipeline. Because this is destructive, the command asks for confirmation unless you pass -`--force`. +Deleting a pipeline is currently not supported, and the `redis-di delete` command is no longer +available. -## Usage - -``` -redis-di delete [pipeline] [flags] -``` - -The pipeline name is an optional argument that defaults to `default`. - -## Options - -| Option | Description | -| :---------- | :-------------------------------------------------------------------------------- | -| `--force` | Skip the confirmation prompt. | -| `--wait` | Wait for the pipeline to reach the expected state (default `true`). | -| `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). | - -This command also accepts the -[global options]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di#global-options" >}}). - -## Example +To stop a pipeline and discard its configuration, deploy an empty configuration instead: ```bash -redis-di delete my-pipeline --force +redis-di deploy --empty ``` + +See [Clear a pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#clear-a-pipeline" >}}) +for what this removes and what it keeps, and +[`redis-di deploy`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-deploy" >}}) +for the command's options. diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-deploy.md b/content/integrate/redis-data-integration/reference/cli/redis-di-deploy.md index c21fec3834..d812e5288c 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-deploy.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-deploy.md @@ -14,6 +14,10 @@ API validates the configuration and rejects an invalid one. By default, the comm pipeline after deploying and waits for it to reach the expected state. `set` is an alias for this command. +Pass `--empty` instead of `--dir` to deploy an empty configuration, which clears the pipeline. See +[Clear a pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#clear-a-pipeline" >}}) for +what clearing a pipeline removes and what it keeps. + ## Usage ``` @@ -27,10 +31,11 @@ The pipeline name is an optional argument that defaults to `default`. | Option | Description | | :------------------ | :----------------------------------------------------------------------------------- | | `--dir` | Directory containing the pipeline configuration (default `.`). | +| `--empty` | Deploy an empty configuration instead of loading one from a directory. Cannot be combined with `--dir`. | | `--dry-run` | Validate the configuration without deploying. | | `--validate-tables` | Validate the configuration against the source and target databases (default `true`). | | `--validate-cdc` | Validate the CDC configuration of the source databases. | -| `--start` | Start the pipeline after deploying (default `true`). | +| `--start` | Start the pipeline after deploying (default `true`, or `false` with `--empty`). | | `--wait` | Wait for the pipeline to reach the expected state (default `true`). | | `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). | @@ -45,4 +50,7 @@ redis-di deploy # Validate a configuration folder without deploying it redis-di deploy --dir /opt/rdi/config --dry-run + +# Clear the pipeline by deploying an empty configuration +redis-di deploy --empty ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di.md b/content/integrate/redis-data-integration/reference/cli/redis-di.md index 608fbd8953..d1db13f7d8 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di.md @@ -56,7 +56,6 @@ authentication modes. Setting both `--cacert` and `--insecure` is also an error. | [`get`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-get" >}}) | Gets a pipeline | | [`describe`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-describe" >}}) | Describes a pipeline with its status (alias `status`) | | [`deploy`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-deploy" >}}) | Deploys a pipeline with the specified configuration (alias `set`) | -| [`delete`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-delete" >}}) | Deletes a pipeline | | [`start`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-start" >}}) | Starts a pipeline | | [`stop`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-stop" >}}) | Stops a pipeline | | [`reset`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-reset" >}}) | Resets a pipeline | From 9d47aa91e28c90cccc851a712dc494aca66fc485 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Wed, 16 Sep 2026 14:06:34 +0300 Subject: [PATCH 25/29] Update release notes --- .../redis-data-integration/release-notes/rdi-2-0-0.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/release-notes/rdi-2-0-0.md b/content/integrate/redis-data-integration/release-notes/rdi-2-0-0.md index c213bafcf7..e49eff03c1 100644 --- a/content/integrate/redis-data-integration/release-notes/rdi-2-0-0.md +++ b/content/integrate/redis-data-integration/release-notes/rdi-2-0-0.md @@ -23,6 +23,8 @@ weight: 967 - A request now applies its changes immediately rather than waiting behind earlier in-flight requests, so the most recent request wins if several are issued in quick succession without waiting for each to complete. Polling `GET /actions/{action_id}` for an action id that a later request has superseded now returns an unknown action error instead of that action's own status. - `POST /pipelines` and `PATCH /pipelines` now also validate target connectivity similarly to their API v2 counterparts, and surface collector API failures as a `422`, `502`, `503`, or `504` error response. A request that previously succeeded despite an unreachable target now fails with a validation error. - **API v1 trace endpoint returns Not Implemented**: `POST /trace/start` now immediately returns `501 Not Implemented` instead of accepting the request and queuing a trace that would never run. +- **API v2 pipeline DELETE no longer empties the pipeline**: `DELETE /api/v2/pipelines/{name}` no longer replaces the pipeline with an empty, inactive one. It now returns `404` when the named pipeline doesn't exist, and `422` for the `default` pipeline, since deleting it isn't supported. Use `redis-di deploy --empty`, or `PUT /api/v2/pipelines/{name}` with an empty configuration, to clear a pipeline instead. +- **API v2 pipeline creation no longer validates against the Collector API**: `POST /pipelines` no longer accepts the `validate_tables` and `validate_cdc` query parameters, and never validates a new pipeline's tables or CDC setup against the Collector API. That validation could never succeed: the Collector API is rendered by the pipeline being created, so it cannot exist yet for the request creating it. `PUT /pipelines/{name}` keeps both parameters, but doesn't use them when creating a new pipeline. - **API v2 pipeline responses omit optional fields that have no value**: The pipeline, pipeline status, create, update, patch, start, stop, and reset responses no longer contain optional fields with null values; such fields are left out instead. This affects `status_changed_at` and an error's `remediation`. The OpenAPI schema is unchanged, since neither field was ever required, so clients generated from it are unaffected; a client that reads either key directly has to treat it as absent rather than null. - **`topic.prefix` is no longer accepted in `advanced.source`**: The `topic.prefix` property is now rejected in a source's `advanced.source` section, since RDI derives the topic prefix from the source name. Remove the property from your configuration and set each job's `server_name` to the derived prefix: the source name, `rdi` for the source of an upgraded single-source pipeline handled by the Debezium collector, and the Spanner instance ID for the source of such an upgraded pipeline handled by the Flink collector. - **API v2 metric collections data stream names contain the qualified table name**: The `data_streams.streams` keys of a metric collection are now the source-qualified table name (`mysql.inventory.addresses`), matching what the dead-letter queue endpoints return, instead of the Redis stream name (`{rdi}:inventory.addresses`). A client that uses the previous form has to be updated. API v1 statistics are unchanged. @@ -31,7 +33,6 @@ weight: 967 ### New Features - **Multiple sources in one pipeline**: A pipeline can ingest data from several source databases, of the same or different types, into one Redis target. Each source has its own name, connection settings, credentials, and collector. Use API v2 or the `redis-di` CLI to manage multi-source pipelines; API v1 supports only single-source pipelines. To reset one source while preserving the other sources' internal data, use the optional `source` query parameter on `POST /pipelines/{name}/reset` or the `redis-di reset --source` flag. Source reset temporarily stops and restarts the whole pipeline and does not support `external` sources. See [Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}). - - **Transformation jobs can select several tables (Flink processor)**: A job's `server_name`, `db`, `schema`, and `table` source selectors now accept a list of values in addition to a single one, and an entry prefixed with `regex:` is matched as an anchored regular expression, so one job can handle many tables. An entry without the prefix is matched literally, so existing jobs are unaffected, including table names containing regular expression characters. Two jobs whose selectors select the same table are rejected. Only the Flink processor supports this syntax, so a job using it is rejected for the classic processor. - **Database-scoped pipeline secret keys**: Pipeline secrets in API v2 and the `redis-di` CLI now use database-independent keys (`USERNAME`, `PASSWORD`, `CACERT`, `CERT`, `KEY`, `KEY_PASSWORD`) together with a `db` parameter (`--db` in the CLI) that specifies the database the secret belongs to: a source name, or `target`. The previous scope-prefixed keys (`SOURCE_DB_*`, `TARGET_DB_*`) are still accepted for single-source and target secrets, used without the `db` parameter. - **Richer validation errors from API v1 pipeline configuration endpoints**: A `422` response from `/pipelines/undeploy`, `/pipelines/sources/*`, `/pipelines/targets/*`, `/pipelines/processors/*`, and `/pipelines/secret-providers/*` may now include an `errors` array with structured per-field detail, in addition to the existing `detail` message, matching the shape already returned by `POST /pipelines` and `PATCH /pipelines`. Existing clients that only read `detail` are unaffected. @@ -43,6 +44,7 @@ weight: 967 - **Pipeline components report their externally-reachable endpoints**: Each component in an API v2 pipeline or pipeline status response now has an `external_endpoints` field listing the URLs at which that component is reachable from outside the cluster. - **Source mTLS without Debezium keystore settings**: A MySQL, MariaDB, or MongoDB source no longer requires setting `database.ssl.keystore`/`mongodb.ssl.keystore` and the matching password in `advanced.source` for mTLS to work correctly. - **Removing a source from a pipeline deletes its internal RDI data**: Removing a source with `PUT` or `PATCH /pipelines/{name}` now deletes its internal RDI data, including its change streams, offsets, schema history, dead letter queue entries, statistics, deduplication state, and record counters. That data used to be kept indefinitely, and only resetting the pipeline removed it. Records already written to the target Redis database are not deleted. +- **`redis-di deploy --empty` clears a pipeline's configuration**: The new `--empty` flag, mutually exclusive with `--dir`, deploys an empty configuration instead of loading one from a directory. An empty pipeline is left stopped unless `--start` is passed explicitly. See [Clear a pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#clear-a-pipeline" >}}). ### Bug Fixes @@ -53,6 +55,8 @@ weight: 967 - **Unreachable source database reported as a validation error**: When RDI cannot list the source tables during pipeline validation, for example because the source database refuses the connection, the failure is now reported as a `422` validation error in the `errors` array, instead of a `422`, `502`, or `504` response carrying only a `detail` message. An unavailable collector API is still reported as a `503` response so the request can be retried. - **Redis configuration template requests return `501 Not Implemented`**: `GET /pipelines/config/templates/ingest/redis` now returns the documented `501 Not Implemented`, instead of a `200` response with an incorrect configuration. Redis cannot be used as a pipeline source. - **A database flavor that does not match the database type is rejected**: `GET /pipelines/config/templates/ingest/{db_type}` now returns a `422` response when `db_flavor` names a MongoDB flavor and `db_type` is not `mongodb`, instead of a `200` response with an incorrect configuration. +- **Clearing a whole configuration section with `PATCH` is rejected**: `PATCH /pipelines/{name}` now returns a `422` validation error when `processors`, `sources`, or `targets` is set to null. Remove a section's entries individually, as in `{"sources": {"mysql": null}}`, or replace the whole configuration with `PUT`. +- **A missing pipeline's 404 error message contains its name**: A `404` response for a pipeline that doesn't exist is now `Pipeline '' not found`, instead of `Pipeline not found`, consistent with the other error messages. ### Improvements From cd95746f62c53726bb5436d20745a0a661f85442 Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Wed, 16 Sep 2026 15:00:24 +0300 Subject: [PATCH 26/29] Rebase on latest release notes and adapt --- .../migration-classic-to-flink.md | 33 +++------------- .../installation/upgrade.md | 38 +++++++------------ 2 files changed, 20 insertions(+), 51 deletions(-) diff --git a/content/integrate/redis-data-integration/installation/migration-classic-to-flink.md b/content/integrate/redis-data-integration/installation/migration-classic-to-flink.md index 82b695e30a..92435ffc28 100644 --- a/content/integrate/redis-data-integration/installation/migration-classic-to-flink.md +++ b/content/integrate/redis-data-integration/installation/migration-classic-to-flink.md @@ -99,11 +99,9 @@ redis-di deploy --dir Wait for the deployment to finish and the source collector to stop. Keep the pipeline active so the classic processor can process the remaining input records. -Do not use `redis-di stop` for this step, because it also stops the processor. Applications can continue writing to the source database while collection -is disabled. Make sure the database change log retains the whole paused -interval. When the collector restarts, it resumes from the saved source +is disabled. When the collector restarts, it resumes from the saved source position and processes those changes. ## Step 3: Wait for the input streams to empty @@ -118,14 +116,11 @@ Do not include DLQ streams. Use any of the following methods. Run: ```bash -redis-di describe default +redis-di describe ``` In the **Statistics** table, the **Pending** value for each classic processor -stream is its current length. Confirm that every input stream is listed and -that the values agree with the Redis command checks below. This **Pending** -value is different from consumer-group pending entries. `XPENDING` or group -lag of `0` alone does not prove that a stream is empty. +stream is its current length. Confirm that every input stream is listed. ### Check with Redis commands @@ -156,24 +151,6 @@ XLEN Run a complete `SCAN` and all `XLEN` commands in each of the three checks. -### Check with Redis Insight - -1. Connect Redis Insight to the RDI database and open **Browse**. -1. Filter by the pipeline's input stream pattern. For the default pipeline, - use `data:{rdi}:*`. Confirm that all input streams are listed. -1. Open each stream, select **Stream Data**, and use the refresh button. - Confirm that **Entries** is `0`. -1. Repeat the complete inventory and entry check three times, five seconds - apart. - -You can also open the built-in **CLI** and run the `SCAN` and `XLEN` commands -shown above. The Browser and CLI results must contain the same streams and -lengths. - -If records remain, keep the classic processor running and resolve its -processing errors before continuing. Do not delete stream entries, reset the -pipeline, or move consumer-group positions to make the count reach `0`. - ## Step 4: Switch processors and resume collection After the drain check passes, remove the source's `active: false` setting @@ -186,7 +163,9 @@ processors: type: flink ``` -RDI 1.19.0 requires this setting because its default processor is `classic`. +RDI 1.19.x requires this setting because its default processor is `classic`. +On RDI 2.0.0 the default processor is `flink`, so this setting is no longer +required. Keep the remaining configuration and jobs, then redeploy the complete configuration directory: diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index 01f75e588c..28ffb51df3 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -21,23 +21,6 @@ Before upgrading to RDI 2.0.0, review the [processor default change](#upgrading-to-rdi-200). {{< /note >}} -## Upgrading to RDI 2.0.0 - -RDI 2.0.0 changes the default processor from `classic` to `flink`. This default -applies when the pipeline's `config.yaml` omits `processors.type`. - -For an existing classic pipeline, choose one of these options before upgrading: - -- To migrate to Flink, first follow - [Migrate from the classic processor to the Flink processor]({{< relref "/integrate/redis-data-integration/installation/migration-classic-to-flink" >}}) - on RDI 1.19.0. This stops the collector and drains the input streams before - switching processors. Then upgrade RDI. -- To keep the classic processor, set `processors.type: classic` in the - pipeline's `config.yaml` and deploy it before upgrading. - -If your pipeline already uses `processors.type: flink`, no processor change -is needed. Continue with the upgrade instructions for your installation. - ## Upgrading a VM installation Follow the steps below to upgrade an existing @@ -196,19 +179,26 @@ described in [Uninstall RDI]({{< relref "/integrate/redis-data-integration/insta and then install the old version. {{< /note >}} -## Upgrading to RDI 2.0.0 {#rdi-200} +## Upgrading to RDI 2.0.0 RDI 2.0.0 changes several behaviors that affect a pipeline upgraded from RDI 1.19.x or earlier. ### The Flink processor becomes the default -A pipeline whose `config.yaml` does not set `processors.type` deploys the Flink processor instead -of the classic one from RDI 2.0.0 on. See -[Enabling the Flink processor](#enabling-the-flink-processor) below. +RDI 2.0.0 changes the default processor from `classic` to `flink`. This default +applies when the pipeline's `config.yaml` omits `processors.type`. -If your pipeline uses the classic processor, we recommend migrating it to the Flink processor deliberately either -before or after the upgrade to 2.0.0, rather than letting this happen during the upgrade. See -[Migrate from the classic processor to the Flink processor]({{< relref "/integrate/redis-data-integration/installation/migration-classic-to-flink" >}}). +For an existing pipeline that uses the classic processor, choose one of these options before upgrading: + +- To migrate to the Flink processor, first follow + [Migrate from the classic processor to the Flink processor]({{< relref "/integrate/redis-data-integration/installation/migration-classic-to-flink" >}}) + on RDI 1.19.0. This stops the collector and drains the input streams before + switching processors. Then upgrade RDI. +- To keep the classic processor, set `processors.type: classic` in the + pipeline's `config.yaml` and deploy it before upgrading. + +If your pipeline already uses `processors.type: flink`, no processor change +is needed. Continue with the upgrade instructions for your installation. ### Clearing a pipeline replaces deleting it From 26058e0fc6585deb2a4086baeede53a38b368dcf Mon Sep 17 00:00:00 2001 From: Stoyan Rachev Date: Wed, 16 Sep 2026 15:11:52 +0300 Subject: [PATCH 27/29] Update config-yaml-reference.md --- .../reference/config-yaml-reference.md | 44 ++++++++++++++++--- 1 file changed, 39 insertions(+), 5 deletions(-) diff --git a/content/integrate/redis-data-integration/reference/config-yaml-reference.md b/content/integrate/redis-data-integration/reference/config-yaml-reference.md index cd352702ec..d63df53a7e 100644 --- a/content/integrate/redis-data-integration/reference/config-yaml-reference.md +++ b/content/integrate/redis-data-integration/reference/config-yaml-reference.md @@ -111,7 +111,7 @@ Connection configuration for a supported SQL database. |**type**
(Database type)|`string`|SQL database engine.
Enum: `"mariadb"`, `"mysql"`, `"oracle"`, `"postgresql"`, `"sqlserver"`
|| |**host**
(Database host)|`string`|Hostname or IP address of the SQL database server.
|| |**port**
(Database port)|`integer`|Network port on which the SQL database server is listening.
Minimum: `1`
Maximum: `65535`
|| -|**database**
(Database name)|`string`|Name of the database to connect to.
|| +|**database**
(Database name)|`string`|Name of the database to connect to.
Minimal Length: `1`
|| |**user**
(Database user)|`string`|Username for authentication to the SQL database.
|| |**password**
(Database password)|`string`|Password for authentication to the SQL database.
|| @@ -373,13 +373,14 @@ sink: redis.oom.retry.max.delay.ms: 10000 redis.oom.retry.backoff.multiplier: 2 redis.wait.enabled: false + redis.wait.timeout.ms: 1000 redis.wait.write.timeout.ms: 1000 redis.wait.retry.enabled: false redis.wait.retry.delay.ms: 1000 source: + snapshot.mode: initial snapshot.max.threads: 1 poll.interval.ms: 500 - snapshot.fetch.size: 10000 max.batch.size: 2048 max.queue.size: 8192 heartbeat.interval.ms: 0 @@ -389,6 +390,16 @@ source: publication.autocreate.mode: all_tables publication.name: dbz_publication slot.name: debezium + capture.mode: change_streams_update_full + capture.mode.full.update.type: lookup + cursor.oversize.handling.mode: fail + cursor.oversize.skip.threshold: 0 + mongodb.authsource: admin + mongodb.ssl.invalid.hostname.allowed: false + mongodb.connect.timeout.ms: 10000 + mongodb.socket.timeout.ms: 0 + mongodb.server.selection.timeout.ms: 30000 + mongodb.heartbeat.frequency.ms: 10000 spanner.version.retention.period.hours: 1 spanner.fetch.timeout.ms: 500 spanner.fetch.heartbeat.ms: 100 @@ -434,6 +445,7 @@ Advanced configuration properties for the RDI Collector stream writer connection |**redis\.oom\.retry\.max\.delay\.ms**
(Sink OOM retry max delay)|`integer`|Maximum delay in milliseconds between retry attempts after a Redis out-of-memory error.
Default: `10000`
Minimum: `1`
|| |**redis\.oom\.retry\.backoff\.multiplier**
(Sink OOM retry backoff multiplier)|`number`|Exponential backoff multiplier between retry attempts after a Redis out-of-memory error.
Default: `2`
Minimum: `1`
|| |**redis\.wait\.enabled**
(Sink replica wait enabled)|`boolean`|When `true`, the collector verifies that each write has been replicated to the configured number of Redis replica shards before acknowledging it.
Default: `false`
|| +|**redis\.wait\.timeout\.ms**
(Sink replica wait timeout)|`integer`|Maximum time in milliseconds to wait for replica write acknowledgements.
Default: `1000`
Minimum: `1`
|| |**redis\.wait\.write\.timeout\.ms**
(Sink replica wait timeout)|`integer`|Maximum time in milliseconds to wait for replica write acknowledgements.
Default: `1000`
Minimum: `1`
|| |**redis\.wait\.retry\.enabled**
(Sink replica wait retry enabled)|`boolean`|When `true`, the collector keeps retrying a write until replica acknowledgement succeeds; when `false`, it gives up after the first failure.
Default: `false`
|| |**redis\.wait\.retry\.delay\.ms**
(Sink replica wait retry delay)|`integer`|Delay in milliseconds between replica wait retry attempts.
Default: `1000`
Minimum: `1`
|| @@ -460,6 +472,7 @@ redis.oom.retry.initial.delay.ms: 1000 redis.oom.retry.max.delay.ms: 10000 redis.oom.retry.backoff.multiplier: 2 redis.wait.enabled: false +redis.wait.timeout.ms: 1000 redis.wait.write.timeout.ms: 1000 redis.wait.retry.enabled: false redis.wait.retry.delay.ms: 1000 @@ -477,9 +490,10 @@ Advanced configuration properties for the source database connection and CDC beh |Name|Type|Description|Required| |----|----|-----------|--------| |**record\.processing\.threads**
(Record processing threads)|`integer`|Controls how many worker threads process captured records before they are written downstream.
Minimum: `1`
|| +|**snapshot\.mode**
(Snapshot mode)|`string`|Determines when the collector snapshots the source database. A pipeline reset clears the recorded offset, so this also decides whether the collector snapshots again after a reset.
Default: `"initial"`
Enum: `"always"`, `"initial"`, `"initial_only"`, `"never"`, `"no_data"`, `"when_needed"`
|| |**snapshot\.max\.threads**
(Snapshot max threads)|`integer`|Sets the maximum number of threads used while taking the initial snapshot.
Default: `1`
Minimum: `1`
|| |**poll\.interval\.ms**
(Poll interval ms)|`integer`|Defines how often the collector polls the source for new changes.
Default: `500`
Minimum: `1`
|| -|**snapshot\.fetch\.size**
(Snapshot fetch size)|`integer`|Defines how many rows are fetched per batch during the initial snapshot.
Default: `10000`
Minimum: `1`
|| +|**snapshot\.fetch\.size**
(Snapshot fetch size)|`integer`|Defines how many rows are fetched per batch during the initial snapshot.
Minimum: `0`
|| |**max\.batch\.size**
(Max batch size)|`integer`|Caps how many records are processed together in a single batch.
Default: `2048`
Minimum: `1`
|| |**max\.queue\.size**
(Max queue size)|`integer`|Limits how many records can be buffered in memory before processing catches up.
Default: `8192`
Minimum: `1`
|| |**heartbeat\.interval\.ms**
(Heartbeat interval ms)|`integer`|Sets how often heartbeat events are emitted to keep change tracking active. Use 0 to disable them.
Default: `0`
Minimum: `0`
|| @@ -491,6 +505,16 @@ Advanced configuration properties for the source database connection and CDC beh |**publication\.autocreate\.mode**
(Publication autocreate mode)|`string`|Controls whether and how the PostgreSQL publication is created or updated automatically.
Default: `"all_tables"`
Enum: `"all_tables"`, `"filtered"`, `"disabled"`
|| |**publication\.name**
(Publication name)|`string`|Sets the PostgreSQL logical replication publication name used by the collector.
Default: `"dbz_publication"`
|| |**slot\.name**
(Slot name)|`string`|Sets the PostgreSQL replication slot name the collector reads from.
Default: `"debezium"`
|| +|**capture\.mode**
(MongoDB capture mode)|`string`|Determines whether an update event carries the full document and its pre-image. Jobs that set an output key, and the set output types, need a `*_with_pre_image` mode and pre-images on the collections.
Default: `"change_streams_update_full"`
Enum: `"change_streams"`, `"change_streams_update_full"`, `"change_streams_update_full_with_pre_image"`, `"change_streams_with_pre_image"`
|| +|**capture\.mode\.full\.update\.type**
(MongoDB full update type)|`string`|Determines how the collector fetches the full document for an update event. `lookup` queries the source per event and can yield a null `after`; `post_image` reads the post-image and needs MongoDB 6.0.
Default: `"lookup"`
Enum: `"lookup"`, `"post_image"`
|| +|**cursor\.oversize\.handling\.mode**
(MongoDB oversize event handling)|`string`|Determines what happens to a change event above the MongoDB 16 MB limit: `fail` stops the collector, `skip` drops it, `split` reassembles it. The limit covers the document and its pre-image.
Default: `"fail"`
Enum: `"fail"`, `"skip"`, `"split"`
|| +|**cursor\.oversize\.skip\.threshold**
(MongoDB oversize skip threshold)|`integer`|Maximum size in bytes of the document, and of its pre-image, for which change events are processed. Only applies when `cursor.oversize.handling.mode` is `skip`, which requires a value above 0.
Default: `0`
Minimum: `0`
|| +|**mongodb\.authsource**
(MongoDB authentication database)|`string`|Database that holds the credentials the collector authenticates with. Only applies to the connection's `user` and `password`; an `authSource` in `connection_string` takes precedence.
Default: `"admin"`
|| +|**mongodb\.ssl\.invalid\.hostname\.allowed**
(Allow invalid MongoDB hostname)|`boolean`|Controls whether the RDI collector skips hostname verification of the MongoDB TLS certificate. Enable this only when the certificate cannot cover the hostnames the replica set advertises.
Default: `false`
|| +|**mongodb\.connect\.timeout\.ms**
(MongoDB connect timeout)|`integer`|Time in milliseconds the driver waits before it aborts a new connection attempt. The `connectTimeoutMS` option of `connection_string` takes precedence.
Default: `10000`
Minimum: `0`
|| +|**mongodb\.socket\.timeout\.ms**
(MongoDB socket timeout)|`integer`|Time in milliseconds a send or receive on the socket may take before it times out, where 0 disables the timeout. The `socketTimeoutMS` option of `connection_string` takes precedence.
Default: `0`
Minimum: `0`
|| +|**mongodb\.server\.selection\.timeout\.ms**
(MongoDB server selection timeout)|`integer`|Time in milliseconds the driver waits to select a server before it fails with an error. The `serverSelectionTimeoutMS` option of `connection_string` takes precedence.
Default: `30000`
Minimum: `0`
|| +|**mongodb\.heartbeat\.frequency\.ms**
(MongoDB heartbeat frequency)|`integer`|Interval in milliseconds at which the cluster monitor reaches each MongoDB server. The `heartbeatFrequencyMS` option of `connection_string` takes precedence.
Default: `10000`
Minimum: `0`
|| |**spanner\.version\.retention\.period\.hours**
(Spanner version retention period)|`integer`|Retention period in hours for Spanner change stream versions. Determines how far back the collector can resume after an outage.
Default: `1`
Minimum: `1`
|| |**spanner\.fetch\.timeout\.ms**
(Spanner fetch timeout)|`integer`|Timeout in milliseconds for a single change stream fetch request to Spanner.
Default: `500`
Minimum: `1`
|| |**spanner\.fetch\.heartbeat\.ms**
(Spanner fetch heartbeat interval)|`integer`|Interval in milliseconds at which Spanner sends heartbeat records when no data changes are available.
Default: `100`
Minimum: `1`
|| @@ -507,9 +531,9 @@ Advanced configuration properties for the source database connection and CDC beh **Example** ```yaml +snapshot.mode: initial snapshot.max.threads: 1 poll.interval.ms: 500 -snapshot.fetch.size: 10000 max.batch.size: 2048 max.queue.size: 8192 heartbeat.interval.ms: 0 @@ -519,6 +543,16 @@ lob.enabled: false publication.autocreate.mode: all_tables publication.name: dbz_publication slot.name: debezium +capture.mode: change_streams_update_full +capture.mode.full.update.type: lookup +cursor.oversize.handling.mode: fail +cursor.oversize.skip.threshold: 0 +mongodb.authsource: admin +mongodb.ssl.invalid.hostname.allowed: false +mongodb.connect.timeout.ms: 10000 +mongodb.socket.timeout.ms: 0 +mongodb.server.selection.timeout.ms: 30000 +mongodb.heartbeat.frequency.ms: 10000 spanner.version.retention.period.hours: 1 spanner.fetch.timeout.ms: 500 spanner.fetch.heartbeat.ms: 100 @@ -737,7 +771,7 @@ Settings that control how data is processed, including batch sizes, error handli |Name|Type|Description|Required| |----|----|-----------|--------| -|**type**
(Processor type)|`string`|Processor implementation to run. `classic` runs the classic processor; `flink` runs the Apache Flink-based processor.
Default: `"classic"`
Enum: `"classic"`, `"flink"`
|| +|**type**
(Processor type)|`string`|Processor implementation to run. Use `flink` (default) for the Apache Flink-based processor. Use `classic` for the classic processor.
Default: `"flink"`
Enum: `"classic"`, `"flink"`
|| |**read\_batch\_size**|`integer`, `string`|Maximum number of records read from the source streams in a single batch.
Default: `2000`
Pattern: `^\${.*}$`
Minimum: `1`
|| |**read\_batch\_timeout\_ms**
(Read batch timeout)|`integer`|Maximum time in milliseconds to wait for a batch to fill before processing it.
Default: `100`
Minimum: `1`
|| |**duration**
(Batch duration limit)|`integer`, `string`|(DEPRECATED)
This property has no effect; use `read_batch_timeout_ms` instead.
Default: `100`
Pattern: `^\${.*}$`
Minimum: `1`
|| From eb6ff3c19f32c792bb71a52ff5f263da72d66c8c Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Wed, 16 Sep 2026 23:47:30 +0300 Subject: [PATCH 28/29] RDSC-6038: Add OpenShift regression fix and security release notes --- .../redis-data-integration/release-notes/rdi-2-0-0.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/release-notes/rdi-2-0-0.md b/content/integrate/redis-data-integration/release-notes/rdi-2-0-0.md index e49eff03c1..9fd7715c30 100644 --- a/content/integrate/redis-data-integration/release-notes/rdi-2-0-0.md +++ b/content/integrate/redis-data-integration/release-notes/rdi-2-0-0.md @@ -48,6 +48,7 @@ weight: 967 ### Bug Fixes +- **Collector API startup on OpenShift**: Fixed a regression introduced in RDI 1.19.1 that caused pod creation to be rejected by standard OpenShift security policies when `global.openShift` is enabled, while preserving source certificate permissions. - **RDI database client certificates isolated from source certificates**: The Debezium collector now presents its RDI database client certificate from a dedicated keystore instead of the shared default keystore, so the source database and RDI database client identities can no longer interfere with each other under mTLS. - **Nested processor properties displayed correctly in `redis-di describe`**: Nested objects and arrays in a processor's advanced properties are now shown as JSON instead of a raw internal format, so the `redis-di describe` output is valid and easy to read. - **Bulk inserts after a snapshot no longer restart the Flink processor**: Fixed an issue where a bulk insert arriving just after a snapshot could lead to a Flink job restart with `Have records for a split that was not registered` during the transition to CDC. @@ -69,7 +70,7 @@ weight: 967 ### Security - **Hardened API log redaction**: Ensured secrets, credentials, tokens, and connection strings are masked in the request and response bodies the RDI API logs, with login and pipeline secret payloads fully redacted. -- **Flink processor and collector security updates**: Resolved `CVE-2024-57699`, `CVE-2025-12183`, `CVE-2025-27820`, `CVE-2025-55163`, `CVE-2025-66566`, `CVE-2025-68973`, `CVE-2026-35194`, `CVE-2026-42198`, `CVE-2026-42583`, `CVE-2026-44249`, `CVE-2026-45416`, `CVE-2026-45447`, `CVE-2026-50010`, `CVE-2026-54291`, `CVE-2026-54512`, `CVE-2026-54513`, `CVE-2026-59901`, and `GHSA-r7wm-3cxj-wff9`. +- **Flink processor and collector security updates**: Resolved `CVE-2024-57699`, `CVE-2025-12183`, `CVE-2025-27820`, `CVE-2025-55163`, `CVE-2025-66566`, `CVE-2025-68973`, `CVE-2025-69720`, `CVE-2026-5260`, `CVE-2026-8376`, `CVE-2026-8925`, `CVE-2026-11822`, `CVE-2026-11824`, `CVE-2026-15146`, `CVE-2026-33845`, `CVE-2026-35194`, `CVE-2026-42010`, `CVE-2026-42013`, `CVE-2026-42198`, `CVE-2026-42496`, `CVE-2026-42583`, `CVE-2026-44249`, `CVE-2026-45416`, `CVE-2026-45447`, `CVE-2026-50010`, `CVE-2026-50813`, `CVE-2026-54291`, `CVE-2026-54399`, `CVE-2026-54428`, `CVE-2026-54512`, `CVE-2026-54513`, `CVE-2026-58472`, `CVE-2026-59901`, and `GHSA-r7wm-3cxj-wff9`. - **Monitor security updates**: Resolved `CVE-2024-6345` and `CVE-2025-47273`. - **Collector API security updates**: Resolved `CVE-2025-66566`, `CVE-2026-10050`, `CVE-2026-40973`, `CVE-2026-40983`, `CVE-2026-40984`, `CVE-2026-42198`, `CVE-2026-42579`, `CVE-2026-42583`, `CVE-2026-44249`, `CVE-2026-45416`, `CVE-2026-45674`, `CVE-2026-47691`, `CVE-2026-50010`, `CVE-2026-54291`, `CVE-2026-54512`, `CVE-2026-54513`, `CVE-2026-59901`, and `GHSA-r7wm-3cxj-wff9`. - **Shared utilities security updates**: Resolved `CVE-2026-21441` and `CVE-2026-33154`. From dcc84dbfbc2e22a3335d92bd0808abe438dd80f2 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Thu, 17 Sep 2026 02:36:04 +0300 Subject: [PATCH 29/29] RDSC-6038: Set the published API version to 2.0.0 --- .../redis-data-integration/reference/api-reference/openapi.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/reference/api-reference/openapi.json b/content/integrate/redis-data-integration/reference/api-reference/openapi.json index d7bff387c8..a8be3ddddd 100644 --- a/content/integrate/redis-data-integration/reference/api-reference/openapi.json +++ b/content/integrate/redis-data-integration/reference/api-reference/openapi.json @@ -3,7 +3,7 @@ "info": { "title": "Redis Data Integration API", "description": "> **NOTE:** RDI API v1 is deprecated as of RDI 1.19.0. Use RDI API v2 for new integrations and migrate existing clients. API v1 will not be extended with new RDI features and may be removed in a future RDI version. See the [RDI API migration guide](https://redis.io/docs/latest/integrate/redis-data-integration/reference/api-migration/).\n\nAPI for Redis Data Integration services", - "version": "1.19.0" + "version": "2.0.0" }, "paths": { "/": {