diff --git a/content/integrate/redis-data-integration/1.19.1/_index.md b/content/integrate/redis-data-integration/1.19.1/_index.md
new file mode 100644
index 0000000000..2c544c5ff8
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/_index.md
@@ -0,0 +1,107 @@
+---
+Title: Redis Data Integration
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+hideListLinks: false
+linkTitle: 1.19.1
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 1
+bannerText: This documentation applies to version 1.19.1.
+bannerChildren: true
+url: '/integrate/redis-data-integration/1.19.1/'
+---
+
+Redis Data Integration (RDI) keeps your Redis cache in sync with a primary system-of-record database in near real time.
+
+RDI's purpose is to help Redis customers sync Redis Enterprise with live data from their slow disk based databases in order to:
+
+- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
+- Save resources and time when building pipelines and coding data transformations.
+- Reduce the total cost of ownership by saving money on expensive database read replicas.
+
+If you use a relational database as the system of record for your app,
+you may eventually find
+that its performance doesn't scale well as your userbase grows. It may be
+acceptable for a few thousand users but for a few million, it can become a
+major problem. If you don't have the option of abandoning the relational
+database, you should consider using a fast
+database, such as Redis, to cache data from read queries. Since read queries
+are typically many times more common than writes, the cache will greatly
+improve performance and let your app scale without a major redesign.
+
+RDI keeps a Redis cache up to date with changes in the primary database, using a
+[*Change Data Capture (CDC)*](https://en.wikipedia.org/wiki/Change_data_capture) mechanism.
+It also lets you *transform* the data from relational tables into convenient
+and fast data structures that match your app's requirements. You specify the
+transformations using a configuration system, so no coding is necessary. RDI supports both standard Redis databases and [Active-Active (CRDB)](https://redis.io/active-active/) replication targets.
+
+## RDI in Redis Cloud
+
+RDI is also available as a fully managed service on Redis Cloud, removing the need to install or maintain the underlying infrastructure. Redis manages the compute, scaling, and upgrades for you. You define the source connection and pipeline configuration using the Redis Cloud console.
+
+The Cloud service currently supports AWS-hosted source databases (Amazon RDS, Amazon Aurora, and Amazon EC2), as well as MongoDB Atlas and Snowflake, writing to a Redis Cloud Pro target database.
+
+See [Data Integration]({{< relref "/operate/rc/rdi" >}}) in the Redis Cloud documentation for
+full setup instructions, prerequisites, and a quick start guide.
+
+## Features
+
+RDI provides enterprise-grade streaming data pipelines with the following features:
+
+- **Near realtime pipeline** - The CDC system captures changes in very short time intervals,
+ then ships and processes them in *micro-batches* to provide near real time updates to Redis.
+- **At least once guarantee** - RDI will deliver any change to the selected data set at least
+ once to the target Redis database.
+- **Data integrity** - RDI keeps the data change order per source table or unique key.
+- **High availability** - All stateless components have hot failover or quick automatic recovery.
+ RDI state is always highly available using Redis Enterprise replication.
+- **Easy to install and operate** - Use a self-documenting command line interface (CLI)
+ for all installation and day-two operations.
+- **No coding needed** - Create and test your pipelines using Redis Insight.
+- **Data-in-transit encryption** - RDI never persists data to disk. All data in-flight is
+ protected using TLS or mTLS connections.
+- **Observability - Metrics** - RDI collects data processing counters at source table granularity
+ along with data processing performance metrics. These are available via GUI, CLI and
+ [Prometheus](https://prometheus.io/) endpoints.
+- **Observability - logs** - RDI saves rotating logs to a single folder. They are in a JSON format,
+ so you can collect and process them with your favorite observability tool.
+- **Backpressure mechanism** - RDI is designed to backoff writing data when the cache gets
+ disconnected, which prevents cascading failure. Since the change data is persisted in the source
+ database and Redis is very fast, RDI can easily catch up with missed changes after a short period of
+ disconnection. See [Backpressure mechanism]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#backpressure-mechanism">}}) for more information.
+- **Recovering from full failure** - If the cache fails or gets disconnected for a long time,
+ RDI can reconstruct the cache data in Redis using a full snapshot of the defined dataset.
+- **High throughput** - Because RDI uses Redis for staging and writes to Redis as a target,
+ it has very high throughput. With a single processor core and records of about 1KB in size,
+ RDI processes around 10,000 records per second. While taking the initial full *snapshot* of
+ the source database, RDI automatically scales to a configurable number of processing units,
+ to fill the cache as fast as possible.
+
+## When to use RDI
+
+RDI is highly configurable but it is not intended to be a general
+solution for all data integration tasks. See
+[When to use RDI]({{< relref "/integrate/redis-data-integration/1.19.1/when-to-use" >}})
+to find out if your use case is a good fit for RDI's features.
+
+## Supported source databases
+
+RDI can capture data from any of the following sources:
+
+{{< embed-md "rdi-supported-source-versions.md" >}}
+
+## Continue learning with Redis University
+
+* [Redis Data Integration Lab](https://university.redis.io/course/2qa1u1ss21vsy5?tab=details)
+
+## Documentation
+
+Learn more about RDI from the other pages in this section:
\ No newline at end of file
diff --git a/content/integrate/redis-data-integration/1.19.1/architecture/_index.md b/content/integrate/redis-data-integration/1.19.1/architecture/_index.md
new file mode 100644
index 0000000000..d7dc22a54b
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/architecture/_index.md
@@ -0,0 +1,207 @@
+---
+Title: Architecture
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Discover the main components of RDI
+group: di
+headerRange: '[2]'
+hideListLinks: false
+linkTitle: Architecture
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 30
+url: '/integrate/redis-data-integration/1.19.1/architecture/'
+---
+
+## Overview
+
+RDI implements a [change data capture](https://en.wikipedia.org/wiki/Change_data_capture) (CDC) pattern that tracks changes to the data in a
+non-Redis *source* database and makes corresponding changes to a Redis
+*target* database. You can use the target as a cache to improve performance
+because it will typically handle read queries much faster than the source.
+
+To use RDI, you define a *dataset* that specifies which data items
+you want to capture from the source and how you want to
+represent them in the target. For example, if the source is a
+relational database then you specify which table columns you want
+to capture but you don't need to store them in an equivalent table
+structure in the target. This means you can choose whatever target
+representation is most suitable for your app. To convert from the
+source to the target representation, RDI applies *transformations*
+to the data after capture.
+
+RDI synchronizes the dataset between the source and target using
+a *data pipeline* that implements several processing steps
+in sequence:
+
+1. A *CDC collector* captures changes to the source database. RDI
+ currently uses an open source collector called
+ [Debezium](https://debezium.io/) for this step.
+
+1. The collector records the captured changes using
+[Redis streams]({{< relref "/develop/data-types/streams" >}})
+ in the RDI database.
+
+1. A *stream processor* reads data from the streams and applies
+ any transformations that you have defined (if you don't need
+ any custom transformations then it uses defaults).
+ It then writes the data to the target database for your app to use.
+
+Note that the RDI control processes run on dedicated virtual machines (VMs)
+outside the Redis
+Enterprise cluster where the target database is kept. However, RDI keeps
+its state and configuration data and also the change data streams in a Redis database on the same cluster as the target. The following diagram shows the pipeline steps and the path the data takes on its way from the source to the target:
+
+{{< image filename="images/rdi/ingest/ingest-dataflow.webp" >}}
+
+When you first start RDI, the target database is empty and so all
+of the data in the source database is essentially "change" data.
+RDI collects this data in a phase called *initial cache loading*,
+which can take minutes or hours to finish, depending on the size
+of the source data. Once the initial cache loading is complete,
+there is a *snapshot* dataset in the target that will gradually
+change when new data gets captured from the source. At this point,
+RDI automatically enters a second phase called *change streaming*, where
+changes in the data are captured as they happen. Changes are usually
+added to the target within a few seconds after capture.
+
+## At-least-once delivery guarantee
+
+RDI guarantees *at-least-once delivery* to the target. This means that
+a given change will never be lost, but it might be added to the target
+more than once. Apart from a slight performance overhead, adding a
+change multiple times is harmless because the multiple writes
+are [*idempotent*](https://en.wikipedia.org/wiki/Idempotence) (that is
+to say that all writes after the first one make no change to the
+overall state).
+
+## Checkpointing
+
+RDI uses Redis streams to store the sequence of change events
+captured from the source. The events are then retrieved in order
+from the streams, processed, and written to the target. The stream
+processor uses a *checkpoint* mechanism to keep track of the last
+event in the sequence that it has successfully processed and stored. If the processor fails
+for any reason, it can restart from the last checkpoint and
+re-process any events that might not have been written to the target.
+This ensures that all changes are eventually recorded, even in the
+face of failures.
+
+## Backpressure mechanism
+
+Sometimes, data records can get added to the streams faster than RDI can
+process them. This can happen if the target is slowed or disconnected
+or simply if the source quickly generates a lot of change data.
+If this continues, then the streams will eventually occupy all the
+available memory. When RDI detects this situation, it applies a
+*backpressure* mechanism to slow or stop the flow of incoming data.
+Change data is held at the source until RDI clears the backlog and has
+enough free memory to resume streaming.
+
+{{}}The Debezium log sometimes reports that RDI has run out
+of memory (usually while creating the initial snapshot). This is not
+an error, just an informative message to note that RDI has applied
+the backpressure mechanism.
+{{ }}
+
+## Supported sources
+
+RDI supports the following database sources using [Debezium Server](https://debezium.io/documentation/reference/stable/operations/debezium-server.html) connectors:
+
+{{< embed-md "rdi-supported-source-versions.md" >}}
+
+## How RDI is deployed
+
+RDI is designed with three *planes* that provide its services.
+
+The *control plane* contains the processes that keep RDI active.
+It includes:
+
+- An *API server* process that exposes a REST API to observe and control RDI.
+- An *operator* process that manages the *data plane* processes.
+- A *metrics exporter* process that reads metrics from the RDI database
+ and exports them as [Prometheus](https://prometheus.io/) metrics.
+
+The *data plane* contains the processes that actually move the data.
+It includes the *CDC collector* and the *stream processor* that implement
+the two phases of the pipeline lifecycle (initial cache loading and change streaming).
+
+The *management plane* provides tools that let you interact
+with the control plane.
+
+- Use the CLI tool to install and administer RDI and to deploy
+ and manage a pipeline.
+- Use the pipeline editor included in Redis Insight to design
+ or edit a pipeline.
+
+The diagram below shows all RDI components and the interactions between them:
+
+{{< image filename="images/rdi/ingest/ingest-control-plane.webp" >}}
+
+## Stream processor implementations
+
+RDI provides two implementations of the stream processor, *classic* and
+*Flink*. You select the implementation per pipeline through the
+[`processors.type`]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config#processors" >}})
+property in `config.yaml`. The default is `classic`, so existing pipelines
+keep their behavior unchanged.
+
+See
+[Differences between the classic and Flink processors]({{< relref "/integrate/redis-data-integration/1.19.1/architecture/classic-vs-flink" >}})
+for a side-by-side comparison and
+[Migrate from the classic processor to the Flink processor]({{< relref "/integrate/redis-data-integration/1.19.1/installation/migration-classic-to-flink" >}})
+for guidance on migrating an existing pipeline to the Flink processor.
+
+## VM and Kubernetes deployments
+
+The following sections describe the VM configurations you can use to
+deploy RDI.
+
+### RDI on your own VMs
+
+For this deployment, you must provide two VMs. The collector and stream processor
+are active on one VM, while on the other they are in standby to provide high availability.
+The two operators running on both VMs use a leader election algorithm to decide which
+VM is the active one (the "leader").
+The diagram below shows this configuration:
+
+{{< image filename="images/rdi/ingest/ingest-active-passive-vms.webp" >}}
+
+See [Install on VMs]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-vm" >}})
+for more information.
+
+### RDI on Kubernetes
+
+You can use the RDI [Helm chart](https://helm.sh/docs/topics/charts/) to install
+on [Kubernetes (K8s)](https://kubernetes.io/), including Red Hat
+[OpenShift](https://docs.openshift.com/). This creates:
+
+- A K8s [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) named `rdi`.
+ You can also use a different namespace name if you prefer.
+- [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) and
+ [services](https://kubernetes.io/docs/concepts/services-networking/service/) for the
+ [RDI operator]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#how-rdi-is-deployed" >}}),
+ [metrics exporter]({{< relref "/integrate/redis-data-integration/1.19.1/observability" >}}), and API server.
+- A [service account](https://kubernetes.io/docs/concepts/security/service-accounts/)
+ and [RBAC resources](https://kubernetes.io/docs/reference/access-authn-authz/rbac) for the RDI operator.
+- A [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) with RDI database details.
+- [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
+ with the RDI database credentials and TLS certificates.
+- Other optional K8s resources such as [ingresses](https://kubernetes.io/docs/concepts/services-networking/ingress/)
+ that can be enabled depending on your K8s environment and needs.
+
+See [Install on Kubernetes]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-k8s" >}})
+for more information.
+
+### Secrets and security considerations
+
+The credentials for the database connections, as well as the certificates
+for [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security) and
+[mTLS](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS) are saved in K8s secrets.
+RDI stores all state and configuration data inside the Redis Enterprise cluster
+and does not store any other data on your RDI VMs or anywhere else outside the cluster.
diff --git a/content/integrate/redis-data-integration/1.19.1/architecture/classic-vs-flink.md b/content/integrate/redis-data-integration/1.19.1/architecture/classic-vs-flink.md
new file mode 100644
index 0000000000..330d8af13e
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/architecture/classic-vs-flink.md
@@ -0,0 +1,145 @@
+---
+Title: Differences between the classic and Flink processors
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Compare the classic and Flink stream processor implementations.
+group: di
+linkTitle: Classic vs. Flink processor
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 10
+url: '/integrate/redis-data-integration/1.19.1/architecture/classic-vs-flink/'
+---
+
+RDI ships with two stream processor implementations. Both consume the same
+source streams, share the same job-level configuration model, and write to
+the same Redis target, but they differ in architecture, supported features,
+configuration, observability, error handling, and performance.
+
+This page summarizes those differences. See
+[Which processor should I use?]({{< relref "/integrate/redis-data-integration/1.19.1/faq#which-processor-should-i-use" >}})
+in the FAQ for the recommendation, and
+[Migrate from the classic processor to the Flink processor]({{< relref "/integrate/redis-data-integration/1.19.1/installation/migration-classic-to-flink" >}})
+for a step-by-step migration guide.
+
+## At a glance
+
+| Aspect | Classic processor | Flink processor |
+|---|---|---|
+| Implementation | Python | Java on top of [Apache Flink](https://flink.apache.org/) |
+| Scaling | Single replica | Horizontal: TaskManager replicas × task slots per TaskManager |
+| Fault tolerance | Source-stream consumer-group replay | Source-stream consumer-group replay plus Flink checkpointing |
+| Metrics endpoint | `rdi-metrics-exporter` service | Flink JobManager `/metrics` (no metrics exporter) |
+| Metric naming | `rdi_*` (e.g., `rdi_incoming_entries`) | `flink_*` (e.g., `flink_jobmanager_job_operator_coordinator_stream_type_rdiRecords`) |
+| End-to-end latency | Bounded by the per-batch read-process-write cycle | Records flow through pipelined operator chains without a per-batch barrier |
+| Snapshot throughput | Limited by single shared reader and writer | Parallelized across all task slots |
+| Expression and `redis.lookup` result caching | Not supported | Optional, opt-in per transformation |
+
+## Architecture and deployment
+
+The classic processor runs as a single pod managed by the operator
+and can be deployed on either VMs or Kubernetes through the RDI Helm
+chart.
+
+The Flink processor runs as an Apache Flink application cluster managed by
+RDI: one JobManager pod plus one or more TaskManager pods. Source,
+transformation, and sink operators run as parallel subtasks across
+all task slots in the cluster. The Flink processor scales
+horizontally by changing the number of TaskManager replicas
+(`advanced.resources.taskManager.replicas`); with adaptive
+parallelism, the default parallelism is the product of TaskManager
+replicas and task slots per TaskManager.
+
+Both processors retain at-least-once delivery semantics; the Flink
+processor adds Flink checkpointing on top of the shared
+consumer-group replay mechanism.
+
+See
+[Configure the Flink processor]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-k8s#configure-the-flink-processor" >}})
+for the Kubernetes Helm settings, and
+[Configure the Flink processor]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-vm#configure-the-flink-processor" >}})
+for VM installations.
+
+## Configuration
+
+The two processors share the same `config.yaml` envelope and the same
+`connections`, `sources`, `targets`, and `jobs` sections. The only
+differences are inside the `processors:` block, which is selected via
+`processors.type` (`classic` or `flink`, default `classic`). Properties
+that apply to only one implementation are annotated with
+**Classic processor only.** or **Flink processor only.** in the
+[pipeline configuration reference]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config#processors" >}}),
+and are silently ignored by the other implementation. The Flink
+processor exposes additional fine-grained tuning under
+`processors.advanced.*`.
+
+## Transformation extensions
+
+The two processors support the same set of transformation blocks
+(`filter`, `map`, `add_field`, `remove_field`, `rename_field`,
+`redis.lookup`), the same expression languages (JMESPath and SQL),
+and the same data types in output blocks: `hash`, `json`, `set`,
+`sorted_set`, `stream`, and `string`. Pipelines written for one processor
+generally execute on the other without changes.
+
+The Flink processor adds three optional, performance-oriented
+extensions that are not available with the classic processor:
+
+- **Expression result caching** through a per-expression `cache:`
+ block on `filter`, `map`, `add_field`, and `redis.lookup` arguments.
+- **`redis.lookup` result caching** through a `lookup_cache:` block.
+- **`redis.lookup` batching**, which groups lookups into a single
+ Redis pipeline. Batching is enabled by default with sensible
+ defaults; the optional `batch:` block lets you override them.
+
+See
+[Caching expression results]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/caching-expression-results" >}})
+for examples and
+[`redis.lookup`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/lookup" >}})
+for the full property list.
+
+## Metrics
+
+The two processors expose different Prometheus metric sets and use
+different naming schemes, so dashboards and alerts cannot be reused
+as-is between them. The classic processor exposes its metrics through
+the `rdi-metrics-exporter` service. The Flink processor emits metrics
+directly from the JobManager and TaskManager pods through Flink's
+native Prometheus reporter; no metrics exporter is deployed.
+
+See
+[Observability — Flink processor metrics]({{< relref "/integrate/redis-data-integration/1.19.1/observability#flink-processor-metrics" >}})
+for the customer-facing list of metrics.
+
+## Error handling and DLQ
+
+Both processors implement a dead-letter queue (DLQ) at
+`dlq:{stream_name}` and honor the same top-level `error_handling`
+(`dlq` or `ignore`) and `dlq_max_messages` properties. The Flink
+processor surfaces a few corner cases as DLQ entries that the classic
+processor logs and skips (for example, missing parent
+keys in nested writes and exceptions thrown by `when` expressions on
+`redis.lookup`). The DLQ entry field set and value encoding also
+differ: the classic processor uses Python-stringified values,
+while the Flink processor uses JSON.
+
+## Performance
+
+The Flink processor delivers significantly higher throughput during
+the initial snapshot and lower end-to-end latency in steady state.
+The classic processor uses a sequential read-process-write batching
+cycle, so each record waits for its batch to complete before being
+written to the target. The Flink processor pipelines records through
+operator chains without a per-batch barrier, and parallelizes work
+across all task slots, which both lowers per-record latency and
+raises throughput.
+
+The Flink processor has a larger baseline memory footprint (JVM plus
+Flink runtime overhead per TaskManager) but, for most pipelines, the
+performance gains and the additional features (horizontal scaling, caching)
+outweigh that cost.
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/_index.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/_index.md
new file mode 100644
index 0000000000..67dcc08380
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/_index.md
@@ -0,0 +1,159 @@
+---
+Title: Data pipelines
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Learn how to configure RDI for data capture and transformation.
+group: di
+hideListLinks: false
+linkTitle: Data pipelines
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 40
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/'
+---
+
+RDI uses *pipelines* to implement
+[change data capture](https://en.wikipedia.org/wiki/Change_data_capture) (CDC). (See the
+[architecture overview]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#overview" >}})
+for an introduction to pipelines.)
+The sections below explain how pipelines work and give an overview of how to configure and
+deploy them.
+
+## How a pipeline works
+
+An RDI pipeline captures change data records from the source database, and transforms them
+into Redis data structures. It writes each of these new structures to a Redis target
+database under its own key.
+
+By default, RDI transforms the source data into
+[hashes]({{< relref "/develop/data-types/hashes" >}}) or
+[JSON objects]({{< relref "/develop/data-types/json" >}}) for the target with a
+standard data mapping and a standard format for the key.
+However, you can also provide your own custom transformation [jobs](#job-files)
+for each source table, using your own data mapping and key pattern. You specify these
+jobs declaratively with YAML configuration files that require no coding.
+
+Data transformation involves two stages:
+
+1. The data ingested during CDC is automatically transformed to an intermediate JSON
+ change event format.
+1. RDI passes this JSON change event data to your custom transformation for further
+ processing.
+
+The diagram below shows the flow of data through the pipeline:
+
+{{< image filename="/images/rdi/ingest/RDIPipeDataflow.webp" >}}
+
+You can provide a job file for each source table that needs a custom
+transformation. You can also add a *default job file* for any tables that don't have their own.
+You must specify the full name of the source table in the job file (or the special
+name "*" in the default job) and you
+can also include filtering logic to skip data that matches a particular condition.
+As part of the transformation, you can specify any of the following data types
+to store the data in Redis:
+
+- [JSON]({{< relref "/develop/data-types/json" >}})
+- [Hashes]({{< relref "/develop/data-types/hashes" >}})
+- [Sets]({{< relref "/develop/data-types/sets" >}})
+- [Streams]({{< relref "/develop/data-types/streams" >}})
+- [Sorted sets]({{< relref "/develop/data-types/sorted-sets" >}})
+- [Strings]({{< relref "/develop/data-types/strings" >}})
+
+### Pipeline lifecycle
+
+After you deploy a pipeline, it goes through the following phases:
+
+1. *Deploy* - when you deploy the pipeline, RDI first validates it before use.
+Then, the [operator]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#how-rdi-is-deployed">}}) creates and configures the collector and stream processor that will run the pipeline.
+1. *Snapshot* - The collector starts the pipeline by creating a snapshot of the full
+dataset. This involves reading all the relevant source data, transforming it and then
+writing it into the Redis target. This phase typically takes minutes to
+hours if you have a lot of data.
+1. *CDC* - Once the snapshot is complete, the collector starts listening for updates to
+the source data. Whenever a change is committed to the source, the collector captures
+it and adds it to the target through the pipeline. This phase continues indefinitely
+unless you change the pipeline configuration.
+1. *Update* - If you update the pipeline configuration, the operator applies it
+to the collector and the stream processor. Note that the changes only affect newly-captured
+data unless you reset the pipeline completely. Once RDI has accepted the updates, the
+pipeline returns to the CDC phase with the new configuration.
+1. *Reset* - There are circumstances where you might want to rebuild the dataset
+completely. For example, you might want to apply a new transformation to all the source
+data or refresh the dataset if RDI is disconnected from the
+source for a long time. In situations like these, you can *reset* the pipeline back
+to the snapshot phase. When this is complete, the pipeline continues with CDC as usual.
+
+## Using a pipeline
+
+Follow the steps described in the sections below to prepare and run an RDI pipeline.
+
+### 1. Prepare the source database
+
+Before using the pipeline you must first prepare your source database to use
+the Debezium connector for *change data capture (CDC)*. See the
+[architecture overview]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#overview" >}})
+for more information about CDC.
+Each database type has a different set of preparation steps. You can
+find the preparation guides for the databases that RDI supports in the
+[Prepare source databases]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs" >}})
+section.
+
+### 2. Configure the pipeline
+
+RDI uses a set of [YAML](https://en.wikipedia.org/wiki/YAML)
+files to configure each pipeline. The folder structure of the
+configuration is shown below:
+
+```hierarchy {type="filesystem"}
+"(root)":
+ "config.yaml":
+ _meta:
+ description: "\"config.yaml\" is the main pipeline configuration file."
+ "jobs":
+ _meta:
+ description: "The 'jobs' folder containing optional job files."
+ "default-job.yaml":
+ _meta:
+ description: "A default job."
+ "job1.yaml":
+ _meta:
+ description: "Each job file must have a unique name."
+ "...":
+ _meta:
+ ellipsis: true
+ description: "Other job files, if required."
+```
+
+The main configuration for the pipeline is in the `config.yaml` file.
+This specifies the connection details for the source database (such
+as host, username, and password) and also the queries that RDI will use
+to extract the required data. You should place job files in the `Jobs`
+folder if you want to specify your own data transformations.
+
+See
+[Pipeline configuration file]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config" >}})
+for a full description of the `config.yaml` file and some example configurations.
+
+### 3. Create job files (optional)
+
+You can use one or more job files to configure which fields from the source tables
+you want to use, and which data structure you want to write to the target. You
+can also optionally specify a transformation to apply to the data before writing it
+to the target. See the
+[Job files]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples" >}})
+section for full details of the file format and examples of common tasks for job files.
+
+### 4. Deploy the pipeline
+
+When your configuration is ready, you must deploy it to start using the pipeline. See
+[Deploy a pipeline]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/deploy" >}})
+to learn how to do this.
+
+## More information
+
+See the other pages in this section for more information and examples:
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization.md
new file mode 100644
index 0000000000..fb8071d5fe
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization.md
@@ -0,0 +1,229 @@
+---
+Title: Data denormalization
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Learn about denormalization strategies
+group: di
+linkTitle: Data denormalization
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 30
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization/'
+---
+
+The data in the source database is often
+[*normalized*](https://en.wikipedia.org/wiki/Database_normalization).
+This means that columns can't have composite values (such as arrays) and relationships between entities
+are expressed as mappings of primary keys to foreign keys between different tables.
+Normalized data models reduce redundancy and improve data integrity for write queries but this comes
+at the expense of speed.
+A Redis cache, on the other hand, is focused on making *read* queries fast, so RDI provides data
+*denormalization* to help with this.
+
+The supported denormalization techniques are joining
+[one-to-one relationships](#joining-one-to-one-relationships) (using `merge`) and
+joining [one-to-many relationships](#joining-one-to-many-relationships) (using nesting),
+both described below.
+
+{{< note >}}
+The [`redis.lookup`]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-lookup-example" >}})
+transformation is *not* a supported way to denormalize data that RDI ingests. RDI
+can't guarantee that a key written by one job is present or up to date when another
+job looks it up, so lookups against pipeline-populated data can miss or return stale
+values. Use the techniques on this page instead. See
+[Reading Redis data with redis.lookup]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-lookup-example" >}})
+for details.
+{{< /note >}}
+
+## Joining one-to-one relationships
+
+You can join one-to-one relationships by making more than one job to write to the same Redis key.
+
+First, you must configure the parent entity to use `merge` as the `on_update` strategy.
+
+```yaml
+# jobs/customers.yaml
+source:
+ table: customers
+
+output:
+ - uses: redis.write
+ with:
+ data_type: json
+ on_update: merge
+```
+
+Then, you can configure the child entity to write to the same Redis key as the parent entity. You can do this by using the `key` attribute in the `with` block of the job, as shown in this example:
+
+```yaml
+# jobs/addresses.yaml
+source:
+ table: addresses
+
+transform:
+ - uses: add_field
+ with:
+ field: customer_address
+ language: jmespath
+ # You can use the following JMESPath expression to create a JSON object and combine the address fields into a single object.
+ expression: |
+ {
+ "street": street,
+ "city": city,
+ "state": state,
+ "zip": zip
+ }
+
+output:
+ - uses: redis.write
+ with:
+ data_type: json
+ # We specify the key to write to the same key as the parent entity.
+ key:
+ expression: concat(['customers:id:', customer_id])
+ language: jmespath
+ on_update: merge
+ mapping:
+ # You can specify one or more fields to write to the parent entity.
+ - customer_address: customer_address
+```
+
+The joined data will look like this in Redis:
+
+```json
+{
+ "id": "1",
+ "first_name": "John",
+ "last_name": "Doe",
+ "customer_address": {
+ "street": "123 Main St",
+ "city": "Anytown",
+ "state": "CA",
+ "zip": "12345"
+ }
+}
+```
+
+{{< note >}}
+If you don't set `merge` as the `on_update` strategy for all jobs targeting the same key, the entire parent record in Redis will be overwritten whenever any related record in the source database is updated. This will result in the loss of values written by other jobs.
+{{< /note >}}
+
+When using this approach, you must ensure that the `key` expression in the child job matches the key expression in the parent job. If you use a different key expression, the child data will not be written to the same Redis key as the parent data.
+
+In the example above, the `addresses` job uses the default key pattern to write to the same Redis key as the `customers` job. You can find more information about the default key pattern [here]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-set-key-name" >}}).
+
+You can also use custom keys for the parent entity, as long as you use the same key for all jobs that write to the same Redis key.
+
+{{< note >}}
+If you are using the same key for different jobs, deleting any of the entities will result in the key being removed from the target.
+For an example workaround, see [Write to the same key from multiple jobs]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-write-same-key" >}}).
+{{< /note >}}
+
+
+## Joining one-to-many relationships
+
+To join one-to-many relationships, you can use the *Nesting* strategy.
+With this, the parent object (the "one") is represented as a JSON document with the children (the "many") nested inside it as a JSON map attribute. The diagram below shows a nesting with the child objects in a map called `InvoiceLineItems`:
+
+{{< image filename="/images/rdi/ingest/nest-flow.webp" width="500px" >}}
+
+
+To configure normalization, you must first configure the parent entity to use JSON as the target data type. Add `data_type: json` to the parent job as shown in the example below:
+
+```yaml
+# jobs/invoice.yaml
+source:
+ schema: public
+ table: Invoice
+
+output:
+ - uses: redis.write
+ with:
+ # Setting the data type to json ensures that the parent object will be created in a way that supports nesting.
+ data_type: json
+ # Important: do not set a custom key for the parent entity.
+ # When nesting the child object under the parent, the parent key is automatically calculated based on
+ # the parent table name and the parent key field and if a custom key is set, it will cause a mismatch
+ # between the key used to write the parent and the key used to write the child.
+
+```
+
+After you have configured the parent entity, you can then configure the child entities to be nested under it, based on their relation type. To do this, use the `nest` block, as shown in this example:
+
+```yaml
+# jobs/invoice_line.yaml
+source:
+ schema: public
+ table: InvoiceLine
+output:
+ - uses: redis.write
+ with:
+ nest: # cannot co-exist with other parameters such as 'key'
+ parent:
+ # schema: public
+ table: Invoice
+ nesting_key: InvoiceLineId # the unique key in the composite structure under which the child data will be stored
+ parent_key: InvoiceId
+ child_key: InvoiceId # optional, if different from parent_key
+ path: $.InvoiceLineItems # path must start from document root ($)
+ structure: map # only map supported for now
+ on_update: merge # only merge supported for now
+ data_type: json # only json supported for now
+```
+
+The job has a `with` section under `output` that includes the `nest` block.
+The job must include the following attributes in the `nest` block:
+
+- `parent`: This specifies the config of the parent entities. You only
+ need to supply the parent `table` name. Note that this attribute refers to a Redis *key* that will be added to the target
+ database, not to a table you can access from the pipeline. See [Using nesting](#using-nesting) below
+ for the format of the key that is generated.
+- `nesting_key`: The unique key of each child entry in the JSON map that will be created under the path.
+- `parent_key`: The field in the parent entity that stores the unique ID (foreign key) of the parent entity. This can't be a composite key.
+- `child_key`: The field in the child entity that stores the unique ID (foreign key) to the parent entity. You only need to add this attribute if the name of the child's foreign key field is different from the parent's. This can't be a composite key.
+- `path`: The [JSONPath](https://goessner.net/articles/JsonPath/)
+ for the map where you want to store the child entities. The path must start with the `$` character, which denotes
+ the document root.
+- `structure`: (Optional) The type of JSON nesting structure for the child entities. Currently, only a JSON map
+ is supported so if you supply this attribute then the value must be `map`.
+
+### Using nesting
+
+There are several important things to note when you use nesting:
+
+- When you specify `nest` in the job, you must also set the `data_type` attribute to `json` and
+ the `on_update` attribute to `merge` in the surrounding `output` block.
+- Key expressions are *not* supported for the `nest` output blocks. The parent key is always calculated
+ using the following template:
+
+ ```bash
+ ::
+ ```
+
+ For example:
+
+ ```bash
+ Invoice:InvoiceId:1
+ ```
+
+- If you specify `expire` in the `nest` output block then this will set the expiration on the *parent* object.
+- You can only use one level of nesting.
+- If you are using PostgreSQL then you must make the following change for all child tables that you want to nest:
+
+ ```sql
+ ALTER TABLE REPLICA IDENTITY FULL;
+ ```
+
+ This configuration affects the information written to the write-ahead log (WAL) and whether it is available
+ for RDI to capture. By default, PostgreSQL only records
+ modified fields in the log, which means that it might omit the `parent_key`. This can cause incorrect updates to the
+ Redis key in the destination database.
+ See the
+ [Debezium PostgreSQL Connector Documentation](https://debezium.io/documentation/reference/connectors/postgresql.html#postgresql-replica-identity)
+ for more information about this.
+- Prior to RDI v1.12.1, there is a known limitation if you change the foreign key of a child object. In that scenario, the child object will be added to the new parent, but the old parent will not be updated.
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/deploy.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/deploy.md
new file mode 100644
index 0000000000..9744418e20
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/deploy.md
@@ -0,0 +1,384 @@
+---
+Title: Deploy a pipeline
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Learn how to deploy an RDI pipeline
+group: di
+linkTitle: Deploy
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 50
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/deploy/'
+---
+
+The sections below explain how to deploy a pipeline after you have created the required
+[configuration]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines" >}}).
+
+## Set secrets
+
+Before you deploy your pipeline, you must set the authentication secrets for the
+source and target databases. Each secret has a name that you pass to the
+[`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-secret" >}})
+command to set the secret value.
+You can then refer to these secrets in the `config.yaml` file using the syntax "`${SECRET_NAME}`"
+(the sample
+[config.yaml file]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config#example" >}})
+shows these secrets in use).
+
+The table below lists all valid secret names. Note that the
+username and password are required for the source and target, but the other
+secrets are only relevant for TLS/mTLS connections.
+
+| Secret name | Description |
+| :-- | :-- |
+| `SOURCE_DB_USERNAME` | Username for the source database |
+| `SOURCE_DB_PASSWORD` | Password for the source database |
+| `SOURCE_DB_CACERT` | (For TLS only) Source database CA certificate |
+| `SOURCE_DB_CERT` | (For mTLS only) Source database client certificate |
+| `SOURCE_DB_KEY` | (For mTLS only) Source database private key |
+| `SOURCE_DB_KEY_PASSWORD` | (For mTLS only) Source database private key password |
+| `TARGET_DB_USERNAME` | Username for the target database |
+| `TARGET_DB_PASSWORD` | Password for the target database |
+| `TARGET_DB_CACERT` | (For TLS only) Target database CA certificate |
+| `TARGET_DB_CERT` | (For mTLS only) Target database client certificate |
+| `TARGET_DB_KEY` | (For mTLS only) Target database private key |
+| `TARGET_DB_KEY_PASSWORD` | (For mTLS only) Target database private key password |
+
+{{< note >}}
+{{< embed-md "rdi-tls-secrets.md" >}}
+{{< /note >}}
+
+### Set secrets with the CLI
+
+Use [`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-secret" >}})
+to set secrets for any installation type (VM, Kubernetes, or Redis Cloud).
+
+The specific command lines for source secrets are as follows:
+
+```bash
+# For username and password
+redis-di set-secret SOURCE_DB_USERNAME yourUsername
+redis-di set-secret SOURCE_DB_PASSWORD yourPassword
+
+# With source TLS, in addition to the above
+redis-di set-secret SOURCE_DB_CACERT /path/to/myca.crt
+
+# With source mTLS, in addition to the above
+redis-di set-secret SOURCE_DB_CERT /path/to/myclient.crt
+redis-di set-secret SOURCE_DB_KEY /path/to/myclient.key
+# Use this only if SOURCE_DB_KEY is password-protected
+redis-di set-secret SOURCE_DB_KEY_PASSWORD yourKeyPassword
+```
+
+The corresponding command lines for target secrets are:
+
+```bash
+# For username and password
+redis-di set-secret TARGET_DB_USERNAME yourUsername
+redis-di set-secret TARGET_DB_PASSWORD yourPassword
+
+# With target TLS, in addition to the above
+redis-di set-secret TARGET_DB_CACERT /path/to/myca.crt
+
+# With target mTLS, in addition to the above
+redis-di set-secret TARGET_DB_CERT /path/to/myclient.crt
+redis-di set-secret TARGET_DB_KEY /path/to/myclient.key
+# Use this only if TARGET_DB_KEY is password-protected
+redis-di set-secret TARGET_DB_KEY_PASSWORD yourKeyPassword
+```
+
+By default, `set-secret` waits for the pipeline to apply the change before returning. When you set
+several secrets at once, set all but the last one with `--wait=false` to avoid a timeout while the
+pipeline is only partially updated. See [Wait for changes to complete](#wait) below for details.
+
+### Manage secrets with the CLI
+
+Along with `set-secret`, the CLI has commands to list, inspect, and delete secrets. Because the API
+never returns secret values, these commands show only the secret keys and whether they are set, not
+the stored values.
+
+```bash
+# List all the secrets of a pipeline and whether each one is set
+redis-di list-secrets
+
+# Show a single secret and whether it is set
+redis-di describe-secret SOURCE_DB_PASSWORD
+
+# Delete a secret (prompts for confirmation unless you add --force)
+redis-di delete-secret SOURCE_DB_CACERT
+```
+
+See the reference pages for
+[`list-secrets`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-secrets" >}}),
+[`get-secret`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-secret" >}}),
+[`describe-secret`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-secret" >}}),
+and [`delete-secret`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete-secret" >}})
+for the full list of options.
+
+### Set secrets for K8s/Helm deployment using Kubectl command
+
+{{< note >}}It is strongly recommended to manage secrets with the `redis-di` CLI rather than with
+`kubectl` directly. The CLI applies the correct labels automatically, validates the secret keys, and
+works the same way across all installation types.{{< /note >}}
+
+For a Kubernetes/Helm deployment, you can also use [`kubectl create secret generic`](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_create/kubectl_create_secret_generic/)
+to set secrets instead of the CLI. The general pattern of the commands is:
+
+```bash
+kubectl create secret generic \
+--namespace=rdi \
+--from-literal==
+```
+
+Where `` is either `source-db` for source secrets or `target-db` for target secrets.
+
+If you use TLS or mTLS for either the source or target databases, you also need to create the `source-db-ssl` and/or `target-db-ssl` K8s secrets that contain the certificates used to establish secure connections. The general pattern of the commands is:
+
+```bash
+kubectl create secret generic -ssl \
+--namespace=rdi \
+--from-file==
+```
+
+The specific command lines for source secrets are as follows:
+
+```bash
+# Without source TLS
+# Create or update source-db secret
+kubectl create secret generic source-db --namespace=rdi \
+--from-literal=SOURCE_DB_USERNAME=yourUsername \
+--from-literal=SOURCE_DB_PASSWORD=yourPassword \
+--save-config --dry-run=client -o yaml | kubectl apply -f -
+
+# With source TLS
+# Create of update source-db secret
+kubectl create secret generic source-db --namespace=rdi \
+--from-literal=SOURCE_DB_USERNAME=yourUsername \
+--from-literal=SOURCE_DB_PASSWORD=yourPassword \
+--from-literal=SOURCE_DB_CACERT=/etc/certificates/source_db/ca.crt \
+--save-config --dry-run=client -o yaml | kubectl apply -f -
+# Create or update source-db-ssl secret
+kubectl create secret generic source-db-ssl --namespace=rdi \
+--from-file=ca.crt=/path/to/myca.crt \
+--save-config --dry-run=client -o yaml | kubectl apply -f -
+
+# With source mTLS
+# Create or update source-db secret
+kubectl create secret generic source-db --namespace=rdi \
+--from-literal=SOURCE_DB_USERNAME=yourUsername \
+--from-literal=SOURCE_DB_PASSWORD=yourPassword \
+--from-literal=SOURCE_DB_CACERT=/etc/certificates/source_db/ca.crt \
+--from-literal=SOURCE_DB_CERT=/etc/certificates/source_db/client.crt \
+--from-literal=SOURCE_DB_KEY=/etc/certificates/source_db/client.key \
+--from-literal=SOURCE_DB_KEY_PASSWORD=yourKeyPassword \ # add this only if SOURCE_DB_KEY is password-protected
+--save-config --dry-run=client -o yaml | kubectl apply -f -
+# Create or update source-db-ssl secret
+kubectl create secret generic source-db-ssl --namespace=rdi \
+--from-file=ca.crt=/path/to/myca.crt \
+--from-file=client.crt=/path/to/myclient.crt \
+--from-file=client.key=/path/to/myclient.key \
+--save-config --dry-run=client -o yaml | kubectl apply -f -
+```
+
+The corresponding command lines for target secrets are:
+
+```bash
+# Without target TLS
+# Create or update target-db secret
+kubectl create secret generic target-db --namespace=rdi \
+--from-literal=TARGET_DB_USERNAME=yourUsername \
+--from-literal=TARGET_DB_PASSWORD=yourPassword \
+--save-config --dry-run=client -o yaml | kubectl apply -f -
+
+# With target TLS
+# Create of update target-db secret
+kubectl create secret generic target-db --namespace=rdi \
+--from-literal=TARGET_DB_USERNAME=yourUsername \
+--from-literal=TARGET_DB_PASSWORD=yourPassword \
+--from-literal=TARGET_DB_CACERT=/etc/certificates/target_db/ca.crt \
+--save-config --dry-run=client -o yaml | kubectl apply -f -
+# Create or update target-db-ssl secret
+kubectl create secret generic target-db-ssl --namespace=rdi \
+--from-file=ca.crt=/path/to/myca.crt \
+--save-config --dry-run=client -o yaml | kubectl apply -f -
+
+# With target mTLS
+# Create or update target-db secret
+kubectl create secret generic target-db --namespace=rdi \
+--from-literal=TARGET_DB_USERNAME=yourUsername \
+--from-literal=TARGET_DB_PASSWORD=yourPassword \
+--from-literal=TARGET_DB_CACERT=/etc/certificates/target_db/ca.crt \
+--from-literal=TARGET_DB_CERT=/etc/certificates/target_db/client.crt \
+--from-literal=TARGET_DB_KEY=/etc/certificates/target_db/client.key \
+--from-literal=TARGET_DB_KEY_PASSWORD=yourKeyPassword \ # add this only if TARGET_DB_KEY is password-protected
+--save-config --dry-run=client -o yaml | kubectl apply -f -
+# Create or update target-db-ssl secret
+kubectl create secret generic target-db-ssl --namespace=rdi \
+--from-file=ca.crt=/path/to/myca.crt \
+--from-file=client.crt=/path/to/myclient.crt \
+--from-file=client.key=/path/to/myclient.key \
+--save-config --dry-run=client -o yaml | kubectl apply -f -
+```
+
+Note that the certificate paths contained in the secrets `SOURCE_DB_CACERT`, `SOURCE_DB_CERT`, and `SOURCE_DB_KEY` (for the source database) and `TARGET_DB_CACERT`, `TARGET_DB_CERT`, and `TARGET_DB_KEY` (for the target database) are internal to RDI, so you *must* use the values shown in the example above. You should only change the certificate paths when you create the `source-db-ssl` and `target-db-ssl` secrets.
+
+Secrets that you create directly with `kubectl` must also be labeled so that the RDI operator
+discovers them as pipeline secrets. Each secret needs the following labels, where the
+`app.kubernetes.io/instance` label is the pipeline name (`default` for the default pipeline):
+
+| Label | Value |
+| :-- | :-- |
+| `app.kubernetes.io/name` | `pipeline` |
+| `app.kubernetes.io/instance` | `default` |
+| `product` | `rdi` |
+
+Apply the labels to each secret with [`kubectl label`](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_label/):
+
+```bash
+kubectl label secret source-db --namespace=rdi --overwrite \
+ app.kubernetes.io/name=pipeline \
+ app.kubernetes.io/instance=default \
+ product=rdi
+kubectl label secret target-db --namespace=rdi --overwrite \
+ app.kubernetes.io/name=pipeline \
+ app.kubernetes.io/instance=default \
+ product=rdi
+
+# With source TLS or mTLS
+kubectl label secret source-db-ssl --namespace=rdi --overwrite \
+ app.kubernetes.io/name=pipeline \
+ app.kubernetes.io/instance=default \
+ product=rdi
+
+# With target TLS or mTLS
+kubectl label secret target-db-ssl --namespace=rdi --overwrite \
+ app.kubernetes.io/name=pipeline \
+ app.kubernetes.io/instance=default \
+ product=rdi
+```
+
+## Deploy a pipeline
+
+When you have created your configuration, including the [jobs]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples" >}}), you are
+ready to deploy. Use the
+[`redis-di deploy`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-deploy" >}})
+command to deploy a pipeline:
+
+```bash
+redis-di deploy --dir
+```
+
+RDI first validates the configuration and then deploys it if it is correct. You can control the
+validation and what happens after deployment with the following options:
+
+- `--dry-run`: Validate the configuration without deploying it. Off by default.
+- `--validate-tables`: Validate the configuration against the source and target databases, for
+ example that the tables it references exist. On by default; pass `--validate-tables=false` to skip
+ this check, which is useful when the databases are not reachable at deploy time.
+- `--validate-cdc`: Additionally validate that the source database is correctly configured for
+ [change data capture (CDC)]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#overview" >}}).
+ Off by default; enable it with `--validate-cdc`.
+- `--start`: Start the pipeline as soon as it is deployed. On by default; pass `--start=false` to
+ deploy the pipeline without starting it, then start it later with
+ [`redis-di start`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-start" >}}).
+
+See the [`redis-di deploy`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-deploy" >}})
+reference page for the full list of options.
+
+You can also use [Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}})
+to configure and deploy pipelines for both VM and K8s installations.
+
+## Display the pipeline status
+
+Once a pipeline is deployed, use the
+[`redis-di describe`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe" >}})
+command (also available as `redis-di status`) to display its status. This combines the pipeline
+configuration with its runtime status, showing its overall state, its sources and targets, its jobs
+and components, and its per-stream statistics and performance metrics.
+
+```bash
+redis-di describe
+```
+
+To watch the status update live, pair the command with `watch`:
+
+```bash
+watch -n 1 redis-di describe
+```
+
+For a shorter overview, [`redis-di list`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list" >}})
+prints a one-line summary of the pipeline, and
+[`redis-di get`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get" >}})
+does the same for a single pipeline. See the
+[`redis-di describe`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe" >}})
+reference page for details.
+
+## Start and stop a pipeline
+
+Use [`redis-di stop`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-stop" >}})
+to pause a running pipeline and
+[`redis-di start`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-start" >}})
+to resume it. Stopping a pipeline halts data processing without deleting the pipeline or its
+configuration, so you can start it again later from where it left off.
+
+```bash
+redis-di stop
+redis-di start
+```
+
+## Reset a pipeline
+
+Use [`redis-di reset`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset" >}})
+to return a pipeline to initial full-sync mode. This reloads a fresh
+[snapshot]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#overview" >}}) of the source
+data and then resumes change data capture (CDC), which is useful when the source and target have
+drifted out of sync.
+
+```bash
+redis-di reset
+```
+
+## Undeploy a pipeline
+
+To remove a pipeline, use the
+[`redis-di delete`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete" >}})
+command. This stops the pipeline and deletes it, along with its configuration and status, from RDI.
+The secrets you set for the pipeline are not affected.
+
+```bash
+redis-di delete
+```
+
+Because deleting a pipeline is destructive, the command asks for confirmation unless you add the
+`--force` option. If you omit the pipeline name, the `default` pipeline is deleted.
+
+## Wait for changes to complete {#wait}
+
+The commands that change a pipeline's state, namely `deploy`, `delete`, `start`, `stop`, `reset`,
+`set-secret`, and `delete-secret`, do not return as soon as the API accepts the request. By default,
+they wait for the pipeline to finish transitioning to the expected state, polling its status until it
+succeeds, reaches an error, or the `--timeout` (2 minutes by default) elapses. This is usually what
+you want: the command reflects the real outcome, so a script can rely on the change having taken
+effect and can fail fast if it did not.
+
+In some cases, though, a pipeline needs *several* changes before it can transition to a healthy state,
+and waiting after each individual change would time out. The clearest example is rotating both the
+username and the password of a database: if you set only the username with the default `--wait=true`,
+the pipeline tries to reconnect with the new username and the old password, fails, and the command
+times out after two minutes with the pipeline in a broken state.
+
+To avoid this, set all the related secrets, or at least all of them except the last, with
+`--wait=false`, so the pipeline applies them together and only the final command waits for it to
+become healthy:
+
+```bash
+redis-di set-secret SOURCE_DB_USERNAME newUsername --wait=false
+redis-di set-secret SOURCE_DB_PASSWORD newPassword
+```
+
+The same applies to any set of changes that are only valid together.
\ No newline at end of file
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md
new file mode 100644
index 0000000000..b202b05c60
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md
@@ -0,0 +1,423 @@
+---
+Title: Pipeline configuration file
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Learn how to specify the main configuration details for an RDI pipeline.
+group: di
+linkTitle: Pipeline configuration file
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 3
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config/'
+---
+
+The main configuration details for an RDI pipeline are in the `config.yaml` file.
+This file specifies the connection details for the source and target databases,
+and also the set of tables you want to capture. You can also add one or more
+[job files]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples" >}})
+if you want to apply custom transformations to the captured data.
+
+Each section explains one part of the file. Start with the minimal example, then
+add only the optional properties that you need. See the
+[configuration file reference]({{< relref "/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference" >}})
+for all supported properties.
+
+## Before you start
+
+Before you create `config.yaml`:
+
+1. [Prepare the source database]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs" >}}) for change data capture.
+1. [Install RDI]({{< relref "/integrate/redis-data-integration/1.19.1/installation" >}}).
+1. [Set the secrets]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/deploy#set-secrets" >}}) that the file references.
+
+## Start with a minimal file
+
+The following example shows the required structure of a `config.yaml` file. Values of the
+form "`${name}`" refer to secrets that you should set as described in
+[Set secrets]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/deploy#set-secrets" >}}).
+In particular, you should normally use secrets as shown to set the source
+and target username and password rather than storing them in plain text in this file.
+
+```yaml
+sources:
+ mysql:
+ type: cdc
+ logging:
+ level: info
+ connection:
+ type: mysql
+ host:
+ port: 3306
+ user: ${SOURCE_DB_USERNAME}
+ password: ${SOURCE_DB_PASSWORD}
+
+targets:
+ target:
+ connection:
+ type: redis
+ host:
+ port:
+ password: ${TARGET_DB_PASSWORD}
+
+processors:
+ type: flink
+ target_data_type: hash
+```
+
+Keep `type: flink` for new pipelines. The other processor properties have defaults,
+so add them only when you need to change the default behavior.
+
+## Build the file with an AI assistant
+
+Copy the following prompt into your AI assistant. The prompt tells the assistant
+to use the RDI documentation as its source of truth and to flag unsupported requests.
+
+```text
+Help me create a valid Redis Data Integration (RDI) config.yaml file.
+
+Use only these pages as sources for configuration properties and behavior:
+- https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config/
+- https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference/
+- https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/
+
+Do not invent property names. If a requested property is not documented, tell me.
+Use ${NAME} secret references for credentials and certificates. Do not include secret
+values in the file. Configure one target Redis database named `target`. Always use the
+Flink processor by setting `processors.type` to `flink`.
+
+Ask me for the following information one question at a time:
+1. Source database type, host, and port.
+2. Databases or schemas to capture.
+3. Tables and columns to capture, including keys for tables without a primary key
+ or unique constraint.
+4. Whether the initial snapshot needs a row filter.
+5. Target Redis host and port, and whether the connection uses TLS or mTLS.
+6. Redis hash or JSON output.
+
+After I answer, generate config.yaml. Then list the required secrets and link me to
+the documented commands to set the secrets and deploy the pipeline.
+```
+
+## Sections
+
+The main sections of the file configure [`sources`](#sources), [`targets`](#targets),
+and [`processors`](#processors).
+
+### Sources
+
+The `sources` section has a subsection for the source that
+you need to configure. The source section starts with a unique name
+to identify the source (in the example, there is a source
+called `mysql` but you can choose any name you like). The example
+configuration contains the following data:
+
+- `type`: The collector to use for the pipeline. Use `cdc` for MariaDB, MySQL,
+ MongoDB, Oracle, PostgreSQL, or SQL Server. Use `flink` for Google Cloud
+ Spanner. Use `riotx` for Snowflake. Use `external` when you provide and manage
+ the collector. RDI doesn't create collector resources for an `external` source,
+ so omit the other properties in the source section.
+- `connection`: The connection details for the source database: `type`, `host`, `port`,
+ and credentials (`user` and `password`).
+ See the [configuration file reference]({{< relref "/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference#sourcesconnection" >}})
+ for the required fields for each source database type.
+ - If you use [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security)/
+ or [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS) to connect
+ to the source database, you may need to specify additional properties in the
+ `advanced` section with references to the corresponding certificates depending
+ on the source database type. Note that these properties **must** be references to
+ secrets that you should set as described in [Set secrets]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/deploy#set-secrets" >}}).
+- `databases`: List of all databases to collect data from for source database types
+ that support multiple databases, such as `mysql` and `mariadb`.
+- `schemas`: List of all schemas to collect data from for source database types
+ that support multiple schemas, such as `oracle`, `postgresql`, and `sqlserver`.
+- `tables`: List of all tables to collect data from. Each table is identified by its
+ full name, including a database or schema prefix. If there is a single
+ database or schema, this prefix can be omitted.
+ For each table, you can specify:
+ - `columns`: A list of the columns you are interested in (the default is to
+ include all columns)
+ - `keys`: A list of columns to create a composite key if your table
+ doesn't already have a [`PRIMARY KEY`](https://www.w3schools.com/sql/sql_primarykey.asp) or
+ [`UNIQUE`](https://www.w3schools.com/sql/sql_unique.asp) constraint.
+ - `snapshot_sql`: A query to be used when performing the initial snapshot.
+ By default, a query that contains all listed columns of all listed tables will be used.
+- `advanced`: These optional properties configure other Debezium-specific features.
+ The available sub-sections are:
+ - `source`: Properties for reading from the source database.
+ See the Debezium [Source connectors](https://debezium.io/documentation/reference/stable/connectors/)
+ pages for more information about the properties available for each database type.
+ - `sink`: Properties for writing to Redis streams in the RDI database.
+ See the Debezium [Redis stream properties](https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream)
+ page for the full set of available properties.
+ - `quarkus`: Properties for the Debezium server, such as the log level. See the
+ Quarkus [Configuration options](https://quarkus.io/guides/all-config)
+ docs for the full set of available properties.
+ - `java_options`: controls the JAVA_OPTS environment variable (for RDI 1.15.1 and above). Use it to modify the default values for Java heap size and other Java options for the Debezium server.
+ For example, set it to `"-Xmx2g -Xms512m"` to set the maximum heap size to 2 GB and the initial heap size to 512 MB.
+
+### Targets
+
+Use this section to provide the connection details for the target Redis
+database. RDI supports one target database. Name the target `target`.
+In the `connection` section, you can specify the
+`type` of the target database, which must be `redis`, along with
+connection details such as `host`, `port`, and credentials (`user` and `password`).
+If you use [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security)/
+or [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS) to connect
+to the target database, you must specify the CA certificate (for TLS),
+and the client certificate and private key (for mTLS) in `cacert`, `cert`, and `key`.
+Note that these certificates **must** be references to secrets
+that you should set as described in [Set secrets]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/deploy#set-secrets" >}})
+(it is not possible to include these certificates as plain text in the file).
+
+### Processors
+
+The `processors` section selects the stream processor and configures its behavior.
+Use the Flink processor for new pipelines:
+
+```yaml
+processors:
+ type: flink
+```
+
+See [Differences between the classic and Flink processors]({{< relref "/integrate/redis-data-integration/1.19.1/architecture/classic-vs-flink" >}})
+and [Migrate from the classic processor to the Flink processor]({{< relref "/integrate/redis-data-integration/1.19.1/installation/migration-classic-to-flink" >}})
+for existing pipelines.
+
+### Tune Classic processor performance
+
+The Classic processor uses the top-level batch, queue, initial-sync, and stream
+polling properties. Larger batches can improve throughput but use more memory and
+can increase latency while RDI waits for a batch to fill.
+
+```yaml
+processors:
+ type: classic
+ read_batch_size: 2000
+ read_batch_timeout_ms: 100
+ write_batch_size: 200
+ enable_async_processing: true
+ batch_queue_size: 3
+ ack_queue_size: 10
+ initial_sync_processes: 4
+ idle_sleep_time_ms: 200
+ idle_streams_check_interval_ms: 1000
+ busy_streams_check_interval_ms: 5000
+```
+
+### Tune Flink processor performance
+
+The Flink processor uses `processors.advanced` for batch behavior, parallelism,
+and memory. Don't use Classic queue and initial-sync properties to tune Flink.
+
+```yaml
+processors:
+ type: flink
+ advanced:
+ source:
+ batch.size: 2000
+ batch.timeout.ms: 100
+ discovery.interval.ms: 1000
+ target:
+ batch.size: 200
+ flush.interval.ms: 100
+ flink:
+ taskmanager.numberOfTaskSlots: 1
+ taskmanager.memory.process.size: 2048m
+ resources:
+ taskManager:
+ replicas: 2
+```
+
+For Kubernetes installations, the number of available task slots is the number
+of TaskManager replicas multiplied by `taskmanager.numberOfTaskSlots`. When you
+omit `parallelism.default`, Flink uses the available task slots. Adding task slots
+can increase initial snapshot throughput. Size `taskmanager.memory.process.size`
+for the work done by each TaskManager, especially when jobs use transformations.
+
+The `advanced.source.batch.size`, `advanced.source.batch.timeout.ms`, and
+`advanced.target.batch.size` properties override their top-level aliases when
+both forms are present. Change other Flink settings only when instructed by Redis
+support. See the [configuration file reference]({{< relref "/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference#processorsadvanced" >}})
+for all Flink processor properties.
+
+### Choose the Redis data type
+
+Set `target_data_type` to `hash` (the default) or `json`. The `json` option
+requires JSON support in the target database. A job file can override this
+setting for its output.
+
+```yaml
+processors:
+ type: flink
+ target_data_type: hash
+```
+
+See [Job files]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples" >}})
+for the data types available to job outputs.
+
+### Confirm writes reached a replica
+
+Use these properties only when target database replication is enabled and a
+healthy replica is available:
+
+```yaml
+processors:
+ type: flink
+ wait_enabled: true
+ wait_timeout: 1000
+ retry_on_replica_failure: true
+```
+
+For the Flink processor, the corresponding properties under
+`processors.advanced.target` take priority over these top-level properties.
+
+See also the
+[RDI configuration file reference]({{< relref "/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference#processors" >}})
+for full details of the other available properties.
+
+## Extended configuration example
+
+This example combines the commonly used options from this page. Remove properties
+that you don't need. See the
+[configuration file reference]({{< relref "/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference" >}})
+for every supported property.
+
+```yaml
+sources:
+ mysql:
+ type: cdc
+ logging:
+ level: info
+ connection:
+ type: mysql
+ host: # e.g. localhost
+ port: 3306
+ # User and password are injected from the secrets.
+ user: ${SOURCE_DB_USERNAME}
+ password: ${SOURCE_DB_PASSWORD}
+ # Additional properties for the source collector:
+ # List of databases to include (optional).
+ # databases:
+ # - database1
+ # - database2
+
+ # List of tables to be synced (optional).
+ # tables:
+ # If only one database is specified in the databases property above,
+ # then tables can be defined without the database prefix.
+ # .:
+ # List of columns to be synced (optional).
+ # columns:
+ # -
+ # -
+ # List of columns to be used as keys (optional).
+ # keys:
+ # -
+
+ # Example: Sync specific tables.
+ # tables:
+ # Sync a specific table with all its columns:
+ # redislabscdc.account: {}
+ # Sync a specific table with selected columns:
+ # redislabscdc.emp:
+ # columns:
+ # - empno
+ # - fname
+ # - lname
+
+ # Advanced collector properties (optional):
+ # advanced:
+ # Sink collector properties - see the full list at
+ # https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream
+ # sink:
+ # Optional hard limits on memory usage of RDI streams.
+ # redis.memory.limit.mb: 300
+ # redis.memory.threshold.percentage: 85
+
+ # Uncomment for production so RDI Collector will wait on replica
+ # when writing entries.
+ # redis.wait.enabled: true
+ # redis.wait.timeout.ms: 1000
+ # redis.wait.retry.enabled: true
+ # redis.wait.retry.delay.ms: 1000
+
+ # Source specific properties - see the full list at
+ # https://debezium.io/documentation/reference/stable/connectors/
+ # source:
+ # snapshot.mode: initial
+ # Uncomment if you want a snapshot to include only a subset of the rows
+ # in a table. This property affects snapshots only.
+ # snapshot.select.statement.overrides: .
+ # The specified SELECT statement determines the subset of table rows to
+ # include in the snapshot.
+ # snapshot.select.statement.overrides..:
+
+ # Example: Snapshot filtering by order status.
+ # To include only orders with non-pending status from customers.orders
+ # table:
+ # snapshot.select.statement.overrides: customer.orders
+ # snapshot.select.statement.overrides.customer.orders: SELECT * FROM customers.orders WHERE status != 'pending' ORDER BY order_id DESC
+
+ # Quarkus framework properties - see the full list at
+ # https://quarkus.io/guides/all-config
+ # quarkus:
+ # banner.enabled: "false"
+
+ # `java_options` (for RDI 1.15.1 and above) controls the JAVA_OPTS environment variable. Use it to modify the default values for
+ # Java heap size and other Java options for the Debezium server.
+ # java_options: "-Xmx2g -Xms512m"
+
+targets:
+ # Redis target database connection.
+ # RDI supports one target database. Name it 'target'.
+ target:
+ connection:
+ type: redis
+ # Host of the Redis database to which RDI will
+ # write the processed data.
+ host: # e.g. localhost
+ # Port for the Redis database to which RDI will
+ # write the processed data.
+ port: # e.g. 12000
+ # User of the Redis database to which RDI will write the processed data.
+ # Uncomment if you are not using the default user.
+ # user: ${TARGET_DB_USERNAME}
+ # Password for Redis target database.
+ password: ${TARGET_DB_PASSWORD}
+ # SSL/TLS configuration: Uncomment to enable secure connections.
+ # key: ${TARGET_DB_KEY}
+ # key_password: ${TARGET_DB_KEY_PASSWORD}
+ # cert: ${TARGET_DB_CERT}
+ # cacert: ${TARGET_DB_CACERT}
+processors:
+ type: flink
+ # Target data type: hash or json.
+ # target_data_type: hash
+ # Enable merge as the default strategy for writing JSON documents.
+ # json_update_strategy: merge
+ # Confirm that writes reached a target database replica.
+ # wait_enabled: false
+ # wait_timeout: 1000
+ # retry_on_replica_failure: true
+ # Flink processor performance settings.
+ # advanced:
+ # source:
+ # batch.size: 2000
+ # batch.timeout.ms: 100
+ # discovery.interval.ms: 1000
+ # target:
+ # batch.size: 200
+ # flush.interval.ms: 100
+ # flink:
+ # taskmanager.numberOfTaskSlots: 1
+ # taskmanager.memory.process.size: 2048m
+ # resources:
+ # taskManager:
+ # replicas: 2
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/_index.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/_index.md
new file mode 100644
index 0000000000..f51d5845fd
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/_index.md
@@ -0,0 +1,30 @@
+---
+Title: Prepare source databases
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Enable CDC features in your source databases
+group: di
+hideListLinks: false
+linkTitle: Prepare source databases
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 1
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/'
+---
+
+Each database uses a different mechanism to track changes to its data and
+generally, these mechanisms are not switched on by default.
+RDI's Debezium collector uses these mechanisms for change data capture (CDC),
+so you must prepare your source database before you can use it with RDI.
+
+RDI supports the following source databases:
+
+{{< embed-md "rdi-supported-source-versions.md" >}}
+
+The pages in this section give detailed instructions to get your source
+database ready for Debezium to use:
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/_index.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/_index.md
new file mode 100644
index 0000000000..5e498b7f6d
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/_index.md
@@ -0,0 +1,22 @@
+---
+Title: Prepare AWS RDS and Aurora databases for RDI
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Learn how to prepare AWS RDS and Aurora databases for RDI.
+group: di
+linkTitle: Prepare AWS RDS and Aurora
+summary: Prepare AWS Aurora and AWS RDS databases to work with Redis Data Integration.
+hideListLinks: false
+type: integration
+weight: 5
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/'
+---
+
+You can use RDI with databases on [AWS Relational Database Service (RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) and [AWS Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html).
+
+The pages in this section give detailed instructions to get your source
+database ready for Debezium to use:
\ No newline at end of file
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md
new file mode 100644
index 0000000000..195653ba25
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md
@@ -0,0 +1,170 @@
+---
+Title: Prepare AWS Aurora MySQL/AWS RDS MySQL for RDI
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Enable CDC features in your source databases
+group: di
+hideListLinks: false
+linkTitle: Prepare AWS Aurora/RDS MySQL
+summary: Prepare AWS Aurora MySQL and AWS RDS MySQL databases to work with Redis Data Integration.
+type: integration
+weight: 2
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql/'
+---
+
+Follow the steps in the sections below to prepare an [AWS Aurora MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.Aurora.html) or [AWS RDS MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.MySQL.html) database.
+database to work with RDI.
+
+Select the steps for your database type.
+
+{{< multitabs id="rds-aur-mysql"
+ tab1="AWS Aurora MySQL"
+ tab2="AWS RDS MySQL" >}}
+
+```checklist {id="auroramysql" nointeractive="true" }
+- [ ] [Add an Aurora reader node](#add-an-aurora-reader-node)
+- [ ] [Create and apply parameter group](#aurora-create-and-apply-parameter-group)
+- [ ] [Create Debezium user](#aurora-create-debezium-user)
+```
+
+## Add an Aurora reader node
+
+RDI requires that your Aurora MySQL database has at least one replica or reader node.
+
+To add a reader node to an existing database, select **Add reader** from the **Actions** menu of the database and [add a reader node](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-replicas-adding.html).
+
+You can also create one during database creation by selecting **Create an Aurora Replica or Reader node in a different AZ (recommended for scaled availability)** under **Availability & durability > Multi-AZ deployment**.
+
+## Create and apply parameter group {#aurora-create-and-apply-parameter-group}
+
+RDI requires some changes to database parameters. On AWS Aurora, you change these parameters via a parameter group.
+
+```checklist {id="auroramysql-param-group" nointeractive="true" }
+- [ ] [Create/modify a parameter group](#aurora-create-a-parameter-group)
+- [ ] [Apply the parameter group](#aurora-apply-the-parameter-group)
+- [ ] [Apply the parameter group to the database](#aurora-apply-the-parameter-group-to-the-database)
+- [ ] [Reboot the database instance](#aurora-reboot-the-database-instance)
+```
+
+1. WITH LOGIN PASSWORD '' VALID UNTIL 'infinity';
+ ```
+
+ Replace `` and `` with a username and password for the new user.
+
+1. ;
+ ```
+
+ Replace `` with the username of the Debezium user.
+
+1. TO ;
+ ```
+
+ Replace `` with the username of the Debezium user and `` with the schema name.
+
+1. TO ;
+ ```
+
+ Replace `` with the name of the database and `` with the username of the Debezium user.
+
+1. TO ;
+ ```
+
+ Replace `` with the schema name and `` with the username of the Debezium user.">
+
+1.
+ GRANT SELECT ON TABLES TO ;
+ ```
+
+ Replace `` with the schema name and `` with the username of the Debezium user.
+
+1. WITH PASSWORD = ''
+ GO
+ USE
+ GO
+ CREATE USER FOR LOGIN
+ GO
+ ```
+
+ Replace `` and `` with a username and password for the new user and replace `` with the name of your database.
+
+1.
+ GO
+ USE
+ GO
+ EXEC sp_addrolemember N'db_datareader', N''
+ GO
+ ```
+
+ Replace `` with the username of the Debezium user and replace `` with the name of your database.
+
+## Enable CDC on the database
+
+Change Data Capture (CDC) must be enabled for the database and for each table you want to capture.
+
+```checklist {id="rds-sqlserver-enable-cdc" nointeractive="true" }
+- [ ] [Enable CDC for the database](#enable-cdc-for-the-database)
+- [ ] [Enable CDC for each table you want to capture](#enable-cdc-for-each-table-you-want-to-capture)
+- [ ] [Add the Debezium user to the CDC role](#add-the-debezium-user-to-the-cdc-role)
+```
+
+1. '
+ GO
+ ```
+
+ Replace `` with the name of your database.
+
+1.
+ GO
+ EXEC sys.sp_cdc_enable_table
+ @source_schema = N'',
+ @source_name = N'',
+ @role_name = N'',
+ @supports_net_changes = 0
+ GO
+ ```
+
+ Replace `` with the name of your database, `` with the name of the schema containing the table, `` with the name of the table, and `` with the name of a new role that will be created to manage access to the CDC data.
+
+ {{< note >}}
+The value for `@role_name` can’t be a fixed database role, such as `db_datareader`.
+Specifying a new name will create a corresponding database role that has full access to the
+captured change data.
+ {{< /note >}}
+
+1.
+ GO
+ EXEC sp_addrolemember N'', N''
+ GO
+ ```
+
+ Replace `` with the name of the role you created in the previous step and replace `` with the username of the Debezium user.
\ No newline at end of file
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/mongodb.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/mongodb.md
new file mode 100644
index 0000000000..b765591805
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/mongodb.md
@@ -0,0 +1,177 @@
+---
+Title: Prepare MongoDB for RDI
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Prepare MongoDB databases to work with RDI
+group: di
+linkTitle: Prepare MongoDB
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 2
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/mongodb/'
+---
+
+This guide describes the steps required to prepare a MongoDB database as a source for Redis Data Integration (RDI) pipelines.
+
+## Prerequisites
+- **MongoDB version:** 6.0 or later (replica set, sharded cluster, or MongoDB Atlas).
+- **User privileges:** You must have a MongoDB user with sufficient privileges to read the oplog and collections, and to use change streams.
+- **Network access:** The RDI Collector must be able to connect to all MongoDB nodes in your deployment.
+
+{{< note >}}The MongoDB connector is not capable of monitoring the changes of a standalone MongoDB server, since standalone servers do not have an oplog. The connector will work if the standalone server is converted to a replica set with one member.{{< /note >}}
+## Summary
+
+The following table summarizes the considerations to prepare a MongoDB database for RDI.
+
+| Requirement | Description |
+|---------------------|-----------------------------------------------------------------------------|
+| MongoDB Topology | Replica Set, Sharded Cluster, or MongoDB Atlas |
+| User Roles | readAnyDatabase, clusterMonitor |
+| Oplog | Sufficient size for snapshot and streaming |
+| Pre/Post Images | Enable on collections **only if using a custom key** |
+| Connection String | Must include all hosts, replicaSet (if applicable), authSource, credentials |
+| MongoDB Atlas | **[SSL required](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-property-mongodb-ssl-enabled)**, provide root CA as `SOURCE_DB_CACERT` secret in RDI |
+| MongoDB mTLS | X.509 authentication requires source TLS secrets and MongoDB SSL properties |
+| Network | RDI Collector must reach all MongoDB nodes on required ports |
+
+The following checklist shows the steps to prepare a MongoDB database for RDI,
+with links to the sections that explain the steps in full detail.
+You may find it helpful to track your progress with the checklist as you
+complete each step.
+
+```checklist {id="mongodblist"}
+- [ ] [Configure oplog size](#1-configure-oplog-size)
+- [ ] [Create a MongoDB user for RDI](#2-create-a-mongodb-user-for-rdi)
+- [ ] [Connection string format](#3-connection-string-format)
+- [ ] [Enable change streams and pre/post images (only if using a custom key)](#4-enable-change-streams-and-prepost-images-only-if-using-a-custom-key)
+- [ ] [MongoDB Atlas specific requirements](#5-mongodb-atlas-specific-requirements)
+- [ ] [Self-hosted MongoDB mTLS and X.509 authentication](#6-self-hosted-mongodb-mtls-and-x509-authentication)
+- [ ] [Network and security](#7-network-and-security)
+```
+
+## 1. Configure oplog size
+
+The Debezium MongoDB connector relies on the [oplog](https://www.mongodb.com/docs/manual/core/replica-set-oplog/) to capture changes from a replica set. The oplog is a fixed-size, capped collection. When it reaches its maximum size, it overwrites the oldest entries. If the connector is stopped and restarted, it attempts to resume from its last recorded position in the oplog. If that position has been overwritten, the connector may fail to start and report an invalid resume token error.
+
+To prevent this, ensure the oplog retains enough history for Debezium to resume streaming after interruptions. You can do this by:
+
+- **Increasing the oplog size:** Set the oplog size based on your workload, ensuring it can store more than the peak number of oplog entries generated per hour.
+- **Setting a minimum oplog retention period (MongoDB 4.4+):** Configure MongoDB to retain oplog entries for a minimum number of hours, guaranteeing availability even if the oplog reaches its maximum size. This is generally preferred, but for high-throughput clusters nearing capacity, you may need to increase the oplog size instead.
+
+For detailed guidance, see the Debezium [oplog configuration documentation](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-optimal-oplog-config).
+
+## 2. Create a MongoDB user for RDI
+
+Create a user with the following roles on the source database:
+- `readAnyDatabase` (optional) OR grant `read` for the specific database(s) you will use with RDI
+- `clusterMonitor`
+
+Example:
+```javascript
+use admin;
+db.createUser({
+ user: "rdi_user",
+ pwd: "rdi_password",
+ roles: [
+ // You can have multiple read roles, one per database.
+ { role: "read", db: "your_database" },
+ // Use the role below if you don't want to grant the `read` role for each database.
+ // { role: "readAnyDatabase", db: "admin" },
+ { role: "clusterMonitor", db: "admin" }
+ ]
+});
+```
+
+## 3. Connection string format
+
+The RDI Collector requires a MongoDB connection string that includes all relevant hosts and authentication details.
+
+Example (Replica Set):
+```
+mongodb://${SOURCE_DB_USERNAME}:${SOURCE_DB_PASSWORD}@host1:27017,host2:27017,host3:27017/?replicaSet=rs0&authSource=admin
+```
+Example (Sharded Cluster):
+```
+mongodb://${SOURCE_DB_USERNAME}:${SOURCE_DB_PASSWORD}@host:30000
+```
+- For Atlas, adjust the connection string accordingly (see example below).
+- Set `replicaSet` and `authSource` as appropriate for your deployment.
+
+## 4. Enable change streams and pre/post images (only if using a custom key)
+
+Change Streams are required only if you are using a custom key in your RDI pipeline. Change streams are available by default on replica sets, sharded clusters, and MongoDB Atlas.
+
+If your RDI pipeline uses a custom key, you must enable pre- and post-images on the relevant collections to capture the document state before and after updates or deletes. This allows RDI to access both the previous and updated versions of documents during change events, ensuring accurate synchronization.
+
+Use the command below to enable change streams and pre/post images:
+
+```javascript
+db.runCommand({
+ collMod: "your_collection",
+ changeStreamPreAndPostImages: { enabled: true }
+});
+```
+
+## 5. MongoDB Atlas specific requirements
+
+MongoDB Atlas only supports secure connections via SSL.
+The root CA certificate for MongoDB Atlas must be added as a SOURCE_DB_CACERT secret in RDI.
+
+- Download the MongoDB Atlas root CA certificate.
+- In RDI, add this certificate as a secret named SOURCE_DB_CACERT.
+- Ensure that the `mongodb.ssl.enabled: true` setting is present in your RDI configuration.
+
+Example connection string for Atlas:
+```
+mongodb+srv://${SOURCE_DB_USERNAME}:${SOURCE_DB_PASSWORD}@cluster0.mongodb.net/?authSource=admin
+```
+
+## 6. Self-hosted MongoDB mTLS and X.509 authentication
+
+For self-hosted MongoDB deployments that require TLS, set the source CA certificate
+as the `SOURCE_DB_CACERT` secret. For X.509 client certificate authentication, also
+set the `SOURCE_DB_CERT` and `SOURCE_DB_KEY` secrets. See
+[Set secrets]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/deploy#set-secrets" >}})
+for the full list of source database TLS and mTLS secrets.
+
+When you use MongoDB X.509 authentication, include all of the following
+properties in the source `advanced.source` section:
+
+```yaml
+advanced:
+ source:
+ mongodb.ssl.enabled: true
+ mongodb.ssl.keystore: /debezium/certs/source_db_keystore
+ mongodb.ssl.keystore.password: debezium
+```
+
+The RDI Collector builds `/debezium/certs/source_db_keystore` from the source
+database client certificate and private key secrets. Debezium requires the
+`mongodb.ssl.keystore` and `mongodb.ssl.keystore.password` properties to present
+the client certificate to MongoDB.
+
+For X.509 authentication, the MongoDB connection string must also include the
+required authentication options, such as `authMechanism=MONGODB-X509` and
+`authSource=%24external`.
+
+## 7. Network and security
+
+- Ensure the RDI Collector can connect to all MongoDB nodes on the required ports (default: 27017, or as provided by Atlas).
+- If using TLS/SSL, provide the necessary certificates and connection options in the connection string.
+
+## 8. Configuration is complete
+Once you have followed the steps above, your MongoDB database is ready for Debezium to use.
+
+## See also
+
+- [MongoDB Replica Set Documentation](https://www.mongodb.com/docs/manual/replication/)
+- [MongoDB Sharded Cluster Documentation](https://www.mongodb.com/docs/manual/sharding/)
+- [MongoDB Change Streams](https://www.mongodb.com/docs/manual/changeStreams/)
+- [MongoDB User Management](https://www.mongodb.com/docs/manual/core/security-users/)
+- [Debezium MongoDB Connector Documentation](https://debezium.io/documentation/reference/stable/connectors/mongodb.html)
+- [MongoDB Atlas SSL Setup](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-in-the-cloud)
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/my-sql-mariadb.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/my-sql-mariadb.md
new file mode 100644
index 0000000000..f62aa12f7e
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/my-sql-mariadb.md
@@ -0,0 +1,230 @@
+---
+Title: Prepare MySQL/MariaDB for RDI
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Prepare MySQL and MariaDB databases to work with RDI
+group: di
+linkTitle: Prepare MySQL/MariaDB
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 2
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/my-sql-mariadb/'
+---
+
+The following checklist summarizes the steps to prepare a MySQL or MariaDB
+database for RDI, with links to the sections that explain the steps in
+full detail. You may find it helpful to track your progress with the
+checklist as you complete each step.
+
+```checklist {id="mysqlmariadblist"}
+- [ ] [Create a CDC user](#1-create-a-cdc-user)
+- [ ] [Enable the binlog](#2-enable-the-binlog)
+- [ ] [Enable GTIDs](#3-enable-gtids)
+- [ ] [Configure session timeouts](#4-configure-session-timeouts)
+- [ ] [Enable query log events](#5-enable-query-log-events)
+- [ ] [Check binlog_row_value_options](#6-check-binlog_row_value_options)
+```
+
+## 1. Create a CDC user
+
+The Debezium connector needs a user account to connect to MySQL/MariaDB. This
+user must have appropriate permissions on all databases where you want Debezium
+to capture changes.
+
+Run the [MySQL CLI client](https://dev.mysql.com/doc/refman/8.3/en/mysql.html)
+and then run the following commands:
+
+```checklist {id="mysqlmariadb-create-cdc-user" nointeractive="true" }
+- [ ] [Create the CDC user](#create-the-cdc-user)
+- [ ] [Grant the user the necessary permissions](#grant-the-user-the-necessary-permissions)
+- [ ] [Finalize the user's permissions](#finalize-the-users-permissions)
+```
+
+1. GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'user' IDENTIFIED BY 'password';
+
+ # MySQL v8.0 and above
+ mysql> GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'user'@'localhost';
+ ```
+
+1.
+mysql> wait_timeout=
+```
+
+## 5. Enable query log events
+
+If you want to see the original SQL statement for each binlog event then you should
+enable `binlog_rows_query_log_events` (MySQL configuration) or
+`binlog_annotate_row_events` (MariaDB configuration):
+
+```
+mysql> binlog_rows_query_log_events=ON
+
+mariadb> binlog_annotate_row_events=ON
+```
+
+This option is available in MySQL 5.6 and later.
+
+## 6. Check `binlog_row_value_options`
+
+You should check the value of the `binlog_row_value_options` variable
+to ensure it is not set to `PARTIAL_JSON`. If it *is* set to
+`PARTIAL_JSON` then Debezium might not be able to see `UPDATE` events.
+
+Check the current value of the variable with the following command:
+
+```sql
+mysql> show global variables where variable_name = 'binlog_row_value_options';
+
+>>> Result:
+
++--------------------------+-------+
+| Variable_name | Value |
++--------------------------+-------+
+| binlog_row_value_options | |
++--------------------------+-------+
+```
+
+If the value is `PARTIAL_JSON` then you should unset the variable:
+
+```sql
+mysql> set @@global.binlog_row_value_options="" ;
+```
+
+## 7. Configuration is complete
+
+After following the steps above, your MySQL/MariaDB database is ready
+for Debezium to use.
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/neon.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/neon.md
new file mode 100644
index 0000000000..0968ad0a93
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/neon.md
@@ -0,0 +1,132 @@
+---
+Title: Prepare Neon for RDI
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Prepare Neon databases to work with RDI
+group: di
+linkTitle: Prepare Neon
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 10
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/neon/'
+---
+
+[Neon](https://neon.com/) is a serverless PostgreSQL platform. To use Neon as a source database for Redis Data Integration (RDI), you must enable logical replication for the compute endpoint that serves your branch, create a dedicated replication role, grant it the minimum required privileges, and allow inbound network traffic from the RDI connector host.
+
+The following checklist summarizes the steps to prepare a Neon database for
+RDI, with links to the sections that explain the steps in full detail. You may
+find it helpful to track your progress with the checklist as you complete each
+step.
+
+```checklist {id="neonlist"}
+- [ ] [Enable logical replication in Neon](#1-enable-logical-replication-in-neon)
+- [ ] [Create a replication role for RDI](#2-create-a-replication-role-for-rdi)
+- [ ] [Grant database and table privileges](#3-grant-database-and-table-privileges)
+- [ ] [Allow inbound traffic to Neon](#4-allow-inbound-traffic-to-neon)
+```
+
+## 1. Enable logical replication in Neon
+
+To capture changes from Neon, the compute endpoint for your branch must have
+logical replication enabled. This sets the PostgreSQL `wal_level` parameter to
+`logical` for that compute.
+
+1. Sign in to the Neon Console and select the project that hosts the database
+ you want to use with RDI.
+1. On the **Branches** page, choose the branch you want to capture from.
+1. Either create a new compute endpoint for that branch or edit an existing
+ one.
+1. In the compute configuration, enable **Logical replication**.
+1. Save the changes and wait for the compute to restart if required.
+
+After the compute is running, connect with a SQL client and confirm that
+logical replication is enabled by running:
+
+```sql
+SHOW wal_level;
+```
+
+The query should return `logical`.
+
+{{< note >}}
+Enabling logical replication increases the volume of write-ahead log (WAL)
+data that Neon retains. Monitor the project for any impact on storage usage and
+cost.
+{{< /note >}}
+
+## 2. Create a replication role for RDI
+
+It is strongly recommended to create a dedicated database role for the
+connection that RDI uses, rather than reusing an existing superuser or
+application role.
+
+Connect to the Neon database as a user with sufficient privileges and create a
+role similar to the following (replace the identifiers with values that match
+your environment):
+
+```sql
+CREATE ROLE rdi_replication
+ WITH LOGIN REPLICATION PASSWORD 'Strong_Password';
+```
+
+This role:
+
+- can sign in to the database (`LOGIN`)
+- can consume logical replication streams (`REPLICATION`)
+- does not have superuser privileges
+
+## 3. Grant database and table privileges
+
+The replication role must be able to connect to the database, access the
+schemas you want to capture, and read from the tables in those schemas.
+
+Run commands like the following, replacing `mydb` and `rdi_replication` with
+your database name and replication role:
+
+```sql
+GRANT CONNECT ON DATABASE mydb TO rdi_replication;
+
+GRANT USAGE ON SCHEMA public TO rdi_replication;
+GRANT SELECT ON ALL TABLES IN SCHEMA public TO rdi_replication;
+
+ALTER DEFAULT PRIVILEGES IN SCHEMA public
+ GRANT SELECT ON TABLES TO rdi_replication;
+```
+
+If you use multiple schemas or need access to sequences, repeat the `GRANT`
+and `ALTER DEFAULT PRIVILEGES` statements for each schema and include
+`SEQUENCES` where appropriate.
+
+## 4. Allow inbound traffic to Neon
+
+RDI connects to Neon over the public Internet using the connection string for
+your compute endpoint. You must ensure that Neon allows inbound traffic from
+the IP addresses that the RDI connector uses.
+
+1. In the Neon Console, open the project that contains your database.
+1. Go to the **Settings** or **Networking** section for the project and locate
+ the IP allow list or **Allowed IPs** configuration.
+1. Add the public IP address or address range that the RDI connector uses to
+ reach Neon. For production systems, restrict this to the smallest set of
+ IPs possible.
+1. Save the configuration.
+
+For development or testing, you can temporarily allow access from your own
+client machine or a broader IP range, but for production environments you
+should always restrict access to known connector IP addresses.
+
+You will also need the Neon connection string for use when you configure the
+source in RDI. In the Neon Console, copy the PostgreSQL connection URI for the
+compute endpoint you prepared above. It has the form:
+
+```text
+postgresql://:@:5432/?sslmode=require
+```
+
+Use this URI, together with the replication role you created, when you set up
+the Neon source connection in RDI.
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/oracle.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/oracle.md
new file mode 100644
index 0000000000..81b59c358f
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/oracle.md
@@ -0,0 +1,1411 @@
+---
+Title: Prepare Oracle and Oracle RAC for RDI
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Prepare Oracle and Oracle RAC databases to work with RDI
+group: di
+linkTitle: Prepare Oracle
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 1
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/oracle/'
+---
+
+Oracle provides two main systems that Debezium can use to capture data changes:
+
+- [LogMiner](#logminer)
+- [XStream](#xstream)
+
+The sections below explain how to configure each system for use with Debezium and RDI.
+The checklists summarize the steps you should follow to configure each system.
+You may find it helpful to use them to track your progress as you work through the steps.
+
+**LogMiner**
+
+```checklist {id="oraclelogminerlist"}
+- [ ] [Configure Oracle LogMiner](#1-configure-oracle-logminer)
+- [ ] [Enable supplemental logging](#supp-logging)
+- [ ] [Check the redo log sizing](#3-check-the-redo-log-sizing)
+- [ ] [Set the Archive log destination](#4-set-the-archive-log-destination)
+- [ ] [Create a user for the connector](#create-dbz-user)
+```
+
+**XStream**
+
+```checklist {id="oraclexstreamlist"}
+- [ ] [Configure recovery area](#1-configure-recovery-area)
+- [ ] [Enable GoldenGate replication](#2-enable-goldengate-replication)
+- [ ] [Configure XStream](#3-configure-xstream)
+- [ ] [Enable supplemental logging](#4-enable-supplemental-logging)
+- [ ] [Create XStream users](#5-create-xstream-users)
+- [ ] [Create an XStream outbound server](#6-create-an-xstream-outbound-server)
+- [ ] [Add a custom Docker image for the Debezium server](#7-add-a-custom-docker-image-for-the-debezium-server)
+- [ ] [Make RDI use the custom image](#8-make-rdi-use-the-custom-image-vm-installation)
+- [ ] [Enable the Oracle configuration in RDI](#9-enable-the-oracle-configuration-in-rdi)
+```
+
+**Optional: XMLTYPE Support**
+
+```checklist {id="oraclexmltypelist"}
+- [ ] [Create a custom Debezium Server image](#create-a-custom-debezium-server-image)
+- [ ] [Configure RDI for XMLTYPE support](#configure-rdi-for-xmltype-support)
+- [ ] [Test XMLTYPE support](#test-xmltype-support)
+```
+
+## LogMiner
+
+Follow the steps below to configure
+[LogMiner](https://docs.oracle.com/en/database/oracle/oracle-database/19/sutil/oracle-logminer-utility.html)
+and prepare your database for use with RDI.
+
+### 1. Configure Oracle LogMiner
+
+The following example shows the configuration for Oracle LogMiner.
+
+{{< note >}}[Amazon RDS for Oracle](https://aws.amazon.com/rds/oracle/)
+doesn't let you execute the commands
+in the example below or let you log in as `sysdba`. See the
+separate example below to [configure Amazon RDS for Oracle](#config-aws).
+{{< /note >}}
+
+```sql
+ORACLE_SID=ORACLCDB dbz_oracle sqlplus /nolog
+
+CONNECT sys/top_secret AS SYSDBA
+alter system set db_recovery_file_dest_size = 10G;
+alter system set db_recovery_file_dest = '/opt/oracle/oradata/recovery_area' scope=spfile;
+-- ======================================================================================================================================================
+-- !!!IMPORTANT!!!:
+-- In order to avoid Oracle downtime, please check if the LOG_MODE on your database is already set to `ARCHIVELOG` before executing the following commands:
+-- SELECT log_mode FROM v$database;
+-- If the LOG_MODE is already `ARCHIVELOG`, then you can skip the rest of the commands in this script
+-- ======================================================================================================================================================
+shutdown immediate
+startup mount
+alter database archivelog;
+alter database open;
+-- You should now see "Database log mode: Archive Mode"
+archive log list
+
+exit;
+```
+
+#### Configure Amazon RDS for Oracle {#config-aws}
+
+AWS provides its own set of commands to configure LogMiner.
+
+{{< note >}}Before executing these commands,
+you must enable backups on your Oracle AWS RDS instance.
+{{< /note >}}
+
+Check that Oracle has backups enabled with the following command:
+
+```sql
+SQL> SELECT LOG_MODE FROM V$DATABASE;
+
+LOG_MODE
+------------
+ARCHIVELOG
+```
+
+The `LOG_MODE` should be set to `ARCHIVELOG`. If it isn't then you
+should reboot your Oracle AWS RDS instance.
+
+Once `LOG_MODE` is correctly set to ARCHIVELOG, execute the following
+commands to complete the LogMiner configuration. The first command enables
+archive logging and the second adds [supplemental logging](#supp-logging).
+
+```sql
+exec rdsadmin.rdsadmin_util.set_configuration('archivelog retention hours',24);
+exec rdsadmin.rdsadmin_util.alter_supplemental_logging('ADD');
+```
+
+### 2. Enable supplemental logging {#supp-logging}
+
+You must enable supplemental logging for the tables you want to capture or
+for the entire database. This lets Debezium capture the state of
+database rows before and after changes occur.
+
+The following example shows how to configure supplemental logging for all columns
+in a single table called `inventory.customers`:
+
+```sql
+ALTER TABLE inventory.customers ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+```
+
+{{< note >}}If you enable supplemental logging for *all* table columns, you will
+probably see the size of the Oracle redo logs increase dramatically. Avoid this
+by using supplemental logging only when you need it. {{< /note >}}
+
+You must also enable minimal supplemental logging at the database level with
+the following command:
+
+```sql
+ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
+```
+
+### 3. Check the redo log sizing
+
+Before you use the Debezium connector, you should check with your
+database administrator that there are enough
+redo logs with enough capacity to store the data dictionary for your
+database. In general, the size of the data dictionary increases with the number
+of tables and columns in the database. If you don't have enough capacity in
+the logs then you might see performance problems with both the database and
+the Debezium connector.
+
+### 4. Set the Archive log destination
+
+You can configure up to 31 different destinations for archive logs
+(you must have administrator privileges to do this). You can set parameters for
+each destination to specify its purpose, such as log shipping for physical
+standbys, or external storage to allow for extended log retention. Oracle reports
+details about archive log destinations in the `V$ARCHIVE_DEST_STATUS` view.
+
+The Debezium Oracle connector only uses destinations that have a status of
+`VALID` and a type of `LOCAL`. If you only have one destination with these
+settings then Debezium will use it automatically.
+If you have more than one destination with these settings,
+then you should consult your database administrator about which one to
+choose for Debezium.
+
+Use the `log.mining.archive.destination.name` property in the connector configuration
+to select the archive log destination for Debezium.
+
+For example, suppose you have two archive destinations, `LOG_ARCHIVE_DEST_2` and
+`LOG_ARCHIVE_DEST_3`, and they both have status set to `VALID` and type set to
+`LOCAL`. Debezium could use either of these destinations, so you must select one
+of them explicitly in the configuration. To select `LOG_ARCHIVE_DEST_3`, you would
+use the following setting:
+
+```json
+{
+ "log.mining.archive.destination.name": "LOG_ARCHIVE_DEST_3"
+}
+```
+
+### 5. Create a user for the connector {#create-dbz-user}
+
+The Debezium Oracle connector must run as an Oracle LogMiner user with specific permissions.
+
+Typically, when you create the Oracle account for the connector,
+you grant the account a level of access that permits the connector to detect changes from all tables in the database.
+However, in some environments, security policies might prohibit you from granting such a broad level of access.
+
+The following example shows some SQL that creates an Oracle user account for the connector in a multi-tenant database model.
+The grant settings in the example permit the Debezium user to access all user tables in the database.
+
+To comply with security policies, you can modify the `SELECT ANY TABLE` and `FLASHBACK ANY TABLE` grants
+so that the connector can access only those tables that you intend to capture.
+
+Do not modify other grants, such as the `SELECT ANY TRANSACTION` grant,
+or the set of `SELECT ON V_$` grants, which provide access to dynamic performance views (`V_$`).
+These grants are required for the connector to function.
+
+{{< note >}}To prevent data loss, if you restrict the scope of the SELECT and FLASHBACK grants,
+be sure that the modified scope is compatible with the settings in the connector’s include configuration.
+The privileges that you set for the account must permit reading from all of the tables that you want the connector to capture.{{< /note >}}
+
+{{< note >}}This example uses `ORCLCDB` as the container database (CDB) name and `ORCLPDB1` as the pluggable database (PDB) name. Replace these with the CDB and PDB names from your own environment.{{< /note >}}
+
+```sql
+sqlplus sys/top_secret@//localhost:1521/ORCLCDB as sysdba
+CREATE TABLESPACE logminer_tbs DATAFILE '/opt/oracle/oradata/ORCLCDB/logminer_tbs.dbf'
+ SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
+exit;
+
+sqlplus sys/top_secret@//localhost:1521/ORCLPDB1 as sysdba
+CREATE TABLESPACE logminer_tbs DATAFILE '/opt/oracle/oradata/ORCLCDB/ORCLPDB1/logminer_tbs.dbf'
+ SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
+exit;
+
+sqlplus sys/top_secret@//localhost:1521/ORCLCDB as sysdba
+
+CREATE USER c##dbzuser IDENTIFIED BY dbz
+ DEFAULT TABLESPACE logminer_tbs
+ QUOTA UNLIMITED ON logminer_tbs
+ CONTAINER=ALL;
+
+GRANT CREATE SESSION TO c##dbzuser CONTAINER=ALL;
+GRANT SET CONTAINER TO c##dbzuser CONTAINER=ALL;
+GRANT SELECT ON V_$DATABASE to c##dbzuser CONTAINER=ALL;
+
+-- See `Limiting privileges` below if the privileges
+-- granted by these two commands raise security concerns.
+GRANT FLASHBACK ANY TABLE TO c##dbzuser CONTAINER=ALL;
+GRANT SELECT ANY TABLE TO c##dbzuser CONTAINER=ALL;
+--
+
+GRANT SELECT_CATALOG_ROLE TO c##dbzuser CONTAINER=ALL;
+GRANT EXECUTE_CATALOG_ROLE TO c##dbzuser CONTAINER=ALL;
+GRANT SELECT ANY TRANSACTION TO c##dbzuser CONTAINER=ALL;
+GRANT LOGMINING TO c##dbzuser CONTAINER=ALL;
+
+-- See `Limiting privileges` below if the privileges
+-- granted by these two commands raise security concerns.
+GRANT CREATE TABLE TO c##dbzuser CONTAINER=ALL;
+GRANT LOCK ANY TABLE TO c##dbzuser CONTAINER=ALL;
+--
+
+GRANT CREATE SEQUENCE TO c##dbzuser CONTAINER=ALL;
+
+GRANT EXECUTE ON DBMS_LOGMNR TO c##dbzuser CONTAINER=ALL;
+GRANT EXECUTE ON DBMS_LOGMNR_D TO c##dbzuser CONTAINER=ALL;
+
+GRANT SELECT ON V_$LOG TO c##dbzuser CONTAINER=ALL;
+GRANT SELECT ON V_$LOG_HISTORY TO c##dbzuser CONTAINER=ALL;
+GRANT SELECT ON V_$LOGMNR_LOGS TO c##dbzuser CONTAINER=ALL;
+GRANT SELECT ON V_$LOGMNR_CONTENTS TO c##dbzuser CONTAINER=ALL;
+GRANT SELECT ON V_$LOGMNR_PARAMETERS TO c##dbzuser CONTAINER=ALL;
+GRANT SELECT ON V_$LOGFILE TO c##dbzuser CONTAINER=ALL;
+GRANT SELECT ON V_$ARCHIVED_LOG TO c##dbzuser CONTAINER=ALL;
+GRANT SELECT ON V_$ARCHIVE_DEST_STATUS TO c##dbzuser CONTAINER=ALL;
+GRANT SELECT ON V_$TRANSACTION TO c##dbzuser CONTAINER=ALL;
+
+GRANT SELECT ON V_$MYSTAT TO c##dbzuser CONTAINER=ALL;
+GRANT SELECT ON V_$STATNAME TO c##dbzuser CONTAINER=ALL;
+
+exit;
+```
+
+| Role name | Description |
+|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| CREATE SESSION | Enables the connector to connect to Oracle. |
+| SET CONTAINER | Enables the connector to switch between pluggable databases. This is only required when the Oracle installation has container database support (CDB) enabled. |
+| SELECT ON V_$DATABASE | Enables the connector to read the V_$DATABASE table. |
+| FLASHBACK ANY TABLE | Enables the connector to perform Flashback queries, which is how the connector performs the initial snapshot of data. Optionally, rather than granting FLASHBACK permission on all tables, you can grant the FLASHBACK privilege for specific tables only. |
+| SELECT ANY TABLE | Enables the connector to read any table. Optionally, rather than granting SELECT permission on all tables, you can grant the SELECT privilege for specific tables only. |
+| SELECT_CATALOG_ROLE | Enables the connector to read the data dictionary, which is needed by Oracle LogMiner sessions. |
+| EXECUTE_CATALOG_ROLE | Enables the connector to write the data dictionary into the Oracle redo logs, which is needed to track schema changes. |
+| SELECT ANY TRANSACTION | Enables the snapshot process to perform a Flashback snapshot query against any transaction so that the connector can read past changes from LogMiner. When FLASHBACK ANY TABLE is granted, this should also be granted. This grant is optional for Oracle 19c and later. In those later releases, the connector obtains the required privileges through the EXECUTE_CATALOG_ROLE and LOGMINING grants. |
+| LOGMINING | This role was added in newer versions of Oracle as a way to grant full access to Oracle LogMiner and its packages. On older versions of Oracle that don’t have this role, you can ignore this grant. |
+| CREATE TABLE | Enables the connector to create its flush table in its default tablespace. The flush table allows the connector to explicitly control flushing of the LGWR internal buffers to disk. |
+| LOCK ANY TABLE | Enables the connector to lock tables during schema snapshot. If snapshot locks are explicitly disabled via configuration, this grant can be safely ignored. |
+| CREATE SEQUENCE | Enables the connector to create a sequence in its default tablespace. |
+| EXECUTE ON DBMS_LOGMNR | Enables the connector to run methods in the DBMS_LOGMNR package. This is required to interact with Oracle LogMiner. On newer versions of Oracle this is granted via the LOGMINING role but on older versions, this must be explicitly granted. |
+| EXECUTE ON DBMS_LOGMNR_D | Enables the connector to run methods in the DBMS_LOGMNR_D package. This is required to interact with Oracle LogMiner. On newer versions of Oracle this is granted via the LOGMINING role but on older versions, this must be explicitly granted. |
+| SELECT ON V_$…. | Enables the connector to read these tables. The connector must be able to read information about the Oracle redo and archive logs, and the current transaction state, to prepare the Oracle LogMiner session. Without these grants, the connector cannot operate. |
+
+#### Limiting privileges
+
+The privileges granted in the example above are convenient,
+but you may prefer to restrict them further to improve security. In particular,
+you might want to prevent the Debezium user from creating tables, or
+selecting or locking any table.
+
+The Debezium user needs the `CREATE TABLE` privilege to create the
+`LOG_MINING_FLUSH` table when it connects for the first
+time. After this point, it doesn't need to create any more tables,
+so you can safely revoke this privilege with the following command:
+
+```sql
+REVOKE CREATE TABLE FROM c##dbzuser container=all;
+```
+
+[The example above](#create-dbz-user) grants the `SELECT ANY TABLE` and
+`FLASHBACK ANY TABLE` privileges for convenience, but only the tables synced to RDI
+and the `V_$XXX` tables strictly need these privileges.
+You can replace the `GRANT SELECT ANY TABLE` command with explicit
+commands for each table. For example, you would use commands like the
+following for the tables in our sample
+[`chinook`](https://github.com/Redislabs-Solution-Architects/rdi-quickstart-postgres)
+database. (Note that Oracle 19c requires you to run a separate `GRANT`
+command for each table individually.)
+
+```sql
+GRANT SELECT ON chinook.album TO c##dbzuser;
+GRANT SELECT ON chinook.artist TO c##dbzuser;
+GRANT SELECT ON chinook.customer TO c##dbzuser;
+...
+```
+
+Similarly, instead of `GRANT FLASHBACK ANY TABLE`, you would use the following
+commands:
+
+```sql
+GRANT FLASHBACK ON chinook.album TO c##dbzuser;
+GRANT FLASHBACK ON chinook.artist TO c##dbzuser;
+GRANT FLASHBACK ON chinook.customer TO c##dbzuser;
+...
+```
+
+The `LOCK` privilege is automatically granted by the `SELECT`
+privilege, so you can omit this command if you have granted `SELECT`
+on specific tables.
+
+#### Revoking existing privileges
+
+If you initially set the Debezium user's privileges on all tables,
+but you now want to restrict them, you can revoke the existing
+privileges before resetting them as described in the
+[Limiting privileges](#limiting-privileges) section.
+
+Use the following commands to revoke and reset the `SELECT` privileges:
+
+```sql
+REVOKE SELECT ANY TABLE FROM c##dbzuser container=all;
+ALTER SESSION SET container=orclpdb1;
+
+GRANT SELECT ON chinook.album TO c##dbzuser;
+-- ...etc
+```
+
+The equivalent commands for `FLASHBACK` are:
+
+```sql
+REVOKE FLASHBACK ANY TABLE FROM c##dbzuser container=all;
+ALTER SESSION SET container=orclpdb1;
+GRANT FLASHBACK ON chinook.album TO c##dbzuser;
+```
+
+The `SELECT` privilege automatically includes the `LOCK`
+privilege, so when you grant `SELECT` for specific tables
+you should also revoke `LOCK` on all tables:
+
+```sql
+REVOKE LOCK ANY TABLE FROM c##dbzuser container=all;
+```
+
+### 6. Configuration is complete {#logminer-complete}
+
+Once you have followed the steps above, your Oracle database is ready
+for Debezium to use.
+
+## XStream
+
+[XStream](https://docs.oracle.com/en/database/oracle/oracle-database/19/xstrm/introduction-to-xstream.html#GUID-5939CB6C-8BA9-4594-8F96-B0453D246722)
+is a set of database components and APIs to communicate change data to and
+from an Oracle database. RDI specifically uses
+[XStream Out](https://docs.oracle.com/en/database/oracle/oracle-database/19/xstrm/xstream-out.html)
+to capture changes.
+
+Follow the steps in the sections below to configure XStream to work with
+Debezium and RDI.
+
+{{< note >}}You should run all database commands shown below as the `sysdba` user.
+{{< /note >}}
+
+```sql
+sqlplus sys/ as sysdba
+```
+
+### 1. Configure recovery area
+
+{{< multitabs id="oracle-recovery-area"
+ tab1="non-RAC (single-instance)"
+ tab2="Oracle RAC (multitenant environment)" >}}
+
+Create a directory for the recovery area on the Oracle host (or Oracle container if you are using a containerized Oracle).
+
+For single-instance (non-RAC) deployments on local disk, you can use:
+
+```bash
+mkdir -p /opt/oracle/oradata/recovery_area
+```
+
+ ```sql
+ ALTER SYSTEM SET db_recovery_file_dest_size = 10G; -- Adjust size as needed
+ ALTER SYSTEM SET db_recovery_file_dest = '/opt/oracle/oradata/recovery_area' SCOPE=BOTH;
+ ```
+
+-tab-sep-
+
+For Oracle RAC (cluster) environments, the Fast Recovery Area (FRA) must be on **shared storage** (ASM or a shared filesystem).
+
+- **If using ASM (recommended for RAC):**
+
+ ```sql
+ ALTER SYSTEM SET db_recovery_file_dest_size = 10G; -- Adjust size as needed
+ ALTER SYSTEM SET db_recovery_file_dest = '+FRA' SCOPE=BOTH;
+ ```
+
+ Replace `+FRA` with the ASM disk group used for your FRA. No `mkdir` is required; ASM manages the storage.
+
+- **If using a shared filesystem (NFS/OCFS2/etc.):**
+
+ ```sql
+ ALTER SYSTEM SET db_recovery_file_dest_size = 10G; -- Adjust size as needed
+ ALTER SYSTEM SET db_recovery_file_dest = '/u02/oradata/recovery_area' SCOPE=BOTH;
+ ```
+
+ The path you use must:
+
+ - Be on shared storage visible from all RAC nodes
+ - Be mounted at the **same path** on all nodes
+ - Be writable by the Oracle user
+
+ Create the directory once on the shared filesystem (it will be visible from all nodes):
+
+ ```bash
+ mkdir -p /u02/oradata/recovery_area
+ chown oracle:oinstall /u02/oradata/recovery_area
+ chmod 755 /u02/oradata/recovery_area
+ ```
+
+{{< /multitabs >}}
+
+### 2. Enable GoldenGate replication
+
+Check if `enable_goldengate_replication` is already set to `true`:
+
+{{< multitabs id="oracle-goldengate-replication"
+ tab1="Container database (CDB)"
+ tab2="Non-container database (Non-CDB)" >}}
+
+For a CDB, check the parameter in the root container:
+
+```sql
+SELECT VALUE FROM V$PARAMETER WHERE NAME = 'enable_goldengate_replication';
+```
+
+If it is not set to `true`, you need to set it and **restart the database**:
+
+```sql
+ALTER SYSTEM SET enable_goldengate_replication=true SCOPE=BOTH CONTAINER=ALL;
+```
+
+-tab-sep-
+
+For a non-CDB, check the parameter in the database instance:
+
+```sql
+SELECT VALUE FROM V$PARAMETER WHERE NAME = 'enable_goldengate_replication';
+```
+
+If it is not set to `true`, you need to set it and **restart the database**:
+
+```sql
+ALTER SYSTEM SET enable_goldengate_replication=true SCOPE=BOTH;
+```
+
+{{< /multitabs >}}
+
+### 3. Configure XStream
+
+Use the following SQL commands to configure XStream (using the
+[`chinook`](https://github.com/Redislabs-Solution-Architects/rdi-quickstart-postgres/tree/main)
+schema as an example):
+
+{{< multitabs id="oracle-xstream-rac"
+ tab1="non-RAC (single-instance)"
+ tab2="Oracle RAC (multitenant environment)" >}}
+
+```sql
+-- ======================================================================================================================================================
+-- !!!IMPORTANT!!!:
+-- In order to avoid Oracle downtime, please check if the LOG_MODE on your database is already set to `ARCHIVELOG` before executing the following commands:
+-- SELECT log_mode FROM v$database;
+-- If the LOG_MODE is already `ARCHIVELOG`, then you can skip the rest of the commands in this script
+-- ======================================================================================================================================================
+SQL> shutdown immediate
+Database closed.
+Database dismounted.
+ORACLE instance shut down.
+SQL> startup mount
+ORACLE instance started.
+
+Total System Global Area 1476391776 bytes
+Fixed Size 9134944 bytes
+Variable Size 1006632960 bytes
+Database Buffers 452984832 bytes
+Redo Buffers 7639040 bytes
+Database mounted.
+SQL> alter database archivelog;
+SQL> alter database open;
+SQL> archive log list
+Database log mode Archive Mode
+Automatic archival Enabled
+Archive destination USE_DB_RECOVERY_FILE_DEST
+Oldest online log sequence 10
+Next log sequence to archive 12
+Current log sequence 12
+
+-- Confirm the database is in ARCHIVELOG mode.
+SQL> SELECT LOG_MODE FROM V$DATABASE;
+```
+
+-tab-sep-
+
+```sql
+-- ======================================================================================================================================================
+-- !!!IMPORTANT!!!:
+-- In order to avoid Oracle downtime, please check if the LOG_MODE on your database is already set to `ARCHIVELOG` before executing the following commands:
+-- SELECT log_mode FROM v$database;
+-- If the LOG_MODE is already `ARCHIVELOG`, then you can skip the rest of the commands in this script
+-- ======================================================================================================================================================
+
+-- Stop all database instances
+SQL> srvctl stop database -d
+
+-- Start the database in mount state.
+SQL> srvctl start database -d -o mount
+
+-- Enable archive log mode
+SQL> alter database archivelog;
+
+-- Restart all database instances.
+SQL> srvctl stop database -d
+SQL> srvctl start database -d
+
+-- Confirm the database is in ARCHIVELOG mode.
+SQL> SELECT LOG_MODE FROM V$DATABASE;
+```
+
+{{< /multitabs >}}
+
+### 4. Enable supplemental logging
+
+{{< multitabs id="oracle-xstream-supplemental-logging"
+ tab1="Container database (CDB)"
+ tab2="Non-container database (Non-CDB)" >}}
+
+Enable supplemental logging for the tables you want to capture or
+for the entire database. This lets Debezium capture the state of
+database rows before and after changes occur.
+
+```sql
+SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
+
+SQL> alter session set container=orclpdb1;
+
+SQL> ALTER TABLE CHINOOK.ALBUM ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.ARTIST ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.CUSTOMER ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.EMPLOYEE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.GENRE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.INVOICE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.INVOICELINE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.MEDIATYPE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.PLAYLIST ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.PLAYLISTTRACK ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.TRACK ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+```
+
+{{< note >}}The example above uses `orclpdb1` as the PDB name. Replace it with the name of the pluggable database (PDB) that you use in your own environment.{{< /note >}}
+
+-tab-sep-
+
+Enable supplemental logging for the tables you want to capture or
+for the entire database. This lets Debezium capture the state of
+database rows before and after changes occur.
+
+```sql
+SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
+
+SQL> ALTER TABLE CHINOOK.ALBUM ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.ARTIST ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.CUSTOMER ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.EMPLOYEE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.GENRE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.INVOICE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.INVOICELINE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.MEDIATYPE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.PLAYLIST ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.PLAYLISTTRACK ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+SQL> ALTER TABLE CHINOOK.TRACK ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+```
+
+{{< /multitabs >}}
+
+{{< note >}}You must configure supplemental logging explicitly for each table as shown
+above. Otherwise, only the original sync will be performed and no change data will be
+captured for the table.
+{{< /note >}}
+
+### 5. Create XStream users
+
+{{< multitabs id="oracle-xstream-user-creation"
+ tab1="Container database (CDB)"
+ tab2="Non-container database (Non-CDB)" >}}
+
+{{< note >}}The XStream user examples below use `ORCLCDB` as the CDB name and `ORCLPDB1`/`orclpdb1` as the PDB name in file paths and `ALTER SESSION SET CONTAINER` commands. Replace these with the CDB and PDB names from your own Oracle deployment.{{< /note >}}
+
+Create an XStream administrator user with the following SQL:
+
+```sql
+sqlplus sys/ as sysdba
+
+SQL> ALTER SESSION SET CONTAINER=CDB$ROOT;
+SQL> CREATE TABLESPACE xstream_adm_tbs DATAFILE '/opt/oracle/oradata/ORCLCDB/xstream_adm_tbs.dbf'
+ SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
+
+SQL> alter session set container=orclpdb1;
+
+SQL> CREATE TABLESPACE xstream_adm_tbs DATAFILE '/opt/oracle/oradata/ORCLCDB/ORCLPDB1/xstream_adm_tbs.dbf'
+ SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
+
+SQL> alter session set container=cdb$root;
+
+SQL> CREATE USER c##dbzadmin IDENTIFIED BY dbz
+ DEFAULT TABLESPACE xstream_adm_tbs
+ QUOTA UNLIMITED ON xstream_adm_tbs
+ CONTAINER=ALL;
+
+SQL> GRANT CREATE SESSION, SET CONTAINER TO c##dbzadmin CONTAINER=ALL;
+
+SQL> BEGIN
+ DBMS_XSTREAM_AUTH.GRANT_ADMIN_PRIVILEGE(
+ grantee => 'c##dbzadmin',
+ privilege_type => 'CAPTURE',
+ grant_select_privileges => TRUE,
+ container => 'ALL'
+ );
+ END;
+ /
+```
+
+Then, create the XStream user:
+
+```sql
+sqlplus sys/ as sysdba
+
+SQL> CREATE TABLESPACE xstream_tbs DATAFILE '/opt/oracle/oradata/ORCLCDB/xstream_tbs.dbf'
+ SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
+
+SQL> alter session set container=orclpdb1;
+
+SQL> CREATE TABLESPACE xstream_tbs DATAFILE '/opt/oracle/oradata/ORCLCDB/ORCLPDB1/xstream_tbs.dbf'
+ SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
+
+SQL> alter session set container=cdb$root;
+
+SQL> CREATE USER c##dbzxsuser IDENTIFIED BY dbz
+ DEFAULT TABLESPACE xstream_tbs
+ QUOTA UNLIMITED ON xstream_tbs
+ CONTAINER=ALL;
+
+SQL> GRANT CREATE SESSION TO c##dbzxsuser CONTAINER=ALL;
+SQL> GRANT SET CONTAINER TO c##dbzxsuser CONTAINER=ALL;
+SQL> GRANT SELECT ON V_$DATABASE to c##dbzxsuser CONTAINER=ALL;
+SQL> GRANT FLASHBACK ANY TABLE TO c##dbzxsuser CONTAINER=ALL;
+SQL> GRANT SELECT_CATALOG_ROLE TO c##dbzxsuser CONTAINER=ALL;
+SQL> GRANT EXECUTE_CATALOG_ROLE TO c##dbzxsuser CONTAINER=ALL;
+SQL> GRANT SELECT ANY TABLE TO c##dbzxsuser CONTAINER=ALL;
+SQL> GRANT LOCK ANY TABLE TO c##dbzxsuser CONTAINER=ALL;
+```
+
+{{< note >}}If you are using the
+[Debezium XStream documentation](https://debezium.io/documentation/reference/stable/connectors/oracle.html#creating-xstream-users-for-the-connector),
+you should note that it misses out the last two GRANT statements shown above:
+
+```sql
+GRANT SELECT ANY TABLE TO c##dbzxsuser CONTAINER=ALL;
+GRANT LOCK ANY TABLE TO c##dbzxsuser CONTAINER=ALL;
+```
+
+However, without these, no tables can be read by Debezium, so neither the initial snapshot nor any subsequent updates will produce any data.
+{{< /note >}}
+
+-tab-sep-
+
+{{< note >}}The non-CDB architecture is deprecated in Oracle Database 12c and discontinued in Oracle Database 20c.{{< /note >}}
+
+For a non-container (non-CDB) Oracle database:
+
+- You have a single database with no containers.
+- Users are regular database users without the `C##` prefix.
+- Application data is stored directly in the database.
+- You do not use `ALTER SESSION SET CONTAINER` statements or `CONTAINER=ALL` clauses.
+
+**Create XStream users and tablespaces (non-CDB)** as sysdba
+-- =====================================================
+
+-- Step 1: Create XStream Administrator Tablespace
+CREATE TABLESPACE xstream_adm_tbs DATAFILE '/u01/app/oracle/oradata/ORCL/xstream_adm_tbs.dbf'
+ SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
+
+PROMPT Tablespace xstream_adm_tbs created
+
+-- Step 2: Create XStream Administrator User
+CREATE USER dbzadmin IDENTIFIED BY dbz
+ DEFAULT TABLESPACE xstream_adm_tbs
+ QUOTA UNLIMITED ON xstream_adm_tbs;
+
+GRANT CREATE SESSION TO dbzadmin;
+
+PROMPT User dbzadmin created
+
+-- Step 3: Grant XStream Admin Privileges
+BEGIN
+ DBMS_XSTREAM_AUTH.GRANT_ADMIN_PRIVILEGE(
+ grantee => 'dbzadmin',
+ privilege_type => 'CAPTURE',
+ grant_select_privileges => TRUE
+ );
+END;
+/
+
+PROMPT XStream admin privileges granted to dbzadmin
+
+-- Step 4: Create XStream User Tablespace
+CREATE TABLESPACE xstream_tbs DATAFILE '/u01/app/oracle/oradata/ORCL/xstream_tbs.dbf'
+ SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
+
+PROMPT Tablespace xstream_tbs created
+
+-- Step 5: Create XStream User
+CREATE USER dbzxsuser IDENTIFIED BY dbz
+ DEFAULT TABLESPACE xstream_tbs
+ QUOTA UNLIMITED ON xstream_tbs;
+
+PROMPT User dbzxsuser created
+
+-- Step 6: Grant necessary privileges to XStream User
+GRANT CREATE SESSION TO dbzxsuser;
+GRANT SELECT ON V_$DATABASE TO dbzxsuser;
+GRANT FLASHBACK ANY TABLE TO dbzxsuser;
+GRANT SELECT_CATALOG_ROLE TO dbzxsuser;
+GRANT EXECUTE_CATALOG_ROLE TO dbzxsuser;
+GRANT SELECT ANY TABLE TO dbzxsuser;
+GRANT LOCK ANY TABLE TO dbzxsuser;
+
+PROMPT Privileges granted to dbzxsuser
+
+-- Verification: Check users were created
+SELECT username, account_status, default_tablespace
+FROM dba_users
+WHERE username IN ('DBZADMIN', 'DBZXSUSER');
+
+PROMPT
+PROMPT =====================================================
+PROMPT XStream Users Created Successfully!
+PROMPT =====================================================
+PROMPT Created users:
+PROMPT - dbzadmin (XStream Administrator)
+PROMPT - dbzxsuser (XStream User)
+PROMPT
+PROMPT Next step:
+PROMPT Run 03_create_xstream_outbound.sql to create outbound server
+PROMPT =====================================================
+```
+
+This script creates:
+
+- The `xstream_adm_tbs` tablespace and `dbzadmin` XStream administrator user.
+- The `xstream_tbs` tablespace and `dbzxsuser` XStream connect user.
+
+{{< /multitabs >}}
+
+### 6. Create an XStream outbound server
+
+Create the outbound server with the following SQL.
+
+{{< multitabs id="oracle-xstream-outbound-server"
+ tab1="Container database (CDB)"
+ tab2="Non-container database (Non-CDB)" >}}
+
+{{< note >}}In this example, `ORCLCDB` is the CDB service name and `orclpdb1` is the PDB name. Replace them with the appropriate service and PDB names for your own environment.{{< /note >}}
+
+Note that you must connect as the `c##dbzadmin` user created in the previous step,
+not the `sys` user:
+
+```bash
+sqlplus c##dbzadmin/dbz@localhost:1521/ORCLCDB
+```
+
+```sql
+-- =====================================================
+-- Create XStream Outbound Server
+-- =====================================================
+-- This script creates the XStream outbound server for CDC
+-- Run as: sqlplus c##dbzadmin/dbz@localhost:1521/ORCLCDB
+-- =====================================================
+
+-- Step 1: Create XStream Outbound Server
+DECLARE
+ tables DBMS_UTILITY.UNCL_ARRAY;
+ schemas DBMS_UTILITY.UNCL_ARRAY;
+BEGIN
+ tables(1) := NULL;
+ schemas(1) := 'C##DBZUSER';
+
+ DBMS_XSTREAM_ADM.CREATE_OUTBOUND(
+ server_name => 'dbzxout',
+ source_container_name => 'XEPDB1',
+ table_names => tables,
+ schema_names => schemas
+ );
+END;
+/
+
+-- Step 2: Configure XStream User to connect to outbound server
+-- This must be run as sys user
+CONNECT sys/oracle AS SYSDBA
+
+BEGIN
+ DBMS_XSTREAM_ADM.ALTER_OUTBOUND(
+ server_name => 'dbzxout',
+ connect_user => 'c##dbzxsuser'
+ );
+END;
+/
+
+-- Verification: Check outbound server was created
+SELECT server_name, source_database, capture_name, capture_user, connect_user, queue_owner, queue_name
+FROM dba_xstream_outbound;
+
+-- Verification: Check capture process
+SELECT capture_name, status, capture_type, source_database
+FROM dba_capture;
+
+PROMPT
+PROMPT =====================================================
+PROMPT XStream Outbound Server Created Successfully!
+PROMPT =====================================================
+PROMPT Server name: dbzxout
+PROMPT Schema: C##DBZUSER
+PROMPT PDB: XEPDB1
+PROMPT =====================================================
+```
+
+--tab-sep-
+
+Note that you must connect as the `dbzadmin` user created in the previous step,
+not the `sys` user:
+
+```bash
+sqlplus dbzadmin/dbz@localhost:1521/ORCL
+```
+
+```sql
+-- =====================================================
+-- Create XStream Outbound Server - FOR NON-CDB ENVIRONMENT
+-- =====================================================
+-- This script creates the XStream outbound server for CDC
+-- Run as: sqlplus dbzadmin/dbz@localhost:1521/ORCL
+-- =====================================================
+
+-- Step 1: Create XStream Outbound Server
+DECLARE
+ tables DBMS_UTILITY.UNCL_ARRAY;
+ schemas DBMS_UTILITY.UNCL_ARRAY;
+BEGIN
+ tables(1) := NULL;
+ schemas(1) := 'chinook'; -- Replace with actual schema name
+
+ DBMS_XSTREAM_ADM.CREATE_OUTBOUND(
+ server_name => 'dbzxout',
+ table_names => tables,
+ schema_names => schemas
+ );
+END;
+/
+
+PROMPT XStream outbound server created
+
+-- Step 2: Configure XStream User to connect to outbound server
+-- This must be run as sys user
+CONNECT sys/ AS SYSDBA
+
+BEGIN
+ DBMS_XSTREAM_ADM.ALTER_OUTBOUND(
+ server_name => 'dbzxout',
+ connect_user => 'dbzxsuser'
+ );
+END;
+/
+
+PROMPT Connect user configured
+
+-- Verification: Check outbound server was created
+SELECT server_name, source_database, capture_name, capture_user, connect_user, queue_owner, queue_name
+FROM dba_xstream_outbound;
+
+-- Verification: Check capture process
+SELECT capture_name, status, capture_type, source_database
+FROM dba_capture;
+
+PROMPT
+PROMPT =====================================================
+PROMPT XStream Outbound Server Created Successfully!
+PROMPT =====================================================
+PROMPT Server name: dbzxout
+PROMPT =====================================================
+```
+
+{{< /multitabs >}}
+
+### 7. Add a custom Docker image for the Debezium server
+
+To support XStream connector, you must create a custom [Docker](https://www.docker.com/) image that includes the required Instant Client package libraries for Linux x64 from the Oracle website.
+
+1. On the Docker machine, download the Instant Client package for Linux x64 from the Oracle website
+
+ ```bash
+ wget https://download.oracle.com/otn_software/linux/instantclient/2380000/instantclient-basic-linux.x64-23.8.0.25.04.zip
+ ```
+
+1. Unzip it to the `./dbz-ora` directory:
+
+ ```bash
+ unzip instantclient-basic-linux.x64-23.8.0.25.04.zip -d ./dbz-ora
+ ```
+
+1. Create a `Dockerfile` in the `./dbz-ora` directory with the following contents:
+
+ ```docker
+ FROM debezium/server:3.0.8.Final
+
+ USER root
+
+ RUN microdnf -y install libaio \
+ && microdnf clean all \
+ && mkdir -p /opt/oracle/instant_client \
+ && rm -f /debezium/lib/ojdbc11*.jar
+
+ COPY instantclient_23_8/* /opt/oracle/instant_client
+
+ USER jboss
+
+ COPY instantclient_23_8/xstreams.jar /debezium/lib
+ COPY instantclient_23_8/ojdbc11.jar /debezium/lib
+
+ ENV LD_LIBRARY_PATH=/opt/oracle/instant_client
+ ```
+
+1. Create the custom image:
+
+ ```bash
+ docker build -t dbz-ora dbz-ora
+ ```
+
+1. Add the image to the K3s image registry using the following commands:
+
+ ```bash
+ docker tag dbz-ora quay.io/debezium/server:3.0.8.Final
+ docker image save quay.io/debezium/server:3.0.8.Final -o dbz3.0.8-xstream-linux-amd.tar
+ sudo k3s ctr images import dbz3.0.8-xstream-linux-amd.tar all
+ ```
+
+### 8. Make RDI use the custom image (VM installation)
+
+Edit the `rdi-operator` configmap:
+
+```bash
+kubectl edit configmap rdi-operator -n rdi
+```
+
+In the editor, find the collector section and change the image settings:
+
+```yaml
+ collector:
+ image:
+ pullPolicy: IfNotPresent
+ registry: docker.io # change this to `quay.io`
+ repository: redislabs/debezium-server # Change this to `debezium/server`
+ tag: 3.0.8.Final-rdi.1 # Change this to `3.0.8.Final`
+```
+
+Save the configmap. Once it is saved, the operator will restart automatically and will apply the changes.
+
+{{< note >}}After upgrading to another RDI version,
+the changes to the configmap will be lost. You must repeat the above steps after each upgrade.
+{{< /note >}}
+
+### 9. Enable the Oracle configuration in RDI
+
+Finally, you must update your `config.yaml` file to enable XStream.
+The example below shows the relevant parts of the `sources` section:
+
+```yaml
+sources:
+ oracle:
+ type: cdc
+ logging:
+ level: info
+ connection:
+ type: oracle
+ host: host.docker.internal
+ port: 1521
+ user: ${SOURCE_DB_USERNAME}
+ password: ${SOURCE_DB_PASSWORD}
+ advanced:
+ source:
+ database.dbname: ORCLCDB
+ database.pdb.name: ORCLPDB1
+ database.connection.adapter: xstream
+ database.out.server.name: dbzxout
+```
+
+{{< note >}}The values `ORCLCDB` and `ORCLPDB1` in the example above are sample CDB and PDB names. Set `database.dbname` and `database.pdb.name` to the CDB and PDB names for your own Oracle database.{{< /note >}}
+
+See the
+[Debezium Oracle documentation](https://debezium.io/documentation/reference/stable/connectors/oracle.html#oracle-connector-properties)
+for a full list of properties you can use in the `advanced.source` subsection.
+
+### 10. Configuration is complete {#xstream-complete}
+
+After you have followed the steps above, your Oracle database is ready
+for Debezium to use.
+
+## Support for Oracle XMLTYPE columns (optional)
+
+If your Oracle database contains tables with columns of type
+[`XMLTYPE`](https://docs.oracle.com/en/database/oracle/oracle-database/21/arpls/XMLTYPE.html),
+you must configure additional libraries for Debezium Server to process these columns correctly.
+
+### Create a custom Debezium Server image
+
+To support `XMLTYPE` columns, you must create a custom [Docker](https://www.docker.com/) image
+that includes the required Oracle XML libraries.
+
+1. Download the required libraries from Maven Central:
+
+ ```bash
+ mkdir xml
+ cd xml
+ wget https://repo.maven.apache.org/maven2/com/oracle/database/xml/xdb/19.27.0.0/xdb-19.27.0.0.jar
+ wget https://repo.maven.apache.org/maven2/com/oracle/database/xml/xmlparserv2/19.27.0.0/xmlparserv2-19.27.0.0.jar
+ mv xdb-19.27.0.0.jar xdb.jar
+ mv xmlparserv2-19.27.0.0.jar xmlparserv2.jar
+ ```
+
+2. Create a `Dockerfile` in the same directory:
+
+ ```dockerfile
+ FROM quay.io/debezium/server:3.0.8.Final
+
+ USER root
+
+ COPY xdb.jar /debezium/lib
+ COPY xmlparserv2.jar /debezium/lib
+ ```
+
+3. Build the custom image:
+
+ ```bash
+ cd ..
+ docker build -t dbz-xml xml
+ docker tag dbz-xml quay.io/debezium/server:3.0.8.Final
+ docker image save quay.io/debezium/server:3.0.8.Final -o dbz3.0.8-xml-linux-amd.tar
+ ```
+
+4. Add the image to your K3s image repository:
+
+ ```bash
+ sudo k3s ctr images import dbz3.0.8-xml-linux-amd.tar all
+ ```
+
+### Configure RDI for XMLTYPE support
+
+In your RDI configuration file, set the `lob.enabled` property to `true` in the
+`advanced.source` section:
+
+```yaml
+sources:
+ oracle:
+ type: cdc
+ logging:
+ level: info
+ connection:
+ type: oracle
+ host: oracle
+ port: 1521
+ user: ${SOURCE_DB_USERNAME}
+ password: ${SOURCE_DB_PASSWORD}
+ database: ORCLCDB
+ advanced:
+ source:
+ database.pdb.name: ORCLPDB1
+ lob.enabled: true
+```
+
+{{< note >}}The XMLTYPE configuration example uses `ORCLCDB` as the CDB name and `ORCLPDB1` as the PDB name. Replace these with your actual CDB and PDB names when configuring XMLTYPE support.{{< /note >}}
+
+### Test XMLTYPE support
+
+You can create a test table to verify that `XMLTYPE` columns work correctly
+(using the
+[`CHINOOK`](https://github.com/Redislabs-Solution-Architects/rdi-quickstart-postgres/tree/main)
+schema as an example):
+
+```sql
+CREATE TABLE tab1 (
+ xmlid INT NOT NULL,
+ col1 SYS.XMLTYPE,
+ CONSTRAINT PK_tab1 PRIMARY KEY (xmlid)
+);
+
+DECLARE
+ v_xml SYS.XMLTYPE;
+ v_doc CLOB;
+BEGIN
+ -- XMLTYPE created from a CLOB
+ v_doc := '' || Chr(10) || ' MY_TABLE ';
+ v_xml := SYS.XMLTYPE.createXML(v_doc);
+
+ INSERT INTO tab1 (xmlid, col1) VALUES (1, v_xml);
+
+ -- XMLTYPE created from a query
+ SELECT SYS_XMLGEN(table_name)
+ INTO v_xml
+ FROM user_tables
+ WHERE rownum = 1;
+
+ INSERT INTO tab1 (xmlid, col1) VALUES (2, v_xml);
+
+ COMMIT;
+END;
+/
+
+ALTER TABLE CHINOOK.TAB1 ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
+```
+
+After you run an initial
+[snapshot]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/#pipeline-lifecycle" >}}),
+the XML data appears in your Redis target database:
+
+{{< image filename="/images/rdi/ingest/xmltype-example.webp" >}}
+
+## FAQ
+
+### How does CDB differ from Non-CDB?
+
+**CDB (Container Database – multitenant)**
+
+- Root container `CDB$ROOT` plus one or more pluggable databases (PDBs).
+- Common users (for example `C##DBZADMIN`, `C##DBZXSUSER`) exist across all containers.
+- Application data lives in PDBs.
+- Many XStream-related operations use `CONTAINER=ALL` and `ALTER SESSION SET CONTAINER`.
+
+**Non-CDB (traditional)**
+
+- Single database, no containers.
+- Users are regular database users without the `C##` prefix (for example `DBZADMIN`, `DBZXSUSER`).
+- Application data lives directly in that single database.
+- No `CONTAINER=ALL` or `ALTER SESSION SET CONTAINER` clauses are required.
+
+From an RDI/Debezium point of view, both models are supported; the main differences are in the user names you create and whether you need container-related clauses in the SQL.
+
+### How does single-instance differ from RAC?
+
+**Single-instance**
+
+- One Oracle instance running on one server.
+- You run the LogMiner/XStream setup scripts once.
+- All configuration (users, tablespaces, outbound server, logging) is stored in that database instance.
+
+**RAC (Real Application Clusters)**
+
+- Multiple instances (nodes) share the same database and storage.
+- You still run the XStream setup only once on any one node.
+- All nodes share the same:
+ - Database files
+ - Archive logs
+ - XStream outbound server
+ - Capture process
+
+Important RAC points:
+
+- Archive log mode is enabled at the database level and applies to all nodes.
+- Tablespaces you create are in shared storage and visible from all nodes.
+- Users you create exist in the shared database and can connect through any node.
+- Debezium/RDI can connect to any RAC node or to a SCAN address; it does not need a specific node.
+
+### In RAC setups, do you need to execute steps on all nodes?
+
+No. The XStream configuration is database-level, so you execute the setup once on any node:
+
+- Enable archive log mode: once (affects the whole database).
+- Create XStream tablespaces: once (on shared storage).
+- Create XStream users: once (in the shared database).
+- Create the XStream outbound server: once (visible from all nodes).
+- Enable supplemental logging: once (database-wide).
+
+You may run client connection tests from multiple nodes, but the actual administrative steps only need to be executed once.
+
+### Why are two XStream tablespaces required?
+
+XStream is not just reading your tables - it's storing its own data:
+
+**Customer's existing tablespace**:
+
+```mermaid
+graph TB
+ USERS["USERS XSTREAM_ADM_TBS XSTREAM_TBS , , READY
+```
+
+#### Size Comparison
+
+Customer's Application Tablespaces:
+
+```sql
+-- Typical size: Large
+SELECT tablespace_name,
+ ROUND(SUM(bytes)/1024/1024/1024, 2) AS size_gb
+FROM dba_data_files
+WHERE tablespace_name = 'USERS'
+GROUP BY tablespace_name;
+
+-- Result: 50 GB, 100 GB, 500 GB, etc.
+```
+
+XStream Tablespaces:
+
+```sql
+-- Typical size: Small
+SELECT tablespace_name,
+ ROUND(SUM(bytes)/1024/1024, 2) AS size_mb
+FROM dba_data_files
+WHERE tablespace_name IN ('XSTREAM_ADM_TBS', 'XSTREAM_TBS')
+GROUP BY tablespace_name;
+
+-- Result:
+-- XSTREAM_ADM_TBS: 100-500 MB
+-- XSTREAM_TBS: 500 MB - 2 GB (depends on queue size)
+```
+
+#### General Recommendations
+
+This guide uses two separate tablespaces:
+
+- `xstream_adm_tbs` – XStream administrator tablespace.
+- `xstream_tbs` – XStream user (connect) tablespace.
+
+They serve different purposes and follow the principle of least privilege.
+
+**1. Separation of concerns and security**
+
+- *Administrator tablespace (`xstream_adm_tbs`)*
+ - Used by the XStream admin user (for example `C##DBZADMIN` or `DBZADMIN`).
+ - Stores XStream metadata and control structures:
+ - Capture process metadata
+ - Outbound server configuration
+ - Queue tables and queues
+ - Other XStream internal objects
+ - Belongs to a highly privileged user that can create and manage XStream objects.
+
+- *User tablespace (`xstream_tbs`)*
+ - Used by the XStream connect user (for example `C##DBZXSUSER` or `DBZXSUSER`).
+ - Lower-privileged account used by Debezium/RDI to read changes.
+ - Keeps operational data and temporary objects separate from admin metadata.
+
+**2. Resource management**
+
+Having two tablespaces lets you:
+
+- Monitor space usage separately for admin vs user workloads.
+- Allow for different growth patterns (admin metadata typically grows more slowly).
+- Apply separate quotas per user/tablespace.
+- Tune storage parameters differently for each tablespace if needed.
+
+**3. Operational benefits**
+
+- Backup and recovery:
+ - You can back up or restore admin and user data independently.
+ - Critical XStream metadata is isolated in its own tablespace.
+- Maintenance and troubleshooting:
+ - You can perform maintenance on one tablespace without affecting the other.
+ - Easier to diagnose which part of the XStream setup is consuming space.
+- Security and auditing:
+ - Clear separation of administrative vs operational data.
+ - Easier to audit which users access which objects.
+
+**4. Can you use a single tablespace?**
+
+Technically you could place both users in a single tablespace, but this is not recommended for production:
+
+- It weakens separation of duties and least-privilege design.
+- It makes monitoring and capacity planning harder.
+- It mixes admin metadata and application-facing data in one place.
+
+Oracle and Debezium best practices recommend separating admin and user workloads into different tablespaces.
+
+### What is the XStream outbound server?
+
+The XStream outbound server is the component that streams database changes out of Oracle in a supported way. It:
+
+- Captures committed changes (INSERT, UPDATE, DELETE) from redo logs.
+- Converts them into logical change records (LCRs).
+- Exposes an API that external consumers such as Debezium/RDI use.
+
+**High-level architecture**
+
+```mermaid {width="100%"}
+graph LR
+ RedoLogs["Redo Logs "]
+ Capture["Capture Process "]
+ Queue["Queue "]
+ Outbound["XStream Outbound "]
+ Consumer["Debezium/RDI "]
+
+ RedoLogs --> Capture
+ Capture --> Queue
+ Queue --> Outbound
+ Outbound --> Consumer
+```
+
+**Key components**
+
+- *Redo logs*: Contain the physical change records for the database.
+- *Capture process*: Mines redo logs and turns them into logical change records.
+- *Queue*: Temporarily stores captured changes.
+- *XStream outbound server* (for example `DBZXOUT`):
+ - Provides the streaming interface for consumers.
+ - Applies filters (schemas/tables) configured when you create it.
+- *Connect user* (for example `C##DBZXSUSER` or `DBZXSUSER`):
+ - The user Debezium/RDI connects as.
+ - Reads changes from the outbound server.
+
+**Why it is needed**
+
+Without an XStream outbound server:
+
+- There is no supported, structured API for consuming change events.
+- You would have to parse redo logs directly, which is complex and unsupported.
+- You lose built-in filtering, transaction grouping, and restart/resume semantics.
+
+With an XStream outbound server:
+
+- Changes are available in near real time.
+- You can capture only selected schemas/tables.
+- Debezium/RDI uses Oracle’s supported XStream API.
+- Transactions are preserved and grouped correctly.
+- Offsets/positions allow clean resume after restarts.
+- Multiple consumers can be attached to the same outbound server if needed.
\ No newline at end of file
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md
new file mode 100644
index 0000000000..461f790115
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md
@@ -0,0 +1,285 @@
+---
+Title: Prepare PostgreSQL for RDI
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Prepare PostgreSQL databases to work with RDI
+group: di
+linkTitle: Prepare PostgreSQL
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 2
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql/'
+---
+
+PostgreSQL supports several
+[logical decoding plug-ins](https://wiki.postgresql.org/wiki/Logical_Decoding_Plugins)
+to enable CDC. If you don't want to use the native `pgoutput` logical replication stream support
+then you must install your preferred plug-in into the PostgreSQL server. Once you have done this,
+you must enable a replication slot, and configure a user with privileges to perform the replication.
+
+If you are using a service like [Heroku Postgres](https://www.heroku.com/postgres) to host
+your database then this might restrict the plug-ins you can use. If you can't use your preferred
+plug-in then could try the `pgoutput` decoder if you are using PostgreSQL 10 or above.
+If this doesn't work for you then you won't be able to use RDI with your database.
+
+The following checklist summarizes the steps to prepare a PostgreSQL
+database for RDI, with links to the sections that explain the steps in
+full detail. You may find it helpful to track your progress with the
+checklist as you complete each step.
+
+```checklist {id="postgreslist"}
+- [ ] [Install the logical decoding output plug-in](#install-the-logical-decoding-output-plug-in)
+- [ ] [Configure the PostgreSQL server](#configure-the-postgresql-server)
+- [ ] [Set up permissions](#set-up-permissions)
+- [ ] [Set privileges for Debezium to create PostgreSQL publications with pgoutput](#set-privileges-for-debezium-to-create-postgresql-publications-with-pgoutput)
+- [ ] [Configure PostgreSQL for replication with the Debezium connector host](#configure-postgresql-for-replication-with-the-debezium-connector-host)
+```
+
+## Amazon RDS for PostgreSQL
+
+Follow the steps below to enable CDC with [Amazon RDS for PostgreSQL](https://aws.amazon.com/rds/postgresql/):
+
+```checklist {id="postgresawslist" nointeractive="true" }
+- [ ] [Set the instance parameter rds.logical_replication to 1](#1-set-the-instance-parameter-rdslogicalreplication-to-1)
+- [ ] [Check that the wal_level parameter is set to logical](#2-check-that-the-wal_level-parameter-is-set-to-logical)
+- [ ] [Set the Debezium plugin.name parameter to pgoutput](#3-set-the-debezium-pluginname-parameter-to-pgoutput)
+- [ ] [Initiate logical replication from an AWS account that has the rds_replication role](#4-initiate-logical-replication-from-an-aws-account-that-has-the-rdsreplication-role)
+```
+
+1.
+ ```
+
+ To enable accounts other than the master account to create an initial snapshot, you must grant `SELECT`
+ permission to the accounts on the tables to be captured. See the documentation about
+ [security for PostgreSQL logical replication](https://www.postgresql.org/docs/current/logical-replication-security.html)
+ for more information.
+
+
+## Azure Database for PostgreSQL
+If you are using [Azure Database for PostgreSQL](https://azure.microsoft.com/en-us/services/postgresql/) you need to
+manually set the `wal_level` parameter in the Azure portal for your PostgreSQL server. Go to the `server parameters`
+section, search for `wal_level` and set it to logical. Then save and restart the server.
+
+
+## Install the logical decoding output plug-in
+
+As of PostgreSQL 9.4, the only way to read changes to the write-ahead-log is to
+[install a logical decoding output plug-in](https://debezium.io/documentation/reference/stable/postgres-plugins.html).
+These plug-ins are written in C using PostgreSQL-specific APIs, as described in the
+[PostgreSQL documentation](https://www.postgresql.org/docs/current/logicaldecoding-output-plugin.html).
+The PostgreSQL connector uses one of Debezium’s supported logical decoding
+plug-ins to receive change events from the database in either the default
+[`pgoutput`](https://github.com/postgres/postgres/blob/master/src/backend/replication/pgoutput/pgoutput.c) format (supplied with PostgreSQL) or the
+[`Protobuf`](https://github.com/protocolbuffers/protobuf) format.
+See the
+[decoderbufs Protobuf plug-in documentation](https://github.com/debezium/postgres-decoderbufs)
+for more details about how to compile it and also its requirements and limitations.
+
+For simplicity, Debezium also provides a container image that compiles and installs the plug-ins
+on top of the upstream PostgreSQL server image. Use this image as an example of the steps
+involved in the installation.
+
+{{< note >}} The Debezium logical decoding plug-ins have been tested on Linux machines, but if you are
+using Windows or other operating systems, the installation steps might be different from
+those listed here. {{< /note >}}
+
+### Plug-in differences
+
+Plug-ins don't all behave in exactly the same way. All of them refresh information about
+the database schema when they detect that it has changed, but the `pgoutput` plug-in is
+more "eager" than some other plug-ins to do this. For example, `pgoutput` will refresh
+when it detects a change to the default value of a column but other plug-ins won't
+notice this until another, more significant change happens (such as adding a new table
+column).
+
+The Debezium project maintains a
+[Java class](https://github.com/debezium/debezium/blob/main/debezium-connector-postgres/src/test/java/io/debezium/connector/postgresql/DecoderDifferences.java) that tracks the known differences between plug-ins.
+
+
+## Configure the PostgreSQL server
+
+If you want to use a logical decoding plug-in other than the default `pgoutput` then
+you must first configure it in the `postgresql.conf` file. Set the `shared_preload_libraries`
+parameter to load your plug-in at startup. For example, to load the `decoderbufs`
+plug-in, you would add the following line:
+
+```
+# MODULES
+shared_preload_libraries = 'decoderbufs'
+```
+
+Add the line below to configure the replication slot (for any plug-in).
+This instructs the server to use logical decoding with the write-ahead log.
+
+```
+# REPLICATION
+wal_level = logical
+```
+
+You can also set other PostgreSQL streaming replication parameters if you need them.
+For example, you can use `max_wal_senders` and `max_replication_slots` to increase
+the number of connectors that can access the sending server concurrently,
+and `wal_keep_size` to limit the maximum WAL size that a replication slot retains.
+The
+[configuration parameters](https://www.postgresql.org/docs/current/runtime-config-replication.html#RUNTIME-CONFIG-REPLICATION-SENDER)
+documentation describes all the parameters you can use.
+
+PostgreSQL’s logical decoding uses replication slots. These are guaranteed to retain all the WAL
+segments that Debezium needs even when Debezium suffers an outage. You should monitor replication
+slots carefully to avoid excessive disk consumption and other conditions such as catalog bloat that can arise
+if a replication slot is used infrequently. See the PostgreSQL documentation about
+[replication slots](https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION-SLOTS)
+for more information.
+If you are using a `synchronous_commit` setting other than `on`, then you should set `wal_writer_delay`
+to a value of about 10 milliseconds to ensure a low latency for change events. If you don't set this then
+the default value of about 200 milliseconds will apply.
+
+{{< note >}}This guide summarizes the operation of the PostgreSQL write-ahead log, but we strongly
+recommend you consult the [PostgreSQL write-ahead log](https://www.postgresql.org/docs/current/wal-configuration.html)
+documentation to get a better understanding.{{< /note >}}
+
+## Set up permissions
+
+The Debezium connector needs a database user that has the REPLICATION and LOGIN roles so that it
+can perform replications. By default, a superuser has these roles but for security reasons, you
+should give the minimum necessary permissions to the Debezium user rather than full superuser
+permissions.
+
+If you have administrator privileges then you can create a role for your Debezium user
+using a query like the following. Note that these are the *minimum* permissions the user
+needs to perform replications, but you might also need to grant other permissions.
+
+```sql
+CREATE ROLE REPLICATION LOGIN;
+```
+
+## Set privileges for Debezium to create PostgreSQL publications with `pgoutput`
+
+The Debezium user needs specific permissions to work with the `pgoutput` plug-in.
+The plug-in captures change events from the
+[*publications*](https://www.postgresql.org/docs/current/logical-replication-publication.html)
+that PostgreSQL produces for your chosen source tables. A publication contains change events from
+one or more tables that are filtered using criteria from a *publication specification*.
+
+If you have administrator privileges, you can create the publication specification
+manually or you can grant the Debezium user the privileges to create the specification
+automatically. The required privileges are:
+
+- Replication privileges in the database to add the table to a publication.
+- `CREATE` privileges on the database to add publications.
+- `SELECT` privileges on the tables to copy the initial table data. Table owners
+ automatically have `SELECT` permission for the table.
+
+To add a table to a publication, the user must be an owner of the table. However, in
+this case, the source table already exists, so you must use a PostgreSQL replication
+group to share ownership between the Debezium user and the original owner. Configure
+the replication group using the following commands:
+
+```checklist {id="postgrespgoutputlist" nointeractive="true" }
+- [ ] [Create the replication group](#1-create-the-replication-group)
+- [ ] [Add the original owner of the table to the group](#2-add-the-original-owner-of-the-table-to-the-group)
+- [ ] [Add the Debezium replication user to the group](#3-add-the-debezium-replication-user-to-the-group)
+- [ ] [Transfer ownership of the table to the replication group](#4-transfer-ownership-of-the-table-to-the-replication-group)
+```
+
+1. ` locally,
+on the server machine, you would add a line like the following:
+
+```
+local replication trust
+```
+
+To allow `` on localhost to receive replication changes using IPV4,
+add the line:
+
+```
+host replication 127.0.0.1/32 trust
+```
+
+To allow `` on localhost to receive replication changes using IPV6,
+add the line:
+
+```
+host replication ::1/128 trust
+```
+
+Find out more from the PostgreSQL pages about
+[`pg_hba.conf`](https://www.postgresql.org/docs/10/auth-pg-hba-conf.html)
+and
+[network address types](https://www.postgresql.org/docs/current/datatype-net-types.html).
+
+## Supported PostgreSQL topologies
+
+You can use the Debezium PostgreSQL connector with a standalone PostgreSQL server or
+with a cluster of servers.
+For versions 12 and below, PostgreSQL supports logical replication slots on only primary servers.
+This means that Debezium can only connect to a primary server for CDC and the connection will
+stop if this server fails. If the same server is promoted to primary when service resumes
+then you can simply restart the Debezium connector. However, if a different server is
+promoted to primary, then you must reconfigure Debezium to use the new server
+before restarting. Also, make sure the new server has the correct plug-in and configuration
+for Debezium.
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/snowflake.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/snowflake.md
new file mode 100644
index 0000000000..51a239f651
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/snowflake.md
@@ -0,0 +1,389 @@
+---
+Title: Prepare Snowflake for RDI
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Prepare Snowflake databases to work with RDI
+group: di
+linkTitle: Prepare Snowflake
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+bannerText: Snowflake source support for Redis Data Integration is currently in private preview. Features and behavior are subject to change. General private preview terms apply.
+type: integration
+weight: 20
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/snowflake/'
+---
+
+This guide describes the steps required to prepare a Snowflake database as a source for Redis Data Integration (RDI) pipelines.
+
+During both the [snapshot]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines#pipeline-lifecycle" >}}) and
+[Change data capture (CDC)]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines#pipeline-lifecycle" >}})
+phases, RDI uses [Snowflake Streams](https://docs.snowflake.com/en/user-guide/streams) to read data from the monitored
+tables. For the initial snapshot, RDI creates the stream with `SHOW_INITIAL_ROWS = TRUE` so it can read the current
+table contents before continuing with ongoing CDC. RDI automatically creates and manages the required streams.
+
+## Setup
+
+The following checklist shows the steps to prepare a Snowflake database for RDI,
+with links to the sections that explain the steps in full detail.
+You may find it helpful to track your progress with the checklist as you
+complete each step.
+
+{{< note >}}
+Snowflake is only supported with RDI deployed on Kubernetes/Helm. RDI VM mode does not support Snowflake as a source database.
+{{< /note >}}
+
+```checklist {id="snowflakelist"}
+- [ ] [Set up Snowflake permissions](#1-set-up-snowflake-permissions)
+- [ ] [Configure authentication](#2-configure-authentication)
+- [ ] [Set up secrets for Kubernetes deployment](#3-set-up-secrets-for-kubernetes-deployment)
+- [ ] [Configure RDI for Snowflake](#4-configure-rdi-for-snowflake)
+```
+
+## 1. Set up Snowflake permissions
+
+The following are the minimum runtime permissions for the RDI role to read the source tables and create the Snowflake
+objects RDI uses for CDC:
+
+- `USAGE`, `OPERATE` on the warehouse used for RDI reads
+- `USAGE` on the source database and source schema
+- `SELECT` on the source tables
+- `USAGE` on the CDC schema used by RDI
+- `CREATE STREAM`, `CREATE TABLE` on the CDC schema used by RDI
+
+If you configure `cdcDatabase` and `cdcSchema`, grant the CDC permissions there. Otherwise, grant them in the source
+schema. If your Snowflake setup requires it, also grant any additional cross-database privileges needed for the CDC
+schema to reference the source tables.
+
+{{< note >}}
+RDI manages the Snowflake streams it uses for snapshot and CDC. The collector creates the stream in the configured CDC
+schema and later issues `CREATE OR REPLACE STREAM` statements to keep the stream aligned with the expected offset, so
+the RDI role must be able to create and own those stream objects in the CDC schema.
+
+There is one stricter bootstrap requirement for the first stream created on a source table: if Snowflake change
+tracking is not already enabled on that table, only the table owner can create that initial stream. If the source
+tables are not owned by the RDI role, ask a Snowflake administrator or table owner to enable change tracking first:
+
+```sql
+ALTER TABLE MYDB.PUBLIC.customers SET CHANGE_TRACKING = TRUE;
+ALTER TABLE MYDB.PUBLIC.orders SET CHANGE_TRACKING = TRUE;
+```
+{{< /note >}}
+
+Grant the required permissions to your RDI user:
+
+```sql
+-- Grant usage on the warehouse
+GRANT USAGE, OPERATE ON WAREHOUSE COMPUTE_WH TO ROLE rdi_role;
+
+-- Grant usage on the source database and schema
+GRANT USAGE ON DATABASE MYDB TO ROLE rdi_role;
+GRANT USAGE ON SCHEMA MYDB.PUBLIC TO ROLE rdi_role;
+
+-- Grant SELECT on tables to capture
+GRANT SELECT ON TABLE MYDB.PUBLIC.customers TO ROLE rdi_role;
+GRANT SELECT ON TABLE MYDB.PUBLIC.orders TO ROLE rdi_role;
+
+-- Grant permissions on the schema RDI uses for CDC objects
+GRANT USAGE ON SCHEMA MYDB.RDI_CDC TO ROLE rdi_role;
+GRANT CREATE STREAM, CREATE TABLE ON SCHEMA MYDB.RDI_CDC TO ROLE rdi_role;
+
+-- Assign the role to your RDI user
+GRANT ROLE rdi_role TO USER rdi_user;
+```
+
+If you use centralized grant management, you can also add future grants in the CDC schema so newly created tables and
+streams automatically receive the desired privileges. These grants are optional and are not part of the minimum runtime
+permissions:
+
+```sql
+GRANT SELECT ON FUTURE TABLES IN SCHEMA MYDB.RDI_CDC TO ROLE rdi_role;
+GRANT SELECT ON FUTURE STREAMS IN SCHEMA MYDB.RDI_CDC TO ROLE rdi_role;
+```
+
+## 2. Configure authentication
+
+RDI supports two authentication methods for Snowflake. You must configure one of these methods.
+
+### Password authentication
+
+Use standard username and password credentials. Store these securely using Kubernetes secrets (see step 3).
+
+{{< note >}}
+Many Snowflake accounts require MFA for password-based sign-ins. If you want to use password authentication for RDI,
+configure the Snowflake user as a service user that is allowed to authenticate non-interactively. Otherwise, use
+private key authentication instead. For more information, see the Snowflake
+[MFA rollout documentation](https://docs.snowflake.com/en/user-guide/security-mfa-rollout).
+{{< /note >}}
+
+### Private key authentication
+
+For enhanced security, use key-pair authentication:
+
+1. Generate a private key:
+
+ ```bash
+ openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt
+ ```
+
+1. Generate the public key:
+
+ ```bash
+ openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub
+ ```
+
+1. Register the public key with your Snowflake user:
+
+ ```sql
+ ALTER USER rdi_user SET RSA_PUBLIC_KEY='';
+ ```
+
+## 3. Set up secrets for Kubernetes deployment
+
+Before deploying the RDI pipeline, configure the necessary secrets.
+
+### Password authentication
+
+```bash
+kubectl create secret generic source-db \
+ --namespace=rdi \
+ --from-literal=SOURCE_DB_USERNAME=your_username \
+ --from-literal=SOURCE_DB_PASSWORD=your_password
+```
+
+### Private key authentication
+
+Create a secret with the private key file:
+
+```bash
+kubectl create secret generic source-db-ssl \
+ --namespace=rdi \
+ --from-file=client.key=/path/to/rsa_key.p8
+```
+
+Also create the source-db secret with the username:
+
+```bash
+kubectl create secret generic source-db \
+ --namespace=rdi \
+ --from-literal=SOURCE_DB_USERNAME=your_username
+```
+
+## 4. Configure RDI for Snowflake
+
+Use the following example configuration in your `config.yaml` file:
+
+```yaml
+sources:
+ snowflake:
+ type: riotx
+ connection:
+ type: snowflake
+ url: "jdbc:snowflake://myaccount.snowflakecomputing.com/"
+ user: "${SOURCE_DB_USERNAME}"
+ password: "${SOURCE_DB_PASSWORD}" # Omit for key-pair auth
+ database: "MYDB"
+ warehouse: "COMPUTE_WH"
+ # role: "RDI_ROLE" # Optional: Snowflake role
+ # cdcDatabase: "CDC_DB" # Optional: Separate database for CDC streams
+ # cdcSchema: "CDC_SCHEMA" # Optional: Separate schema for CDC streams
+ schemas:
+ - PUBLIC
+ tables:
+ PUBLIC.customers: {}
+ PUBLIC.orders: {}
+ advanced:
+ riotx:
+ poll: "30s"
+ snapshot: "INITIAL" # Or "NEVER" to skip initial snapshot
+ # streamPrefix: "data:" # Optional: Redis stream prefix
+ # streamLimit: 100000 # Optional: Max stream length
+ # keyColumns: # Recommended: stable key columns
+ # - "id"
+ # clearOffset: false # Optional: Clear offset on start
+
+targets:
+ target:
+ connection:
+ type: redis
+ host: ${TARGET_DB_HOST}
+ port: ${TARGET_DB_PORT}
+ user: ${TARGET_DB_USERNAME}
+ password: ${TARGET_DB_PASSWORD}
+
+processors:
+ target_data_type: json
+```
+
+{{< note >}}
+Snowflake uses one configured `database` and one or more source-level `schemas`. In the `tables` section, specify each
+table as `SCHEMA.table`. Even when you configure only one schema, explicit `SCHEMA.table` names are recommended for
+clarity.
+{{< /note >}}
+
+### Snowflake connection properties
+
+| Property | Type | Required | Description |
+|---------------|--------|----------|----------------------------------------------------------------|
+| `type` | string | Yes | Must be `"snowflake"` |
+| `url` | string | Yes | JDBC URL: `jdbc:snowflake://.snowflakecomputing.com/` |
+| `user` | string | Yes | Snowflake user |
+| `password` | string | No* | Snowflake password |
+| `database` | string | Yes | Snowflake database name |
+| `warehouse` | string | Yes | Snowflake warehouse name |
+| `role` | string | No | Snowflake role name |
+| `cdcDatabase` | string | No | Database for CDC streams (if different from source) |
+| `cdcSchema` | string | No | Schema for CDC streams (if different from source) |
+
+* Either `password` or private key authentication is required. See [Configure authentication](#2-configure-authentication) for details.
+
+### Snowflake source properties
+
+| Property | Type | Required | Description |
+|------------|--------|----------|------------------------------------------------------------------|
+| `schemas` | array | Yes | Schema names to capture from |
+| `tables` | object | Yes | Tables to capture, keyed as `SCHEMA.table` |
+
+### Advanced configuration options
+
+Configure under `sources..advanced.riotx`:
+
+| Property | Type | Default | Description |
+|----------------|---------|-------------|----------------------------------------------|
+| `poll` | string | `"30s"` | Polling interval for stream changes |
+| `snapshot` | string | `"INITIAL"` | Snapshot mode: `INITIAL` or `NEVER` |
+| `streamPrefix` | string | `"data:"` | Prefix for the Redis stream written by RDI |
+| `streamLimit` | integer | - | Maximum stream length (XTRIM MAXLEN) |
+| `keyColumns` | array | - | Stable source columns to use as message keys |
+| `clearOffset` | boolean | `false` | Clear existing offset on start |
+| `count` | integer | `0` | Limit records per poll (0 = unlimited) |
+
+For reliable update and delete handling, define `keyColumns` with a stable business key or surrogate key when possible.
+
+## Troubleshooting
+
+### Connection issues
+
+**Error: "Failed to connect to Snowflake"**
+
+- Verify the account URL is correct (format: `.snowflakecomputing.com`)
+- Check network connectivity to Snowflake
+- Verify the warehouse is running and accessible
+- Check firewall rules allow outbound HTTPS (port 443)
+
+**Error: "Authentication failed"**
+
+- For password auth: verify username and password are correct
+- For key-pair auth: verify the private key matches the public key registered in Snowflake
+- Ensure the user has appropriate permissions
+
+**Error: "Warehouse not found"**
+
+- Verify the warehouse name is correct
+- Ensure the user has USAGE permission on the warehouse
+
+**Error: "Network policy is required"**
+
+If the collector logs show an error like the following:
+
+```
+Failed to open new session for user: USERNAME, host: .snowflakecomputing.com. Error: Fail : Network policy is required.
+Failed to initialize pool: Fail : Network policy is required.
+```
+
+Your Snowflake account enforces a network policy, so you must whitelist the RDI egress IP addresses in Snowflake.
+
+The following example creates a network policy and applies it to the RDI user. Replace the example IP addresses with your actual RDI egress IPs and replace `"USERNAME"` with your RDI user:
+
+```sql
+USE ROLE SECURITYADMIN;
+
+CREATE NETWORK POLICY rdi_policy
+ ALLOWED_IP_LIST = (
+ '203.0.113.10/32',
+ '203.0.113.20/32',
+ '198.51.100.30/32',
+ '198.51.100.40/32'
+ );
+
+ALTER USER "USERNAME"
+SET NETWORK_POLICY = rdi_policy;
+```
+
+### CDC issues
+
+**No data appearing in Redis**
+
+1. Verify Snowflake Streams exist in the CDC schema:
+
+ ```sql
+ SHOW STREAMS IN SCHEMA my_cdc_database.my_cdc_schema;
+ ```
+
+1. Check the polling interval configuration
+1. Verify Redis connection is working
+1. Check the collector logs:
+
+ ```bash
+ kubectl get deployments -n rdi | grep riotx-collector
+ kubectl logs -n rdi deployment/
+ ```
+
+**Stale or missing changes**
+
+- Snowflake Streams depend on Snowflake change tracking and retention settings
+- If the collector was offline longer than the available retention window, changes may be lost
+- Consider using `clearOffset: true` to restart from current state
+
+### Performance tuning
+
+**High Snowflake warehouse usage**
+
+- Increase `poll` interval (e.g., `"60s"` or `"120s"`)
+- Use a dedicated warehouse for CDC operations
+- Each poll first calls Snowflake's `SYSTEM$STREAM_HAS_DATA` function to check whether the stream has new data. This
+ check does not start the warehouse; warehouse compute starts only when RDI reads rows from the stream.
+
+**Redis memory concerns**
+
+- Set `streamLimit` to cap stream length
+- Use `count` to limit records per poll batch
+
+**Initial snapshot too slow**
+
+- Use `snapshot: "NEVER"` to skip initial snapshot
+- Pre-load data using other methods if needed
+
+### Enable debug logging
+
+Enable debug logging in the source configuration:
+
+```yaml
+sources:
+ snowflake:
+ type: riotx
+ logging:
+ level: debug
+ # ... rest of configuration
+```
+
+View collector logs:
+
+```bash
+kubectl get deployments -n rdi | grep riotx-collector
+kubectl logs -n rdi deployment/ -f
+```
+
+## 5. Configuration is complete
+
+Once you have followed the steps above, your Snowflake database is ready for RDI to use.
+
+## See also
+
+- [Snowflake Streams Documentation](https://docs.snowflake.com/en/user-guide/streams)
+- [Snowflake Key Pair Authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth)
+- [Snowflake MFA rollout documentation](https://docs.snowflake.com/en/user-guide/security-mfa-rollout)
+- [RDI Deployment Guide]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/deploy" >}})
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/spanner.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/spanner.md
new file mode 100644
index 0000000000..f68acc9156
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/spanner.md
@@ -0,0 +1,249 @@
+---
+Title: Prepare Spanner for RDI
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Prepare Google Cloud Spanner databases to work with RDI
+group: di
+linkTitle: Prepare Spanner
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 2
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/spanner/'
+---
+
+Google Cloud Spanner requires specific configuration to enable change data capture (CDC) with RDI.
+RDI operates in two phases with Spanner: snapshot (initial sync) and streaming. During the snapshot
+phase, RDI uses the JDBC driver to connect directly to Spanner and read the current state of the
+database. In the streaming phase, RDI uses [Spanner's Change Streams](https://cloud.google.com/spanner/docs/change-streams) to capture changes related to
+the monitored schemas and tables.
+
+{{< note >}}
+Spanner is only supported with RDI deployed on Kubernetes/Helm. RDI VM mode does not support Spanner as a source database.
+{{< /note >}}
+
+The following checklist summarizes the steps to prepare a Spanner
+database for RDI, with links to the sections that explain the steps in
+full detail. You may find it helpful to track your progress with the
+checklist as you complete each step.
+
+```checklist {id="spannerlist"}
+- [ ] [Prepare for snapshot](#1-prepare-for-snapshot)
+- [ ] [Prepare for streaming](#2-prepare-for-streaming)
+- [ ] [Create a service account](#3-create-a-service-account)
+- [ ] [Set up secrets for Kubernetes deployment (optional)](#4-set-up-secrets-for-kubernetes-deployment-optional)
+- [ ] [Configure RDI for Spanner](#5-configure-rdi-for-spanner)
+- [ ] [Additional Kubernetes configuration](#6-additional-kubernetes-configuration)
+```
+
+## 1. Prepare for snapshot
+
+During the snapshot phase, RDI executes multiple transactions to capture data at an exact point
+in time that remains consistent across all queries. This is achieved using a Spanner feature called
+[Timestamp bounds with exact staleness](https://cloud.google.com/spanner/docs/timestamp-bounds#exact_staleness).
+
+This feature relies on the
+[version_retention_period](https://cloud.google.com/spanner/docs/reference/rest/v1/projects.instances.databases#Database.FIELDS.version_retention_period),
+which is set to one hour by default. Depending on the database tier, the volume of data to be
+ingested into RDI, and the load on the database, this setting may need to be increased. You can
+update it using [this method](https://cloud.google.com/spanner/docs/use-pitr#set-period).
+
+## 2. Prepare for streaming
+
+To enable streaming, you must create a change stream in Spanner at the database level. Use the
+option `value_capture_type = 'NEW_ROW_AND_OLD_VALUES'` to capture both the previous and updated
+row values.
+
+Be sure to specify only the tables you want to ingest from and, optionally, the specific columns
+you're interested in. Here's an example using Google SQL syntax:
+
+```sql
+CREATE CHANGE STREAM change_stream_table1_and_table2
+ FOR table1, table2
+ OPTIONS (
+ value_capture_type = 'NEW_ROW_AND_OLD_VALUES'
+ );
+```
+
+Refer to the [official documentation](https://cloud.google.com/spanner/docs/change-streams/manage#googlesql)
+for more details, including additional configuration options and dialect-specific syntax.
+
+## 3. Create a service account
+
+To allow RDI to access the Spanner instance, you'll need to create a service account with the
+appropriate permissions. By default, RDI uses Google Cloud Workload Identity authentication. In this case RDI will assume the [service account is assigned to the GKE cluster](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#enable_on_clusters_and_node_pools). Alternatively, you can provide the
+service account credentials as a Kubernetes secret (see step 4 for details).
+
+```checklist {id="spanner-service-account" nointeractive="true" }
+- [ ] [Create the service account](#create-the-service-account)
+- [ ] [Grant required roles](#grant-required-roles)
+- [ ] [Download the service account key](#download-the-service-account-key)
+```
+
+1. /:
+ env:
+ - name: SQL_HOST
+ value: .database.windows.net
+ - name: SQL_DATABASE
+ value:
+ - name: SCAN_INTERVAL_MS
+ value: "2000"
+ envFrom:
+ - secretRef:
+ name:
+```
+
+The secret referenced by `envFrom` must provide the credentials of the
+`db_owner` identity created for the scan worker — not the RDI source
+credentials.
+
+##### Verifying the workaround
+
+After the worker has been running for a few minutes, confirm that the
+effective scan cadence has dropped to the worker's interval by querying the
+`sys.dm_cdc_log_scan_sessions` dynamic management view. This DMV records both
+the automatic scheduler's scans and the worker's manual scans, so the gap
+between successive `start_time` values should now match the worker's interval:
+
+```sql
+-- Recent CDC log-scan sessions (manual and automatic combined)
+SELECT TOP (10)
+ session_id, start_time, end_time, duration, scan_phase,
+ latency, tran_count, last_commit_cdc_time
+FROM sys.dm_cdc_log_scan_sessions
+WHERE session_id > 0
+ORDER BY session_id DESC
+```
+
+To check the commit time of the latest change captured for a specific table,
+map the highest captured LSN back to a time using `sys.fn_cdc_map_lsn_to_time`:
+
+```sql
+-- Replace with the capture instance name shown by
+-- sys.sp_cdc_help_change_data_capture
+SELECT sys.fn_cdc_map_lsn_to_time(MAX(__$start_lsn)) AS latest_captured_commit_time
+FROM cdc._CT
+```
+
+The difference between that value and the current time is an upper bound on
+how stale the captured stream is for that table.
+
+You can also confirm that end-to-end change propagation through RDI now meets
+your latency target by measuring ` → `
+on a representative table.
+
+#### Azure SQL Managed Instance
+
+Follow the on-premises instructions for
+[enabling CDC on the database](#2-enable-cdc-on-the-database) and
+[enabling CDC on the tables you want to capture](#3-enable-cdc-for-the-tables-you-want-to-capture).
+The procedures are identical on Managed Instance.
+
+### Create a database user for Debezium
+
+RDI only reads from the source database, so the Debezium user only needs read access.
+Do not grant `db_datawriter` or any other write permissions.
+
+You can authenticate to Azure SQL using either SQL authentication or Microsoft Entra ID.
+Microsoft Entra authentication with a service principal is the validated path for RDI;
+use it for production deployments. SQL authentication is supported and is the simplest
+option for development or proof-of-concept setups.
+
+#### Option A: SQL authentication
+
+Follow the on-premises instructions for
+[creating the Debezium user](#1-create-a-debezium-user), with one Azure-specific
+change: on Azure SQL Database, omit the `master`-database step and create a contained
+user in the user database. Connect to your user database as the server admin and run:
+
+```sql
+CREATE USER WITH PASSWORD = ''
+GO
+ALTER ROLE db_datareader ADD MEMBER
+GO
+GRANT VIEW DATABASE STATE TO
+GO
+```
+
+After enabling CDC on the tables you want to capture, add the user to the CDC role:
+
+```sql
+EXEC sp_addrolemember N'', N''
+GO
+```
+
+{{< note >}}Use `VIEW DATABASE STATE` rather than `VIEW SERVER STATE`. The server-scoped
+permission does not exist on Azure SQL Database.{{< /note >}}
+
+#### Option B: Microsoft Entra service principal
+
+1. **Register an application in Microsoft Entra ID.**
+ In the Azure portal, go to **Microsoft Entra ID > App registrations > New registration**.
+ Note the **Application (client) ID** — you'll use it as the RDI `user` value. Create
+ a client secret under **Certificates & secrets** and note its value — you'll use it
+ as the RDI `password` value.
+
+1. **Set a Microsoft Entra admin on the Azure SQL logical server.**
+ In the Azure portal, open the logical SQL server and set a Microsoft Entra admin (a
+ user or group you can sign in as). You will connect as this admin to create the
+ contained user in the next step. (The permission to create contained users mapped
+ to Microsoft Entra principals can also be delegated to other database principals;
+ see Microsoft's [Microsoft Entra authentication for Azure SQL](https://learn.microsoft.com/en-us/azure/azure-sql/database/authentication-aad-overview)
+ documentation.)
+
+1. **Create a contained database user for the service principal.**
+ Connect to the user database as the Microsoft Entra admin (for example, using
+ `sqlcmd -G` or Azure Data Studio) and run:
+
+ ```sql
+ CREATE USER [] FROM EXTERNAL PROVIDER
+ GO
+ ALTER ROLE db_datareader ADD MEMBER []
+ GO
+ GRANT VIEW DATABASE STATE TO []
+ GO
+ ```
+
+ After enabling CDC on the tables you want to capture, add the principal to the CDC role:
+
+ ```sql
+ EXEC sp_addrolemember N'', N''
+ GO
+ ```
+
+ {{< note >}}`` is the **display name** of the app registration — the
+ value shown in the **Name** column on the **App registrations** page — not its client
+ ID. The client ID is used by the RDI connector (see the next section), but the
+ database user must be created from the display name. If the display name is not
+ unique in your Microsoft Entra tenant (display names are not guaranteed unique),
+ disambiguate by adding the `WITH OBJECT_ID = ''` clause to the
+ `CREATE USER` statement.{{< /note >}}
+
+### Configure the RDI source for Azure SQL
+
+Use a `cdc` source with `type: sqlserver`. The example below shows the validated
+configuration for Azure SQL Database with Microsoft Entra service-principal
+authentication:
+
+```yaml
+sources:
+ sqlserver:
+ type: cdc
+ connection:
+ type: sqlserver
+ host: .database.windows.net
+ port: 1433
+ database:
+ user: ${SOURCE_DB_USERNAME}
+ password: ${SOURCE_DB_PASSWORD}
+ logging:
+ level: info
+ schemas:
+ - dbo
+ tables:
+ :
+ columns:
+ -
+ -
+ keys:
+ -
+ advanced:
+ source:
+ driver.authentication: ActiveDirectoryServicePrincipal
+ database.encrypt: "true"
+ database.hostNameInCertificate: "*.database.windows.net"
+ database.trustServerCertificate: "false"
+ database.applicationIntent: ReadOnly
+ snapshot.mode: initial
+```
+
+The properties under `advanced.source` are passed straight through to the underlying
+Debezium SQL Server connector and JDBC driver. The Azure-specific values are:
+
+| Property | Purpose | Value for Azure SQL Database |
+| --- | --- | --- |
+| `driver.authentication` | Selects the JDBC Microsoft Entra authentication mode. | `ActiveDirectoryServicePrincipal` (validated). See [other Microsoft Entra authentication modes](#other-microsoft-entra-authentication-modes) for alternatives. |
+| `database.encrypt` | Enforces TLS on the JDBC connection. | `"true"`. Azure SQL rejects unencrypted connections. |
+| `database.trustServerCertificate` | If `true`, the driver skips certificate validation. | `"false"`. Azure SQL presents a valid certificate; never disable validation in production. |
+| `database.hostNameInCertificate` | Tells the JDBC driver which hostname pattern to expect in the server's TLS certificate. Set explicitly when the certificate's subject does not match the connection hostname directly. | `"*.database.windows.net"` (used in the RDI-validated configuration to match Azure SQL's wildcard certificate). |
+| `database.applicationIntent` | When set to `ReadOnly`, routes the connection to a read-only replica on tiers that support [read scale-out](https://learn.microsoft.com/en-us/azure/azure-sql/database/read-scale-out). | `ReadOnly`. Recommended because RDI only reads. On tiers where Azure SQL read scale-out is available (Business Critical and Hyperscale), this routes the RDI read connection to a read-only replica. On General Purpose, which has no read scale-out, the setting has no effect. |
+| `snapshot.mode` | The Debezium snapshot strategy. | `initial`. Captures a snapshot of the existing rows, then streams subsequent changes from the CDC tables. |
+
+For SQL authentication, omit the `driver.authentication` line and set
+`${SOURCE_DB_USERNAME}` and `${SOURCE_DB_PASSWORD}` to the SQL user's credentials.
+Keep the other Azure-specific properties.
+
+#### Secret mapping
+
+For Microsoft Entra service-principal authentication, the RDI source secret must
+provide:
+
+| Secret key | Value |
+| --- | --- |
+| `SOURCE_DB_USERNAME` | The service principal's **Application (client) ID** (a GUID). |
+| `SOURCE_DB_PASSWORD` | The service principal's **client secret**. |
+
+{{< warning >}}The `SOURCE_DB_USERNAME` value is the client ID (a GUID), but the contained
+database user created in the previous section uses the service principal's **display
+name**. These are two different identifiers for the same principal — mixing them up is
+the most common cause of `Login failed for user ''` errors
+at connection time.{{< /warning >}}
+
+#### Other Microsoft Entra authentication modes
+
+The Microsoft JDBC driver supports several other Microsoft Entra modes. The following
+are technically usable but are not currently validated by RDI — check with Redis
+support before using them in production:
+
+- **`ActiveDirectoryServicePrincipalCertificate`** — service principal authenticated by
+ a certificate instead of a secret. Useful when organizational policy forbids
+ long-lived shared secrets.
+- **`ActiveDirectoryManagedIdentity`** — for RDI installations running on an Azure
+ resource (such as an Azure VM or Azure Kubernetes Service node) that has a system-
+ or user-assigned managed identity.
+
+The deprecated `ActiveDirectoryPassword` mode and the interactive
+`ActiveDirectoryInteractive` mode are not suitable for a server-side connector and are
+not supported.
+
+See Microsoft's
+[Connect using Microsoft Entra authentication](https://learn.microsoft.com/en-us/sql/connect/jdbc/connecting-using-azure-active-directory-authentication?view=sql-server-ver17)
+for the full list of modes and their connection-string syntax.
+
+### Verify the connection
+
+Connect to the database as the Debezium user (the SQL user or the Microsoft Entra
+service principal) and run `sys.sp_cdc_help_change_data_capture` to confirm that the
+user can see the captured tables. The query is the same as for
+[on-premises SQL Server](#4-check-that-you-have-access-to-the-cdc-table).
+
+You can also confirm the database-level and table-level CDC state directly from the
+catalog views:
+
+```sql
+-- Check whether CDC is enabled on the database
+SELECT name, is_cdc_enabled FROM sys.databases WHERE name = ''
+GO
+
+-- Check which tables in the current database have CDC enabled
+SELECT name, is_tracked_by_cdc FROM sys.tables WHERE is_tracked_by_cdc = 1
+GO
+```
+
+### Troubleshooting
+
+- **`Login failed for user ''`** — the contained database
+ user was not created for this service principal, or it was created with the wrong
+ identifier. Verify that the `CREATE USER ... FROM EXTERNAL PROVIDER` statement used
+ the service principal's display name, and that `SOURCE_DB_USERNAME` contains its
+ client ID. Query `sys.database_principals` on the source database to see which
+ principals exist.
+- **`SSL Server certificate validation failed` or hostname mismatch** —
+ `database.hostNameInCertificate` is missing or has the wrong value. For Azure SQL
+ Database, set it to `"*.database.windows.net"` to match the wildcard certificate.
+- **`Change data capture is not supported for this edition of SQL Server`** — the
+ Azure SQL Database is on an unsupported service tier. In the DTU purchasing model,
+ scale up to S3 or higher. In the vCore purchasing model, CDC is supported on all
+ tiers, so check that you are connecting to a standard Azure SQL Database (CDC is
+ not supported on Azure SQL Edge or other variants).
+- **Connection timeouts** — the RDI connector's source IP is not allowed by the Azure
+ SQL firewall, or the private endpoint is not reachable from the connector's network.
+ Verify firewall rules in the Azure portal and that DNS resolves to the expected
+ (public or private) endpoint.
+
+## Handling changes to the schema
+
+RDI can't adapt automatically when you change the schema of a CDC table in SQL Server. For example,
+if you add a new column to a table you are capturing then RDI will generate errors
+instead of capturing the changes correctly. See Debezium's
+[SQL Server schema evolution](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#sqlserver-schema-evolution)
+docs for more information.
+
+If you have administrator privileges, you can follow the steps below to update RDI after
+a schema change and resume CDC. See the
+[online schema updates](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#online-schema-updates)
+documentation for further details.
+
+```checklist {id="sqlserver-schema-changes" nointeractive="true" }
+- [ ] [Make your changes to the source table schema](#make-your-changes-to-the-source-table-schema)
+- [ ] [Create a new capture table for the updated source table](#create-a-new-capture-table-for-the-updated-source-table)
+- [ ] [Drop the old capture table](#drop-the-old-capture-table)
+```
+
+1. .supabase.co
+```
+
+You should generally use port `5432`, but you can use a private hostname or
+address instead if you have configured private connectivity between the RDI
+deployment and Supabase.
+Don't use a Supavisor transaction or session pooler connection string because
+these endpoints don't support logical replication.
+
+Supabase direct connections use IPv6 by default. If your RDI deployment
+doesn't have IPv6 egress, enable the
+[dedicated IPv4 add-on](https://supabase.com/docs/guides/platform/ipv4-address).
+The add-on requires a paid Supabase plan.
+
+If you use the public endpoint and enable
+[Supabase Network Restrictions](https://supabase.com/docs/guides/platform/network-restrictions),
+add the public egress address of the RDI host or cluster to the allowlist. Use
+a `/32` CIDR for an individual IPv4 address. For private connectivity, make
+sure the RDI host or cluster can resolve and route to the private endpoint.
+
+## 3. Create a dedicated RDI role
+
+In the Supabase SQL editor, create a dedicated login for RDI. Replace the
+example name and password with your own values:
+
+```sql
+CREATE ROLE rdi_replication
+ WITH LOGIN REPLICATION PASSWORD '';
+```
+
+{{< warning >}}
+Don't use the Supabase `postgres` administrator account for the RDI connection.
+The RDI role's credentials provide continuous access to captured data, so grant
+the role only the permissions it needs.
+{{< /warning >}}
+
+## 4. Grant access to source tables
+
+The RDI role needs to connect to the database and read every table included in
+the initial snapshot. For example:
+
+```sql
+GRANT CONNECT ON DATABASE postgres TO rdi_replication;
+
+GRANT USAGE ON SCHEMA public TO rdi_replication;
+GRANT SELECT ON ALL TABLES IN SCHEMA public TO rdi_replication;
+
+ALTER DEFAULT PRIVILEGES IN SCHEMA public
+ GRANT SELECT ON TABLES TO rdi_replication;
+```
+
+Repeat the schema grants for every schema you want RDI to capture.
+
+If RLS is enabled on a source table, the initial snapshot only contains rows
+visible to the RDI role. To capture all rows, define appropriate RLS policies
+for the role or grant `BYPASSRLS`:
+
+```sql
+ALTER ROLE rdi_replication BYPASSRLS;
+```
+
+`BYPASSRLS` applies to every table in the database. Grant it only to a
+dedicated RDI role and protect that role's credentials.
+
+### Create a publication
+
+By default, RDI uses the PostgreSQL `pgoutput` logical decoding plug-in, a
+publication named `dbz_publication`, and a replication slot named `debezium`.
+These defaults work with Supabase if the RDI role has permission to create the
+publication and manage its source tables.
+
+It is recommended that a database administrator create a publication
+containing only the tables RDI should capture:
+
+```sql
+CREATE PUBLICATION rdi_publication
+ FOR TABLE public.customers, public.orders;
+```
+
+Creating the publication explicitly avoids granting table ownership or broad
+publication-creation permissions to the RDI role and limits the publication's
+table scope.
+
+## 5. Configure TLS
+
+In the Supabase dashboard, go to
+[**Database settings** > **SSL configuration**](https://supabase.com/docs/guides/platform/ssl-enforcement):
+
+1. Enable **Enforce SSL on incoming connections**.
+1. Download the Supabase CA certificate.
+
+Store the database username, password, and CA certificate as RDI secrets:
+
+```bash
+redis-di set-secret SOURCE_DB_USERNAME rdi_replication
+redis-di set-secret SOURCE_DB_PASSWORD ''
+redis-di set-secret SOURCE_DB_CACERT /path/to/prod-ca-2021.crt
+```
+
+RDI verifies that the direct endpoint hostname matches the certificate.
+
+## 6. Configure RDI
+
+Add a PostgreSQL source to `config.yaml`. Replace the project reference and
+table names with your values:
+
+```yaml
+sources:
+ supabase:
+ type: cdc
+ connection:
+ type: postgresql
+ host: db..supabase.co
+ port: 5432
+ database: postgres
+ user: ${SOURCE_DB_USERNAME}
+ password: ${SOURCE_DB_PASSWORD}
+ schemas:
+ - public
+ tables:
+ public.customers: {}
+ public.orders: {}
+ advanced:
+ source:
+ plugin.name: pgoutput
+ publication.name: rdi_publication
+ publication.autocreate.mode: disabled
+ slot.name: rdi_supabase
+```
+
+Use a unique replication slot name for each active pipeline that connects to
+the project.
+
+## 7. Monitor replication slots
+
+RDI creates a logical replication slot that retains write-ahead log (WAL)
+records while the pipeline is stopped or disconnected. Use a query like the
+following to monitor inactive slots and retained WAL to prevent unexpected
+storage growth:
+
+```sql
+SELECT
+ slot_name,
+ active,
+ restart_lsn,
+ confirmed_flush_lsn
+FROM pg_replication_slots;
+```
+
+[Supabase requires logical replication slots to be removed](https://supabase.com/docs/guides/platform/upgrading)
+before a PostgreSQL major-version upgrade. Before upgrading:
+
+1. Stop the RDI pipeline.
+1. Record the pipeline configuration and slot name.
+1. Drop the RDI replication slot.
+1. Upgrade the Supabase project.
+1. Reset and start the RDI pipeline to create a new slot and initial snapshot.
+
+Allow time for the new initial snapshot to complete, and monitor the pipeline
+until pending records return to zero.
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/rejected-records.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/rejected-records.md
new file mode 100644
index 0000000000..5a2c9d7327
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/rejected-records.md
@@ -0,0 +1,123 @@
+---
+Title: Rejected records
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Learn how RDI stores records that cannot be processed.
+group: di
+linkTitle: Rejected records
+summary: Redis Data Integration stores records that cannot be processed in a dead letter queue.
+type: integration
+weight: 45
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/rejected-records/'
+---
+
+Redis Data Integration (RDI) sends records that it cannot process to a dead letter
+queue (DLQ). In the Redis Cloud UI, these records are called **rejected records**.
+
+Rejected records help you understand which source tables are affected and why
+specific records could not continue through the pipeline. They are intended for
+troubleshooting and support, not for normal pipeline operation.
+
+## When records are rejected
+
+RDI can reject a record when it cannot safely transform or write the change event.
+Common causes include:
+
+- The incoming change event is malformed or missing required metadata.
+- A transformation job fails while processing the record.
+- A target write fails, for example because the target key already has an incompatible data type.
+
+By default, RDI stores rejected records instead of silently dropping them. You can
+change this behavior with the `processors.error_handling` setting. See the
+[pipeline configuration file]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config" >}})
+for more information.
+
+## How RDI stores rejected records
+
+RDI stores rejected records in the RDI database as capped Redis streams. Each DLQ
+stream corresponds to a source table and tracks the records rejected for that
+table.
+
+DLQ stream names use the `dlq:` prefix followed by the source data stream name.
+In current RDI versions, the stream name is typically:
+
+```text
+dlq:data:{rdi}:.
+```
+
+For example, rejected records for the `public.users` table are stored in:
+
+```text
+dlq:data:{rdi}:public.users
+```
+
+For sources that include the source name in the stream qualifier, the final part
+can contain three components:
+
+```text
+dlq:data:{rdi}:..
+```
+
+Some RDI versions or configurations can use a hash-tagged variant such as
+`dlq:{data:rdi:.}`. To find all DLQ streams in the
+RDI database, scan for stream keys that start with `dlq:`.
+
+The maximum number of records stored per DLQ stream is controlled by
+`processors.dlq_max_messages`. When the stream reaches the configured limit,
+older entries are evicted as newer entries are added.
+
+## What to inspect
+
+Start with the rejected count for each affected table, then inspect a sample
+record from the table with the highest count or the table that matters most to
+your application.
+
+Useful fields include:
+
+- The affected table.
+- The rejection time.
+- The rejected operation. RDI stores this as an `opcode` value such as `c`,
+ `u`, `d`, `r`, `t`, or `m`. See [Using the operation code]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-opcode-example" >}})
+ for the operation labels.
+- The rejection reason.
+- The transformation job or operation, when the failure happened during transformation.
+
+{{< note >}}
+Rejected records can contain source data when you inspect them directly in the
+RDI database. Treat DLQ contents as sensitive customer data. The Redis Cloud UI
+uses the RDI DLQ API and shows a sanitized set of troubleshooting metadata. It
+does not show the original record payload or every field stored in the
+corresponding DLQ stream.
+{{< /note >}}
+
+## Resolve rejected records
+
+Use the rejection reason to identify the likely fix:
+
+- If a transformation job failed, update the job configuration and deploy the pipeline change.
+- If target writes failed because of incompatible existing keys, update the target data or key mapping.
+- If records are malformed, inspect the source connector and source database change data capture configuration.
+
+RDI does not automatically replay records from the DLQ after you fix the cause.
+If you need existing source data to be processed again, reset the pipeline after
+applying the fix. See [Reset data pipeline]({{< relref "/operate/rc/rdi/view-edit#reset-data-pipeline" >}})
+for Redis Cloud, or use the appropriate self-managed RDI reset workflow.
+
+## CLI and API access
+
+For self-managed RDI, use the [`redis-di list-dlqs`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlqs" >}})
+command to see the dead-letter queues and the
+[`redis-di list-dlq-records`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlq-records" >}})
+command (also available as `redis-di get-rejected`) to inspect the rejected records of a queue.
+
+For Redis Cloud RDI, connect to the RDI database and inspect the corresponding
+DLQ streams directly when you need details that are not shown in the Redis Cloud
+UI.
+
+RDI API v2 also includes DLQ inspection endpoints. See the
+[API reference]({{< relref "/integrate/redis-data-integration/1.19.1/reference/api-reference" >}})
+for endpoint details.
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/supported-types.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/supported-types.md
new file mode 100644
index 0000000000..d37e6ee0ff
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/supported-types.md
@@ -0,0 +1,523 @@
+---
+Title: Supported data types by source
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Learn about supported data types for each source database.
+group: di
+linkTitle: Supported data types
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 80
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/supported-types/'
+---
+
+This page describes the source data types that RDI captures for the
+[Oracle](#oracle), [MySQL/MariaDB](#mysql-and-mariadb),
+[PostgreSQL](#postgresql-supabase-and-alloydb), [SQL Server](#sql-server),
+[MongoDB](#mongodb), and [Spanner](#spanner) source databases, and how they are
+represented in Redis. There are also some
+[cross-cutting considerations](#cross-cutting-considerations) that apply to all
+source databases.
+
+## How RDI captures and represents data
+
+For most source databases, RDI uses an embedded
+[Debezium](https://debezium.io/) connector as its change data capture (CDC)
+*collector*. RDI ships a Debezium 3.x–based collector, so the collector-level
+mappings on this page follow the
+[Debezium connector reference](https://debezium.io/documentation/reference/).
+[Google Cloud Spanner](#spanner) is the exception: it uses a Flink-based collector
+that reads Spanner change streams rather than Debezium (see the
+[Spanner section](#spanner) for details).
+
+{{< note >}}**RDI does not always pass the collector value through unchanged.** RDI's
+processors normalize several Debezium logical types before they reach your jobs and
+Redis, and drop a few that they cannot represent. The collector-level representations
+in the tables below are therefore the *input* to RDI's processing, not always the
+final Redis value — the per-type notes and [cross-cutting considerations](#cross-cutting-considerations)
+call out where RDI transforms or drops a value.{{< /note >}}
+
+It helps to think of the data flow in two layers:
+
+1. **What the collector emits.** Debezium converts each source column to a
+ Kafka Connect value with a *literal type* (for example, `STRING`, `INT64`,
+ `BYTES`, `STRUCT`) and an optional *semantic type* (for example,
+ `io.debezium.time.MicroTimestamp`). Several of these conversions are controlled
+ by connector properties such as
+ [`decimal.handling.mode`](#decimal-and-numeric-values),
+ [`binary.handling.mode`](#binary-values), and
+ [`time.precision.mode`](#temporal-values). The tables below show the
+ representation that each connector produces with its **default** settings.
+2. **How RDI writes it to Redis.** RDI's processors take the collector value,
+ normalize some logical types (and drop a few unsupported ones), then write each
+ record to a Redis [Hash]({{< relref "/develop/data-types/hashes" >}}) (the default)
+ or, if you set `target_data_type: json`, to a
+ [JSON]({{< relref "/develop/data-types/json" >}}) document. For Hash targets,
+ every field value is stored as a string; for JSON targets, numbers and booleans
+ are stored as native JSON values.
+
+When you need to reformat a value in a job, see
+[Formatting date and time values]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/formatting-date-and-time-values" >}})
+for worked examples.
+
+### Setting collector properties
+
+Where the sections below recommend a Debezium property (for example,
+`decimal.handling.mode` or `lob.enabled`), set it in the `advanced.source` block of
+the source in your pipeline `config.yaml` file. These properties are passed
+through to the underlying Debezium connector. For example:
+
+```yaml
+sources:
+ my-source:
+ # ...connection details...
+ advanced:
+ source:
+ decimal.handling.mode: double
+ binary.handling.mode: base64
+```
+
+See [Pipeline configuration]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config" >}})
+for more about the `advanced` section.
+
+## Quick configuration summary
+
+The lists below summarize the extra configuration you may need for each source
+database. Each database has its own section with full detail.
+
+[**Oracle**](#oracle)
+
+- Enable [supplemental logging]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/oracle" >}}) on the tables/schemas you capture.
+- Set `lob.enabled: true` if you need `CLOB`, `NCLOB`, `BLOB`, or `XMLTYPE`.
+- Choose `binary.handling.mode` for `RAW`/`BLOB` (default is `bytes`).
+- `decimal.handling.mode` defaults to `string` in RDI; set `double` for numeric `NUMBER`/`DECIMAL` values.
+- Avoid unsupported types (`LONG`, `LONG RAW`, `BFILE`, `UROWID`, `VECTOR`, UDTs, spatial) or cast them upstream.
+
+[**MySQL/MariaDB**](#mysql-and-mariadb)
+
+- Enable the binary log in **ROW** mode.
+- Choose `decimal.handling.mode` to balance precision against convenience.
+- For `BOOLEAN`/`TINYINT(1)` fidelity, consider the `TinyIntOneToBooleanConverter`.
+- Be aware that spatial and `VECTOR` types arrive as structured values, not scalars.
+
+[**PostgreSQL/Supabase/AlloyDB**](#postgresql-supabase-and-alloydb)
+
+- Ensure WAL/logical replication settings match the connector's needs.
+- `decimal.handling.mode` defaults to `string` in RDI; set `double` for numeric values.
+- Use a RedisJSON target to get the most value from `JSON`/`JSONB`.
+
+[**SQL Server**](#sql-server)
+
+- Enable CDC at both the database and table level.
+- Choose `decimal.handling.mode` for `MONEY`/`DECIMAL` precision.
+- Choose `time.precision.mode` if you need predictable temporal precision.
+
+[**MongoDB**](#mongodb)
+
+- Ensure a replica set and change streams are configured.
+- Use a RedisJSON target to preserve document structure.
+- Choose `capture.mode` to control whether updates include the full document.
+
+[**Spanner**](#spanner)
+
+- Spanner uses the Flink-based collector (not Debezium) and is supported only on Kubernetes/Helm.
+
+## Cross-cutting considerations
+
+The settings below apply to all of the Debezium-based source connectors. They are
+the most common cause of "the value in Redis doesn't look like the value in my
+database", so review them before reading the per-database sections.
+
+### Decimal and numeric values
+
+`DECIMAL`, `NUMERIC`, `MONEY`, and similar types are controlled by
+`decimal.handling.mode`. **RDI's effective default is `string`** — the RDI collector
+templates set `debezium.source.decimal.handling.mode=string` before your
+`advanced.source` overrides apply, so decimals reach Redis as readable strings rather
+than Debezium's own `precise` binary default:
+
+| `decimal.handling.mode` | Representation |
+|-------------------------|----------------------------------------------------------------------------|
+| `string` (RDI default) | The exact decimal as a `STRING`. |
+| `double` | A `FLOAT64` number (may lose precision for very large/precise values). |
+| `precise` | A Kafka Connect `Decimal` (`BYTES`) — a base64-encoded, scaled binary value. This is Debezium's default but not RDI's. |
+
+Leave the default (`string`) for exact decimal fidelity, or set
+`decimal.handling.mode: double` if you want numeric values and can accept double
+precision.
+
+### Temporal values
+
+Temporal types are controlled by `time.precision.mode`. The default is `adaptive`
+for Oracle, PostgreSQL, and SQL Server, and `adaptive_time_microseconds` for MySQL
+and MariaDB. In adaptive modes, the precision of the emitted value depends on the
+column's declared precision:
+
+- `DATE` columns are emitted as **days since epoch** (an `INT32`), *not*
+ milliseconds at midnight.
+- `TIME`/`DATETIME`/`TIMESTAMP` columns are emitted as milliseconds, **microseconds**,
+ or **nanoseconds** since epoch (or since midnight for time-of-day types) depending
+ on their precision. For example, an Oracle `TIMESTAMP(6)` or a PostgreSQL
+ `timestamp` is emitted as microseconds.
+
+RDI's processors may normalize these Debezium temporal types before they reach your
+jobs and Redis (for example, converting a `Date` to epoch milliseconds). See
+[Formatting date and time values]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/formatting-date-and-time-values" >}})
+for worked examples of converting temporal values in an RDI job.
+
+### Time zones
+
+Time zone–aware types (for example, Oracle `TIMESTAMP WITH TIME ZONE`, PostgreSQL
+`timestamptz`, SQL Server `datetimeoffset`, and MySQL `TIMESTAMP`) are **not**
+converted to epoch milliseconds. They are emitted as **ISO 8601 strings**
+(semantic type `io.debezium.time.ZonedTimestamp`), normalized to UTC/GMT — for
+example, `2025-06-07T10:15:00.000000Z`.
+
+### Binary values
+
+Binary columns are controlled by `binary.handling.mode`. The default is **`bytes`**
+(raw bytes), *not* base64. The options are:
+
+- `bytes` (default) — raw byte array.
+- `base64` — base64-encoded string.
+- `base64-url-safe` — URL-safe base64 string.
+- `hex` — hex string.
+
+Set `binary.handling.mode: base64` (or `hex`) if your consumers expect an encoded
+string rather than raw bytes. Make sure your consumer understands the encoding you
+choose.
+
+### Large objects (LOBs) and unavailable values
+
+When a connector captures large objects (for example, Oracle `CLOB`/`BLOB`), an
+update event never contains the value of an *unchanged* LOB column. Instead, the
+column carries a placeholder. The default placeholder is `__debezium_unavailable_value`,
+which you can change with `advanced.source.unavailable.value.placeholder` on the
+source (the Helm chart exposes this as `processor.lob.placeholder`). RDI skips these
+placeholder values rather than writing them to Redis as user data.
+
+### Nullability
+
+- **Redis Hashes**: null values are not stored (the field is absent).
+- **RedisJSON**: null values become JSON `null`. Note that if you use the native
+ `JSON.MERGE` command (the default from RDI 1.15.0, controlled by
+ `use_native_json_merge`), merging a `null` value *removes* the field rather than
+ storing it, following [RFC 7396](https://datatracker.ietf.org/doc/html/rfc7396).
+ See [Pipeline configuration]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config" >}}).
+
+### Structured values (structs, arrays, and maps)
+
+Some source types are emitted as Kafka Connect `STRUCT`, `ARRAY`, or `MAP` values
+rather than scalars — for example, spatial types (a struct of `srid` + `wkb`) and
+vector types (an array of floats).
+
+**RDI does not support every complex logical type.** In particular, RDI treats
+`io.debezium.data.Bits` (from `BIT(>1)`/`BIT VARYING`) and the interval logical types
+as **unsupported**: the classic processor maps them to `None` and the Flink processor
+removes the field (via `RemovalConverter`), so they do not reach Redis. Other
+structured values (spatial `Geometry`, pgvector) pass through to RDI's processors,
+but how they are rendered into a Redis Hash or JSON document is noted per type below.
+
+## Oracle
+
+RDI captures Oracle changes via the
+[Debezium Oracle connector](https://debezium.io/documentation/reference/3.0/connectors/oracle.html).
+See [Prepare Oracle for RDI]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/oracle" >}})
+for the required supplemental-logging setup.
+
+### Supported types
+
+| Oracle type | Collector representation (default settings) | Notes |
+|-------------|---------------------------------------------|-------|
+| `NUMBER`, `DECIMAL`, `NUMERIC`, `INT`, `INTEGER`, `SMALLINT` | Kafka Connect `Decimal` (`BYTES`) | Controlled by `decimal.handling.mode`. See [Decimal and numeric values](#decimal-and-numeric-values). |
+| `NUMBER(p,*)`, `FLOAT`, `REAL`, `DOUBLE PRECISION` | `VariableScaleDecimal` (`STRUCT`) | Variable-scale decimal. |
+| `BINARY_FLOAT` | `FLOAT32` | |
+| `BINARY_DOUBLE` | `FLOAT64` | |
+| `CHAR`, `VARCHAR`, `VARCHAR2`, `NCHAR`, `NVARCHAR2` | `STRING` | UTF-8 preserved. |
+| `DATE` | `Timestamp` (`INT64`, ms since epoch) | |
+| `TIMESTAMP(0-3)` | `Timestamp` (ms) | Precision depends on the column; see [Temporal values](#temporal-values). |
+| `TIMESTAMP(4-6)` | `MicroTimestamp` (µs) | A bare `TIMESTAMP` defaults to precision 6 (microseconds). |
+| `TIMESTAMP(7-9)` | `NanoTimestamp` (ns) | |
+| `TIMESTAMP WITH TIME ZONE` | `ZonedTimestamp` (`STRING`, ISO 8601) | See [Time zones](#time-zones). |
+| `TIMESTAMP WITH LOCAL TIME ZONE` | `ZonedTimestamp` (`STRING`, UTC) | |
+| `INTERVAL YEAR TO MONTH`, `INTERVAL DAY TO SECOND` | `MicroDuration` (`INT64`) | **Not supported by RDI** — interval types are dropped before reaching Redis. See [Structured values](#structured-values-structs-arrays-and-maps). |
+| `CLOB`, `NCLOB` | `STRING` | Requires `lob.enabled: true`. |
+| `BLOB` | `BYTES` | Requires `lob.enabled: true`; encoded per `binary.handling.mode`. |
+| `RAW` | `BYTES` | Encoded per `binary.handling.mode`. |
+| `XMLTYPE` | `Xml` (`STRING`) | **Incubating** in Debezium. Requires `lob.enabled: true` and a non-hybrid mining strategy. |
+| `ROWID` | `STRING` | Supported in LogMiner mode only; not exposed when using XStream. |
+
+### Configuration notes
+
+- **LOBs**: set `lob.enabled: true` (default `false`) to capture `CLOB`, `NCLOB`,
+ `BLOB`, and `XMLTYPE`. You cannot use the *hybrid* mining strategy with
+ `lob.enabled: true` — use `online_catalog` or `redo_log_catalog` instead.
+- **Extended strings**: if the database parameter `max_string_size` is `EXTENDED`,
+ set `lob.enabled: true` to capture `VARCHAR2`/`NVARCHAR2` values over 4000 bytes
+ and `RAW` values over 2000 bytes.
+- **XMLTYPE**: requires `lob.enabled: true` and a non-hybrid mining strategy
+ (`online_catalog` or `redo_log_catalog`). The connector emits the XML as text
+ (`STRING`). XMLTYPE support also requires the Oracle **XDB library** and the
+ **`xmlparserv2`** dependency in addition to the standard `ojdbc11.jar` driver. If
+ the runtime selects Oracle's `xmlparserv2` SAX parser, you may need to set the JVM
+ option `-Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl`.
+
+### Not captured
+
+The Debezium Oracle connector does not support `LONG`, `LONG RAW`, `BFILE`,
+`UROWID`, `VECTOR`, the native Oracle 23 `BOOLEAN` column type, user-defined/object
+types (objects, `REF`, `VARRAY`, nested tables), or Oracle spatial types. Cast these
+to a supported type upstream if you need them. A `NumberOneToBooleanConverter` is
+available to map `NUMBER(1)` columns to booleans.
+
+## MySQL and MariaDB
+
+RDI captures both `mysql` and `mariadb` sources with the
+[Debezium MySQL connector](https://debezium.io/documentation/reference/stable/connectors/mysql.html)
+(`io.debezium.connector.mysql.MySqlConnector`) — it does not use Debezium's separate
+MariaDB connector. The mappings below therefore apply to both source types. See
+[Prepare MySQL/MariaDB for RDI]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/my-sql-mariadb" >}})
+for setup. Enable the binary log in **ROW** mode.
+
+### Supported types
+
+| MySQL/MariaDB type | Collector representation (default settings) | Notes |
+|--------------------|---------------------------------------------|-------|
+| `TINYINT`, `SMALLINT`, `MEDIUMINT`, `INT`, `BIGINT` | `INT8`/`INT16`/`INT32`/`INT64` | |
+| `BIT(1)` | `BOOLEAN` | A single bit is mapped to a boolean. |
+| `BIT(>1)` | `Bits` (`BYTES`) | **Not supported by RDI** — dropped before reaching Redis. See [Structured values](#structured-values-structs-arrays-and-maps). |
+| `DECIMAL`, `NUMERIC` | Kafka Connect `Decimal` (`BYTES`) | Controlled by `decimal.handling.mode`. See [Decimal and numeric values](#decimal-and-numeric-values). |
+| `FLOAT(0-23)`, `REAL` | `FLOAT32` | |
+| `FLOAT(24-53)`, `DOUBLE` | `FLOAT64` | |
+| `CHAR`, `VARCHAR`, `TINYTEXT`, `TEXT`, `MEDIUMTEXT`, `LONGTEXT` | `STRING` | |
+| `BINARY`, `VARBINARY`, `TINYBLOB`, `BLOB`, `MEDIUMBLOB`, `LONGBLOB` | `BYTES` | Encoded per `binary.handling.mode`. Up to 2 GB; use the claim-check pattern for large values. |
+| `DATE` | `Date` (days since epoch) | See [Temporal values](#temporal-values). |
+| `TIME` | `MicroTime` (µs since midnight) | Default `time.precision.mode` is `adaptive_time_microseconds`. |
+| `DATETIME` | `Timestamp`/`MicroTimestamp` by precision | |
+| `TIMESTAMP` | `ZonedTimestamp` (`STRING`, ISO 8601, UTC) | Not epoch ms. See [Time zones](#time-zones). |
+| `YEAR` | `io.debezium.time.Year` (`INT32`) | |
+| `BOOLEAN`, `BOOL` | `BOOLEAN` | During snapshots the connector sees `TINYINT(1)`; use `TinyIntOneToBooleanConverter` for consistent boolean fidelity. |
+| `ENUM` | `io.debezium.data.Enum` (`STRING`) | The `allowed` schema parameter lists the permitted values. |
+| `SET` | `io.debezium.data.EnumSet` (`STRING`) | Comma-separated selected values. |
+| `JSON` | `io.debezium.data.Json` (`STRING`) | Parsed into a nested structure on a RedisJSON target. |
+| `VECTOR` | `ARRAY (FLOAT32)`, `io.debezium.data.FloatVector` | See [Structured values](#structured-values-structs-arrays-and-maps). |
+| Spatial: `GEOMETRY`, `POINT`, `LINESTRING`, `POLYGON`, `MULTIPOINT`, `MULTILINESTRING`, `MULTIPOLYGON`, `GEOMETRYCOLLECTION` | `io.debezium.data.geometry.Geometry` (`STRUCT`) | A struct with `srid` (`INT32`) and `wkb` (`BYTES`, Well-Known Binary). |
+
+### Booleans
+
+MySQL and MariaDB both represent `BOOLEAN`/`BOOL` as `TINYINT(1)`. Because RDI uses
+the MySQL connector for both, the connector may report these columns as `TINYINT(1)`
+rather than `BOOLEAN` (especially during snapshots). Use the
+`TinyIntOneToBooleanConverter` for consistent boolean fidelity.
+
+## PostgreSQL, Supabase, and AlloyDB
+
+RDI captures PostgreSQL changes via the
+[Debezium PostgreSQL connector](https://debezium.io/documentation/reference/3.0/connectors/postgresql.html)
+using logical replication. **Supabase** and **AlloyDB** are PostgreSQL-compatible
+and use the same connector, so the mappings below apply to all three. See
+[Prepare PostgreSQL for RDI]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql" >}}).
+
+### Supported types
+
+| PostgreSQL type | Collector representation (default settings) | Notes |
+|-----------------|---------------------------------------------|-------|
+| `SMALLINT`, `INTEGER`, `BIGINT` | `INT16`/`INT32`/`INT64` | `SMALLSERIAL`/`SERIAL`/`BIGSERIAL` map the same as their integer base. |
+| `OID` | `INT64` | |
+| `NUMERIC`, `DECIMAL` | Kafka Connect `Decimal` (`BYTES`), or `VariableScaleDecimal` when unscaled | Controlled by `decimal.handling.mode`. See [Decimal and numeric values](#decimal-and-numeric-values). |
+| `MONEY` | Kafka Connect `Decimal` (`BYTES`) | Scale set by `money.fraction.digits`. |
+| `REAL` | `FLOAT32` | |
+| `DOUBLE PRECISION` | `FLOAT64` | |
+| `BOOLEAN` | `BOOLEAN` | |
+| `BIT(1)` | `BOOLEAN` | |
+| `BIT(>1)`, `BIT VARYING` | `Bits` (`BYTES`) | **Not supported by RDI** — dropped before reaching Redis. See [Structured values](#structured-values-structs-arrays-and-maps). |
+| `CHAR`, `VARCHAR`, `TEXT`, `CITEXT` | `STRING` | |
+| `BYTEA` | `BYTES` | Encoded per `binary.handling.mode`. Requires `bytea_output = hex` in PostgreSQL. |
+| `DATE` | `Date` (days since epoch) | See [Temporal values](#temporal-values). |
+| `TIME` | `MicroTime` (µs since midnight) | |
+| `TIME WITH TIME ZONE` (`TIMETZ`) | `ZonedTime` (`STRING`, GMT) | For example, `07:15:00Z`. |
+| `TIMESTAMP` | `MicroTimestamp` (µs since epoch) | See [Temporal values](#temporal-values). |
+| `TIMESTAMP WITH TIME ZONE` (`TIMESTAMPTZ`) | `ZonedTimestamp` (`STRING`, GMT) | See [Time zones](#time-zones). |
+| `INTERVAL` | `MicroDuration` (`INT64`) | **Not supported by RDI** — interval types are dropped before reaching Redis. See [Structured values](#structured-values-structs-arrays-and-maps). |
+| `UUID` | `io.debezium.data.Uuid` (`STRING`) | |
+| `INET`, `CIDR`, `MACADDR`, `MACADDR8` | `STRING` | |
+| `JSON`, `JSONB` | `io.debezium.data.Json` (`STRING`) | Parsed into a nested structure on a RedisJSON target. |
+| `HSTORE` | `io.debezium.data.Json` (`STRING`) | Default `hstore.handling.mode` is `json` (for example, `{"key":"val"}`); set `map` for a `MAP` value. |
+| `XML` | `io.debezium.data.Xml` (`STRING`) | |
+| `LTREE` | `io.debezium.data.Ltree` (`STRING`) | |
+| `TSVECTOR` | `io.debezium.data.Tsvector` (`STRING`) | |
+| Range types (`INT4RANGE`, `INT8RANGE`, `NUMRANGE`, `TSRANGE`, `TSTZRANGE`, `DATERANGE`) | `STRING` | |
+| `ENUM` | `io.debezium.data.Enum` (`STRING`) | |
+| pgvector `VECTOR` | `ARRAY (FLOAT64)`, `io.debezium.data.DoubleVector` | Supabase and AlloyDB commonly enable pgvector. |
+| pgvector `HALFVEC` | `ARRAY (FLOAT32)`, `io.debezium.data.FloatVector` | |
+| pgvector `SPARSEVEC` | `STRUCT`, `io.debezium.data.SparseVector` | `dimensions` (`INT16`) + `vector` (`MAP(INT16, FLOAT64)`). |
+| PostGIS `GEOMETRY` | `io.debezium.data.geometry.Geometry` (`STRUCT`) | `srid` (`INT32`) + `wkb` (`BYTES`). |
+| PostGIS `GEOGRAPHY` | `io.debezium.data.geometry.Geography` (`STRUCT`) | |
+| Native `POINT` | `io.debezium.data.geometry.Point` (`STRUCT`) | Two `FLOAT64` fields (`x`, `y`). |
+
+Domain types (user-defined types based on an underlying type) are captured using
+their base type's representation.
+
+The Debezium 3.0 PostgreSQL reference does not explicitly document how native array
+columns (for example, `int[]` or `text[]`) are captured — the Kafka Connect `ARRAY`
+literal type is used in the reference only for the pgvector types above. In practice,
+the connector represents arrays of supported primitive types as `ARRAY` values, but
+this is not stated in the reference.
+
+### Not captured
+
+The connector does not capture the native geometric types `LINE`, `LSEG`, `BOX`,
+`PATH`, `POLYGON`, and `CIRCLE`, or true composite/row types. Cast these upstream if
+you need them.
+
+## SQL Server
+
+RDI captures SQL Server changes via the
+[Debezium SQL Server connector](https://debezium.io/documentation/reference/3.0/connectors/sqlserver.html).
+CDC must be enabled at both the database and table level. See
+[Prepare SQL Server for RDI]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/sql-server" >}}).
+
+### Supported types
+
+| SQL Server type | Collector representation (default settings) | Notes |
+|-----------------|---------------------------------------------|-------|
+| `TINYINT`, `SMALLINT`, `INT`, `BIGINT` | `INT16`/`INT16`/`INT32`/`INT64` | |
+| `BIT` | `BOOLEAN` | |
+| `DECIMAL`, `NUMERIC` | Kafka Connect `Decimal` (`BYTES`) | Controlled by `decimal.handling.mode`. See [Decimal and numeric values](#decimal-and-numeric-values). |
+| `MONEY`, `SMALLMONEY` | Kafka Connect `Decimal` (`BYTES`) | |
+| `REAL` | `FLOAT32` | |
+| `FLOAT[(N)]` | `FLOAT64` | |
+| `CHAR`, `VARCHAR`, `NCHAR`, `NVARCHAR`, `TEXT`, `NTEXT` | `STRING` | |
+| `XML` | `io.debezium.data.Xml` (`STRING`) | |
+| `DATE` | `Date` (days since epoch) | Not "ms at midnight". See [Temporal values](#temporal-values). |
+| `TIME(0-3)` | `Time` (ms since midnight) | |
+| `TIME(4-6)` | `MicroTime` (µs since midnight) | |
+| `TIME(7)` | `NanoTime` (ns since midnight) | |
+| `DATETIME`, `SMALLDATETIME` | `Timestamp` (ms since epoch) | |
+| `DATETIME2(0-3)` | `Timestamp` (ms) | |
+| `DATETIME2(4-6)` | `MicroTimestamp` (µs) | |
+| `DATETIME2(7)` | `NanoTimestamp` (ns) | |
+| `DATETIMEOFFSET` | `ZonedTimestamp` (`STRING`, GMT) | See [Time zones](#time-zones). |
+| `BINARY`, `VARBINARY` | `BYTES` | Encoded per `binary.handling.mode` (default `bytes`). Not in the reference's mapping tables, but handled via the `binary.handling.mode` property. |
+
+### Types requiring confirmation
+
+The previous version of this page documented `UNIQUEIDENTIFIER`,
+`ROWVERSION`/`TIMESTAMP` (the row-version column type), `sql_variant`, `hierarchyid`,
+`IMAGE`, and the spatial types (`geometry`, `geography`) for SQL Server. None of these
+appear in the Debezium 3.0 SQL Server connector reference's data type mapping tables.
+
+Note that *absence from the reference's tables does not necessarily mean a type is
+unsupported* — `BINARY` and `VARBINARY`, for example, are handled via the
+`binary.handling.mode` property even though they have no mapping-table row. So these
+types should be confirmed empirically rather than assumed unsupported.
+
+## MongoDB
+
+RDI captures MongoDB changes via the
+[Debezium MongoDB connector](https://debezium.io/documentation/reference/3.0/connectors/mongodb.html),
+which works differently from the relational connectors. See
+[Prepare MongoDB for RDI]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/mongodb" >}}).
+
+### What the collector emits
+
+The MongoDB connector does **not** map each BSON field to a separate typed value.
+Instead, it emits the whole document as a **single JSON string** using MongoDB
+[extended JSON, strict mode](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/).
+BSON values appear inside that string using extended-JSON wrappers, for example:
+
+| BSON type | Extended-JSON representation |
+|-----------|------------------------------|
+| `ObjectId` | `{"$oid": "596e275826f08b2730779e1f"}` |
+| `Int32` / `Int64` | `1234` / `{"$numberLong": "1234"}` |
+| `Double` | a JSON number |
+| `Decimal128` | `{"$numberDecimal": "..."}` |
+| `Date` | `{"$date": ...}` |
+| `Timestamp` (BSON) | `{"$timestamp": {"t": ..., "i": ...}}` |
+| `Binary` | `{"$binary": "...", "$type": "00"}` |
+| `Boolean` | `true` / `false` |
+| `Null` | `null` |
+| Regular expression | `{"$regularExpression": {"pattern": "...", "options": "..."}}` |
+| JavaScript | `{"$code": "..."}` |
+| `MinKey` / `MaxKey` | `{"$minKey": 1}` / `{"$maxKey": 1}` |
+
+The document's `_id` is placed in the change event **key** (as an extended-JSON
+string). It can be any BSON type — it is only a 24-character hex value when it is an
+`ObjectId`.
+
+What is available for updates depends on `capture.mode`:
+
+- A *create* event always includes the full document.
+- An *update* event includes the full document only when `capture.mode` is
+ `change_streams_update_full`; otherwise it carries only the changed fields
+ (`updatedFields`/`removedFields`). A `*_with_pre_image` mode is required to include
+ the prior document state.
+
+Documents larger than the 16 MB BSON limit require `oversize.handling.mode` (and
+MongoDB 6.0.9+).
+
+### How RDI maps it to Redis
+
+RDI parses the collector's JSON string and writes the result to your Redis target:
+
+- With a RedisJSON target, the document structure (nested objects and arrays) is
+ preserved.
+- With a Hash target, nested objects and arrays are stored as stringified JSON.
+- RDI typically derives the Redis key (in whole or in part) from the document's
+ `_id`.
+
+## Spanner
+
+RDI supports [Google Cloud Spanner](https://cloud.google.com/spanner) as a source,
+but **Spanner does not use Debezium**. During the snapshot phase RDI reads Spanner
+directly over JDBC, and during streaming it consumes
+[Spanner change streams](https://cloud.google.com/spanner/docs/change-streams) via a
+Flink-based collector (`type: flink`). Spanner is supported only when RDI is deployed
+on Kubernetes/Helm. See
+[Prepare Spanner for RDI]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/spanner" >}})
+for setup.
+
+Because Spanner uses a different collector, its data type handling is not governed by
+the Debezium settings described elsewhere on this page. There is no Debezium or Flink
+type-mapping reference to consult: Flink is only the stream-processing runtime, and
+neither Flink core nor Flink CDC provides a Spanner connector. The representation of
+each value comes from Spanner itself.
+
+### Supported types
+
+During the streaming phase, values arrive in Spanner's
+[change stream record format](https://cloud.google.com/spanner/docs/change-streams/details),
+in which each value is JSON-encoded according to the Spanner
+[`TypeCode`](https://cloud.google.com/spanner/docs/reference/rest/v1/Type) reference
+(the same encoding that the record's `column_types` metadata points to). The table
+below uses GoogleSQL type names; the PostgreSQL dialect uses different type names
+(for example, `bigint`, `bytea`, `timestamptz`, `jsonb`) but the same value encoding.
+
+| Spanner type (GoogleSQL) | Change-stream representation |
+|--------------------------|------------------------------|
+| `BOOL` | JSON `true`/`false`. |
+| `INT64` | A `STRING` in decimal format (not a JSON number). |
+| `FLOAT32`, `FLOAT64` | A JSON number, or the strings `"NaN"`, `"Infinity"`, `"-Infinity"`. |
+| `NUMERIC` | A `STRING` in decimal or scientific notation. |
+| `STRING` | A `STRING`. |
+| `BYTES` | A base64-encoded `STRING` (RFC 4648). |
+| `JSON` | A JSON-formatted `STRING` (RFC 7159). |
+| `TIMESTAMP` | A `STRING` in RFC 3339 format, time zone `Z` (UTC). |
+| `DATE` | A `STRING` in RFC 3339 date format. |
+| `UUID` | A lower-case hexadecimal `STRING` (RFC 9562). |
+| `ENUM` | A `STRING` in decimal format. |
+| `ARRAY` | A JSON list of elements encoded per the element type. |
+| `STRUCT` | A JSON list of field values encoded per the field types. |
+
+This table shows the **raw change-stream representation** only. RDI's Spanner
+collector parses some of these values before they become change events — for
+example, the string-encoded `INT64` values are parsed to numbers — and the snapshot
+phase reads through the Spanner JDBC path, which may not match the change-stream
+encoding. The RDI job/target representation is therefore not always identical to the
+raw encoding above.
+
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/_index.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/_index.md
new file mode 100644
index 0000000000..aa30ee930a
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/_index.md
@@ -0,0 +1,158 @@
+---
+Title: Job files
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Learn how to configure job files for data transformation.
+group: di
+hideListLinks: false
+linkTitle: Job files
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 5
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/'
+---
+
+You can optionally supply one or more job files that specify how you want to
+transform the captured data before writing it to the target.
+Each job file contains a YAML
+configuration that controls the transformation for a particular table from the source
+database. You can also add a `default-job.yaml` file to provide
+a default transformation for tables that don't have a specific job file of their own.
+
+The job files have a structure like the following example. This configures a default
+job that:
+
+- Writes the data to a Redis hash
+- Adds a field `app_code` to the hash with a value of `foo`
+- Adds a prefix of `aws` and a suffix of `gcp` to the key
+
+```yaml
+name: Default job with prefix and suffix
+source:
+ table: "*"
+ row_format: full
+transform:
+ - uses: add_field
+ with:
+ fields:
+ - field: after.app_code
+ expression: "`foo`"
+ language: jmespath
+output:
+ - uses: redis.write
+ with:
+ data_type: hash
+ key:
+ expression: concat(['aws', '#', table, '#', keys(key)[0], '#', values(key)[0], '#gcp'])
+ language: jmespath
+```
+
+The main sections of these files are:
+
+- `source`: This is a mandatory section that specifies the data items that you want to
+ use. You can add the following properties here:
+ - `server_name`: Logical server name (optional).
+ - `db`: Database name (optional). This refers to a database name you supplied in
+ [config.yaml]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config" >}}).
+ - `schema`: Database schema (optional). This refers to a schema name you supplied in
+ [config.yaml]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config" >}}).
+ - `table`: Database table name. This refers to a table name you supplied in `config.yaml`. The default
+ job doesn't apply to a specific table, so use "*" in place of the table name for this job only.
+ - `row_format`: Format of the data to be transformed. This can take the values `partial` (default) to
+ use only the payload data, or `full` to use the complete change record. See the `transform` section below
+ for details of the extra data you can access when you use the `full` option.
+ - `case_insensitive`: This applies to the `server_name`, `db`, `schema`, and `table` properties
+ and is set to `true` by default. Set it to `false` if you need to use case-sensitive values for these
+ properties.
+
+- `transform`: This is an optional section describing the transformation that the pipeline
+ applies to the data before writing it to the target. The `uses` property specifies a
+ *transformation block* that will use the parameters supplied in the `with` section. See the
+ [data transformation reference]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation" >}})
+ for more details about the supported transformation blocks. See also the
+ [JMESPath custom functions]({{< relref "/integrate/redis-data-integration/1.19.1/reference/jmespath-custom-functions" >}})
+ reference and [JMESPath functions proposal](https://jmespath.org/proposals/functions.html) for
+ full details of the functions available for the `expression` field.
+ You can test your transformation logic using the
+ [dry run]({{< relref "/integrate/redis-data-integration/1.19.1/reference/api-reference/#tag/secure/operation/job_dry_run_api_v1_pipelines_jobs_dry_run_post" >}})
+ feature in the API.
+
+ {{< note >}}If you set `row_format` to `full` under the `source` settings, you can access extra data from the
+ change record in the transformation:
+ - Use the `key` object to access the attributes of the key. For example, `key.id` will give you the value of the `id` column as long as it is part of the primary key.
+ - Use `before.` to get the value of a field *before* it was updated in the source database
+ - Use `after.` to get the value of a field *after* it was updated in the source database
+ - Use `after.` when adding new fields during transformations
+
+ See [Row Format]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-row-format#full" >}}) for a more detailed explanation of the full format.
+ {{< /note >}}
+
+- `output`: This is a mandatory section to specify the data structure(s) that
+ RDI will write to
+ the target along with the text pattern for the key(s) that will access it.
+ Note that you can map one record to more than one key in Redis or nest
+ a record as a field of a JSON structure (see
+ [Data denormalization]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization" >}})
+ for more information about nesting). You can add the following properties in the `output` section:
+ - `uses`: This must have the value `redis.write` to specify writing to a Redis data
+ structure. You can add more than one block of this type in the same job.
+ - `with`:
+ - `connection`: Connection name as defined in `config.yaml` (by default, the connection named `target` is used).
+ - `data_type`: Target data structure when writing data to Redis. The supported types are `hash`, `json`, `set`,
+ `sorted_set`, `stream` and `string`.
+ - `key`: This lets you override the default key for the data structure with custom logic:
+ - `expression`: Expression to generate the key.
+ - `language`: Expression language, which must be `jmespath` or `sql`.
+ - `expire`: Positive integer value or SQL/JMESPath expression indicating a number of seconds
+ for the key to expire. If you don't specify this property, the key will never expire.
+ See [Set custom expiration times / TTL]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-expiration-example" >}}) for more information and examples.
+
+{{< note >}}In a job file, the `transform` section is optional, but if you don't specify
+a `transform`, you must specify custom key logic in `output.with.key`. You can include
+both of these sections if you want both a custom transform and a custom key.{{< /note >}}
+
+Another example below shows how you can rename the `fname` field to `first_name` in the table `emp`
+using the
+[`rename_field`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/rename_field" >}}) block. It also demonstrates how you can set the key of this record instead of relying on
+the default logic.
+
+```yaml
+name: Rename field example
+source:
+ server_name: redislabs
+ schema: dbo
+ table: emp
+transform:
+ - uses: rename_field
+ with:
+ from_field: fname
+ to_field: first_name
+output:
+ - uses: redis.write
+ with:
+ connection: target
+ key:
+ expression: concat(['emp:fname:',fname,':lname:',lname])
+ language: jmespath
+```
+
+See the
+[RDI configuration file]({{< relref "/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference" >}})
+reference for full details about the
+available source, transform, and target configuration options and see
+also the
+[data transformation reference]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation" >}})
+for details of all the available transformation blocks.
+
+{{< note >}}When using the `sql` option as language for the expressions keep in mind that RDI uses the SQL syntax and
+functions supported by SQLite and those may differ from the ANSI-SQL ones. You can find more details in SQLite's
+[official documentation](https://sqlite.org/lang.html).{{< /note >}}
+
+## Examples
+
+The pages listed below show examples of typical job files for different use cases.
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/caching-expression-results.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/caching-expression-results.md
new file mode 100644
index 0000000000..f6e0d15fa2
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/caching-expression-results.md
@@ -0,0 +1,139 @@
+---
+Title: Caching expression results
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Caching expression results
+summary: How to cache expression and lookup results to reduce CPU and Redis load
+type: integration
+weight: 50
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/caching-expression-results/'
+---
+
+The Flink processor can cache the result of any expression that
+produces a value (for example, an
+[`add_field`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/add_field" >}})
+expression, a [`map`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/map" >}})
+expression, the arguments to a
+[`redis.lookup`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/lookup" >}}),
+or a custom output `key`/`expire` expression). Caching is useful when
+the same expression is evaluated repeatedly with the same input field
+values, for example when many incoming records share a common foreign
+key.
+
+{{< note >}}Caching is supported only by the **Flink processor**. The
+classic processor silently ignores `cache:` blocks.{{< /note >}}
+
+## The `cache:` block
+
+You enable caching by adding a `cache:` block next to the expression
+you want to cache. Cache keys are derived from the values of the input
+fields referenced by the expression, not from the full record. See
+[`cache`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/cache" >}})
+for the full property list.
+
+| Property | Type | Description | Default |
+| ------------- | --------- | -------------------------------------------------------------- | ------- |
+| `enabled` | `boolean` | Set to `true` to enable caching. | `false` |
+| `max_size` | `integer` | Maximum number of entries kept in the cache. Must be positive. | `1000` |
+| `ttl_seconds` | `integer` | Time-to-live for each entry, in seconds. Must be positive. | `60` |
+
+## Caching an `add_field` expression
+
+The example below adds a `country` field whose value is derived from
+`country_code` and `country_name`. When the same combination of input
+values appears repeatedly (for example, many customers from the same
+country), caching the result avoids re-evaluating the expression.
+
+```yaml
+name: Cached country field
+source:
+ schema: dbo
+ table: customer
+transform:
+ - uses: add_field
+ with:
+ field: country
+ language: sql
+ expression: country_code || ' - ' || UPPER(country_name)
+ cache:
+ enabled: true
+ max_size: 500
+ ttl_seconds: 300
+```
+
+## Caching a `map` expression
+
+```yaml
+name: Cached map expression
+source:
+ table: customer
+transform:
+ - uses: map
+ with:
+ language: jmespath
+ expression: |
+ {
+ "CustomerId": customer_id,
+ "Country": country_code
+ }
+ cache:
+ enabled: true
+```
+
+## Caching `redis.lookup` arguments and results
+
+`redis.lookup` supports two independent caches. The `cache:` block
+caches the *argument* expressions (the JMESPath or SQL expressions
+that produce the Redis command arguments). The `lookup_cache:` block
+caches the *result* of the Redis command itself, keyed by the
+resolved arguments. Both blocks accept the same properties as the
+`cache:` block above.
+
+```yaml
+name: Cached lookup
+source:
+ table: order
+transform:
+ - uses: redis.lookup
+ with:
+ connection: target
+ cmd: HGETALL
+ args:
+ - concat(['customer:', customer_id])
+ language: jmespath
+ field: customer
+ cache:
+ enabled: true
+ ttl_seconds: 60
+ lookup_cache:
+ enabled: true
+ max_size: 10000
+ ttl_seconds: 300
+```
+
+## Caching `key` and `expire` output expressions
+
+A `cache:` block can also be added to the
+[output `key` and `expire` expressions]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/_index" >}})
+when those are dynamic. The properties are the same as above.
+
+```yaml
+name: Cached key expression
+source:
+ table: order
+output:
+ - uses: redis.write
+ with:
+ data_type: hash
+ key:
+ expression: concat(['order:', order_id])
+ language: jmespath
+ cache:
+ enabled: true
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/formatting-date-and-time-values.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/formatting-date-and-time-values.md
new file mode 100644
index 0000000000..608526df5b
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/formatting-date-and-time-values.md
@@ -0,0 +1,201 @@
+---
+Title: Formatting date and time values
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Formatting date and time values
+summary: Redis Data Integration keeps Redis in sync with a primary database in near
+ real time.
+type: integration
+weight: 40
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/formatting-date-and-time-values/'
+---
+
+The way you format date and time values depends on the source database, the data type of the field, and how it is represented in the incoming record. Below are some examples for different databases and data types.
+
+## Oracle
+
+Oracle supports the following date and time data types:
+
+- `DATE` - represented by Debezium as a 64-bit integer representing the milliseconds since epoch
+ ```yaml
+ name: Format Oracle DATE field
+ transform:
+ - uses: add_field
+ with:
+ fields:
+ - field: formatted_date
+ language: sql
+ # Date is stored as a Unix timestamp in milliseconds so you need to
+ # divide it by 1000 to convert it to seconds.
+ expression: STRFTIME('%Y-%m-%d %H:%M:%S', DATE / 1000, 'unixepoch')
+ # Example: 1749047572000 is transformed to 2025-06-04 14:32:52
+ ```
+- `TIMESTAMP` - the value is represented by Debezium as a 64-bit integer and depends on the number of decimal places of precision of the column, representing fractions of a second. For example, if the column is defined as `TIMESTAMP(6)`, there are six decimal places and so the value is represented as microseconds since epoch (since there are 10^6 microseconds in each second).
+You can format it similarly to `DATE`, but you need to divide the value by the appropriate factor based on the precision.
+
+- `TIMESTAMP WITH TIME ZONE` - the value is represented as a string containing the timestamp and time zone.
+
+- `TIMESTAMP WITH LOCAL TIME ZONE` - the value is represented as a string containing the timestamp and local time zone.
+
+ SQLite supports both `TIMESTAMP WITH TIME ZONE` and `TIMESTAMP WITH LOCAL TIME ZONE`. You can format them using the `STRFTIME` function.
+
+ ```yaml
+ name: Format Oracle TIMESTAMP WITH TIME ZONE
+ transform:
+ - uses: add_field
+ with:
+ fields:
+ - field: seconds_since_epoch
+ language: sql
+ # Convert the timestamp with local time zone to seconds since epoch.
+ expression: STRFTIME('%s', TIMESTAMP_FIELD)
+
+ - field: date_from_timestamp
+ language: sql
+ # Convert the timestamp with local time zone to date and time.
+ expression: STRFTIME('%Y-%m-%d %H:%M:%S', TIMESTAMP_FIELD)
+ ```
+
+----
+
+## SQL Server
+SQL Server supports the following date and time data types:
+
+- `date` - represented by Debezium as number of days since epoch (1970-01-01). You can multiply the value by 86400 (the number of seconds in a day) to convert it to seconds since epoch and then use the `STRFTIME` or `DATE` functions to format it.
+ ```yaml
+ name: Format SQL Server date field
+ transform:
+ - uses: add_field
+ with:
+ fields:
+ - field: with_default_date_format
+ language: sql
+ # Uses the default DATE format
+ expression: DATE(event_date * 86400, 'unixepoch')
+
+ - field: with_custom_date_format
+ language: sql
+ # Uses the default DATE format
+ expression: STRFTIME('%Y/%m/%d', event_date * 86400, 'unixepoch')
+ ```
+
+- `datetime`, `smalldatetime` - represented by Debezium as number of milliseconds since epoch. Divide the value by 1000 to convert it to seconds since epoch and then use the `STRFTIME` function to format it.
+ ```yaml
+ name: Format SQL Server datetime field
+ transform:
+ - uses: add_field
+ with:
+ fields:
+ - field: formatted_datetime
+ language: sql
+ expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_datetime / 1000, 'unixepoch')
+ ```
+
+- `datetime2` - similar to `datetime` but with higher precision. For `datetime2(0-3)`, the representation is the same as for `datetime`. For `datetime2(4-6)`, it is the number of microseconds since epoch. For `datetime2(7)`, it is the number of nanoseconds since epoch. To convert to another time unit, you can use the same approach as for `datetime` but you need to divide by 1000, 1000000 or 1000000000 depending on the precision.
+
+- `time` - the number of milliseconds since midnight.
+ ```yaml
+ name: Format SQL Server time field
+ transform:
+ - uses: add_field
+ with:
+ fields:
+ - field: formatted_time
+ language: sql
+ expression: TIME(event_time, 'unixepoch', 'utc')
+ ```
+
+- `datetimeoffset` - represented as a timestamp with timezone information (for example, `2025-05-27T15:21:42.864Z` or `2025-01-02T14:45:30.123+05:00`).
+ ```yaml
+ name: Format SQL Server datetimeoffset field
+ transform:
+ - uses: add_field
+ with:
+ fields:
+ - field: formatted_datetimeoffset
+ language: sql
+ expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_datetimeoffset)
+ ```
+
+
+
+
+
+
+
+
+----
+
+## PostgreSQL
+
+PostgreSQL supports the following date and time data types:
+
+- `date` - represented by Debezium as number of days since epoch (1970-01-01). You can multiply the value by 86400 (the number of seconds in a day) to convert it to seconds since epoch and then use the `STRFTIME` or `DATE` functions to format it.
+ ```yaml
+ name: Format PostgreSQL date field
+ transform:
+ - uses: add_field
+ with:
+ fields:
+ - field: with_default_date_format
+ language: sql
+ # Uses the default DATE format
+ expression: DATE(event_date * 86400, 'unixepoch')
+ ```
+
+- `time` - the time of microseconds since midnight.
+ ```yaml
+ name: Format PostgreSQL time field
+ transform:
+ - uses: add_field
+ with:
+ fields:
+ - field: formatted_time
+ language: sql
+ # Divide by 1000000 to convert microseconds to seconds
+ expression: TIME(event_time / 1000000, 'unixepoch', 'utc')
+ ```
+
+- `time with time zone` - a string representation of the time with timezone information, where the timezone is GMT (for example, `07:15:00Z`).
+ ```yaml
+ name: Format PostgreSQL time with time zone
+ transform:
+ - uses: add_field
+ with:
+ fields:
+ - field: formatted_time_with_tz
+ language: sql
+ expression: STRFTIME('%H:%M:%S', event_time_with_time_zone)
+ ```
+
+- `timestamp` - represented by Debezium as a 64-bit integer containing the microseconds since epoch. You can use the `STRFTIME` function to format it.
+ ```yaml
+ name: Format PostgreSQL timestamp field
+ transform:
+ - uses: add_field
+ with:
+ fields:
+ - field: formatted_timestamp
+ language: sql
+ # Divide by 1000000 to convert microseconds to seconds
+ expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_timestamp / 1000000, 'unixepoch')
+ ```
+
+- `timestamp with time zone` - represented by Debezium as a string containing the timestamp with time zone information, where the timezone is GMT (for example, `2025-06-07T10:15:00.000000Z`).
+ ```yaml
+ name: Format PostgreSQL timestamp with time zone
+ transform:
+ - uses: add_field
+ with:
+ fields:
+ - field: formatted_timestamp_with_tz
+ language: sql
+ # Divide by 1000000 to convert microseconds to seconds
+ expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_timestamp_with_time_zone)
+ ```
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/map-example.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/map-example.md
new file mode 100644
index 0000000000..5d31f19a31
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/map-example.md
@@ -0,0 +1,161 @@
+---
+Title: Restructure JSON or hash objects
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Restructure objects
+summary: Redis Data Integration keeps Redis in sync with a primary database in near
+ real time.
+type: integration
+weight: 40
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/map-example/'
+---
+
+By default, RDI adds fields to
+[hash]({{< relref "/develop/data-types/hashes" >}}) or
+[JSON]({{< relref "/develop/data-types/json" >}}) objects in the target
+database that closely match the columns of the source table.
+If you just want to limit the set fields in the output and/or rename some of them, you can use the
+[`output mapping`]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/remapping-the-output" >}}) configuration option.
+
+For situations where you want to create a new object structure with multiple levels or use calculations for the field values, you can use the
+[`map`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/map" >}})
+transformation, as described in the following sections.
+
+## Creating multilevel JSON objects
+
+You can use the `map` transformation to create a new structure for the output data, which can include nested objects and calculated fields. The `map` transformation allows you to define a new structure using an expression language, such as SQL or JavaScript.
+
+```yaml
+name: Create multilevel employee JSON
+source:
+ db: chinook
+ table: employee
+
+transform:
+ - uses: map
+ with:
+ expression: |
+ {
+ "id": employeeid,
+ "name": concat([firstname, ' ', upper(lastname)]),
+ "address": {
+ "street": address,
+ "city": city,
+ "state": state,
+ "postalCode": postalcode,
+ "country": country
+ },
+ "contact": {
+ "phone": phone,
+ "safeEmail": replace(replace(email, '@', '_at_'), '.', '_dot_')
+ }
+ }
+ language: jmespath
+
+output:
+ - uses: redis.write
+ with:
+ data_type: json
+ key:
+ expression: concat(['emp:', id])
+ language: jmespath
+```
+
+
+The example above creates a new JSON object with the following structure:
+ - A top-level `id` field that is the same as the `employeeid` field in the source table.
+ - A `name` field that is a concatenation of the `firstname` and `lastname` fields, with the `lastname` converted to uppercase.
+ - An `address` subobject that contains the `address`, `city`, `state`, `postalcode`, and `country` fields.
+ - A `contact` subobject that contains the `phone` field and a modified version of the `email` field, where the '@' sign and dots are replaced with '_at_' and '_dot_' respectively.
+
+The `output` section of the file configures the job to write
+to a JSON object with a custom key. Note that in the `output` section, you must refer to
+fields defined in the `map` transformation, so we use the new name `id`
+for the key instead of `employeeid`.
+
+
+
+If you query one of the new JSON objects, you see output like the following:
+
+```bash
+> JSON.GET emp:1 $
+"[{\"id\":1,\"name\":\"Andrew ADAMS\",\"address\":{\"street\":\"11120 Jasper Ave NW\",\"city\":\"Edmonton\",\"state\":\"AB\",\"postalCode\":\"T5K 2N1\",\"country\":\"Canada\"},\"contact\":{\"phone\":\"+1 (780) 428-9482\",\"safeEmail\":\"andrew_at_chinookcorp_dot_com\"}}]"
+```
+
+Formatted in the usual JSON style, the output looks like the sample below:
+
+```json
+{
+ "id": 1,
+ "name": "Andrew ADAMS",
+ "address": {
+ "street": "11120 Jasper Ave NW",
+ "city": "Edmonton",
+ "state": "AB",
+ "postalCode": "T5K 2N1",
+ "country": "Canada"
+ },
+ "contact": {
+ "phone": "+1 (780) 428-9482",
+ "safeEmail": "andrew_at_chinookcorp_dot_com"
+ }
+}
+```
+
+## Creating hash structure
+
+This example creates a new [hash]({{< relref "/develop/data-types/hashes" >}})
+object structure for items from the `track` table. Here, the `map` transformation uses
+[SQL](https://en.wikipedia.org/wiki/SQL) for the expression because this is often
+more suitable for hashes or "flat"
+JSON objects without subobjects or arrays. The expression renames some of the fields.
+It also calculates more human-friendly representations for the track duration (originally
+stored in the `milliseconds` field) and the storage size (originally stored in the
+`bytes` field).
+
+The full example is shown below:
+
+```yaml
+name: Create track hash with calculated fields
+source:
+ db: chinook
+ table: track
+transform:
+ - uses: map
+ with:
+ expression:
+ id: trackid
+ name: name
+ duration: concat(floor(milliseconds / 60000), ':', floor(mod(milliseconds / 1000, 60)))
+ storagesize: concat(round(bytes / 1048576.0, 2), 'MB')
+ language: sql
+output:
+ - uses: redis.write
+ with:
+ connection: target
+ data_type: hash
+ key:
+ expression: concat('track:', id)
+ language: sql
+```
+
+If you query the data for one of the `track` hash objects, you see output
+like the following:
+
+```bash
+> hgetall track:16
+1) "id"
+2) "16"
+3) "name"
+4) "Dog Eat Dog"
+5) "duration"
+6) "3:35.0"
+7) "storagesize"
+8) "6.71MB"
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-add-field-example.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-add-field-example.md
new file mode 100644
index 0000000000..a9831f2571
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-add-field-example.md
@@ -0,0 +1,180 @@
+---
+Title: Add new fields to a key
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Add new fields
+summary: Redis Data Integration keeps Redis in sync with a primary database in near
+ real time.
+type: integration
+weight: 40
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-add-field-example/'
+---
+
+By default, RDI adds fields to
+[hash]({{< relref "/develop/data-types/hashes" >}}) or
+[JSON]({{< relref "/develop/data-types/json" >}}) objects in the target
+database that match the columns of the source table.
+The examples below show how to add extra fields to the target data with the
+[`add_field`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/add_field" >}}) transformation.
+
+## Add a single field
+
+The first example adds a single field to the data.
+The `source` section selects the `customer` table of the
+[`chinook`](https://github.com/Redislabs-Solution-Architects/rdi-quickstart-postgres)
+database (the optional `db` value here corresponds to the
+`sources..connection.database` value defined in
+[`config.yaml`]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config" >}})).
+
+In the `transform` section, the `add_field` transformation adds an extra field called `localphone`
+to the object, which is created by removing the country and area code from the `phone`
+field with the
+[JMESPath]({{< relref "/integrate/redis-data-integration/1.19.1/reference/jmespath-custom-functions" >}}) function `regex_replace()`.
+You can also specify `sql` as the `language` if you prefer to create the new
+field with an [SQL](https://en.wikipedia.org/wiki/SQL) expression.
+
+The `output` section specifies `hash` as the `data_type` to write to the target, which
+overrides the default setting of `target_data_type` defined in `config.yaml`. Also, the
+`output.with.key` section specifies a custom key format of the form `cust:` where
+the `id` part is generated by the `uuid()` function.
+
+The full example is shown below:
+
+```yaml
+name: Add local phone field to customer
+source:
+ db: chinook
+ table: customer
+transform:
+ - uses: add_field
+ with:
+ expression: regex_replace(phone, '\+[0-9]+ (\([0-9]+\) )?', '')
+ field: localphone
+ language: jmespath
+output:
+ - uses: redis.write
+ with:
+ connection: target
+ data_type: hash
+ key:
+ expression: concat(['cust:', uuid()])
+ language: jmespath
+```
+
+If you queried the generated target data from the default transformation
+using [`redis-cli`]({{< relref "/develop/tools/cli" >}}), you would
+see something like the following:
+
+```
+ 1) "customerid"
+ 2) "27"
+ 3) "firstname"
+ 4) "Patrick"
+ 5) "lastname"
+ 6) "Gray"
+.
+.
+17) "phone"
+18) "+1 (520) 622-4200"
+.
+.
+```
+
+Using the job file above, the data also includes the new `localphone` field:
+
+```
+ 1) "customerid"
+ 2) "27"
+ 3) "firstname"
+ 4) "Patrick"
+ 5) "lastname"
+ 6) "Gray"
+ .
+ .
+23) "localphone"
+24) "622-4200"
+```
+
+## Add multiple fields
+
+The `add_field` transformation can also add multiple fields at the same time
+if you specify them under a `fields` subsection. The example below adds two
+fields to the `track` objects. The first new field, `seconds`, is created using a SQL
+expression to calculate the duration of the track in seconds from the
+`milliseconds` field.
+The second new field, `composerlist`, adds a JSON array using the `split()` function
+to split the `composer` string field wherever it contains a comma.
+
+```yaml
+name: Add multiple fields to track
+source:
+ db: chinook
+ table: track
+transform:
+ - uses: add_field
+ with:
+ fields:
+ - expression: floor(milliseconds / 1000)
+ field: seconds
+ language: sql
+ - expression: split(composer)
+ field: composerlist
+ language: jmespath
+output:
+ - uses: redis.write
+ with:
+ connection: target
+ data_type: json
+ key:
+ expression: concat(['track:', trackid])
+ language: jmespath
+```
+
+You can query the target database to see the new fields in
+the JSON object:
+
+```bash
+> JSON.GET track:1 $
+
+"[{\"trackid\":1,\"name\":\"For Those About To Rock (We Salute You)\",\"albumid\":1,\"mediatypeid\":1,\"genreid\":1,\"composer\":\"Angus Young, Malcolm Young, Brian Johnson\",\"milliseconds\":343719,\"bytes\":11170334,\"unitprice\":\"0.99\",\"seconds\":343,\"composerlist\":[\"Angus Young\",\" Malcolm Young\",\" Brian Johnson\"]}]"
+```
+
+## Using `add_field` with `remove_field`
+
+You can use the `add_field` and
+[`remove_field`]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-remove-field-example" >}})
+transformations together to completely replace fields from the source. For example,
+if you add a new `fullname` field, you might not need the separate `firstname` and
+`lastname` fields. You can remove them with a job file like the following:
+
+```yaml
+name: Add fullname and remove separate name fields
+source:
+ db: chinook
+ table: customer
+transform:
+ - uses: add_field
+ with:
+ expression: concat(firstname, ' ', lastname)
+ field: fullname
+ language: sql
+ - uses: remove_field
+ with:
+ fields:
+ - field: firstname
+ - field: lastname
+output:
+ - uses: redis.write
+ with:
+ connection: target
+ data_type: hash
+ key:
+ expression: concat(['cust:', customerid])
+ language: jmespath
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-expiration-example.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-expiration-example.md
new file mode 100644
index 0000000000..2d2d2cf76a
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-expiration-example.md
@@ -0,0 +1,91 @@
+---
+Title: Set custom expiration times / TTL
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Set expiration times / TTL
+summary: How to set expiration times / TTL for keys
+type: integration
+weight: 40
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-expiration-example/'
+---
+
+
+You can configure custom key expiration times (TTL) for keys written to Redis by using the `expire` parameter in the `output` section of the job file. This parameter specifies the duration, in seconds, that a newly created key will remain in Redis before being automatically deleted. If the `expire` parameter is not provided, the keys will persist indefinitely.
+
+There are two ways to set the expiration time:
+
+- as a static value
+- as a dynamic value using a JMESPath or SQL expression
+
+
+## Static expiration time
+
+The following example sets the expiration time to 100 seconds for all keys:
+
+```yaml
+name: Static expiration example
+output:
+ - uses: redis.write
+ with:
+ data_type: hash
+ expire: 100
+```
+
+## Dynamic expiration time
+
+You can use a JMESPath or SQL expression to set the expiration time dynamically when it is based on a field in the source data. For example, you can set the expiration time to the value of a `ttl` field in the source data:
+
+```yaml
+name: Dynamic expiration from field
+output:
+ - uses: redis.write
+ with:
+ data_type: hash
+ expire:
+ expression: ttl
+ language: jmespath
+```
+
+## Dynamic expiration time based on a date, datetime, or timestamp field
+
+In some cases, you can also set the expiration time based on a field that contains a date, datetime, or timestamp value, but it depends on the source database and the data types it supports. See the examples below for your specific source database and data type.
+
+There are two main approaches you can use to set the expiration time based on a date, datetime, or timestamp field:
+
+- For values representing an elapsed time since epoch start (in milliseconds, for example), you have to convert the value to seconds since epoch and then subtract the current time (also in seconds since epoch). The difference between the two is the time until expiration.
+
+ ```yaml
+ name: Expiration from timestamp in milliseconds
+ output:
+ - uses: redis.write
+ with:
+ data_type: hash
+ expire:
+ # To set the expiration time to a date field, convert the value to
+ # seconds (e.g. divide it by 1000 if the fields has milliseconds precision)
+ # and subtract the current time in seconds since epoch.
+ expression: EXPIRES_TIMESTAMP / 1000 - STRFTIME('%s', 'now')
+ language: sql
+ ```
+
+- For values matching the subset of ISO 8601 supported by SQLite (for example, `2023-10-01T12:00:00`, `2023-10-01T12:00:00Z`, or `2025-06-05T13:40:14.784000+02:00`), you can use the `STRFTIME` function to convert the value to seconds since epoch and subtract the current time in seconds since epoch from it.
+
+ ```yaml
+ name: Expiration from ISO 8601 datetime
+ output:
+ - uses: redis.write
+ with:
+ data_type: hash
+ expire:
+ language: sql
+ expression: STRFTIME('%s', EXPIRATION_TS) - STRFTIME('%s', 'now')
+ ```
+
+For more examples of how to manipulate date and time values, see [Formatting date and time values]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/formatting-date-and-time-values/">}}).
+
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-hash-example.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-hash-example.md
new file mode 100644
index 0000000000..0ca7d0808a
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-hash-example.md
@@ -0,0 +1,39 @@
+---
+Title: Write to a Redis hash
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Write to a Redis hash
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 30
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-hash-example/'
+---
+
+In the following example, the data is captured from the source table named `invoice` and is written to the Redis database as hash keys. The `connection` is an optional parameter that refers to the corresponding connection name defined in `config.yaml`.
+When you specify the `data_type` parameter for the job, it overrides the system-wide setting `target_data_type` defined in `config.yaml`.
+
+In this case, the result will be Redis hashes with key names based on the key expression (for example, `invoice_id:1`) and with an expiration of 100 seconds.
+If you don't supply an `expire` parameter, the keys will never expire.
+
+```yaml
+name: Write invoice to hash
+source:
+ schema: public
+ table: invoice
+output:
+ - uses: redis.write
+ with:
+ connection: target
+ data_type: hash
+ key:
+ expression: concat(['invoice_id:', InvoiceId])
+ language: jmespath
+ expire: 100
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-json-example.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-json-example.md
new file mode 100644
index 0000000000..fc99847bc2
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-json-example.md
@@ -0,0 +1,45 @@
+---
+Title: Write to a Redis JSON document
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Write to a Redis JSON document
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 30
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-json-example/'
+---
+
+{{}}
+You must enable the [RedisJSON]({{< relref "/develop/data-types/json" >}}) module in the target Redis
+database to use this feature.
+{{ }}
+
+In the example below, the data is captured from the source table named `invoice` and is written to the Redis database as a JSON document. The `connection` is an optional parameter that refers to the corresponding connection name defined in `config.yaml`. When you specify the `data_type` parameter for the job, it overrides the system-wide setting `target_data_type` defined in `config.yaml`.
+
+Another optional parameter, `on_update`, specifies the writing strategy. You can set this to either `replace` (the default) or `merge`. This affects the way the document is written to the target. Replacing the document will overwrite it completely, while merging will update it with the fields captured in the source, keeping the rest of the document intact. The `replace` option is usually more performant, while `merge` allows other jobs and applications to set extra fields in the same JSON documents.
+
+In this case, the result will be Redis JSON documents with key names based on the key expression (for example, `invoice_id:1`) and with an expiration of 100 seconds. If you don't supply an `expire` parameter, the keys will never expire.
+
+```yaml
+name: Write invoice to JSON
+source:
+ schema: public
+ table: invoice
+output:
+ - uses: redis.write
+ with:
+ connection: target
+ data_type: json
+ key:
+ expression: concat(['invoice_id:', InvoiceId])
+ language: jmespath
+ on_update: replace
+ expire: 100
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-lookup-example.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-lookup-example.md
new file mode 100644
index 0000000000..66e97a2096
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-lookup-example.md
@@ -0,0 +1,178 @@
+---
+Title: Reading Redis data with redis.lookup
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Reading Redis data with redis.lookup
+summary: Redis Data Integration keeps Redis in sync with a primary database in near
+ real time.
+type: integration
+weight: 40
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-lookup-example/'
+---
+
+You can use the
+[`redis.lookup`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/lookup" >}})
+transformation to read existing data from Redis during the `transform` stage of a
+job. This lets you enrich an incoming record with values that are already present
+in the target database.
+
+For example, a pipeline for the Chinook database might read an `artist` record
+that is already stored in Redis and use `redis.lookup` in an `album` table job to
+add selected artist details to each album record before writing it to the target
+database.
+
+{{< warning >}}
+Do not rely on `redis.lookup` to *denormalize* data that RDI writes from another
+table in the **same pipeline**. RDI can't guarantee that the looked-up data will be
+present or up to date when the lookup runs, for the following reasons:
+
+- **Snapshot order isn't guaranteed.** During the initial snapshot, RDI can't
+ guarantee that the table you look up is ingested before the table that depends
+ on it. If a dependent job runs before the referenced key has been written, the
+ lookup misses.
+- **Change (CDC) order isn't guaranteed.** If a parent and child record are
+ inserted or updated at around the same time, RDI has no way to order these
+ events, so the lookup can still miss.
+- **Parent updates don't refresh existing keys.** Even if the lookup succeeds,
+ updating the source record later does *not* update the keys that already copied
+ its values. The denormalized data becomes stale.
+
+The only case where `redis.lookup` is safe for enrichment is when you can guarantee
+that the looked-up data is present in the target database *independently* of the
+RDI pipeline (for example, a reference table that is loaded and maintained
+separately).
+
+To denormalize data that RDI ingests, use a supported technique instead. See
+[Data denormalization]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization" >}})
+for one-to-one joins (using `merge`) and one-to-many joins (using nesting).
+{{< /warning >}}
+
+## Reading a hash field
+
+The `redis.lookup` transformation works by executing a Redis command and adding the
+result to the record. You specify the command and its arguments in the
+`transform` configuration with the `cmd` and `args` properties. For example, the
+following transformation job uses the
+[`HGET`]({{< relref "/commands/hget" >}}) command to read the `name` field from an
+artist [hash]({{< relref "/develop/data-types/hashes" >}}) and adds it to the
+album record under the `artist` field. A particularly important thing to note
+here is that the `args` elements are all interpreted as [JMESPath](https://jmespath.org/)
+expressions, but YAML syntax allows for each element to be a quoted string. This means that
+you must *double quote* any string arguments that you want to be treated as
+literal strings (as with `name` below), otherwise JMESPath will try to interpret
+them as field names, which will generally give the wrong result. Specifically, use
+a different quote character for the outer quotes and the inner quotes.
+
+```yaml
+source:
+ table: album
+transform:
+ - uses: redis.lookup
+ with:
+ connection: target
+ cmd: HGET
+ args:
+ - concat(['artist:artistid:', artistid])
+ - '`name`'
+ language: jmespath
+ field: artist
+output:
+ - uses: redis.write
+ with:
+ connection: target
+ data_type: hash
+ key:
+ expression: concat(['album:albumid:', albumid])
+ language: jmespath
+```
+
+Before the lookup runs, the album hash object contains only the `artistid` field to
+reference the artist:
+
+```bash
+> hgetall album:albumid:1
+1) "albumid"
+2) "1"
+3) "title"
+4) "For Those About To Rock We Salute You"
+5) "artistid"
+6) "1"
+```
+
+After running the job specified above, querying one of the album hash objects shows the
+extra `artist` field obtained by looking up the artist with the `artistid`:
+
+```bash
+> hgetall album:albumid:1
+1) "albumid"
+2) "1"
+3) "title"
+4) "For Those About To Rock We Salute You"
+5) "artistid"
+6) "1"
+7) "artist"
+8) "AC/DC"
+```
+
+## Embedding a JSON document
+
+If you are using [JSON]({{< relref "/develop/data-types/json" >}}) objects,
+you can read the whole of one object and embed it
+as a field of another. The following example shows how to do this using a temporary field
+to hold the result of the `redis.lookup` command. It then uses
+[`add_field`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/add_field" >}})
+to insert the new field and
+[`remove_field`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/remove_field" >}})
+to remove the temporary field and the now-redundant `artistid` field before writing the album object.
+
+```yaml
+source:
+ table: album
+transform:
+ - uses: redis.lookup
+ with:
+ connection: target
+ cmd: JSON.GET
+ args:
+ - concat(['artist:artistid:', artistid])
+ language: jmespath
+ field: artiststring
+ - uses: add_field
+ with:
+ field: artist
+ language: jmespath
+ expression: json_parse(artiststring)
+ - uses: remove_field
+ with:
+ fields:
+ - field: artistid
+ - field: artiststring
+output:
+ - uses: redis.write
+ with:
+ connection: target
+ data_type: json
+ key:
+ expression: concat(['album:albumid:', albumid])
+ language: jmespath
+```
+
+After running this job, the album JSON object includes the artist object
+in a new `artist` field:
+
+```json
+{
+ "albumid": 239,
+ "title": "War",
+ "artist": {
+ "artistid": 150,
+ "name": "U2"
+ }
+}
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-opcode-example.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-opcode-example.md
new file mode 100644
index 0000000000..fc969d728f
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-opcode-example.md
@@ -0,0 +1,103 @@
+---
+Title: Using the operation code
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Using the operation code
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 100
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-opcode-example/'
+---
+
+The operation code (`opcode`) is a metadata field that indicates the type of operation that generated the change in the source database. It can be useful for tracking changes and understanding the context of the data being processed.
+
+The opcode is only available in the [full row format]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-row-format#full" >}}), and can be accessed in the `transform` and `output` sections of the job file.
+
+It has one of the following values:
+
+| Opcode | Operation | Notes |
+|--------|-----------|-------|
+| `c` | Create row | |
+| `u` | Update row | |
+| `d` | Delete row | |
+| `r` | Read row (snapshot) | Applies only to snapshots. |
+| `t` | Truncate table | PostgreSQL specific. |
+| `m` | Message event | PostgreSQL specific. |
+
+
+You can add the value of the operation code to the output, and also use it in a conditional expression to modify the behavior of the job. The following examples demonstrate the different use-cases.
+
+### Adding the operation code to the output
+
+Use the `add_field` transformation to add a new field that contains the value of the `opcode` field from the source data. Note that the fields must be prefixed with `after` to be included in the output.
+
+
+```yaml
+name: Add operation code to employee
+source:
+ schema: public
+ table: employee
+ row_format: full
+
+transform:
+ # add the operation code to the data
+ - uses: add_field
+ with:
+ field: after.operation_code
+ expression: opcode
+ language: jmespath
+```
+
+
+### Filtering operation by output code.
+
+In some cases you may want to ignore certain operations (for example, you may not be interested in deletions). Use the `filter` transformation to filter out any operations you don't need to process.
+
+```yaml
+name: Filter out delete operations
+source:
+ schema: public
+ table: employee
+ row_format: full
+
+transform:
+ - uses: filter
+ with:
+ expression: opcode != 'd'
+ language: jmespath
+```
+
+### Modifying the output based on the operation code
+
+The previous example filters out specific operations, but you can also modify the output based on the operation code. For example, you can add a new field that tracks the status of the record based on the operation code.
+
+Note that when a source record is deleted, you must modify the value of the `opcode` field if you want to prevent the corresponding record in the target database from being removed automatically.
+
+```yaml
+name: Track status based on operation code
+source:
+ schema: public
+ table: employee
+ row_format: full
+
+transform:
+ - uses: add_field
+ with:
+ fields:
+ # Here you set the value of the field based on the value of the opcode field
+ - field: after.status
+ expression: opcode == 'd' && 'inactive' || 'active'
+ language: jmespath
+
+ # You have to change the value of the opcode field to prevent deletion
+ - field: opcode
+ expression: opcode == 'd' && 'u' || opcode
+ language: jmespath
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-remove-field-example.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-remove-field-example.md
new file mode 100644
index 0000000000..2a73cfa630
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-remove-field-example.md
@@ -0,0 +1,169 @@
+---
+Title: Remove fields from a key
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Remove fields
+summary: Redis Data Integration keeps Redis in sync with a primary database in near
+ real time.
+type: integration
+weight: 40
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-remove-field-example/'
+---
+
+By default, RDI adds fields to
+[hash]({{< relref "/develop/data-types/hashes" >}}) or
+[JSON]({{< relref "/develop/data-types/json" >}}) objects in the target
+database for each of the columns of the source table.
+The examples below show how to omit some of those fields from the target data with the
+[`remove_field`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/remove_field" >}}) transformation.
+
+## Remove a single field
+
+The first example removes a single field from the data.
+The `source` section selects the `employee` table of the
+[`chinook`](https://github.com/Redislabs-Solution-Architects/rdi-quickstart-postgres)
+database (the optional `db` field here corresponds to the
+`sources..connection.database` field defined in
+[`config.yaml`]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config" >}})).
+
+In the `transform` section, the `remove_field` transformation removes the
+`hiredate` field.
+
+The `output` section specifies `hash` as the `data_type` to write to the target, which
+overrides the default setting of `target_data_type` defined in `config.yaml`. Also, the
+`output.with.key` section specifies a custom key format of the form `emp:`.
+Note that any fields you remove in the `transform` section are not available for
+the key calculation in the `output` section.
+
+The full example is shown below:
+
+```yaml
+name: Remove hiredate from employee
+source:
+ db: chinook
+ table: employee
+transform:
+ - uses: remove_field
+ with:
+ field: hiredate
+output:
+ - uses: redis.write
+ with:
+ connection: target
+ data_type: hash
+ key:
+ expression: concat(['emp:', employeeid])
+ language: jmespath
+```
+
+If you queried the generated target data from the default transformation
+using [`redis-cli`]({{< relref "/develop/tools/cli" >}}), you would
+see something like the following:
+
+```bash
+> hgetall emp:8
+ 1) "employeeid"
+ 2) "8"
+ 3) "lastname"
+ 4) "Callahan"
+ 5) "firstname"
+ 6) "Laura"
+ 7) "title"
+ 8) "IT Staff"
+ 9) "reportsto"
+10) "6"
+11) "birthdate"
+12) "-62467200000000"
+13) "hiredate"
+14) "1078358400000000"
+15) "address"
+16) "923 7 ST NW"
+.
+.
+```
+
+Using the job file above, the data omits the `hiredate` field:
+
+```bash
+ > hgetall emp:8
+ 1) "employeeid"
+ 2) "8"
+ 3) "lastname"
+ 4) "Callahan"
+ 5) "firstname"
+ 6) "Laura"
+ 7) "title"
+ 8) "IT Staff"
+ 9) "reportsto"
+10) "6"
+11) "birthdate"
+12) "-62467200000000"
+13) "address"
+14) "923 7 ST NW"
+.
+.
+```
+
+## Remove multiple fields
+
+The `remove_field` transformation can also remove multiple fields at the same time
+if you specify them under a `fields` subsection. The example below is similar
+to the previous one but also removes the `birthdate` field:
+
+```yaml
+name: Remove multiple date fields from employee
+source:
+ db: chinook
+ table: employee
+transform:
+ - uses: remove_field
+ with:
+ fields:
+ - field: hiredate
+ - field: birthdate
+output:
+ - uses: redis.write
+ with:
+ connection: target
+ data_type: hash
+ key:
+ expression: concat(['emp:', employeeid])
+ language: jmespath
+```
+
+If you query the data, you can see that it also omits the
+`birthdate` field:
+
+```bash
+> hgetall emp:8
+ 1) "employeeid"
+ 2) "8"
+ 3) "lastname"
+ 4) "Callahan"
+ 5) "firstname"
+ 6) "Laura"
+ 7) "title"
+ 8) "IT Staff"
+ 9) "reportsto"
+10) "6"
+11) "address"
+12) "923 7 ST NW"
+.
+.
+```
+
+## Using `remove_field` with `add_field`
+
+The `remove_field` transformation is very useful in combination with
+[`add_field`]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-add-field-example" >}}).
+For example, if you use `add_field` to concatenate a person's first
+and last names, you may not need separate `firstname` and `lastname`
+fields, so you can use `remove_field` to omit them.
+See [Using `add_field` with `remove_field`]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-add-field-example#using-add_field-with-remove_field" >}})
+for an example of how to do this.
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-row-format.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-row-format.md
new file mode 100644
index 0000000000..c1b03f0b65
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-row-format.md
@@ -0,0 +1,156 @@
+---
+Title: Row Format
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Row Format
+summary: Explanation of the row formats supported by Redis Data Integration jobs.
+type: integration
+weight: 30
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-row-format/'
+---
+
+
+The RDI pipelines support two separate row formats which you can specify in the `source` section of the job file:
+
+- `partial` - (Default) Contains the current value of the row only.
+- `full` - Contains all information available for the row, including the key, the before and after values, and the operation code.
+
+The `full` row format is useful when you want to access the metadata associated with the row, such as the operation code, and the before and after values.
+The structure of the data passed to the `transform` and `output` sections is different depending on the row format you choose. Consider which row format you are using when you reference keys.
+The following two examples demonstrate the difference between the two row formats.
+
+## Default row format
+
+With the default row format, the input value is a JSON object containing the current value of the row, and fields can be referenced directly by their name.
+
+Usage example:
+
+```yaml
+name: Default row format example
+source:
+ table: addresses
+transform:
+ - uses: add_field
+ with:
+ field: city_state
+ expression: concat([CITY, ', ', STATE])
+ language: jmespath
+ - uses: add_field
+ with:
+ field: op_code_value
+ # Operation code is not available in standard row format
+ # so the following expression will result in `op_code - None`
+ expression: concat(['op_code', ' - ', opcode])
+ language: jmespath
+output:
+ - uses: redis.write
+ with:
+ data_type: hash
+ key:
+ expression: concat(['addresses', '#', ID])
+ language: jmespath
+```
+
+
+## Full row format {#full}
+
+With `row_format: full` the input value is a JSON object with the following structure:
+
+- `key` - An object containing the attributes of the primary key. For example, `key.id` will give you the value of the `id` column as long as it is part of the primary key.
+- `before` - An object containing the previous value of the row.
+- `after` - An object containing the current value of the row.
+- `opcode` - The operation code. See [Using the operation code]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-opcode-example" >}}) for more information about the possible opcode values and how to use them.
+- `db` - The database name.
+- `table` - The table name.
+- `schema` - The schema name.
+
+Note: The `db` and `schema` fields are database-specific and may not be available in all databases. For example, MySQL doesn't use `schema` and uses `db` as the database name.
+
+
+Usage example:
+
+```yaml
+name: Full row format example
+source:
+ table: addresses
+ row_format: full
+transform:
+ - uses: add_field
+ with:
+ # opcode is only available in full row format and can be used in the transformations
+ field: after.op_code_value
+ expression: address
+ language: jmespath
+ - uses: add_field
+ with:
+ field: after.city_state
+ # Note that we need to use the `after` prefix to access the current value of the row
+ # or `before` to access the previous value
+ expression: concat([after.CITY, ', ', after.STATE])
+ language: jmespath
+output:
+ - uses: redis.write
+ with:
+ data_type: hash
+ key:
+ # There are different ways to express the key
+ # If the `ID` column is the primary key the following expressions
+ # are equivalent - `key.ID`, `after.ID`, `values(key)[0]`
+ expression: concat(['addresses-full', '#', values(key)[0]])
+ language: jmespath
+```
+
+### Important notes when using `row_format: full`
+
+- The `before` object will be `null` for `insert` and `create` operations, and the `after` object will be `null` for `delete` operations. If you are building the output key manually, you should account for this and ensure that you are not trying to access fields from a `null` object, as shown in the example below:
+
+ ```yaml
+ name: Handle delete operations with full row format
+ source:
+ table: addresses
+ row_format: full
+
+ output:
+ - uses: redis.write
+ with:
+ key:
+ language: jmespath
+ # The following pattern will fail for delete operations. In those cases `after` is null, the resulting key will
+ # be 'addresses:None' and the key won't be removed from the target
+ # expression: concat(['addresses:', after.ID])
+
+ # This pattern works for all operations, by using the ID from the `after` object if it is available,
+ # and falling back to the ID from the `before` object if not.
+ expression: concat(['addresses:', after.ID || before.ID])
+
+ # Another option is to use the ID from the `key` object
+ # expression: concat(['addresses:', values(key)[0]])
+ ```
+
+ Please note that you should not use `key` in combination with `row_format: full` and more than one output, as the `key` object will be overwritten by the previous output. This is a known limitation of the current implementation and is subject to change in future versions.
+
+
+- The final result of the processing (which is what will be stored in the output) is the value of the `after` object. This means you must reference the fields using the `after` prefix unless you change the output structure in a transformation step. Also, when you add new fields, you must prefix them with `after.` to ensure that they are added to the correct part of the output:
+
+ ```yaml
+ name: Add fields with full row format
+ source:
+ table: addresses
+ row_format: full
+
+ transform:
+ - uses: add_field
+ with:
+ field: after.city_state # use this to add the new field to the final output
+ # field: city_state # use this if you need a temporary field in the transformation steps, but not in the final output
+ expression: concat([after.CITY, ', ', after.STATE])
+ language: jmespath
+ ```
+
+- When using the `full` row format, you should prepend all data fields in transformation and key calculations with `before` or `after` depending on whether you want to access the previous or current value of the row. The only exception to this is the `mapping` section where only the `after` values are available, so you should use the column/value name without the `after` prefix.
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-set-example.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-set-example.md
new file mode 100644
index 0000000000..70274871bb
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-set-example.md
@@ -0,0 +1,40 @@
+---
+Title: Write to a Redis set
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Write to a Redis set
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 30
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-set-example/'
+---
+
+In the example below, data is captured from the source table named `invoice` and is written to a Redis set. The `connection` is an optional parameter that refers to the corresponding connection name defined in `config.yaml`. When you specify the
+`data_type` parameter for the job, it overrides the system-wide setting `target_data_type` defined in `config.yaml`.
+
+When writing to a set, you must supply an extra argument, `member`, which specifies the field that will be written. In this case, the result will be a Redis set with key names based on the key expression (for example, `invoices:Germany`, `invoices:USA`) and with an expiration of 100 seconds. If you don't supply an `expire` parameter, the keys will never expire.
+
+```yaml
+name: Write invoices to set by country
+source:
+ schema: public
+ table: invoice
+output:
+ - uses: redis.write
+ with:
+ connection: target
+ data_type: set
+ key:
+ expression: concat(['invoices:', BillingCountry])
+ language: jmespath
+ args:
+ member: InvoiceId
+ expire: 100
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-set-key-name.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-set-key-name.md
new file mode 100644
index 0000000000..d2d052a3dd
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-set-key-name.md
@@ -0,0 +1,89 @@
+---
+Title: Set the key name in the target database
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Set the key name in the target database
+summary: Learn how to customize Redis key names when synchronizing data from your primary database using Redis Data Integration.
+type: integration
+weight: 40
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-set-key-name/'
+---
+
+## Understanding Default Key Names
+
+When RDI synchronizes data from your primary database to Redis, it automatically generates key names based on a specific pattern.
+By default, RDI creates keys using the following format:
+
+* **Single primary key**: `tablename:primarykeyname:primarykeyvalue`
+* **Composite primary keys**: `tablename:key1name:key1value:key2name:key2value`
+
+Examples
+
+* For a table named `employee` with primary key `employeeid`, a record with `employeeid=1` will have the key `employee:employeeid:1`
+* For a table named `orders` with composite primary keys `orderid` and `customerid`, a record with `orderid=1` and `customerid=2` will have the key `orders:orderid:1:customerid:2`
+
+## Customizing Key Names
+
+While the default key naming convention works for many use cases, you may need custom key formats to:
+
+* Match existing Redis key patterns
+* Create more concise key names
+* Implement application-specific naming schemes
+* Optimize for specific access patterns
+
+To customize key names, use the `key` section within the `redis.write` output configuration. This section requires two parameters:
+
+* `expression`: Defines the custom key format using a supported expression language
+* `language`: Specifies the expression language to use (`jmespath` or `sql`)
+
+### Example
+
+
+```yaml
+name: Custom key name for customers
+source:
+ db: inventory
+ table: customers
+output:
+ - uses: redis.write
+ with:
+ key:
+ expression: concat(['customers', '#', id])
+ language: jmespath
+```
+
+## Special Considerations for Full Row Format
+
+When working with the full row format, you need to handle key generation differently to ensure proper behavior across all operation types (create, update, and delete). You must reference attributes correctly to ensure consistent key generation, especially for delete operations where the "after" state is empty. The example below demonstrates how to handle this:
+
+```yaml
+name: Custom key with full row format
+source:
+ db: inventory
+ table: customers
+ row_format: full
+output:
+ - uses: redis.write
+ with:
+ key:
+ # Here we use the operation code to determine the value of the key to ensure that
+ # delete operations will result in the correct key being deleted
+ expression: concat(['customers', '#', opcode == 'd' && before.id || after.id])
+ language: jmespath
+```
+
+## Summary
+
+Proper key naming is essential for effective data organization in Redis. RDI provides:
+
+1. **Default key naming** that follows a consistent pattern based on table and primary key information
+2. **Custom key naming** through the `key` section with `expression` and `language` parameters
+3. **Special handling for full row format** to ensure consistent key generation across all operation types
+
+By understanding and utilizing these options, you can ensure your Redis keys are optimally structured for your specific use case, making data retrieval more efficient and your Redis implementation more maintainable.
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-sorted-set-example.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-sorted-set-example.md
new file mode 100644
index 0000000000..3b103db4a3
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-sorted-set-example.md
@@ -0,0 +1,47 @@
+---
+Title: Write to a Redis sorted set
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Write to a Redis sorted set
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 30
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-sorted-set-example/'
+---
+
+In the example below, data is captured from the source table named `invoice` and is written to a Redis sorted set. The `connection` is an optional parameter that refers to the corresponding connection name defined in `config.yaml`. When
+you specify the `data_type` parameter for the job, it overrides the system-wide setting `target_data_type` defined in `config.yaml`.
+
+When writing to sorted sets, you must provide two additional arguments, `member` and `score`. These specify the field names that will be used as a member and a score to add an element to a sorted set. In this case, the result will be a Redis sorted set named `invoices:sorted` based on the key expression and with an expiration of 100 seconds for each set member. If you don't supply an `expire` parameter, the keys will never expire.
+
+```yaml
+name: Write invoices to sorted set by total
+source:
+ schema: public
+ table: invoice
+output:
+ - uses: redis.write
+ with:
+ connection: target
+ data_type: sorted_set
+ key:
+ expression: "`invoices:sorted`"
+ language: jmespath
+ args:
+ score: Total
+ member: InvoiceId
+ expire: 100
+```
+
+Since sorted sets in Redis are inherently sorted, you can easily get the top N invoices by total invoice amount using the command below (the range 0..9 gets the top 10 invoices):
+
+```
+ZREVRANGE invoices:sorted 0 9 WITHSCORES
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-sql-case-example.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-sql-case-example.md
new file mode 100644
index 0000000000..1dbe5cecbd
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-sql-case-example.md
@@ -0,0 +1,65 @@
+---
+Title: Using SQL CASE
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Using SQL CASE
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 30
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-sql-case-example/'
+---
+
+The [`CASE`](https://www.w3schools.com/sql/sql_case.asp) statement allows you to specify conditions and return different values based on those conditions. You can use it both to create new fields or filter existing data.
+
+## Using SQL CASE to create a new field
+The example below demonstrates how to use the `CASE` statement to create a new field called `Market` based on the value of the `BillingCountry` field in the `Invoice` table. The new field categorizes countries into regions such as "North America" and "Europe".
+
+```yaml
+name: Add market field using SQL CASE
+source:
+ table: Invoice
+
+transform:
+ - uses: add_field
+ with:
+ field: "Market"
+ language: sql
+ expression: |
+ CASE
+ WHEN BillingCountry = 'USA' THEN 'North America'
+ WHEN BillingCountry = 'Canada' THEN 'North America'
+ WHEN BillingCountry = 'UK' THEN 'Europe'
+ WHEN BillingCountry = 'France' THEN 'Europe'
+ ELSE 'Other'
+ END
+```
+
+## Using SQL CASE to filter data
+
+You can also use the `CASE` statement to filter data based on specific conditions. The example below demonstrates how to filter the `Invoice` table to include only invoices from the USA and Canada that have a `Total` value above their country-specific threshold.
+
+Because the `Total` field is a Decimal in the source table, it is represented as a string in Debezium and so you must cast it to `REAL` to compare it numerically in the `CASE` statement. Without this cast, it will be compared as a string value, which will give the wrong result.
+
+```yaml
+name: Filter invoices by country and total
+source:
+ table: Invoice
+
+transform:
+ - uses: filter
+ with:
+ language: sql
+ expression: |
+ CASE
+ WHEN BillingCountry = 'USA' AND CAST(Total AS REAL) > 11.99 THEN True
+ WHEN BillingCountry = 'Canada' AND CAST(Total AS REAL) > 9.99 THEN True
+ ELSE False
+ END
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-stream-example.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-stream-example.md
new file mode 100644
index 0000000000..e86b92c050
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-stream-example.md
@@ -0,0 +1,48 @@
+---
+Title: Write to a Redis stream
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Write to a Redis stream
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 30
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-stream-example/'
+---
+
+In the example below, data is captured from the source table named `invoice` and is written to a Redis stream. The `connection` is an optional parameter that refers to the corresponding connection name defined in `config.yaml`.
+When you specify the `data_type` parameter for the job, it overrides the system-wide setting `target_data_type` defined in `config.yaml`.
+
+When writing to streams, you can use the optional parameter `mapping` to limit the number of fields sent in a message and to provide aliases for them. If you don't use the `mapping` parameter, all fields captured in the source will be passed as the message payload.
+
+Note that streams are different from other data structures because existing messages are never updated or deleted. Any operation in the source will generate a new message with the corresponding operation code (`op_code` field) that is automatically added to the message payload.
+
+In this case, the result will be a Redis stream with the name based on the key expression (for example, `invoice:events`) and with an expiration of 100 seconds for the whole stream. If you don't supply an `expire` parameter, the keys will never expire.
+
+In the example, only three original fields are passed in the message payload: `InvoiceId` (as `message_id`), `BillingCountry` (as `country`), `Total` (as `Total`, no alias provided) and `op_code`, which is implicitly added to all messages sent to streams.
+
+```yaml
+name: Write invoice events to stream
+source:
+ schema: public
+ table: invoice
+output:
+ - uses: redis.write
+ with:
+ connection: target
+ data_type: stream
+ key:
+ expression: "`invoice:events`"
+ language: jmespath
+ mapping:
+ - InvoiceId: message_id
+ - BillingCountry: country
+ - Total
+ expire: 100
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-string-example.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-string-example.md
new file mode 100644
index 0000000000..e1870b19ae
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-string-example.md
@@ -0,0 +1,52 @@
+---
+Title: Write to a Redis string
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Write to a Redis string
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 30
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-string-example/'
+---
+
+The string data type is useful for capturing a string representation of a single column from
+a source table.
+
+In the example job below, the `title` column is captured from the `album` table in the source.
+The `title` is then written to the Redis target database as a string under a custom key of the
+form `AlbumTitle:42`, where the `42` is the primary key value of the table (the `albumid` column).
+
+The `connection` is an optional parameter that refers to the corresponding connection name defined in
+[`config.yaml`]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config" >}}).
+When you specify the `data_type` parameter for the job, it overrides the system-wide setting `target_data_type` defined in `config.yaml`. Here, the `string` data type also requires an `args` subsection
+with a `value` argument that specifies the column you want to capture from the source table.
+
+The optional `expire` parameter sets the length of time, in seconds, that a new key will
+persist for after it is created (here, it is 86400 seconds, which equals one day).
+After this time, the key will be deleted automatically.
+If you don't supply an `expire` parameter, the keys will never expire.
+
+```yaml
+name: Write album title to string
+source:
+ table: album
+ row_format: full
+output:
+ - uses: redis.write
+ with:
+ connection: target
+ data_type: string
+ key:
+ expression: concat(['AlbumTitle:', values(key)[0]])
+ language: jmespath
+ args:
+ value: title
+ expire: 86400
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-write-same-key.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-write-same-key.md
new file mode 100644
index 0000000000..6f05272bc2
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-write-same-key.md
@@ -0,0 +1,103 @@
+---
+Title: Write to the same key from multiple jobs
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Write to the same key
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 100
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-write-same-key/'
+---
+
+Use this pattern when two or more jobs write related source entities, such as
+`customer` and `address`, to the same Redis JSON document.
+
+When multiple jobs write to the same Redis key, a delete event from any of the
+source entities can delete the key from the target. To work around this, use
+[`row_format: full`]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-row-format#full" >}})
+so the job can inspect the
+[`opcode`]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-opcode-example" >}}),
+convert delete events into update events before writing to Redis, and write JSON
+documents with `on_update: merge`.
+
+{{< note >}}
+Use the same key expression in all jobs that write to the shared Redis key. For
+delete events, read key values from `before` or `key` because `after` is `null`.
+{{< /note >}}
+
+## Customer job
+
+```yaml
+# jobs/customer.yaml
+name: customers
+
+source:
+ table: customers
+
+output:
+ - uses: redis.write
+ with:
+ data_type: json
+ on_update: merge
+ key:
+ expression: concat(['customer:', id])
+ language: jmespath
+
+```
+
+## Address job
+
+For delete events from the `addresses` table, this job sets all fields to `null`
+to instruct RDI to remove them from the target JSON document. This behavior is
+available in RDI 1.15.0 or later when native JSON merge is enabled and the target
+database uses RedisJSON 2.6.0 or later.
+
+```yaml
+# jobs/addresses.yaml
+name: addresses
+
+source:
+ table: addresses
+ row_format: full
+
+transform:
+ - uses: add_field
+ with:
+ fields:
+ # For create/update records, we take the new values as is.
+ # If the record is a deletion, we set all fields to null.
+ - field: after
+ expression: |
+ (opcode != 'd' && after)
+ ||
+ from_entries(to_entries(before)[].{key: key, value: `null`})
+ language: jmespath
+
+ # Treat deletes as updates so that we can use the same output configuration
+ - field: opcode
+ expression: opcode == 'd' && 'u' || opcode
+ language: jmespath
+
+ # If you have overlapping field names (for example, FK and PK have the same name, or both tables have
+ # a field called "id"), you may want to remove the field from the after object to prevent it
+ # from overwriting the PK.
+ - uses: remove_field
+ with:
+ field: after.id
+
+output:
+ - uses: redis.write
+ with:
+ data_type: json
+ on_update: merge
+ key:
+ expression: concat(['customer:', after.customer_id || before.customer_id])
+ language: jmespath
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-writing-to-multiple-keys.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-writing-to-multiple-keys.md
new file mode 100644
index 0000000000..16f898e1bf
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-writing-to-multiple-keys.md
@@ -0,0 +1,43 @@
+---
+Title: Writing to multiple keys
+alwaysopen: false
+categories:
+ - docs
+ - integrate
+ - rs
+ - rdi
+description: null
+group: di
+linkTitle: Writing to multiple keys
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 100
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/redis-writing-to-multiple-keys/'
+---
+
+If you want to write results to multiple keys, you can do so by defining multiple `redis.write` subsections in the `output` section of the job file. Each instance of `redis.write` can specify a different key, data format, and other parameters. For example, you can create two different keys for the same data, one with a default key format and another with a custom key format and mapping.
+
+```yaml
+name: Write to multiple keys with different formats
+output:
+ - uses: redis.write
+ with:
+ # Setting data_type to JSON and using the default key format
+ data_type: json
+
+ - uses: redis.write
+ with:
+ data_type: json
+
+ # Defining a custom key format
+ key:
+ language: jmespath
+ expression: concat(['events-simplified:id:', id])
+
+ # And defining a custom mapping
+ mapping:
+ - id: id
+ - name: name
+ - location: location
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/remapping-the-output.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/remapping-the-output.md
new file mode 100644
index 0000000000..d98bf14f3d
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/remapping-the-output.md
@@ -0,0 +1,39 @@
+---
+Title: Remapping the fields in the output
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Remapping the fields in the output
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 40
+url: '/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/remapping-the-output/'
+---
+
+Sometimes, you may want to remap the fields in the output of a data pipeline. You can do this by defining a `mapping` section in the output configuration.
+
+```yaml
+name: Remap customer fields
+source:
+ table: Customer
+
+output:
+ - uses: redis.write
+ with:
+ data_type: hash
+ mapping:
+ - CustomerId: id
+ - FirstName: first_name
+ - LastName: last_name
+```
+
+The example above remaps the `CustomerId` field to `id`, `FirstName` to `first_name`, and `LastName` to `last_name` in the output. This allows you to customize the field names in the Redis data store according to your application's requirements.
+You can also use `mapping` to include only the fields you need in the output and exclude the rest.
+
+Mapping only allows you to rename fields and limit the output to specific fields and define a single level structure. To create nested structures and/or perform operations on the field values you can use the [map transformation]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/map-example" >}}).
diff --git a/content/integrate/redis-data-integration/1.19.1/faq.md b/content/integrate/redis-data-integration/1.19.1/faq.md
new file mode 100644
index 0000000000..ce494c4f8e
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/faq.md
@@ -0,0 +1,181 @@
+---
+Title: FAQ
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Find answers to common questions about RDI
+group: di
+hideListLinks: false
+linkTitle: FAQ
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 50
+url: '/integrate/redis-data-integration/1.19.1/faq/'
+---
+
+## Which license does RDI use?
+
+You must purchase a commercial license for RDI with Redis Enterprise. This includes two extra
+Redis Enterprise shards (primary and replica) for the staging database.
+
+## How does RDI track data changes in the source database?
+
+RDI uses change data capture (CDC) mechanisms that are specific to each of the
+supported source databases:
+
+- **Oracle**: RDI uses `LogMiner` to read Oracle's `redo logs` and `archive logs`,
+ or, alternatively, `XStream`.
+- **MySQL/MariaDB**: RDI uses `binary log` (binlog) replication to capture all commits.
+- **PostgreSQL**: RDI uses the `pgoutput` logical decoding plugin. The same
+ applies to the PostgreSQL-compatible databases that RDI supports, including
+ Supabase, AlloyDB for PostgreSQL, Amazon Aurora/RDS for PostgreSQL, and Neon.
+- **SQL Server**: RDI uses the database's built-in CDC feature.
+- **MongoDB**: RDI uses `change streams` to read the `oplog`. The source must be
+ a replica set, sharded cluster, or MongoDB Atlas deployment, because a
+ standalone MongoDB server has no oplog.
+- **Google Cloud Spanner**: RDI uses `Spanner change streams` for the streaming
+ phase and the JDBC driver for the initial snapshot. Spanner is supported only
+ when RDI is deployed on Kubernetes with Helm.
+- **Snowflake** (preview): RDI uses `Snowflake Streams`. Snowflake is supported
+ only when RDI is deployed on Kubernetes with Helm.
+
+For the complete list of supported source databases and versions, see
+[Prepare source databases]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs" >}}).
+
+## How much data can RDI process?
+
+RDI uses the concept of *processing units*. Each processing unit uses 1 CPU core and can process
+about 10,000 records per second, assuming the records have a size of about 1KB each. This throughput
+might change slightly depending on the number of columns, the number of data transformations,
+and the speed of the network. Typically, one processing unit is enough for RDI to deal with the
+traffic from a relational database.
+
+## Can RDI work with any Redis database?
+
+No. RDI is designed and tested to work only with Redis Enterprise. The staging database can
+only use version 6.4 or above. The target Redis database can be of any version and can be a
+replica of an Active-Active replication setup or an Auto tiering database.
+
+## Can I use Active-Active for the RDI database?
+
+Yes, starting with RDI 1.16.0, you can use Active-Active for the RDI database. This is
+supported whether or not you also run a disaster recovery (DR) setup for RDI.
+
+If you have two RDI instances sharing a single RDI database then they will use that database for leader election, so
+they need no other lease mechanism. This is how high availability (HA) works for VM
+installations. See
+[Installing with High Availability]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-vm#installing-with-high-availability" >}}).
+
+In a DR setup, each site runs its own RDI instance against its local instance of the
+Active-Active RDI database, so leader election needs an external lease. Google Cloud Storage
+(GCS) is currently the only supported lease mechanism, and you can configure it only for
+[Helm based installations]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-k8s" >}}).
+
+**Important:** Use a DR setup only when both sites capture changes from the same source
+database server. Both RDI instances must point at that same server, not at a replica of it.
+
+## Can I run multiple RDI installations in the same Kubernetes cluster?
+
+No. Only one RDI installation is supported per Kubernetes cluster, even if
+you install into different namespaces. If you need more than one RDI
+deployment, use separate Kubernetes clusters. See
+[Install on Kubernetes]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-k8s" >}})
+for installation details.
+
+## Can RDI automatically track changes to the source database schema?
+
+If you don't configure RDI to capture a specific set of tables in the schema then it will
+detect any new tables when they are added. Similarly, RDI will capture new table columns
+and changes to column names unless you configure it for a specific set of columns.
+Bear in mind that the Redis keys in the target database will change to reflect the
+new or renamed tables and columns.
+
+## Should I be concerned when the log says RDI is out of memory? {#rdi-oom}
+
+Sometimes the Debezium log will contain a message saying that RDI is out of
+memory. This is not an error but an informative message to say that RDI
+is applying *backpressure* to Debezium. See
+[Backpressure mechanism]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#backpressure-mechanism" >}})
+in the Architecture guide for more information.
+
+## What happens when RDI can't write to the target Redis database?
+
+RDI will keep attempting to write the changes to the target and will also attempt
+to reconnect to it, if necessary. While the target is disconnected, RDI
+will keep capturing change events from the source database and adding them to its
+streams in the staging database. This continues until the staging database gets
+low on space to store new events. When RDI detects this, it applies a "back pressure"
+mechanism to capture data from the source less frequently, which reduces the risk of running
+out of space altogether. The systems that the source databases use to record changes can
+retain the change data for at least a few hours, and RDI can catch up with the
+changes as soon as the target connection recovers or the staging database has
+more space available.
+
+## What does RDI do if the data is corrupted or invalid?
+
+The collector reports the data to RDI in a structured JSON format. If
+the structure of the JSON data is invalid or if there is a fatal bug in the transformation
+job then RDI can't transform the data. When this happens, RDI will store the original data
+in a "dead letter queue" along with a message to say why it was rejected. The dead letter
+queue is stored as a capped stream in the RDI staging database. You can see its contents
+with Redis Insight or with the
+[`redis-di list-dlq-records`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlq-records" >}})
+command from the CLI.
+
+See [Rejected records]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/rejected-records" >}}) for more information about DLQ.
+
+## Can I use RDI without persistence enabled?
+
+By default, RDI requires persistence to be enabled on the RDI database. This ensures that RDI can recover both its configuration and the last known state if the cluster crashes.
+
+If you don't have permissions to use persistence due to compliance or other reasons, you can disable
+the persistence check on the RDI database (Helm installation only). If you do this, RDI will not be
+able to recover from a crash, and you will have to perform a new deploy to reinitialize the pipeline.
+
+To disable the persistence check, set the `aofRequired` value to `false` in the `operator.prerequisiteChecks`
+section of the `values.yaml` file.
+
+```yaml
+operator:
+ prerequisiteChecks:
+ aofRequired: false
+```
+
+This option is available in RDI 1.16.2 and later.
+
+## Which processor should I use? {#which-processor-should-i-use}
+
+RDI ships with two stream processor implementations: the *classic*
+processor and the *Flink* processor. Both are fully supported for
+production on VM and Kubernetes installations. The Flink processor
+is generally available as of RDI 1.19.0 and is enabled per pipeline.
+
+The Flink processor delivers significantly higher snapshot throughput,
+lower end-to-end latency, horizontal scaling, and Flink checkpointing
+on top of the same at-least-once delivery guarantees as the classic
+processor. It also adds optional expression and `redis.lookup` result
+caching.
+
+**We strongly recommend using the Flink processor** for new pipelines and
+migrating existing pipelines to it, to benefit from these improvements. The
+*classic* processor is still the default, so pipelines keep using it until
+you opt in, and it remains a fully supported choice — for example, when you
+want to ensure your pipelines continue to work as before until you have
+consciously migrated them. In a future release, however, the Flink processor
+will become the default and the classic processor may be deprecated, so adopting
+the Flink processor now avoids a later migration.
+
+Switch a pipeline to the Flink processor by setting
+[`processors.type`]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config#processors" >}})
+to `flink` (`classic` is the default). You can adopt it per pipeline without
+changing the others.
+
+See
+[Differences between the classic and Flink processors]({{< relref "/integrate/redis-data-integration/1.19.1/architecture/classic-vs-flink" >}})
+for a side-by-side comparison and
+[Migrate from the classic processor to the Flink processor]({{< relref "/integrate/redis-data-integration/1.19.1/installation/migration-classic-to-flink" >}})
+for a step-by-step migration guide.
diff --git a/content/integrate/redis-data-integration/1.19.1/installation/_index.md b/content/integrate/redis-data-integration/1.19.1/installation/_index.md
new file mode 100644
index 0000000000..e320de2008
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/installation/_index.md
@@ -0,0 +1,24 @@
+---
+Title: Install and upgrade
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Learn how to install and upgrade RDI
+group: di
+hideListLinks: false
+linkTitle: Install/upgrade
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 20
+url: '/integrate/redis-data-integration/1.19.1/installation/'
+---
+
+The guides in this section explain the options you have for installing and upgrading RDI on your own servers. See the [Redis Cloud RDI guide]({{< relref "/operate/rc/rdi" >}}) to
+learn how to set up RDI for a cloud database.
+Before you use RDI, you must also configure your source database to enable CDC. See the
+[Prepare source databases]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs" >}})
+section to learn how to do this.
\ No newline at end of file
diff --git a/content/integrate/redis-data-integration/1.19.1/installation/ha-test.md b/content/integrate/redis-data-integration/1.19.1/installation/ha-test.md
new file mode 100644
index 0000000000..f7c76521d4
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/installation/ha-test.md
@@ -0,0 +1,83 @@
+---
+Title: Test HA failover
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Learn how to perform HA failover testing for Redis Data Integration (RDI) to ensure high availability and reliability of your data integration setup.
+group: di
+hideListLinks: false
+linkTitle: Test HA failover
+summary: How to perform HA failover testing
+type: integration
+weight: 100
+url: '/integrate/redis-data-integration/1.19.1/installation/ha-test/'
+---
+
+## Setup
+1. Ensure that RDI is up and running on both primary and secondary nodes.
+ Run the following command and verify and that each instance should show healthy and running `rdi-api` and `rdi-operator` pods.
+```
+kubectl -n rdi get pods
+
+# Example output:
+NAME READY STATUS RESTARTS AGE
+collector-api-577d95bfd8-5wbg6 1/1 Running 0 12m
+collector-source-95f45bcf7-vwn5l 1/1 Running 0 12m
+fluentd-zq2lc 1/1 Running 0 72m
+logrotate-29530445-j729x 0/1 Completed 0 14m
+logrotate-29530450-dprr2 0/1 Completed 0 9m40s
+logrotate-29530455-mfmzw 0/1 Completed 0 4m40s
+processor-f66655469-h7nw2 1/1 Running 0 12m
+rdi-api-f75df6796-qwqjw 1/1 Running 0 72m
+rdi-metrics-exporter-d57cdf8c8-wjzb5 1/1 Running 0 72m
+rdi-operator-7f7f6c7dfd-5qmjd 1/1 Running 0 71m
+rdi-reloader-77df5f7854-lwmvz 1/1 Running 0 71m
+```
+
+2. Identify the leader node - this is the one that has a running `collector-source` pod.
+
+## Performing the HA Failover Testing
+
+To perform HA, you can simulate a connection failure between the leader and the RDI database by blocking the network traffic. You can do this by running the following commands on the leader node:
+
+1. Identify the RDI database IP (replace `` with your own hostname):
+```
+dig +short
+
+# Example:
+# dig +short my.redis.hostname.com
+
+# Example output:
+54.78.220.161
+```
+
+2. For each of the IPs returned by the above command, run the following command to block the traffic:
+
+```
+sudo iptables -I FORWARD -d -j DROP
+
+# With the IP from the example above, the command would be:
+sudo iptables -I FORWARD -d 54.78.220.161 -j DROP
+```
+
+
+The default configuration for the leader lock is 60 seconds, so it may take up to 2 minutes for the failover to occur.
+Meanwhile you can follow the logs of the operator to see the failover process:
+
+```
+kubectl -n rdi logs rdi-operator-7f7f6c7dfd-5qmjd -f
+```
+
+In about 10 seconds you will start seeing log entries from the leader saying that it could not acquire the leadership.
+When the leader lock expires, the second node will acquire the leadership and you will see log entries from the second node indicating that it has become the leader.
+
+## Cleanup
+
+To clean up after the test, remove the `iptables` rule that you added to block the traffic:
+
+```sudo iptables -D FORWARD -d -j DROP```
+
+Use `sudo iptables -S | grep ` to verify that the rule has been removed.
diff --git a/content/integrate/redis-data-integration/1.19.1/installation/install-k8s.md b/content/integrate/redis-data-integration/1.19.1/installation/install-k8s.md
new file mode 100644
index 0000000000..24093cab45
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/installation/install-k8s.md
@@ -0,0 +1,461 @@
+---
+Title: Install on Kubernetes
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Learn how to install RDI on Kubernetes
+group: di
+hideListLinks: false
+linkTitle: Install on K8s
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 20
+url: '/integrate/redis-data-integration/1.19.1/installation/install-k8s/'
+---
+
+This guide explains how to use the RDI [Helm chart](https://helm.sh/docs/topics/charts/)
+to install on [Kubernetes](https://kubernetes.io/) (K8s). You can also
+[Install RDI on VMs]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-vm" >}}).
+
+The installation creates the following K8s objects:
+
+- A K8s [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) named `rdi`.
+ You can also use a different namespace name if you prefer.
+- [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) and
+ [services](https://kubernetes.io/docs/concepts/services-networking/service/) for the
+ [RDI operator]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#how-rdi-is-deployed" >}}),
+ [metrics exporter]({{< relref "/integrate/redis-data-integration/1.19.1/observability" >}}), and API server.
+- A [service account](https://kubernetes.io/docs/concepts/security/service-accounts/)
+ and [RBAC resources](https://kubernetes.io/docs/reference/access-authn-authz/rbac) for the RDI operator.
+- A [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) with RDI database details.
+- [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
+ with the RDI database credentials and TLS certificates.
+- Other optional K8s resources such as [ingresses](https://kubernetes.io/docs/concepts/services-networking/ingress/)
+ that can be enabled depending on your K8s environment and needs.
+
+You can use this installation on [OpenShift](https://docs.openshift.com/) and other K8s distributions
+including cloud providers' K8s managed clusters.
+
+You can configure the RDI Helm chart to pull the RDI images from [dockerhub](https://hub.docker.com/u/redis)
+or from your own [private image registry](#using-a-private-image-registry).
+
+## Before you install
+
+Complete the following steps before installing the RDI Helm chart:
+
+- [Create the RDI database](#create-the-rdi-database) on your Redis Enterprise cluster.
+
+- Create a [user]({{< relref "/operate/rs/security/access-control/create-users" >}})
+ for the RDI database if you prefer not to use the default password (see
+ [Access control]({{< relref "/operate/rs/security/access-control" >}}) for
+ more information).
+
+- Download the RDI Helm chart tar file from the
+ [Redis download center](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-{{< rdi-version >}}.tgz) (in the *Modules, Tools & Integration* category) .
+
+ ```bash
+ export RDI_VERSION={{< rdi-version >}}
+ wget https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-$RDI_VERSION.tgz
+ ```
+
+- If you want to use a private image registry,
+ [prepare it with the RDI images](#using-a-private-image-registry).
+
+- [Download the RDI CLI](#download-the-rdi-cli), which you use to deploy and manage pipelines.
+
+### Create the RDI database
+
+RDI uses a database on your Redis Enterprise cluster to store its state
+information. Use the Redis Enterprise Cluster Manager UI to create the RDI database with the following
+requirements:
+
+{{< embed-md "rdi-db-reqs.md" >}}
+
+You should then provide the details of this database in the [`values.yaml`](#the-valuesyaml-file)
+file as described below.
+
+### Using a private image registry
+
+Add the RDI images from [dockerhub](https://hub.docker.com/u/redis) to your local registry.
+You need the following RDI images with tags matching the RDI version you want to install:
+
+- [redis/rdi-api](https://hub.docker.com/r/redis/rdi-api)
+- [redis/rdi-operator](https://hub.docker.com/r/redis/rdi-operator)
+- [redis/rdi-monitor](https://hub.docker.com/r/redis/rdi-monitor)
+- [redis/rdi-processor](https://hub.docker.com/r/redis/rdi-processor)
+- [redis/rdi-collector-api](https://hub.docker.com/r/redis/rdi-collector-api)
+- [redis/rdi-collector-initializer](https://hub.docker.com/r/redis/rdi-collector-initializer)
+
+If you plan to use the Flink processor for any of your pipelines, you'll also need:
+
+- [redis/rdi-flink-processor](https://hub.docker.com/r/redis/rdi-flink-processor)
+- [redis/rdi-metrics-aggregator](https://hub.docker.com/r/redis/rdi-metrics-aggregator)
+
+If you plan to use the Flink processor exclusively, the `redis/rdi-processor`
+and `redis/rdi-monitor` images are not required.
+
+If you plan to use Spanner as a source for your pipeline, you'll also need
+[redis/rdi-flink-collector](https://hub.docker.com/r/redis/rdi-flink-collector).
+
+If you plan to use Snowflake as a source for any of your pipelines, you'll also need
+[riotx/riotx:v1.8.0](https://hub.docker.com/r/riotx/riotx):
+[RIOT-X](https://redis.github.io/riotx/), a data ingestion and replication tool for Redis.
+
+In addition, the RDI Helm chart uses the following 3rd party images:
+
+- [redislabs/debezium-server:3.5.0.Final-rdi.1](https://hub.docker.com/r/redislabs/debezium-server),
+ based on `quay.io/debezium/server/3.5.0.Final` with minor modifications:
+ [Debezium](https://debezium.io/), an open source distributed platform for change data capture.
+- [redis/reloader:v1.4.13](https://hub.docker.com/r/redis/reloader), originally `ghcr.io/stakater/reloader:v1.4.13`:
+ [Reloader](https://github.com/stakater/Reloader), a K8s controller to watch changes to ConfigMaps
+ and Secrets and do rolling upgrades.
+
+The example below shows how to specify the registry and image pull secret in your
+[`rdi-values.yaml`](#the-valuesyaml-file) file for the Helm chart:
+
+```yaml
+global:
+ # Global image settings.
+ # If using a private image registry, update the default values accordingly.
+ image:
+ registry: your-registry
+ repository: your-repository # If different from "redis"
+
+ # Image pull secrets to be used when using a private image registry.
+ imagePullSecrets:
+ - name: your-secret-name
+
+# ...
+
+# Configuration of the reloader.
+reloader:
+ reloader:
+ # ...
+ deployment:
+ image:
+ name: my-registry.com/my-repo/reloader
+ #...
+```
+
+To pull images from a private image registry, you must provide the image pull secret and in some cases also set the permissions. Follow the links below to learn how to use a private registry with:
+
+- [Rancher](https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-and-docker-registries#using-a-private-registry)
+- [OpenShift](https://docs.openshift.com/container-platform/4.17/openshift_images/managing_images/using-image-pull-secrets.html)
+- [Amazon Elastic Kubernetes Service (EKS)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_on_EKS.html)
+- [Google Kubernetes Engine (GKE)](https://cloud.google.com/artifact-registry/docs/pull-cached-dockerhub-images)
+- [Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/cluster-container-registry-integration?tabs=azure-cli)
+
+### Download the RDI CLI
+
+You manage RDI with the [`redis-di` CLI]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli" >}}),
+which you use to deploy pipelines, set secrets, and inspect status. Unlike the VM installation, which
+bundles the CLI, a Kubernetes installation requires you to download it separately from the Redis
+download center.
+
+The CLI is currently built for the following platforms. Download the binary that matches the operating
+system and architecture of the machine you will run it from. On Linux and macOS, if you are not sure
+which to choose, run `uname -sm`: `Linux x86_64` is Linux amd64, `Linux aarch64` is Linux arm64,
+`Darwin x86_64` is macOS on Intel, and `Darwin arm64` is macOS on Apple silicon.
+
+| Platform | Download |
+| :-- | :-- |
+| Linux, x86-64 (amd64) | [`redis-di`](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/cli/{{< rdi-version >}}/bin/linux-amd64/redis-di) |
+| Linux, ARM64 (aarch64) | [`redis-di`](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/cli/{{< rdi-version >}}/bin/linux-arm64/redis-di) |
+| macOS, Intel (amd64) | [`redis-di`](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/cli/{{< rdi-version >}}/bin/darwin-amd64/redis-di) |
+| macOS, Apple silicon (arm64) | [`redis-di`](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/cli/{{< rdi-version >}}/bin/darwin-arm64/redis-di) |
+| Windows, x86-64 (amd64) | [`redis-di.exe`](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/cli/{{< rdi-version >}}/bin/windows-amd64/redis-di.exe) |
+
+For example, to download the CLI for Linux amd64, make it executable, and put it on your `PATH`:
+
+```bash
+export RDI_VERSION={{< rdi-version >}}
+wget https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/cli/$RDI_VERSION/bin/linux-amd64/redis-di
+chmod +x redis-di
+sudo mv redis-di /usr/local/bin/
+```
+
+{{< note >}}The macOS and Windows binaries are not currently signed or notarized, so the operating
+system may block them the first time you run them. On macOS, allow the binary to run in
+**System Settings > Privacy & Security**, or remove the quarantine attribute with
+`xattr -d com.apple.quarantine ./redis-di`, and then run it again. On Windows, if Microsoft Defender
+SmartScreen blocks it, choose **More info > Run anyway**.{{< /note >}}
+
+## Supported versions of Kubernetes and OpenShift
+
+{{< embed-md "rdi-k8s-reqs.md" >}}
+
+## Install the RDI Helm chart
+
+1. Scaffold the default `values.yaml` file from the chart into a local
+ `rdi-values.yaml` file:
+
+ ```bash
+ helm show values rdi-.tgz > rdi-values.yaml
+ ```
+
+1. Open the `rdi-values.yaml` file you just created, change or add the appropriate
+ values for your installation, and delete the values you have not changed to
+ use their default values.
+ See [The `values.yaml` file](#the-valuesyaml-file) for more details.
+
+1. Run the `helm upgrade --install` command:
+
+ ```bash
+ helm upgrade --install rdi rdi-.tgz -f rdi-values.yaml -n rdi --create-namespace
+ ```
+
+ {{< note >}}The above command will install RDI in a namespace called
+ `rdi`. If you want to use a different namespace, pass the option
+ `-n ` to the `helm install` command instead.
+ {{< /note >}}
+
+ {{< warning >}}Only one RDI installation is supported per Kubernetes
+ cluster. Installing RDI into multiple namespaces in the same cluster is
+ not supported and will fail. If you need more than one RDI deployment,
+ use separate Kubernetes clusters.
+ {{< /warning >}}
+
+### The `values.yaml` file
+
+The [`values.yaml`](https://helm.sh/docs/topics/charts/#templates-and-values) file inside the
+Helm chart contains the values you can set for the RDI Helm installation.
+See the comments by each value for more information about the values you may need to add or change
+depending on your use case.
+
+At a minimum, you must set the values of `connection.host`, `connection.port`, and `connection.password`
+to enable the basic connection to the RDI database.
+You must also set `api.jwtKey`, RDI uses this value to encrypt the
+[JSON web token (JWT)](https://jwt.io/) token used by RDI API. Best practice is
+to generate a value containing 32 random bytes of data (equivalent to 256
+bits) and then encode this value as ASCII characters. Use the following
+command to generate the random key from the
+[`urandom` special file](https://en.wikipedia.org/wiki//dev/random):
+
+```bash
+head -c 32 /dev/urandom | base64
+```
+
+If you use TLS to connect to the RDI database, you must set the
+CA certificate content in `connection.ssl.cacert` (for TLS). In addition, if you
+also use mTLS, you must set the client certificate and private key contents in
+`connection.ssl.cert`, and `connection.ssl.key`.
+
+- You can add the certificate content directly in the `rdi-values.yaml` file
+ as follows:
+
+ ```yaml
+ connection:
+ ssl:
+ enabled: true
+ cacert: |
+ -----BEGIN CERTIFICATE-----
+ ...
+ -----END CERTIFICATE-----
+ cert: |
+ -----BEGIN CERTIFICATE-----
+ ...
+ -----END CERTIFICATE-----
+ key: |
+ -----BEGIN PRIVATE KEY-----
+ ...
+ -----END PRIVATE KEY-----
+ ```
+
+- Alternatively, you can use the `--set-file` argument to set these values to
+ the content of your certificate files as follows:
+
+ ```bash
+ helm upgrade --install rdi rdi-.tar.gz -f rdi-values.yaml -n rdi --create-namespace \
+ --set connection.ssl.enabled=true \
+ --set-file connection.ssl.cacert= \
+ --set-file connection.ssl.cert= \
+ --set-file connection.ssl.key=
+ ```
+
+{{< note >}}
+Please see [these docs]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/spanner#6-additional-kubernetes-configuration" >}}) if this RDI installation is for use with GCP Spanner.
+{{< /note >}}
+
+If you are deploying to [OpenShift](https://docs.openshift.com/), you must
+set `global.openshift` to `true`:
+
+```yaml
+global:
+ # Indicates whether the deployment is intended for an OpenShift environment.
+ openShift: true
+```
+
+Set `global.securityContext.runAsUser` and
+`global.securityContext.runAsGroup` to the appropriate values for your
+OpenShift environment.
+
+```yaml
+global:
+ # Container default security context.
+ # ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container
+ securityContext:
+ runAsNonRoot: true
+ # On OpenShift, user and group 1000 are usually not allowed.
+ # If using OpenShift, set runAsUser and runAsGroup to values in your project's user and group ranges.
+ # You can examine the latter via `oc get projects -o yaml | grep "openshift.io/sa.scc"`
+ runAsUser: 1000701234
+ runAsGroup: 1000701234
+ allowPrivilegeEscalation: false
+```
+
+{{< warning >}}The default OpenShift Security Context Constraints (SCCs)
+will not allow RDI to run if `global.securityContext.runAsUser`
+and `global.securityContext.runAsGroup` have their default values of `1000`.
+You must edit your `rdi-values.yaml` file to ensure these values are
+in the valid range for your OpenShift environment.
+
+Use the following [OpenShift CLI](https://docs.redhat.com/en/documentation/openshift_container_platform/4.19/html/cli_tools/openshift-cli-oc) command
+to find the user and group ranges for your project:
+
+```bash
+oc get projects -o yaml | grep "openshift.io/sa.scc"
+```
+{{< /warning >}}
+
+### Configure the Flink processor
+
+RDI ships with two stream processor implementations: the default *classic*
+processor and the
+[Apache Flink](https://flink.apache.org/)-based *Flink* processor.
+See
+[Stream processor implementations]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#stream-processor-implementations" >}})
+for an overview of the differences and
+[Differences between the classic and Flink processors]({{< relref "/integrate/redis-data-integration/1.19.1/architecture/classic-vs-flink" >}})
+for a side-by-side comparison.
+
+To configure the Flink processor at the Helm chart level, add the
+`operator.dataPlane.flinkProcessor` block to your `rdi-values.yaml` file. The
+snippet below shows a few of the most commonly adjusted values. See the
+`flinkProcessor` block in the Helm chart's `values.yaml` for the full set of
+supported values.
+
+```yaml
+operator:
+ dataPlane:
+ flinkProcessor:
+ jobManager:
+ # JobManager pod resources.
+ cpu: 0.1
+ memory: 1024
+ taskManager:
+ # TaskManager pod resources.
+ cpu: 1
+ memory: 2048
+ # Number of parallel task slots per TaskManager pod.
+ # Total parallelism is `replicas * numberOfTaskSlots`.
+ numberOfTaskSlots: 1
+```
+
+Configuring the Flink processor at the Helm chart level only sets the values
+that the operator will use when deploying the JobManager and TaskManager workloads.
+To run a specific pipeline on the Flink processor, set
+[`processors.type`]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config#processors" >}})
+to `flink` in that pipeline's `config.yaml`. Pipelines without this setting
+continue to use the classic processor. Fine-tune the Flink runtime
+through the `processors.advanced` section of `config.yaml` (see the
+[configuration reference]({{< relref "/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference#processors" >}})).
+
+For migrating existing pipelines to the Flink processor, see
+[Migrate from the classic processor to the Flink processor]({{< relref "/integrate/redis-data-integration/1.19.1/installation/migration-classic-to-flink" >}}).
+
+## Check the installation
+
+To verify the status of the K8s deployment, run the following command:
+
+```bash
+helm list -n rdi
+```
+
+The output looks like the following. Check that the `rdi` release is listed.
+With RDI 1.8.0 or later, check that the `default` release is also listed.
+
+```
+NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
+default rdi 1 2025-05-08 ... deployed pipeline-0.1.0
+rdi rdi 3 2025-05-08 ... deployed rdi-1.0.0
+```
+
+Also, check that all pods have `Running` status:
+
+```bash
+kubectl get pod -n rdi
+
+NAME READY STATUS RESTARTS AGE
+collector-api- 1/1 Running 0 29m
+rdi-api- 1/1 Running 0 29m
+rdi-operator- 1/1 Running 0 29m
+rdi-reloader- 1/1 Running 0 29m
+```
+
+You can verify that the RDI API works by running
+[`redis-di info`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-info" >}})
+against it, or by adding a connection to the RDI API server to
+[Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}}).
+
+## Using ingress controllers
+
+You must ensure that an appropriate
+[ingress controller](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/)
+is available in your K8s cluster to expose the RDI API service via the K8s
+[`Ingress`](https://kubernetes.io/docs/concepts/services-networking/ingress/)
+resource. Follow the documentation of your cloud provider or of
+the ingress controller to install the controller correctly.
+
+### Using the `nginx` ingress controller on AKS
+
+On AKS, if you want to use the open source
+[`nginx`](https://nginx.org/)
+[ingress controller](https://github.com/kubernetes/ingress-nginx/blob/main/README.md#readme)
+rather than the
+[AKS application routing add-on](https://learn.microsoft.com/en-us/azure/aks/app-routing),
+follow the AKS documentation for
+[creating an unmanaged ingress controller](https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/load-bal-ingress-c/create-unmanaged-ingress-controller?tabs=azure-cli).
+Specifically, ensure that one or both of the following Helm chart values is set:
+
+- `controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz`
+- `controller.service.externalTrafficPolicy=Local`
+
+## Prepare your source database
+
+Before deploying a pipeline, you must configure your source database to enable CDC. See the
+[Prepare source databases]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs" >}})
+section to learn how to do this.
+
+## Deploy a pipeline
+
+When the Helm installation is complete and you have prepared the source database for CDC,
+you are ready to start using RDI. See the guides on how to
+[configure]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines" >}}) and
+[deploy]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/deploy" >}})
+RDI pipelines for more information. You can also configure and deploy a pipeline
+using [Redis Insight]({{< relref "/develop/tools/insight" >}}). See
+[RDI in Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}})
+for full details on how to connect to RDI and deploy pipelines.
+
+## Uninstall RDI
+
+If you want to remove your RDI K8s installation, first run
+the following commands. (If you installed RDI into a custom namespace then
+replace `rdi` with the name of your namespace.)
+
+```bash
+kubectl delete pipeline default -n rdi
+helm uninstall rdi -n rdi
+kubectl delete namespace rdi
+```
+
+{{< note >}}The line `kubectl delete pipeline default -n rdi` is only needed for RDI 1.8.0 or above.
+{{< /note >}}
+
+If you also want to delete the keys from your RDI database, connect to it with
+[`redis-cli`]({{< relref "/develop/tools/cli" >}}) and run a
+[`FLUSHALL`]({{< relref "/commands/flushall" >}}) command.
diff --git a/content/integrate/redis-data-integration/1.19.1/installation/install-vm.md b/content/integrate/redis-data-integration/1.19.1/installation/install-vm.md
new file mode 100644
index 0000000000..b2f1c17a3d
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/installation/install-vm.md
@@ -0,0 +1,335 @@
+---
+Title: Install on VMs
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Learn how to install RDI on one or more VMs
+group: di
+hideListLinks: false
+linkTitle: Install on VMs
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 10
+url: '/integrate/redis-data-integration/1.19.1/installation/install-vm/'
+---
+
+This guide explains how to install Redis Data Integration (RDI) on one or more VMs and integrate it with
+your source database. You can also
+[Install RDI on Kubernetes]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-k8s" >}}).
+
+{{< note >}}We recommend you always use the latest version, which is RDI v{{< rdi-version >}}.
+{{< /note >}}
+
+## Create the RDI database
+
+RDI uses a database on your Redis Enterprise cluster to store its state
+information. Use the Redis Enterprise Cluster Manager UI to create the RDI database with the following
+requirements:
+
+{{< embed-md "rdi-db-reqs.md" >}}
+
+## Hardware sizing
+
+RDI is mainly CPU and network bound.
+Each of the RDI VMs should have at least:
+
+{{< embed-md "rdi-vm-reqs.md" >}}
+
+## VM Installation Requirements
+
+You would normally install RDI on two VMs for High Availability (HA) but you can also install
+just one VM if you don't need this. For example, you might not need HA during
+development and testing.
+
+{{< note >}}You can't install RDI on a host where a Redis Enterprise cluster
+is also installed, due to incompatible network rules. If you want to install RDI on a
+host that you have previously used for Redis Enterprise then you must
+use [`iptables`](https://www.netfilter.org/projects/iptables/index.html) to
+"clean" the host before installation with the following command line:
+
+```bash
+ sudo iptables-save | awk '/^[*]/ { print $1 }
+ /^:[A-Z]+ [^-]/ { print $1 " ACCEPT" ; }
+ /COMMIT/ { print $0; }' | sudo iptables-restore
+```
+
+You may encounter problems if you use `iptables` v1.6.1 and earlier in
+`nftables` mode. Use `iptables` versions later than v1.6.1 or enable the `iptables`
+legacy mode with the following commands:
+
+```bash
+sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
+sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
+```
+
+Also, `iptables` versions 1.8.0-1.8.4 have known issues that can prevent RDI
+from working, especially on RHEL 8. Ideally, use `iptables` v1.8.8, which is
+known to work correctly with RDI.
+{{< /note >}}
+
+The supported OS versions for RDI are:
+
+{{< embed-md "rdi-os-reqs.md" >}}
+
+You must run the RDI installer as a privileged user because it installs
+[containerd](https://containerd.io/) and registers services. However, you don't
+need any special privileges to run RDI processes for normal operation.
+
+RDI has a few
+requirements for cloud VMs that you must implement before running the
+RDI installer, or else installation will fail. The following sections
+give full pre-installation instructions for [RHEL](#firewall-rhel) and
+[Ubuntu](#firewall-ubuntu).
+
+### RHEL {#firewall-rhel}
+
+We recommend you turn off
+[`firewalld`](https://firewalld.org/documentation/)
+before installation using the command:
+
+```bash
+sudo systemctl disable firewalld --now
+```
+
+However, if you do need to use `firewalld`, you must add the following rules:
+
+```bash
+sudo firewall-cmd --permanent --add-port=443/tcp # RDI API
+sudo firewall-cmd --permanent --add-port=6443/tcp # kube-apiserver
+sudo firewall-cmd --permanent --zone=trusted --add-source=10.42.0.0/16 # Kubernetes pods
+sudo firewall-cmd --permanent --zone=trusted --add-source=10.43.0.0/16 # Kubernetes services
+sudo firewall-cmd --reload
+```
+
+If you have `nm-cloud-setup.service` enabled, you must disable it and reboot the
+node with the following commands:
+
+```bash
+sudo systemctl disable nm-cloud-setup.service nm-cloud-setup.timer
+sudo reboot
+```
+
+### Ubuntu {#firewall-ubuntu}
+
+We recommend you turn off
+[Uncomplicated Firewall](https://wiki.ubuntu.com/UncomplicatedFirewall) (`ufw`)
+before installation with the command:
+
+```bash
+sudo ufw disable
+```
+
+However, if you do need to use `ufw`, you must add the following rules:
+
+```bash
+sudo ufw allow 443/tcp # RDI API
+sudo ufw allow 6443/tcp # kube-apiserver
+sudo ufw allow from 10.42.0.0/16 to any # Kubernetes pods
+sudo ufw allow from 10.43.0.0/16 to any # Kubernetes services
+sudo ufw reload
+```
+
+## Installation steps
+
+Follow the steps below for each of your VMs.
+
+{{< note >}}RDI installs executables by default in the `/var` partition, so you must
+ensure it is mounted without the `noexec` option. Use the following command to
+find any partitions mounted with the `noexec` option:
+
+```bash
+mount | grep noexec
+```
+
+If your `/var` partition is listed in the output from this command, you must remount
+it without the `noexec` option. See
+[Using the mount command](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/5/html/deployment_guide/chap-using_the_mount_command) in the Red Hat documentation to learn how to remount a partition.
+{{< /note >}}
+
+1. Download the RDI installer from the
+ [Redis download center](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-installation-{{< rdi-version >}}.tar.gz)
+ (from the *Modules, Tools & Integration* category) and extract it to your preferred installation
+ folder.
+
+ ```bash
+ export RDI_VERSION={{< rdi-version >}}
+ wget https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-installation-$RDI_VERSION.tar.gz
+ tar -xvf rdi-installation-$RDI_VERSION.tar.gz
+ ```
+
+1. Go to the installation folder:
+
+ ```bash
+ cd rdi_install/$RDI_VERSION
+ ```
+
+1. Run the `install.sh` script as a privileged user:
+
+ ```bash
+ sudo ./install.sh
+ ```
+
+ {{< note >}}RDI uses [K3s](https://k3s.io/) as part of its implementation.
+ By default, the installer installs K3s in the `/var/lib` directory,
+ but this might be a problem if you have limited space in `/var`
+ or your company policy forbids you to install there. You can
+ select a different directory for the K3s installation using the
+ `--installation-dir` option with `install.sh`:
+```bash
+sudo ./install.sh --installation-dir
+```
+ {{< /note >}}
+
+ **Advanced**: You can also pass custom K3s parameters to the installer using the
+ `INSTALL_K3S_EXEC` environment variable. For example, to set the kubeconfig file
+ permissions to be readable by all users:
+
+ ```bash
+ sudo INSTALL_K3S_EXEC='--write-kubeconfig-mode=644' ./install.sh
+ ```
+
+ You can combine multiple K3s options in the `INSTALL_K3S_EXEC` variable. See the
+ [K3s documentation](https://docs.k3s.io/installation/configuration) for a full list of
+ available options.
+
+ {{< warning >}}Only modify K3s parameters if you understand exactly what you are changing
+ and why. Incorrect K3s configuration can cause RDI installation to fail or result in an
+ unstable deployment. {{< /warning >}}
+
+
+The RDI installer collects all necessary configuration details and alerts you to potential issues,
+offering options to abort, apply fixes, or provide additional information.
+Once complete, it guides you through creating secrets and setting up your pipeline.
+
+{{< note >}}It is strongly recommended to specify a hostname rather than an IP address for
+connecting to your RDI database, for the following reasons:
+
+- Any DNS resolution issues will be detected during the installation rather than
+ later during pipeline deployment.
+- If you use TLS, your RDI database CA certificate must contain the hostname you specified
+ either as a common name (CN) or as a subject alternative name (SAN). CA certificates
+ usually don't contain IP addresses.
+{{< /note >}}
+
+{{< note >}}If you specify `localhost` as the address of the RDI database server during
+installation then the connection will fail if the actual IP address changes for the local
+VM. For this reason, we recommend that you don't use `localhost` for the address. However,
+if you do encounter this problem, you can fix it using the following commands on the VM
+that is running RDI itself:
+
+```bash
+sudo k3s kubectl delete nodes --all
+sudo service k3s restart
+```
+{{< /note >}}
+
+After the installation is finished, RDI is ready for use.
+
+### Supply cloud DNS information
+
+{{< note >}}This section is only relevant if you are installing RDI
+on VMs in a cloud environment.
+{{< /note >}}
+
+If you are using [Amazon Route 53](https://aws.amazon.com/route53/),
+[Google Cloud DNS](https://cloud.google.com/dns?hl=en), or
+[Azure DNS](https://azure.microsoft.com/en-gb/products/dns)
+then you must supply the installer with the nameserver IP address
+during installation. The table below
+shows the appropriate IP address for each cloud provider:
+
+| Platform | Nameserver IP |
+| :-- | :-- |
+| [Amazon Route 53](https://aws.amazon.com/route53/) | 169.254.169.253 |
+| [Google Cloud DNS](https://cloud.google.com/dns?hl=en) | 169.254.169.254 |
+| [Azure DNS](https://azure.microsoft.com/en-gb/products/dns) | 168.63.129.16 |
+
+If you are using Route 53, you should first check that your VPC
+is configured to allow it. See
+[DNS attributes in your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/AmazonDNS-concepts.html#vpc-dns-support)
+in the Amazon docs for more information.
+
+### Installing with High Availability
+
+To install RDI with High Availability (HA), perform the [Installation steps](#installation-steps)
+on two different VMs. The first VM will automatically become the active (primary) instance,
+while the second VM will become the passive (secondary) one.
+When starting the RDI installation on the second VM, the installer will detect that the RDI
+database is already in use and ask you to confirm that you intend to install RDI with HA.
+
+After the installation is complete, you must set the source and target database secrets
+on both VMs as described in [Deploy a pipeline]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/deploy" >}}). If you use `redis-di` to deploy your configuration, you only need to do this on one of the VMs, not both.
+
+In a High Availability setup, the RDI pipeline is only active on the primary instance (VM).
+The two RDI instances will use the RDI database for leader election. If the primary instance fails
+to renew the lease in the RDI database, it will lose the leadership and a failover to the secondary instance
+will take place. After the failover, the secondary instance will become the primary one,
+and the RDI pipeline will be active on that VM.
+
+You may find it useful to trigger a failover deliberately to check that RDI is correctly configured to handle it. See [Test HA failover]({{< relref "/integrate/redis-data-integration/1.19.1/installation/ha-test" >}}) to learn how to do this.
+
+## Prepare your source database
+
+Before deploying a pipeline, you must configure your source database to enable CDC. See the
+[Prepare source databases]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs" >}})
+section to learn how to do this.
+
+## Deploy a pipeline
+
+When the installation is complete, and you have prepared the source database for CDC,
+you are ready to start using RDI. See the guides on how to
+[configure]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines" >}}) and
+[deploy]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/deploy" >}})
+RDI pipelines for more information. You can also configure and deploy a pipeline
+using [Redis Insight]({{< relref "/develop/tools/insight" >}}). See
+[RDI in Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}})
+for full details on how to connect to RDI and deploy pipelines.
+
+{{< note >}}The [`redis-di` CLI]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli" >}})
+is bundled with the VM installer, so it is already available on the VM where RDI is installed. If you
+prefer to run it from your own laptop or desktop instead, you can
+[download it separately]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-k8s#download-the-rdi-cli" >}})
+for your platform.{{< /note >}}
+
+## Configure the Flink processor
+
+RDI ships with two stream processor implementations: the default *classic*
+processor and the
+[Apache Flink](https://flink.apache.org/)-based *Flink* processor.
+See
+[Stream processor implementations]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#stream-processor-implementations" >}})
+for an overview of the differences and
+[Differences between the classic and Flink processors]({{< relref "/integrate/redis-data-integration/1.19.1/architecture/classic-vs-flink" >}})
+for a side-by-side comparison.
+
+To run a specific pipeline on the Flink processor, set
+[`processors.type`]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config#processors" >}})
+to `flink` in that pipeline's `config.yaml` and redeploy it. Pipelines without
+this setting continue to use the classic processor. Fine-tune the Flink runtime
+through the `processors.advanced` section of `config.yaml` (see the
+[configuration reference]({{< relref "/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference#processors" >}})).
+
+For migrating existing pipelines to the Flink processor, see
+[Migrate from the classic processor to the Flink processor]({{< relref "/integrate/redis-data-integration/1.19.1/installation/migration-classic-to-flink" >}}).
+
+## Uninstall RDI
+
+If you want to remove your RDI installation, go to the installation folder and run
+the uninstall script as a privileged user:
+
+```bash
+sudo ./uninstall.sh
+```
+
+The script will ask if you are sure before proceeding:
+
+```
+This will uninstall RDI and its dependencies, are you sure? [y, N]
+```
+
+If you type anything other than "y" here, the script will abort without making any changes
+to RDI or your source database.
diff --git a/content/integrate/redis-data-integration/1.19.1/installation/migration-classic-to-flink.md b/content/integrate/redis-data-integration/1.19.1/installation/migration-classic-to-flink.md
new file mode 100644
index 0000000000..ab2e529efa
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/installation/migration-classic-to-flink.md
@@ -0,0 +1,141 @@
+---
+Title: Migrate from the classic processor to the Flink processor
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Learn how to migrate an existing RDI pipeline from the classic processor to the Apache Flink-based processor.
+group: di
+hideListLinks: false
+linkTitle: Migrate to the Flink processor
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 35
+url: '/integrate/redis-data-integration/1.19.1/installation/migration-classic-to-flink/'
+---
+
+RDI ships with two stream processor implementations. The default *classic*
+processor is implemented in Python. The *Flink* processor is built on top of
+[Apache Flink](https://flink.apache.org/). Both run on VM and Kubernetes
+installations. The Flink processor can achieve much higher throughput
+during snapshots, scales horizontally by changing the number of TaskManager replicas,
+and uses Flink checkpointing for fault tolerance. See [Stream processor implementations]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#stream-processor-implementations" >}})
+for an overview.
+
+This page describes how to migrate an existing pipeline from the classic
+processor to the Flink processor. The steps are the same on VMs and Kubernetes,
+except for the optional Helm-level tuning in [Step 1](#step-1-configure-the-flink-processor-at-the-helm-chart-level-kubernetes),
+which applies to Kubernetes only.
+
+## Before you migrate
+
+Confirm that your pipeline is compatible with the Flink processor:
+
+- `JSON.MERGE` semantics differ from the classic processor's Lua-based merge
+ when null values are involved (see
+ [`use_native_json_merge`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference#processors" >}})).
+ The Flink processor always uses the native `JSON.MERGE` command when the
+ target database supports it.
+- Ensure your Kubernetes cluster or VM has enough capacity for the Flink JobManager
+ and TaskManager pods (see
+ [Configure the Flink processor]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-k8s#configure-the-flink-processor" >}})
+ for the default sizing).
+
+## Step 1: Configure the Flink processor at the Helm chart level (Kubernetes)
+
+This step applies to **Kubernetes** installations only. On VM installations,
+skip it and enable the Flink processor per pipeline in step 2.
+
+The Flink processor is always available — no opt-in is required at the Helm
+chart level. The defaults are sized for typical workloads, so you can skip
+this step if you don't need to override them. To adjust the JobManager and
+TaskManager defaults, add an `operator.dataPlane.flinkProcessor` block to
+your `rdi-values.yaml` file and run `helm upgrade` as described in
+[Configure the Flink processor]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-k8s#configure-the-flink-processor" >}}).
+Existing pipelines continue to run on the classic processor until you switch
+them in step 2.
+
+For VM installations, skip this step. You can configure per-pipeline Flink
+resources in step 4.
+
+## Step 2: Switch the pipeline to the Flink processor
+
+In the pipeline's `config.yaml`, set
+[`processors.type`]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config#processors" >}})
+to `flink`:
+
+```yaml
+processors:
+ type: flink
+ ...
+```
+
+Then redeploy the pipeline. The operator stops the classic processor pods
+and starts the Flink JobManager and TaskManager workloads for the pipeline.
+
+## Step 3: Adapt deprecated and classic-only properties
+
+Some `processors` properties are no-ops, classic-only, or have moved to
+`processors.advanced` for the Flink processor. The following table lists the
+properties that need attention when migrating.
+
+| Property | Action when migrating to Flink |
+| :-- | :-- |
+| `on_failed_retry_interval` | No-op. Remove. |
+| `duration` | No-op. Use `read_batch_timeout_ms` instead. |
+| `dedup`, `dedup_max_size`, `dedup_strategy` | Classic-only. Remove. |
+| `enable_async_processing`, `batch_queue_size`, `ack_queue_size` | Classic-only. Remove. |
+| `initial_sync_processes` | Classic-only. Configure parallelism through `advanced.flink.taskmanager.numberOfTaskSlots` and `advanced.resources.taskManager.replicas` instead. |
+| `idle_streams_check_interval_ms`, `busy_streams_check_interval_ms` | Classic-only. Use `processors.advanced.source.discovery.interval.ms` for a single discovery interval. |
+| `idle_sleep_time_ms` | Classic-only. Remove. |
+| `use_native_json_merge` | Classic-only. The Flink processor always uses `JSON.MERGE` when the target supports it. |
+
+The classic processor silently ignores `processors.advanced`,
+and the Flink processor silently ignores classic-only top-level properties, so keeping
+both top-level properties and their `processors.advanced` equivalents lets
+you switch back without further edits.
+
+## Step 4: Tune the Flink processor (optional)
+
+Fine-tune the Flink processor through the `processors.advanced` section.
+For example:
+
+```yaml
+processors:
+ type: flink
+ advanced:
+ source:
+ # Time between checks for new input streams.
+ discovery.interval.ms: 1000
+ flink:
+ # Number of parallel task slots per TaskManager pod.
+ taskmanager.numberOfTaskSlots: 2
+ # Total memory budget for each TaskManager JVM process.
+ taskmanager.memory.process.size: 4096m
+ resources:
+ taskManager:
+ # Number of TaskManager pods.
+ replicas: 2
+```
+
+See the
+[`processors.advanced` reference]({{< relref "/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference#processors" >}})
+for the full set of available properties.
+
+## Step 5: Update observability
+
+The Flink processor exposes Prometheus metrics directly
+from the Flink JobManager and TaskManager pods.
+See
+[Flink processor metrics]({{< relref "/integrate/redis-data-integration/1.19.1/observability#flink-processor-metrics" >}})
+for the `ServiceMonitor` configuration and the available metrics.
+
+## Rolling back
+
+To revert a pipeline to the classic processor, set `processors.type` back to
+`classic` (or remove the property) and redeploy the pipeline. The
+`processors.advanced` section is silently ignored by the classic processor,
+so you don't need to remove it before switching back.
diff --git a/content/integrate/redis-data-integration/1.19.1/installation/reqsummary.md b/content/integrate/redis-data-integration/1.19.1/installation/reqsummary.md
new file mode 100644
index 0000000000..d30996b58d
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/installation/reqsummary.md
@@ -0,0 +1,41 @@
+---
+Title: Requirements summary
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Requirements and recommendations for RDI installations.
+group: di
+hideListLinks: false
+linkTitle: Requirements summary
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 5
+url: '/integrate/redis-data-integration/1.19.1/installation/reqsummary/'
+---
+
+The sections below summarize the software and hardware requirements for
+an RDI installation.
+
+## Hardware requirements for VM installation
+
+{{< embed-md "rdi-vm-reqs.md" >}}
+
+## OS requirements for VM installation
+
+{{< embed-md "rdi-os-reqs.md" >}}
+
+## Kubernetes/OpenShift supported versions
+
+{{< embed-md "rdi-k8s-reqs.md" >}}
+
+## RDI database requirements
+
+{{< embed-md "rdi-db-reqs.md" >}}
+
+## Supported source databases
+
+{{< embed-md "rdi-supported-source-versions.md" >}}
diff --git a/content/integrate/redis-data-integration/1.19.1/installation/upgrade.md b/content/integrate/redis-data-integration/1.19.1/installation/upgrade.md
new file mode 100644
index 0000000000..13066fe065
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/installation/upgrade.md
@@ -0,0 +1,211 @@
+---
+Title: Upgrading RDI
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Learn how to upgrade an existing RDI installation
+group: di
+hideListLinks: false
+linkTitle: Upgrade
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 30
+url: '/integrate/redis-data-integration/1.19.1/installation/upgrade/'
+---
+
+## Upgrading a VM installation
+
+Follow the steps below to upgrade an existing
+[VM installation]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-vm" >}})
+of RDI:
+
+1. Download the RDI installer from the [Redis download center](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-installation-{{< rdi-version >}}.tar.gz)
+ (in the *Modules, Tools & Integration* category) and extract it to your
+ preferred installation folder.
+
+ ```bash
+ export RDI_VERSION={{< rdi-version >}}
+ wget https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-installation-$RDI_VERSION.tar.gz
+ tar -xvf rdi-installation-$RDI_VERSION.tar.gz
+ ```
+
+1. Go to the installation folder:
+
+ ```bash
+ cd rdi_install/$RDI_VERSION
+ ```
+
+1. Run the `upgrade.sh` script as a privileged user. Note that you must pass
+ your RDI password to the script unless the password is empty.
+
+ ```bash
+ sudo ./upgrade.sh --rdi-password
+ ```
+
+### Recovering from failure during a VM upgrade
+
+If the previous version is v1.4.4 or later, go to the `rdi_install/`
+directory and run `sudo ./upgrade.sh` to revert to that version, as described in the section
+[Upgrading a VM installation](#upgrading-a-vm-installation) above.
+
+If the version you are replacing is earlier than v1.4.4, follow these steps. These steps restore and
+run the CLI binary of the previous RDI version, which still provided the `redis-di upgrade` command.
+(On current versions, upgrades are performed with the `upgrade.sh` script as described above, and
+`redis-di upgrade` is no longer a CLI command.)
+
+1. Run `redis-di --version` to check the current version.
+
+ If the version is the new one, copy the previous version
+ of the RDI CLI to `/usr/local/bin` with the following command:
+
+ ```bash
+ sudo cp rdi_install//deps/rdi-cli//redis-di usr/local/bin
+ ```
+
+1. Check that the CLI version is correct by running `redis-di --version`.
+
+ Then, go to the `rdi_install/` directory and run the
+ following command;
+
+ ```bash
+ sudo redis-di upgrade --rdi-host --rdi-port
+ ```
+
+{{< note >}}If the `collector-source` or the `processor` pods are not in the `Running` state after
+the upgrade, you must run `redis-di deploy` and check again that they are both in the
+`Running` state.
+{{< /note >}}
+
+### Upgrading a VM installation with High Availability
+
+If there is an active pipeline, upgrade RDI on the active VM first.
+This will cause a short pipeline downtime of up to two minutes.
+Afterwards, upgrade RDI on the passive VM. This will not cause any downtime.
+
+{{< warning >}}
+When upgrading from RDI < 1.8.0 to RDI >= 1.8.0 in a VM HA setup, both RDI instances may incorrectly consider themselves active after the upgrade. This occurs because the upgrade process doesn't change the cluster id value from its default `cluster-1`, causing both clusters to assume they are the active cluster.
+
+**Symptoms:**
+
+- The upgraded passive node will start collector and processor components
+- Collector may enter a crash loop as it fails to connect to the source
+- Both clusters will restart in a loop
+
+**Workaround:**
+
+After upgrading, manually set a unique cluster ID for one of the installations (preferably on the passive instance):
+
+1. Locate the RDI configuration file on the VM host. The file is typically located at `/etc/rdi/rdi-sys-config.yaml`.
+2. Open the configuration file in a text editor. For example:
+
+ ```bash
+ sudo nano /etc/rdi/rdi-sys-config.yaml
+ ```
+{{< /warning >}}
+
+## Upgrading a Kubernetes installation
+
+Follow the steps below to upgrade an existing
+[Kubernetes]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-k8s" >}})
+installation of RDI:
+
+1. If you are using a private registry, pull the new versions of all images listed in
+ [Using a private image registry]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-k8s#using-a-private-image-registry" >}})
+ and add them to your local registry.
+
+1. Download the RDI Helm chart tar file from the [Redis download center](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-{{< rdi-version >}}.tgz)
+ (in the *Modules, Tools & Integration* category).
+
+ ```bash
+ export RDI_VERSION={{< rdi-version >}}
+ wget https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/rdi-$RDI_VERSION.tgz
+ ```
+
+1. Adapt your `rdi-values.yaml` file to any changes in the new RDI version if needed.
+ See also [Upgrading to RDI 1.8.0 or later from an earlier version](#upgrading-to-rdi-180-or-later-from-an-earlier-version).
+ Before making any changes, save your existing `rdi-values.yaml` if you need to revert
+ to the old RDI version for any reason.
+
+1. Run the `helm upgrade` command:
+
+ ```bash
+ helm upgrade --install rdi rdi-.tar.gz -f rdi-values.yaml -n rdi
+ ```
+
+Note that you don't need to
+[deploy]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/deploy" >}})
+the RDI configuration again after this step.
+
+### Upgrading to RDI 1.8.0 or later from an earlier version
+
+When upgrading to RDI 1.8.0 or later from an earlier version
+you must adapt your `rdi-values.yaml` file to the following changes:
+
+- All collector and processor values that were previously under `collector`,
+ `collectorSourceMetricsExporter`, and `processor` have been moved to
+ `operator.dataPlane.collector` and `operator.dataPlane.processor`.
+- `global.collectorApiEnabled` has been moved to `operator.dataPlane.collectorApi.enabled`,
+ and is now a boolean value, not `"0"` or `"1"`.
+- `api.authEnabled` is also now a boolean value, not `"0"` or `"1"`.
+- The following values have been deprecated: `rdiMetricsExporter.service.protocol`,
+ `rdiMetricsExporter.service.port`, `rdiMetricsExporter.serviceMonitor.path`,
+ `api.service.name`.
+
+### Verifying the upgrade
+
+Check that all pods have `Running` status:
+
+```bash
+kubectl get all -n rdi
+```
+
+If you find that the upgrade did not work as expected for any reason,
+then run the `helm upgrade` command again (as described in the section
+[Upgrading a Kubernetes installation](#upgrading-a-kubernetes-installation) above),
+but this time with the previous version you were upgrading from, and using
+your saved `rdi-values.yaml` for that version. This will restore your previous working state.
+
+{{< note >}}Downgrading from RDI 1.8.0 or later to an earlier version using `helm upgrade`
+will not work. If you need to perform such an upgrade, uninstall RDI completely first as
+described in [Uninstall RDI]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-k8s#uninstall-rdi" >}}),
+and then install the old version.
+{{< /note >}}
+
+## Enabling the Flink processor
+
+The
+[Apache Flink](https://flink.apache.org/)-based stream processor is
+fully supported on both VM and Kubernetes installations after upgrading to
+RDI 1.19.0. Once the upgrade completes, it is always available —
+no opt-in is required, and the defaults are sized for typical workloads.
+Upgrading does not change the processor used by existing pipelines, which keep
+running on the classic processor until you explicitly switch them by
+setting
+[`processors.type`]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config#processors" >}})
+to `flink` in their `config.yaml`.
+
+On Kubernetes, to override the Flink processor defaults, add an
+`operator.dataPlane.flinkProcessor` block to your `rdi-values.yaml` file as
+described in
+[Configure the Flink processor]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-k8s#configure-the-flink-processor" >}}).
+On VMs, see
+[Configure the Flink processor]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-vm#configure-the-flink-processor" >}}).
+For the per-pipeline migration steps, see
+[Migrate from the classic processor to the Flink processor]({{< relref "/integrate/redis-data-integration/1.19.1/installation/migration-classic-to-flink" >}}).
+
+## What happens during the upgrade?
+
+The upgrade process replaces the current RDI components with their new versions:
+
+- Firstly, the control plane components are replaced. At this point, the pipeline
+ is still active but monitoring will be disconnected.
+- Secondly, the pipeline data plane components are replaced.
+ If a pipeline is active while upgrading, the `collector-source` and `processor`
+ pods will be restarted. The pipeline will pause for up to two minutes but it
+ will catch up very quickly after restarting.
+ The pipeline data and state are both stored in Redis, so data will not
+ be lost during the upgrade.
diff --git a/content/integrate/redis-data-integration/1.19.1/observability.md b/content/integrate/redis-data-integration/1.19.1/observability.md
new file mode 100644
index 0000000000..8aae31c566
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/observability.md
@@ -0,0 +1,405 @@
+---
+Title: Observability
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: Learn how to monitor RDI
+group: di
+hideListLinks: false
+linkTitle: Observability
+summary: Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 40
+url: '/integrate/redis-data-integration/1.19.1/observability/'
+---
+
+RDI reports metrics about its operation using
+[Prometheus exporter endpoints](https://prometheus.io/docs/instrumenting/exporters/).
+You can connect to the endpoints with
+[Prometheus](https://prometheus.io/docs/prometheus/latest/getting_started/)
+to query the metrics and plot simple graphs or with
+[Grafana](https://grafana.com/) to produce more complex visualizations and
+dashboards.
+
+RDI exposes the following endpoints:
+- **Collector metrics**: CDC collector performance and connectivity
+- **Stream processor metrics**: Data processing performance and throughput. The exposed metrics depend on the [stream processor implementation]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#stream-processor-implementations" >}}) used by the pipeline:
+ - The classic processor exposes the metrics described in [Stream processor metrics](#stream-processor-metrics) through the `rdi-metrics-exporter` service.
+ - The Flink processor exposes the metrics described in [Flink processor metrics](#flink-processor-metrics) directly from its JobManager and TaskManager pods. The `rdi-metrics-exporter` service is not deployed for Flink-based pipelines.
+- **Operator metrics**: Kubernetes operator health and Pipeline resource states
+
+The sections below explain these sets of metrics in more detail.
+See the
+[architecture overview]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#overview" >}})
+for an introduction to these concepts.
+
+{{< note >}}If you don't use Prometheus or Grafana, you can still see
+RDI metrics with the RDI monitoring screen in Redis Insight or with the
+[`redis-di describe`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe" >}})
+command from the CLI.{{< /note >}}
+
+## Accessing the metrics
+
+The way you access the metrics endpoints depends on whether you are using a VM installation or a Helm installation for RDI. The sections below describe the correct approach for each installation type.
+
+### VM Installation
+
+For VM installations, the metrics are available by default on the following endpoints:
+- Collector metrics: `https:///collector-source/metrics`
+- Stream processor metrics: `https:///processor/metrics`
+- Operator metrics: `https:///operator/metrics`
+
+Please note that for RDI versions prior to 1.16.0 the collector metrics are not accessible.
+
+### Helm installation
+
+For Helm installations, the metrics are available via autodiscovery in the K8s cluster. Follow the steps below to use them:
+1. Make sure you have the Prometheus Operator installed in your K8s cluster (see the
+ [Prometheus Operator installation guide](https://prometheus-operator.dev/docs/getting-started/installation/) for more information about this).
+
+2. Update your values.yaml file to enable metrics for the operator, collector and stream processor components.
+
+ - For the collector, update the `collector` section, under the `dataPlane` section:
+ ```yaml
+ dataPlane:
+ collector:
+ # Enable service monitor
+ serviceMonitor:
+ enabled: true
+
+ # Make sure to label the ServiceMonitor so that Prometheus can discover it
+ labels:
+ release: prometheus
+ ```
+
+ - For the stream processor, update the `rdiMetricsExporter` section:
+ ```yaml
+ rdiMetricsExporter:
+ # Enable service monitor
+ serviceMonitor:
+ enabled: true
+
+ # Make sure to label the ServiceMonitor so that Prometheus can discover it
+ labels:
+ release: prometheus
+ ```
+
+ - For the operator, update the `operator` section:
+ ```yaml
+ operator:
+ prometheus:
+ enabled: true
+ labels:
+ release: prometheus
+ metrics:
+ enabled: true
+ ```
+
+ - For the Flink processor, enable the JobManager and TaskManager `ServiceMonitor` resources under `operator.dataPlane.flinkProcessor`:
+ ```yaml
+ operator:
+ dataPlane:
+ flinkProcessor:
+ jobManager:
+ serviceMonitor:
+ enabled: true
+ labels:
+ release: prometheus
+ taskManager:
+ serviceMonitor:
+ enabled: true
+ labels:
+ release: prometheus
+ ```
+
+{{< note >}}The Prometheus service discovery loop runs at regular intervals. This means that after deploying or updating RDI with the above configuration, it may take a few minutes for Prometheus to discover the new ServiceMonitors and start scraping metrics from the RDI components.
+{{< /note >}}
+
+## Collector metrics
+
+These metrics are divided into three groups:
+
+- **Pipeline state**: metrics about the pipeline mode and connectivity
+- **Data flow counters**: counters for data breakdown per source table
+- **Processing performance**: processing speed of RDI micro batches
+
+The following table lists all collector metrics and their descriptions:
+
+| Metric | Type | Description | Alerting Recommendations |
+|:--|:--|:--|:--|
+| **Schema History Metrics** | | | |
+| `ChangesApplied` | Counter | Total number of schema changes applied during recovery and runtime | Informational - monitor for trends |
+| `ChangesRecovered` | Counter | Number of changes that were read during the recovery phase | Informational - monitor for trends |
+| `MilliSecondsSinceLastAppliedChange` | Gauge | Number of milliseconds since the last change was applied | Informational - monitor for trends |
+| `MilliSecondsSinceLastRecoveredChange` | Gauge | Number of milliseconds since the last change was recovered from the history store | Informational - monitor for trends |
+| `RecoveryStartTime` | Gauge | Time in epoch milliseconds when recovery started (-1 if not applicable) | Informational - monitor for trends |
+| **Connection and State Metrics** | | | |
+| `Connected` | Gauge | Whether the collector is currently connected to the database (1=connected, 0=disconnected) | **Critical Alert**: Alert if value = 0 (disconnected) |
+| **Queue Metrics** | | | |
+| `CurrentQueueSizeInBytes` | Gauge | Current size of the collector's internal queue in bytes | Informational - monitor for trends |
+| `MaxQueueSizeInBytes` | Gauge | Maximum configured size of the collector's internal queue in bytes | Informational - use for capacity planning |
+| `QueueRemainingCapacity` | Gauge | Remaining capacity of the collector's internal queue | Informational - monitor for trends |
+| `QueueTotalCapacity` | Gauge | Total capacity of the collector's internal queue | Informational - use for capacity planning |
+| **Streaming Performance Metrics** | | | |
+| `MilliSecondsBehindSource` | Gauge | Number of milliseconds the collector is behind the source database (-1 if not applicable) | Informational - monitor for trends and business SLA requirements |
+| `MilliSecondsSinceLastEvent` | Gauge | Number of milliseconds since the collector processed the most recent event (-1 if not applicable) | Informational - monitor for trends in active systems |
+| `NumberOfCommittedTransactions` | Counter | Number of committed transactions processed by the collector | Informational - monitor for trends |
+| `NumberOfEventsFiltered` | Counter | Number of events filtered by include/exclude list rules | Informational - monitor for trends |
+| **Event Counters** | | | |
+| `TotalNumberOfCreateEventsSeen` | Counter | Total number of CREATE (INSERT) events seen by the collector | Informational - monitor for trends |
+| `TotalNumberOfDeleteEventsSeen` | Counter | Total number of DELETE events seen by the collector | Informational - monitor for trends |
+| `TotalNumberOfEventsSeen` | Counter | Total number of events seen by the collector | Informational - monitor for trends |
+| `TotalNumberOfUpdateEventsSeen` | Counter | Total number of UPDATE events seen by the collector | Informational - monitor for trends |
+| `NumberOfErroneousEvents` | Counter | Number of events that caused errors during processing | **Critical Alert**: Alert if > 0 (indicates processing failures) |
+| **Snapshot Metrics** | | | |
+| `RemainingTableCount` | Gauge | Number of tables remaining to be processed during snapshot | Informational - monitor snapshot progress |
+| `RowsScanned` | Counter | Number of rows scanned per table during snapshot (reported per table) | Informational - monitor snapshot progress |
+| `SnapshotAborted` | Gauge | Whether the snapshot was aborted (1=aborted, 0=not aborted) | **Critical Alert**: Alert if value = 1 (snapshot failed) |
+| `SnapshotCompleted` | Gauge | Whether the snapshot completed successfully (1=completed, 0=not completed) | Informational - monitor snapshot completion |
+| `SnapshotDurationInSeconds` | Gauge | Total duration of the snapshot process in seconds | Informational - monitor for performance trends |
+| `SnapshotPaused` | Gauge | Whether the snapshot is currently paused (1=paused, 0=not paused) | Informational - monitor snapshot state |
+| `SnapshotPausedDurationInSeconds` | Gauge | Total time the snapshot was paused in seconds | Informational - monitor snapshot state |
+| `SnapshotRunning` | Gauge | Whether a snapshot is currently running (1=running, 0=not running) | Informational - monitor snapshot state |
+| `TotalTableCount` | Gauge | Total number of tables included in the snapshot | Informational - use for progress calculation |
+
+{{< note >}}
+Many metrics include context labels that specify the phase (`snapshot` or `streaming`), database name, and other contextual information. Metrics with a value of `-1` typically indicate that the measurement is not applicable in the current state.
+{{< /note >}}
+
+## Stream processor metrics
+
+The metrics in this section are reported by the *classic* stream processor and
+exposed through the `rdi-metrics-exporter` service. For pipelines that use
+the [Flink processor]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#stream-processor-implementations" >}}),
+see [Flink processor metrics](#flink-processor-metrics) instead.
+
+RDI reports metrics during the two main phases of the ingest pipeline, the *snapshot*
+phase and the *change data capture (CDC)* phase. (See the
+[pipeline lifecycle]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines" >}})
+docs for more information). The table below shows the full set of metrics that
+RDI reports with their descriptions.
+
+| Metric Name | Metric Type | Metric Description | Alerting Recommendations |
+|-------------|-------------|--------------------|-----------------------|
+| `incoming_records_total` | Counter | Total number of incoming records processed by the system | Informational - monitor for trends |
+| `incoming_records_created` | Gauge | Timestamp when the incoming records counter was created | Informational - no alerting needed |
+| `processed_records_total` | Counter | Total number of records that have been successfully processed | Informational - monitor for trends |
+| `rejected_records_total` | Counter | Total number of records that were rejected during processing | **Critical Alert**: Alert if > 0 (indicates processing failures) |
+| `filtered_records_total` | Counter | Total number of records that were filtered out during processing | Informational - monitor for trends |
+| `rdi_engine_state` | Gauge | Current state of the RDI engine with labels for `state` (e.g., STARTED, RUNNING) and `sync_mode` (e.g., SNAPSHOT, STREAMING) | **Critical Alert**: Alert if state indicates failure or error condition |
+| `rdi_version_info` | Gauge | Version information for RDI components with labels for `cli` and `engine` versions | Informational - use for version tracking |
+| `monitor_time_elapsed_total` | Counter | Total time elapsed (in seconds) since monitoring started | Informational - use for uptime tracking |
+| `monitor_time_elapsed_created` | Gauge | Timestamp when the monitor time elapsed counter was created | Informational - no alerting needed |
+| `rdi_incoming_entries` | Gauge | Count of incoming events by `data_source` and `operation` type (pending, inserted, updated, deleted, filtered, rejected) | Informational - monitor for trends, alert only on "rejected" > 0 |
+| `rdi_stream_event_latency_ms` | Gauge | Latency in milliseconds of the oldest event in each data stream, labeled by `data_source` | Informational - monitor based on business SLA requirements |
+| **Processor Performance Total Metrics** | | | |
+| `rdi_processed_batches_total` | Counter | Total number of processed batches | Informational - use for data ingestion and load tracking |
+| `rdi_processor_batch_size_total` | Counter | Total batch size across all processed batches | Informational - use for throughput analysis |
+| `rdi_processor_read_time_ms_total` | Counter | Total read time in milliseconds across all batches | Informational - use for performance analysis |
+| `rdi_processor_transform_time_ms_total` | Counter | Total transform time in milliseconds across all batches | Informational - use for performance analysis |
+| `rdi_processor_write_time_ms_total` | Counter | Total write time in milliseconds across all batches | Informational - use for performance analysis |
+| `rdi_processor_process_time_ms_total` | Counter | Total process time in milliseconds across all batches | Informational - use for performance analysis |
+| `rdi_processor_ack_time_ms_total` | Counter | Total acknowledgment time in milliseconds across all batches | Informational - use for performance analysis |
+| `rdi_processor_total_time_ms_total` | Counter | Sum of the total `read_time`, `process_time` and `ack_time` values in milliseconds across all batches | Informational - use for performance analysis |
+| `rdi_processor_rec_per_sec_total` | Gauge | Total records per second across all batches | Informational - use for throughput analysis |
+| **Processor Performance Last Batch Metrics** | | | |
+| `rdi_processor_batch_size_last` | Gauge | Last batch size processed | Informational - use for real-time monitoring |
+| `rdi_processor_read_time_ms_last` | Gauge | Last batch read time in milliseconds | Informational - use for real-time performance monitoring |
+| `rdi_processor_transform_time_ms_last` | Gauge | Last batch transform time in milliseconds | Informational - use for real-time performance monitoring |
+| `rdi_processor_write_time_ms_last` | Gauge | Last batch write time in milliseconds | Informational - use for real-time performance monitoring |
+| `rdi_processor_process_time_ms_last` | Gauge | Last batch process time in milliseconds | Informational - use for real-time performance monitoring |
+| `rdi_processor_ack_time_ms_last` | Gauge | Last batch acknowledgment time in milliseconds | Informational - use for real-time performance monitoring |
+| `rdi_processor_total_time_ms_last` | Gauge | Last batch total time in milliseconds | Informational - use for real-time performance monitoring |
+| `rdi_processor_rec_per_sec_last` | Gauge | Last batch records per second | Informational - use for real-time throughput monitoring |
+
+{{< note >}}
+**Additional information about stream processor metrics:**
+
+- Where the metric name has the `rdi_` prefix, this will be replaced by the Kubernetes namespace name if you supplied a custom name during installation. The prefix is always `rdi_` for VM installations.
+- Metrics with the `_created` suffix are automatically generated by Prometheus for counters and gauges to track when they were first created.
+- The `rdi_incoming_entries` metric provides a detailed breakdown for each data source by operation type.
+- The `rdi_stream_event_latency_ms` metric helps monitor data freshness and processing delays.
+- The processor performance metrics are divided into two categories:
+ - **Total metrics**: Accumulate values across all processed batches for historical analysis
+ - **Last batch metrics**: Show real-time performance data for the most recently processed batch
+{{< /note >}}
+
+## Flink processor metrics
+
+The Flink processor exposes Prometheus metrics directly from its JobManager
+and TaskManager pods. The `rdi-metrics-exporter` service is not deployed for
+Flink-based pipelines, and the metrics described in
+[Stream processor metrics](#stream-processor-metrics) are not available.
+
+The full set of metrics returned by the Flink processor is large and includes
+every metric emitted by the underlying Flink runtime (job, task, operator,
+JVM, network, and connector metrics). See the
+[Flink metrics documentation](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/ops/metrics/)
+for the full reference of Flink-emitted metrics, and the
+[Flink Prometheus reporter](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/deployment/metric_reporters/#prometheus)
+docs for the naming scheme.
+
+Configure Prometheus to scrape these metrics by enabling the JobManager and
+TaskManager `ServiceMonitor` resources under `operator.dataPlane.flinkProcessor`,
+as shown in [Helm installation](#helm-installation) above.
+
+### Useful metrics
+
+In addition to the standard Flink metrics, the Flink processor emits a small
+set of RDI-specific metrics that cover record counters, source/target
+connectivity, and stream backlog. These metrics, together with a curated
+subset of native Flink metrics, are surfaced through the
+[RDI API v2 metric collections endpoint]({{< relref "/integrate/redis-data-integration/1.19.1/reference/api-reference" >}})
+and are the recommended starting point for dashboards and alerts.
+
+**RDI-emitted metrics** (per pipeline):
+
+| Metric | Description |
+|---|---|
+| `flink_jobmanager_job_operator_coordinator_stream_type_rdiRecords` | Per-stream record counters. Labels: `stream`, `type` (one of `incoming`, `inserted`, `updated`, `deleted`, `filtered`, `rejected`). |
+| `flink_jobmanager_job_operator_coordinator_enumerator_stream_type_rdiRecords` | Per-stream backlog and freshness. Labels: `stream`, `type` (`pending` for stream length, `lastArrival` for the epoch-millisecond timestamp of the last entry). |
+| `flink_taskmanager_job_task_operator_rdi_connected` | Source or target connection status (`1` = connected, `0` = disconnected). Filter by `operator_name` equal to `Source:_source` for the source and matching the regex `.*:target:_Writer$` for target writers; treat the source or target as connected if any subtask reports `1`. |
+| `flink_taskmanager_job_task_operator_rdi_lastModified` | Epoch-millisecond timestamp of the last successful write to the target Redis database. Filter by `operator_name` matching `.*:target:_Writer$` and take the maximum across subtasks. |
+| `flink_taskmanager_job_task_operator_pendingAck` | Number of records emitted by the source but awaiting checkpoint completion before being acknowledged. Sum across subtasks. |
+
+**Native Flink metrics** used by the API:
+
+| Metric | Description |
+|---|---|
+| `flink_taskmanager_job_task_operator_numRecordsInPerSecond` | Per-operator throughput. For source throughput, filter by `operator_name` equal to `Source:_source` and sum across subtasks. For sink throughput, filter by `operator_name` matching `.*:target:_Writer$` and sum across subtasks and across all target writers. |
+| `flink_taskmanager_job_task_busyTimeMsPerSecond` | Time the task spends actively processing records (ms/s). Average across subtasks of the main chained task; exclude the `dlq:_Writer` task. |
+| `flink_taskmanager_job_task_idleTimeMsPerSecond` | Time the task spends waiting for input (ms/s). Average across subtasks of the main chained task; exclude the `dlq:_Writer` task. |
+| `flink_taskmanager_job_task_backPressuredTimeMsPerSecond` | Time the task spends back-pressured because the downstream cannot keep up (ms/s). Average across subtasks of the main chained task; exclude the `dlq:_Writer` task. |
+| `flink_jobmanager_job_lastCheckpointDuration` | Duration of the most recent checkpoint (ms). |
+| `flink_jobmanager_job_lastCheckpointSize` | Persisted size of the most recent checkpoint (bytes). |
+| `flink_jobmanager_job_numberOfCompletedCheckpoints` | Total number of completed checkpoints. |
+| `flink_jobmanager_job_numberOfFailedCheckpoints` | Total number of failed checkpoints. |
+| `flink_jobmanager_job_Time` | Time spent in each job state (ms), where `` is one of `running`, `restarting`, `failing`, `cancelling`, `initializing`, `created`, or `deploying`. The metric for the current state is non-zero; all others are zero. Use this to derive both the current job status and the time spent in it. |
+| `flink_jobmanager_job_numRestarts` | Total number of job restarts since submission. |
+
+{{< note >}}Flink runtime metric names follow Flink's own naming scheme rather
+than the `rdi_` prefix used by the classic processor. When you build
+dashboards that should work for both processors, query the two metric sets
+separately.{{< /note >}}
+
+## Operator metrics
+
+The RDI operator exposes Prometheus metrics at the `/metrics` endpoint to monitor the health and state of the operator itself and the Pipeline resources it manages.
+
+The endpoint for operator metrics is `https:///operator/metrics` (or the operator service endpoint in Kubernetes environments).
+
+### Operator metric types
+
+Most of the metrics exposed by the RDI operator are standard controller-runtime [metrics](https://book.kubebuilder.io/reference/metrics-reference).
+The metrics that are relevant for RDI operations are listed in the table below:
+
+| Metric Name | Metric Type | Metric Description | Alerting Recommendations |
+|-------------|-------------|-------------------|-------------------------|
+| `rdi_operator_pipeline_phase` | Gauge | Current phase of each Pipeline resource with labels for `namespace`, `name`, and `phase` (Active, Inactive, Pending, Resetting, Error) | **Critical Alert**: Alert if the phase is "Error" for periods longer than 2 minutes |
+| `rdi_operator_is_leader` | Gauge | Leadership status of the operator instance (1 = leader, 0 = not leader) with label for `instance_id` | Informational - monitor to ensure that the correct RDI instance is the leader in HA or DR deployments |
+
+### Understanding operator metrics
+
+**Pipeline phase tracking**: The `rdi_operator_pipeline_phase` metric helps you monitor the lifecycle state of each RDI Pipeline resource. Each pipeline reports its current phase (Active, Inactive, Pending, Resetting, or Error) as a gauge value of `1`, while all other phases for that pipeline are set to `0`. This allows you to track phase transitions and identify pipelines that are stuck in error states.
+
+**Leader election**: In high availability (HA) or disaster recovery (DR) deployments with multiple RDI instances, the `rdi_operator_is_leader` metric indicates which RDI instance is actively managing Pipeline resources. Only one RDI instance should have a value of `1` at any time, while all other instances should report `0`. This metric is useful for troubleshooting leader election issues in HA or DR deployments.
+
+### Accessing operator metrics
+
+In Kubernetes deployments, you can configure Prometheus to scrape operator metrics by enabling the Prometheus ServiceMonitor in your Helm values:
+
+```yaml
+operator:
+ prometheus:
+ enabled: true
+ labels:
+ release: prometheus
+```
+**Note:** The ServiceMonitor resources must be labelled correctly for metrics to be auto-scraped by Prometheus. The correct label is configured in Prometheus, by default it is `release: prometheus`.
+You can also expose the metrics endpoint externally using an Ingress:
+
+```yaml
+operator:
+ ingress:
+ enabled: true
+ hosts:
+ - operator.example.com
+ pathPrefix: ""
+```
+
+Then access metrics at `https://operator.example.com/operator/metrics`.
+
+## Recommended alerting strategy
+
+The alerting strategy described in the sections below focuses on system failures and data integrity issues that require immediate attention. Most other metrics are informational, so you should monitor them for trends rather than trigger alerts.
+
+### Critical alerts (immediate response required)
+
+These are the only alerts that require immediate action:
+
+**Collector alerts:**
+- `Connected = 0`: Database connectivity has been lost. RDI cannot function without a database connection.
+- `NumberOfErroneousEvents > 0`: Errors are occurring during data processing. This indicates data corruption or processing failures.
+- `SnapshotAborted = 1`: The snapshot process has failed, so the initial sync is incomplete.
+
+**Processor alerts:**
+- `rejected_records_total > 0`: Records are being rejected. This indicates data quality issues or processing failures.
+- `rdi_engine_state`: Alert only if the state indicates a clear failure condition (not just "not running").
+
+**Operator alerts:**
+- `rdi_operator_pipeline_phase` with `phase="Error"` for more than 2 minutes: A Pipeline resource has entered an error state and requires investigation.
+- No leader in HA or DR setups: If both RDI instances report `rdi_operator_is_leader = 0` for more than 2 minutes, the RDI pipeline is not active.
+- Multiple leaders in HA or DR setups: If both RDI instances report `rdi_operator_is_leader = 1`, RDI is in a "split brain" state.
+
+### Important monitoring (but not alerts)
+
+You should monitor these metrics on dashboards and review them regularly, but they don't require automated alerts:
+
+- **Queue metrics**: Queue utilization can vary widely and hitting 0% or 100% capacity may be normal during certain operations.
+- **Latency metrics**: Lag and processing times depend heavily on business requirements and normal operational patterns.
+- **Event counters**: Event rates naturally vary based on application usage patterns.
+- **Snapshot progress**: Snapshot duration and progress depend on data size, so you should typically monitor them manually.
+- **Schema changes**: Schema change frequency is highly application-dependent.
+
+### Key principles for RDI alerting
+
+- **Alert on failures, not performance**: Focus alerts on system failures rather than performance degradation.
+- **Business context matters**: Latency and throughput requirements vary significantly between organizations.
+- **Establish baselines first**: Monitor metrics for weeks before you set any threshold-based alerts.
+- **Avoid alert fatigue**: If you see too many non-critical alerts, you are less likely to take truly critical issues seriously.
+- **Use dashboards for trends**: Most metrics are better suited for dashboard monitoring than alerting
+
+### Monitoring best practices
+
+- **Dashboard-first approach**: Use Grafana dashboards to visualize trends and patterns.
+- **Baseline establishment**: Monitor your specific workload for 2-4 weeks before you consider adding more alerts.
+- **Business SLA alignment**: Only create alerts for metrics that directly impact your business SLA requirements.
+- **Manual review**: Don't use automated alerts to review metric trends. Instead, schedule regular business reviews to check them manually.
+
+## RDI logs
+
+RDI uses [fluentd](https://www.fluentd.org/) and
+[logrotate](https://linux.die.net/man/8/logrotate) to ship and rotate logs
+for its Kubernetes (K8s) components.
+So whenever a containerized component is removed by the RDI operator process or by K8s,
+the logs are available for you to inspect.
+By default, RDI stores logs in the host VM file system at `/opt/rdi/logs`.
+The logs are recorded at the minimum `INFO` level and get rotated when they reach a size of 100MB.
+RDI retains the last five log rotated files by default.
+Logs are in a straightforward text format, which lets you analyze them with several different observability tools.
+You can change the default log settings using the
+[`redis-di configure-rdi`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-configure-rdi" >}})
+command.
+
+## Dump support package
+
+If you ever need to send a comprehensive set of forensics data to Redis support then you should
+run the
+[`redis-di dump-support-package`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-dump-support-package" >}})
+command from the CLI. See
+[Troubleshooting]({{< relref "/integrate/redis-data-integration/1.19.1/troubleshooting#dump-support-package" >}})
+for more information.
diff --git a/content/integrate/redis-data-integration/1.19.1/quick-start-guide.md b/content/integrate/redis-data-integration/1.19.1/quick-start-guide.md
new file mode 100644
index 0000000000..37ca49cded
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/quick-start-guide.md
@@ -0,0 +1,153 @@
+---
+Title: Quickstart
+linkTitle: Quickstart
+description: Get started with a simple pipeline example
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/quick-start-guide/'
+---
+
+In this tutorial you will learn how to install RDI and set up a pipeline to ingest live data from a [PostgreSQL](https://www.postgresql.org/) database into a Redis database.
+
+## Prerequisites
+
+- A Redis Enterprise database that will serve as the pipeline target. The dataset that will be ingested is
+ quite small in size, so a single shard database should be enough. RDI also needs to maintain its
+ own database on the cluster to store state information. *This requires Redis Enterprise v6.4 or greater*.
+- [Redis Insight]({{< relref "/develop/tools/insight" >}})
+ to edit your pipeline
+- A virtual machine (VM) with one of the following operating systems:
+ {{< embed-md "rdi-os-reqs.md" >}}
+
+## Overview
+
+The following diagram shows the structure of the pipeline we will create (see
+the [architecture overview]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#overview" >}}) to learn how the pipeline works):
+
+{{< image filename="images/rdi/ingest/ingest-qsg.webp" >}}
+
+Here, the RDI *collector* tracks changes in PostgreSQL and writes them to streams in the
+RDI database in Redis. The *stream processor* then reads data records from the RDI
+database streams, processes them, and writes them to the target.
+
+### Install PostgreSQL
+
+We provide a [Docker](https://www.docker.com/) image for an example PostgreSQL
+database that we will use for the tutorial. Follow the
+[instructions on our Github page](https://github.com/Redislabs-Solution-Architects/rdi-quickstart-postgres/tree/main)
+to download the image and start serving the database. The database, which is
+called `chinook`, has the [schema and data](https://www.kaggle.com/datasets/samaxtech/chinook-music-store-data?select=schema_diagram.png) for an imaginary online music store
+and is already set up for the RDI collector to use.
+
+### Install RDI
+
+Install RDI using the instructions in the
+[VM installation guide]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-vm" >}}).
+
+RDI will create the pipeline template for your chosen source database type at
+`/opt/rdi/config`. You will need this pathname later when you prepare the pipeline for deployment
+(see [Prepare the pipeline](#prepare-the-pipeline) below).
+
+At the end of the installation, RDI CLI will prompt you to set the access secrets
+for both the source PostgreSQL database and the target Redis database. RDI needs these to
+run the pipeline.
+
+Use the Redis Enterprise Cluster Manager UI to create the RDI database with the following requirements:
+
+{{< embed-md "rdi-db-reqs.md" >}}
+
+### Prepare the pipeline
+
+During the installation, RDI placed the pipeline templates at `/opt/rdi/config`.
+If you go to that folder and run the `ll` command, you will see the pipeline
+configuration file, `config.yaml`, and the `jobs` folder (see the page about
+[Pipelines]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines" >}}) for more information). Use Redis Insight to open
+the `config.yaml` file and then edit the following settings:
+
+- Set the `host` to `localhost` and the `port` to 5432.
+- Under `tables`, specify the `Track` table from the source database.
+- Add the details of your target database to the `target` section.
+
+At this point, the pipeline is ready to deploy.
+
+### Create a context (optional) {#create-context}
+
+To manage and inspect RDI, you can use the
+[`redis-di`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli" >}})
+CLI tool, which has several commands for different purposes. Most of these commands connect to
+the RDI API, which you specify with the `--api-url` option. You can avoid typing this and the other
+connection options repeatedly by saving them in a *context*.
+
+When you activate a context, its saved connection options are used automatically whenever
+you use `redis-di`. If you have more than one RDI installation, you can create a context
+for each of them and select the one you want to be active using its unique name.
+
+To create a context, use the
+[`redis-di set-context`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-context" >}})
+command. For a VM installation, the API has the same hostname or IP address as your RDI VM and uses
+the default HTTPS port 443:
+
+```bash
+redis-di set-context --api-url https:// --user
+```
+
+You can save a few other options, such as a CA certificate (`--cacert`) if the API uses a private
+certificate (see the
+[reference page]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-context" >}})
+for details). When you have created a context, use
+[`redis-di use-context`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-use-context" >}})
+to activate it:
+
+```bash
+redis-di use-context
+```
+
+There are also subcommands to
+[list]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-contexts" >}})
+and [delete]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete-context" >}})
+contexts.
+
+### Deploy the pipeline
+
+You can deploy the pipeline with the following command:
+
+```bash
+redis-di deploy --dir
+```
+
+where the path is the one you supplied earlier during the installation. (You may also need
+to supply the `--api-url` option if you are not using a
+[context](#create-context) as described above.) RDI first
+validates your pipeline and then deploys it if the configuration is correct.
+
+You can also use [Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}})
+to deploy the pipeline, by adding a connection to the RDI API
+endpoint (which has the same hostname or IP address as your RDI VM and uses the default HTTPS port 443) and then clicking the **Deploy** button.
+
+Once the pipeline is running, you can use Redis Insight to view the data flow using the
+pipeline metrics. You can also connect to your target database to see the keys that RDI has written there.
+
+See [Deploy a pipeline]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/deploy" >}})
+for more information about deployment settings.
+
+### View RDI's response to data changes
+
+Once the pipeline has loaded a *snapshot* of all the existing data from the source,
+it enters *change data capture (CDC)* mode (see the
+[architecture overview]({{< relref "/integrate/redis-data-integration/1.19.1/architecture#overview" >}})
+and the
+[ingest pipeline lifecycle]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines#pipeline-lifecycle" >}})
+for more information
+).
+
+To see the RDI pipeline working in CDC mode:
+
+- Create a simulated load on the source database
+ (see [Generating load on the database](https://github.com/Redislabs-Solution-Architects/rdi-quickstart-postgres?tab=readme-ov-file#generating-load-on-the-database)
+ to learn how to do this).
+- Run
+ [`redis-di describe`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe" >}})
+ to see the flow of records. To watch it update live, pair the command with `watch`, for example
+ `watch -n 1 redis-di describe`.
+- Use [Redis Insight]({{< relref "/develop/tools/insight" >}}) to look at the data in the target database.
diff --git a/content/integrate/redis-data-integration/1.19.1/rdi-archive.md b/content/integrate/redis-data-integration/1.19.1/rdi-archive.md
new file mode 100644
index 0000000000..7fe07cea98
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/rdi-archive.md
@@ -0,0 +1,17 @@
+---
+Title: Preview version
+alwaysopen: false
+categories:
+- docs
+- operate
+- rc
+description: Describes where to view the preview version for RDI products
+linkTitle: Preview
+weight: 999
+url: '/integrate/redis-data-integration/1.19.1/rdi-archive/'
+---
+
+RDI is now in general availability but you can still access an
+[archived version of the docs for the preview version](https://docs.redis.com/rdi-preview/rdi/)
+if you need to refer to them. Note that these docs will not be updated and
+information in the current docs supersedes the content of the preview docs.
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/_index.md b/content/integrate/redis-data-integration/1.19.1/reference/_index.md
new file mode 100644
index 0000000000..612636220a
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/_index.md
@@ -0,0 +1,22 @@
+---
+Title: Reference
+alwaysopen: false
+categories:
+ - docs
+ - integrate
+ - rs
+ - rdi
+description: View reference material for Redis Data Integration
+group: di
+hideListLinks: false
+linkTitle: Reference
+summary:
+ Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 60
+url: '/integrate/redis-data-integration/1.19.1/reference/'
+---
+
+For API clients, [RDI API v1 is deprecated as of RDI 1.19.0]({{< relref "/integrate/redis-data-integration/1.19.1/reference/api-migration" >}}).
+Use [RDI API v2]({{< relref "/integrate/redis-data-integration/1.19.1/reference/api-reference" >}}) for new integrations and migrate existing clients. API v1 will not be extended with new RDI features and may be removed in a future RDI version.
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/api-migration.md b/content/integrate/redis-data-integration/1.19.1/reference/api-migration.md
new file mode 100644
index 0000000000..0229323d20
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/api-migration.md
@@ -0,0 +1,123 @@
+---
+Title: Migrate from RDI API v1 to v2
+categories:
+ - docs
+ - integrate
+ - rs
+ - rdi
+description: Migrate RDI API clients from API v1 to API v2
+group: di
+linkTitle: API migration
+summary: Migrate existing RDI API clients from the deprecated API v1 to API v2.
+type: integration
+weight: 61
+url: '/integrate/redis-data-integration/1.19.1/reference/api-migration/'
+---
+
+RDI API v1 is deprecated as of RDI 1.19.0. Existing v1 endpoints remain available for backward compatibility, but Redis recommends moving all integrations to API v2. API v1 will not be extended with new RDI features and may be removed in a future RDI version. See the [RDI API reference]({{< relref "/integrate/redis-data-integration/1.19.1/reference/api-reference" >}}) for the current request and response schemas.
+
+## What changes in API v2
+
+API v2 uses the pipeline resource to represent the current state of a pipeline. Operations update that resource and return its current state, so applications no longer need to poll a separate action ID. API v2 scopes related operations under a pipeline name in the request path.
+
+{{< note >}}
+RDI 1.19.0 supports only one pipeline, which must be named `default`. Support for other pipeline names will be added in a future version.
+{{< /note >}}
+
+The API version is part of the URL. Update `/api/v1` requests to use `/api/v2` where a corresponding v2 endpoint is available. You should also review the request and response models, because they can differ between versions.
+
+## Endpoint mapping
+
+| API v1 | API v2 |
+| --- | --- |
+| `GET /api/v1/pipelines` | `GET /api/v2/pipelines` |
+| `POST /api/v1/pipelines` | `POST /api/v2/pipelines` |
+| `PATCH /api/v1/pipelines` | `PATCH /api/v2/pipelines/{name}` |
+| `GET /api/v1/status` | `GET /api/v2/pipelines/{name}/status` |
+| `POST /api/v1/pipelines/start` | `POST /api/v2/pipelines/{name}/start` |
+| `POST /api/v1/pipelines/stop` | `POST /api/v2/pipelines/{name}/stop` |
+| `POST /api/v1/pipelines/reset` | `POST /api/v2/pipelines/{name}/reset` |
+| `GET /api/v1/monitoring/statistics` | `GET /api/v2/pipelines/{name}/metric-collections/{collection_name}` |
+| `GET /api/v1/pipelines/config/schemas` | `GET /api/v2/schemas/config` |
+| `GET /api/v1/pipelines/jobs/functions` | `GET /api/v2/functions` |
+| `GET /api/v1/pipelines/jobs/schemas` | `GET /api/v2/schemas/jobs` |
+| `PUT /api/v1/pipelines/sources` and source subresources | `PATCH /api/v2/pipelines/{name}` with `sources` in the payload |
+| `PUT /api/v1/pipelines/targets` and target subresources | `PATCH /api/v2/pipelines/{name}` with `targets` in the payload |
+| `PUT /api/v1/pipelines/processors` and `PUT /api/v1/pipelines/processors/{prop}` | `PATCH /api/v2/pipelines/{name}` with `processors` in the payload |
+| Secret provider endpoints | `POST`, `PUT`, or `DELETE /api/v2/pipelines/{name}/secrets[/{key}]` |
+| Source metadata, schemas, databases, tables, and columns endpoints | `GET /api/v2/pipelines/{name}/source-schemas/{source_name}` with the appropriate filters |
+| `POST /api/v1/pipelines/sources/dry-run` | `POST /api/v2/pipelines?dry_run=true` |
+| `POST /api/v1/pipelines/targets/dry-run` | `POST /api/v2/pipelines?dry_run=true` |
+| `POST /api/v1/pipelines/undeploy` | `DELETE /api/v2/pipelines/{name}` |
+| `POST /api/v1/trace/start` | `POST /api/v2/pipelines/{name}/traces` |
+
+API v2 also adds endpoints for DLQ inspection, target flushing, metric collections, and API information. See the [API reference]({{< relref "/integrate/redis-data-integration/1.19.1/reference/api-reference" >}}) for the complete list.
+
+## v1 endpoints without a v2 equivalent
+
+Most v1 endpoints have a v2 replacement. However, the following endpoints remain available under v1 because the current API v2 design does not define a corresponding endpoint:
+
+| v1 endpoint | Notes |
+| --- | --- |
+| `GET /api/v1/me` | Returns the authenticated user. |
+| `GET /api/v1/pipelines/strategies` | Returns pipeline strategies. |
+| `POST /api/v1/login` | API v2 continues to use this endpoint for authentication. |
+| `GET /api/v1/pipelines/config/templates/ingest/{db_type}` | Used by the CLI to scaffold pipeline configuration and will remain available. |
+| `GET /api/v1/pipelines/jobs/templates/ingest` | Used to scaffold jobs and will remain available. |
+
+## Replace action polling with pipeline-status polling
+
+In API v1, a pipeline operation returns an action ID. The client then repeatedly requests that action until it finishes:
+
+```bash
+# Start a pipeline with API v1
+action=$(curl -sS -X POST "$RDI_URL/api/v1/pipelines/start" \
+ -H "Authorization: Bearer $RDI_TOKEN" \
+ | jq -r '.action_id')
+
+# Poll the action until it completes
+curl -sS "$RDI_URL/api/v1/actions/$action" \
+ -H "Authorization: Bearer $RDI_TOKEN"
+```
+
+With API v2, you should instead include the pipeline name in the operation URL to read its status. The operation response contains the current pipeline state. You can check the pipeline's status by polling its status endpoint, as shown below:
+
+```bash
+pipeline=default
+
+# Start the pipeline with API v2
+curl -sS -X POST "$RDI_URL/api/v2/pipelines/$pipeline/start" \
+ -H "Authorization: Bearer $RDI_TOKEN"
+
+# Poll the pipeline status while the operation is in progress
+while true; do
+ status=$(curl -sS "$RDI_URL/api/v2/pipelines/$pipeline/status" \
+ -H "Authorization: Bearer $RDI_TOKEN")
+ phase=$(printf '%s' "$status" | jq -r '.status')
+ current=$(printf '%s' "$status" | jq -r '.current')
+ printf '%s\n' "$status"
+
+ case "$current:$phase" in
+ true:started|true:error) break ;;
+ *) sleep 2 ;;
+ esac
+done
+```
+
+Note:
+
+- The successful state depends on the operation. For a stop, reset, update, or create operation, wait for the corresponding successful state returned by the API. For example, wait for `stopped` instead of `started` for a stop operation.
+- Only accept a terminal status when `current` is `true`. When it is `false`, the status is outdated, so continue polling.
+- Always handle `error` as a failed operation. Use the status values documented by the API response for the operation you are performing to check for errors.
+- Do not expect an action ID from API v2.
+
+## Migration steps
+
+1. Find the applications, scripts, and SDKs in your environment that use `/api/v1`.
+2. Add the pipeline name to each v2 request. The only pipeline in 1.19.0 is always named `default`.
+3. Check the pipeline response, or call `GET /api/v2/pipelines/{name}/status`, instead of polling an action ID.
+4. Use `POST /api/v2/pipelines`, `PUT /api/v2/pipelines/{name}`, or `PATCH /api/v2/pipelines/{name}` to update source, target, processor, and secret-provider settings as needed. When using `PATCH`, omit the configuration sections that you do not want to change.
+5. Use `GET /api/v2/pipelines/{name}/metric-collections/{collection_name}` for monitoring and `GET /api/v2/pipelines/{name}/source-schemas/{source_name}` for source metadata.
+6. Test creating, updating, validating, starting, stopping, resetting, and deleting a pipeline on a non-production RDI 1.19.0 or later installation before updating production applications.
+
+Authentication and the API base URL do not change. The migration requires updates to the endpoint paths, pipeline scoping, request models, and operation status handling.
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/api-reference.md b/content/integrate/redis-data-integration/1.19.1/reference/api-reference.md
new file mode 100644
index 0000000000..8ba8113391
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/api-reference.md
@@ -0,0 +1,16 @@
+---
+linkTitle: RDI API Reference
+Title: Redis Data Integration API
+layout: apireference
+type: page
+url: '/integrate/redis-data-integration/1.19.1/reference/api-reference/'
+---
+
+
+
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/api-reference/openapi.json b/content/integrate/redis-data-integration/1.19.1/reference/api-reference/openapi.json
new file mode 100644
index 0000000000..8c3c37d013
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/api-reference/openapi.json
@@ -0,0 +1,7901 @@
+{
+ "openapi": "3.1.0",
+ "info": {
+ "title": "Redis Data Integration API",
+ "description": "> **NOTE:** RDI API v1 is deprecated as of RDI 1.19.0. Use RDI API v2 for new integrations and migrate existing clients. API v1 will not be extended with new RDI features and may be removed in a future RDI version. See the [RDI API migration guide](/integrate/redis-data-integration/reference/api-migration/).\n\nAPI for Redis Data Integration services",
+ "version": "1.19.0"
+ },
+ "paths": {
+ "/": {
+ "get": {
+ "summary": "Redirect To Doc",
+ "description": "Redirects to the API documentation page.",
+ "operationId": "redirect_to_doc__get",
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {}
+ }
+ }
+ }
+ },
+ "security": []
+ }
+ },
+ "/api/v1/login": {
+ "post": {
+ "tags": ["login"],
+ "summary": "Login",
+ "description": "Login route.\n\nArgs:\n credentials (CredentialsInfo): User credentials.\n\nReturns:\n TokenInfo: Token information.",
+ "operationId": "login_api_v1_login_post",
+ "requestBody": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/CredentialsInfo"
+ }
+ }
+ },
+ "required": true
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/TokenInfo"
+ }
+ }
+ }
+ }
+ },
+ "security": []
+ }
+ },
+ "/api/v1/pipelines/start": {
+ "post": {
+ "tags": ["secure"],
+ "summary": "Start Pipeline",
+ "description": "Starts a pipeline.",
+ "operationId": "start_pipeline_api_v1_pipelines_start_post",
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/pipelines/stop": {
+ "post": {
+ "tags": ["secure"],
+ "summary": "Stop Pipeline",
+ "description": "Stops a pipeline.",
+ "operationId": "stop_pipeline_api_v1_pipelines_stop_post",
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/pipelines/reset": {
+ "post": {
+ "tags": ["secure"],
+ "summary": "Reset Pipeline",
+ "description": "Reset pipeline.",
+ "operationId": "reset_pipeline_api_v1_pipelines_reset_post",
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/actions/{action_id}": {
+ "get": {
+ "tags": ["secure"],
+ "summary": "Get Action",
+ "description": "Gets an action status by ID.",
+ "operationId": "get_action_api_v1_actions__action_id__get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "action_id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "title": "Action Id"
+ }
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/ActionResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/me": {
+ "get": {
+ "tags": ["secure"],
+ "summary": "Protected Route",
+ "description": "Protected route.\n\nArgs:\n token_data (dict): Token data.\n\nReturns:\n Dict[str, Any]: Token data.",
+ "operationId": "protected_route_api_v1_me_get",
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Response Protected Route Api V1 Me Get"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ },
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/pipelines/sources": {
+ "put": {
+ "tags": ["secure", "sources"],
+ "summary": "Update Sources",
+ "description": "Updates the sources in RDI settings.",
+ "operationId": "update_sources_api_v1_pipelines_sources_put",
+ "requestBody": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Body",
+ "examples": [
+ {
+ "psql": {
+ "type": "cdc",
+ "logging": {
+ "level": "debug"
+ },
+ "connection": {
+ "type": "mysql",
+ "host": "${HOST_IP}",
+ "port": 3306,
+ "database": "new_schema",
+ "user": "${secret:local-vault:credentials:user}",
+ "password": "${secret:local-vault:credentials:password}"
+ }
+ }
+ }
+ ]
+ }
+ }
+ },
+ "required": true
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ ],
+ "title": "Response Update Sources Api V1 Pipelines Sources Put"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/pipelines/sources/{name}": {
+ "put": {
+ "tags": ["secure", "sources"],
+ "summary": "Upsert Source By Name",
+ "description": "Upserts a source in RDI settings by its name.",
+ "operationId": "upsert_source_by_name_api_v1_pipelines_sources__name__put",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "title": "Name"
+ }
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "object",
+ "additionalProperties": true,
+ "examples": [
+ {
+ "type": "cdc",
+ "logging": {
+ "level": "debug"
+ },
+ "connection": {
+ "type": "mysql",
+ "host": "${HOST_IP}",
+ "port": 3306,
+ "database": "new_schema",
+ "user": "${secret:local-vault:credentials:user}",
+ "password": "${secret:local-vault:credentials:password}"
+ }
+ }
+ ],
+ "title": "Body"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ ],
+ "title": "Response Upsert Source By Name Api V1 Pipelines Sources Name Put"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ },
+ "patch": {
+ "tags": ["secure", "sources"],
+ "summary": "Partial Update Source By Name",
+ "description": "Partially updates a source in RDI settings by name.",
+ "operationId": "partial_update_source_by_name_api_v1_pipelines_sources__name__patch",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "title": "Name"
+ }
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "object",
+ "additionalProperties": true,
+ "examples": [
+ {
+ "active": true,
+ "tables": {
+ "my_table": {
+ "columns": ["id", "name"]
+ }
+ }
+ }
+ ],
+ "title": "Body"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ ],
+ "title": "Response Partial Update Source By Name Api V1 Pipelines Sources Name Patch"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ },
+ "delete": {
+ "tags": ["secure", "sources"],
+ "summary": "Delete Source By Name",
+ "description": "Deletes a source from RDI settings by its name.",
+ "operationId": "delete_source_by_name_api_v1_pipelines_sources__name__delete",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "title": "Name"
+ }
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ ],
+ "title": "Response Delete Source By Name Api V1 Pipelines Sources Name Delete"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/pipelines/targets": {
+ "put": {
+ "tags": ["secure", "targets"],
+ "summary": "Update Targets",
+ "description": "Updates the targets in RDI settings.",
+ "operationId": "update_targets_api_v1_pipelines_targets_put",
+ "requestBody": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Body",
+ "examples": [
+ {
+ "target": {
+ "connection": {
+ "type": "redis",
+ "host": "${HOST_IP}",
+ "port": 12002
+ }
+ }
+ }
+ ]
+ }
+ }
+ },
+ "required": true
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ ],
+ "title": "Response Update Targets Api V1 Pipelines Targets Put"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/pipelines/targets/{name}": {
+ "put": {
+ "tags": ["secure", "targets"],
+ "summary": "Upsert Target By Name",
+ "description": "Upserts a target in RDI settings by its name.",
+ "operationId": "upsert_target_by_name_api_v1_pipelines_targets__name__put",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "title": "Name"
+ }
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "object",
+ "additionalProperties": true,
+ "examples": [
+ {
+ "connection": {
+ "type": "redis",
+ "host": "${HOST_IP}",
+ "port": 12002
+ }
+ }
+ ],
+ "title": "Body"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ ],
+ "title": "Response Upsert Target By Name Api V1 Pipelines Targets Name Put"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ },
+ "patch": {
+ "tags": ["secure", "targets"],
+ "summary": "Partial Update Target By Name",
+ "description": "Partially updates a target in RDI settings by name.",
+ "operationId": "partial_update_target_by_name_api_v1_pipelines_targets__name__patch",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "title": "Name"
+ }
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "object",
+ "additionalProperties": true,
+ "examples": [
+ {
+ "connection": {
+ "port": 12001
+ }
+ }
+ ],
+ "title": "Body"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ ],
+ "title": "Response Partial Update Target By Name Api V1 Pipelines Targets Name Patch"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ },
+ "delete": {
+ "tags": ["secure", "targets"],
+ "summary": "Delete Target By Name",
+ "description": "Deletes a target from RDI settings by its name.",
+ "operationId": "delete_target_by_name_api_v1_pipelines_targets__name__delete",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "title": "Name"
+ }
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ ],
+ "title": "Response Delete Target By Name Api V1 Pipelines Targets Name Delete"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/pipelines/secret-providers/{name}": {
+ "put": {
+ "tags": ["secure", "secret-providers"],
+ "summary": "Upsert Secret Provider By Name",
+ "description": "Upserts a secret-provider in RDI settings by its name.",
+ "operationId": "upsert_secret_provider_by_name_api_v1_pipelines_secret_providers__name__put",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "title": "Name"
+ }
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "object",
+ "additionalProperties": true,
+ "examples": [
+ {
+ "type": "vault",
+ "parameters": {
+ "vaultAddress": "http://vault.default:8200",
+ "roleName": "database",
+ "someField": "abc",
+ "objects": [
+ {
+ "objectName": "password",
+ "secretPath": "secret/data/db-pass",
+ "secretKey": "password"
+ }
+ ]
+ }
+ }
+ ],
+ "title": "Body"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ ],
+ "title": "Response Upsert Secret Provider By Name Api V1 Pipelines Secret Providers Name Put"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ },
+ "patch": {
+ "tags": ["secure", "secret-providers"],
+ "summary": "Partial Update Secret Provider By Name",
+ "description": "Partially updates a secret provider in RDI settings by name.",
+ "operationId": "partial_update_secret_provider_by_name_api_v1_pipelines_secret_providers__name__patch",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "title": "Name"
+ }
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "object",
+ "additionalProperties": true,
+ "examples": [
+ {
+ "type": "aws"
+ }
+ ],
+ "title": "Body"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ ],
+ "title": "Response Partial Update Secret Provider By Name Api V1 Pipelines Secret Providers Name Patch"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ },
+ "delete": {
+ "tags": ["secure", "secret-providers"],
+ "summary": "Delete Secret Provider By Name",
+ "description": "Deletes a secret-provider from RDI settings by its name.",
+ "operationId": "delete_secret_provider_by_name_api_v1_pipelines_secret_providers__name__delete",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "title": "Name"
+ }
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ ],
+ "title": "Response Delete Secret Provider By Name Api V1 Pipelines Secret Providers Name Delete"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/pipelines/sources/dry-run": {
+ "post": {
+ "tags": ["secure", "connection"],
+ "summary": "Sources Dry Run",
+ "description": "Tests the connection to the source database.\n\nArgs:\n body: The connection details according to the OpenAPI Scheme.\n\nReturns:\n Union[ConnectionStatusSingleConnection, Error]: The connection status.",
+ "operationId": "sources_dry_run_api_v1_pipelines_sources_dry_run_post",
+ "requestBody": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Body",
+ "examples": [
+ {
+ "type": "cdc",
+ "logging": {
+ "level": "debug"
+ },
+ "connection": {
+ "type": "mysql",
+ "host": "${HOST_IP}",
+ "port": 3306,
+ "database": "new_schema",
+ "user": "${secret:local-vault:credentials:user}",
+ "password": "${secret:local-vault:credentials:password}"
+ }
+ }
+ ]
+ }
+ }
+ },
+ "required": true
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ConnectionStatusSingleConnection"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ ],
+ "title": "Response Sources Dry Run Api V1 Pipelines Sources Dry Run Post"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/pipelines/targets/dry-run": {
+ "post": {
+ "tags": ["secure"],
+ "summary": "Multiple Targets Dry Run",
+ "description": "Tests connection to the target database(s).",
+ "operationId": "multiple_targets_dry_run_api_v1_pipelines_targets_dry_run_post",
+ "requestBody": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Body",
+ "examples": [
+ {
+ "targets": {
+ "target-one": {
+ "type": "redis",
+ "host": "${HOST_IP}",
+ "port": 12000
+ },
+ "target-two": {
+ "type": "redis",
+ "host": "${HOST_IP}",
+ "port": 12002,
+ "password": "${TARGET_DB_PASSWORD}"
+ }
+ }
+ }
+ ]
+ }
+ }
+ },
+ "required": true
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/TargetsOutput"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/pipelines/target/dry-run": {
+ "post": {
+ "tags": ["secure"],
+ "summary": "Single Target Dry Run",
+ "description": "Tests connection by name to the target database.",
+ "operationId": "single_target_dry_run_api_v1_pipelines_target_dry_run_post",
+ "requestBody": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Body",
+ "examples": [
+ {
+ "type": "redis",
+ "host": "${HOST_IP}",
+ "port": 12002
+ }
+ ]
+ }
+ }
+ },
+ "required": true
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/ConnectionStatusSingleConnection"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/pipelines/sources/{source_name}/columns": {
+ "get": {
+ "tags": ["secure"],
+ "summary": "Get Columns",
+ "description": "Returns metadata for the specified schema and tables, for the specified source.",
+ "operationId": "get_columns_api_v1_pipelines_sources__source_name__columns_get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "source_name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "title": "Source Name"
+ }
+ },
+ {
+ "name": "schema",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Optional schema name",
+ "title": "Schema"
+ },
+ "description": "Optional schema name"
+ },
+ {
+ "name": "tables",
+ "in": "query",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Comma-separated list of table names to filter by",
+ "title": "Tables"
+ },
+ "description": "Comma-separated list of table names to filter by"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/MetadataResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "The specified source is not found in the current pipeline",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "423": {
+ "description": "Collector API is not available for the specified source or is disabled",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/pipelines/sources/{source_name}/schemas": {
+ "get": {
+ "tags": ["secure"],
+ "summary": "Get Schemas",
+ "description": "Returns all available schemas for the specified source.\nFor databases that do not support schemas (such as MySQL), returns available databases instead.",
+ "operationId": "get_schemas_api_v1_pipelines_sources__source_name__schemas_get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "source_name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "title": "Source Name"
+ }
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/SchemaResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "The specified source is not found in the current pipeline",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "423": {
+ "description": "Collector API is not available for the specified source or is disabled",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/pipelines/sources/{source_name}/databases": {
+ "get": {
+ "tags": ["secure"],
+ "summary": "Get Databases",
+ "description": "Returns all available databases for the specified source.",
+ "operationId": "get_databases_api_v1_pipelines_sources__source_name__databases_get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "source_name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "title": "Source Name"
+ }
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/DatabaseResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "The specified source is not found in the current pipeline",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "423": {
+ "description": "Collector API is not available for the specified source or is disabled",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/pipelines/sources/{source_name}/metadata": {
+ "get": {
+ "tags": ["secure"],
+ "summary": "Get Metadata",
+ "description": "Returns metadata for the specified schema and tables, for the specified source.",
+ "operationId": "get_metadata_api_v1_pipelines_sources__source_name__metadata_get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "source_name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "title": "Source Name"
+ }
+ },
+ {
+ "name": "schema",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Optional schema name",
+ "title": "Schema"
+ },
+ "description": "Optional schema name"
+ },
+ {
+ "name": "tables",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Optional comma-separated list of table names to filter by",
+ "title": "Tables"
+ },
+ "description": "Optional comma-separated list of table names to filter by"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/MetadataResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "The specified source is not found in the current pipeline",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "423": {
+ "description": "Collector API is not available for the specified source or is disabled",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/pipelines/sources/{source_name}/tables": {
+ "get": {
+ "tags": ["secure"],
+ "summary": "Get Tables",
+ "description": "Returns available tables in the specified schema for the specified source.",
+ "operationId": "get_tables_api_v1_pipelines_sources__source_name__tables_get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "source_name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "title": "Source Name"
+ }
+ },
+ {
+ "name": "schema",
+ "in": "query",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Schema name",
+ "title": "Schema"
+ },
+ "description": "Schema name"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/TableResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "The specified source is not found in the current pipeline",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "423": {
+ "description": "Collector API is not available for the specified source or is disabled",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/pipelines": {
+ "get": {
+ "tags": ["secure", "deploy"],
+ "summary": "Retrieve Pipelines",
+ "description": "Returns the current pipeline.",
+ "operationId": "retrieve_pipelines_api_v1_pipelines_get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "object",
+ "additionalProperties": true,
+ "title": "Response Retrieve Pipelines Api V1 Pipelines Get"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ },
+ "post": {
+ "tags": ["secure", "deploy"],
+ "summary": "Pipelines",
+ "description": "Updates the current pipeline completely, or creates it if it doesn't exist, using the provided configuration.\nBefore creating or updating, validates the provided configuration for schema compliance and integrity.",
+ "operationId": "pipelines_api_v1_pipelines_post",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "dry_run",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "anyOf": [
+ {
+ "type": "boolean"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Only validate the provided configuration without making changes",
+ "default": false,
+ "title": "Dry Run"
+ },
+ "description": "Only validate the provided configuration without making changes"
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "object",
+ "additionalProperties": true,
+ "examples": [
+ {
+ "sources": {
+ "psql": {
+ "type": "cdc",
+ "logging": {
+ "level": "debug"
+ },
+ "connection": {
+ "type": "postgresql",
+ "host": "host.docker.internal",
+ "port": 5432,
+ "database": "chinook",
+ "user": "postgres",
+ "password": "postgres"
+ },
+ "tables": {
+ "public.invoice": {
+ "columns": [
+ "billingaddress",
+ "billingcity",
+ "billingcountry",
+ "billingpostalcode",
+ "total",
+ "customerid",
+ "billingstate",
+ "invoiceid",
+ "invoicedate"
+ ],
+ "keys": ["invoiceid"]
+ },
+ "public.track": {
+ "columns": [
+ "genreid",
+ "milliseconds",
+ "mediatypeid",
+ "trackid",
+ "composer",
+ "bytes",
+ "name",
+ "albumid",
+ "unitprice"
+ ],
+ "keys": ["trackid"]
+ },
+ "public.mediatype": {
+ "columns": ["mediatypeid", "name"],
+ "keys": ["mediatypeid"]
+ },
+ "public.customer": {
+ "columns": [
+ "country",
+ "firstname",
+ "address",
+ "city",
+ "lastname",
+ "phone",
+ "postalcode",
+ "customerid",
+ "company",
+ "state",
+ "fax",
+ "email",
+ "supportrepid"
+ ],
+ "keys": ["customerid"]
+ },
+ "public.genre": {
+ "columns": ["genreid", "name"],
+ "keys": ["genreid"]
+ },
+ "public.invoiceline": {
+ "columns": [
+ "quantity",
+ "trackid",
+ "invoicelineid",
+ "invoiceid",
+ "unitprice"
+ ],
+ "keys": ["invoicelineid"]
+ },
+ "public.playlist": {
+ "columns": ["playlistid", "name"],
+ "keys": ["playlistid"]
+ },
+ "public.employee": {
+ "columns": [
+ "country",
+ "firstname",
+ "birthdate",
+ "address",
+ "city",
+ "reportsto",
+ "title",
+ "employeeid",
+ "hiredate",
+ "lastname",
+ "phone",
+ "postalcode",
+ "state",
+ "fax",
+ "email"
+ ],
+ "keys": ["employeeid"]
+ },
+ "public.album": {
+ "columns": ["albumid", "artistid", "title"],
+ "keys": ["albumid"]
+ },
+ "public.artist": {
+ "columns": ["name", "artistid"],
+ "keys": ["artistid"]
+ },
+ "public.playlisttrack": {
+ "columns": ["playlistid", "trackid"],
+ "keys": ["playlistid", "trackid"]
+ }
+ }
+ }
+ },
+ "targets": {
+ "target": {
+ "type": "redis",
+ "host": "host.docker.internal",
+ "port": 12002,
+ "password": "test"
+ }
+ },
+ "processors": {
+ "target_data_type": "hash"
+ },
+ "jobs": [
+ {
+ "name": "chinook_customer",
+ "source": {
+ "schema": "public",
+ "table": "Customer"
+ },
+ "output": [
+ {
+ "uses": "redis.write",
+ "with": {
+ "connection": "target",
+ "key": {
+ "expression": "concat(['CustomerId:', CustomerId])",
+ "language": "jmespath"
+ },
+ "expire": 100
+ }
+ }
+ ]
+ }
+ ]
+ }
+ ],
+ "title": "Body"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Pipeline created or updated successfully",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ },
+ "examples": {
+ "dry_run=false": {
+ "summary": "dry_run=false",
+ "value": {
+ "action_id": "1234567890"
+ }
+ },
+ "dry_run=true": {
+ "summary": "dry_run=true",
+ "value": {}
+ }
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "422": {
+ "description": "Configuration validation failed",
+ "content": {
+ "application/json": {
+ "examples": {
+ "Single validation error": {
+ "summary": "Single validation error",
+ "value": {
+ "detail": "Validation failed",
+ "errors": [
+ {
+ "code": "job_validation_error",
+ "message": "Job 'users' table 'dbz.users' was not found in the list of captured source tables.",
+ "details": {
+ "job": "users",
+ "captured_tables": ["customers", "invoices"]
+ }
+ }
+ ]
+ }
+ },
+ "Multiple validation errors": {
+ "summary": "Multiple validation errors",
+ "value": {
+ "detail": "Validation failed",
+ "errors": [
+ {
+ "code": "job_validation_error",
+ "message": "Job name 'user_sync' is not unique",
+ "details": {
+ "job": "user_sync"
+ }
+ },
+ {
+ "code": "validation_error",
+ "message": "Source database 'test1' is not used in any source tables.",
+ "details": {}
+ }
+ ]
+ }
+ }
+ },
+ "schema": {
+ "$ref": "#/components/schemas/ExtendedErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ },
+ "patch": {
+ "tags": ["secure", "deploy"],
+ "summary": "Patch Pipelines",
+ "description": "Updates the current pipeline partially by merging with the provided configuration using JSON Merge Patch semantics.\nBefore updating, validates the merged configuration for schema compliance and integrity.",
+ "operationId": "patch_pipelines_api_v1_pipelines_patch",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "dry_run",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "anyOf": [
+ {
+ "type": "boolean"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Only validate the provided configuration without making changes",
+ "default": false,
+ "title": "Dry Run"
+ },
+ "description": "Only validate the provided configuration without making changes"
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "object",
+ "additionalProperties": true,
+ "examples": [
+ {
+ "sources": {
+ "psql": {
+ "type": "cdc",
+ "logging": {
+ "level": "debug"
+ },
+ "connection": {
+ "type": "postgresql",
+ "host": "host.docker.internal",
+ "port": 5432,
+ "database": "chinook",
+ "user": "postgres",
+ "password": "postgres"
+ },
+ "tables": {
+ "public.invoice": {
+ "columns": [
+ "billingaddress",
+ "billingcity",
+ "billingcountry",
+ "billingpostalcode",
+ "total",
+ "customerid",
+ "billingstate",
+ "invoiceid",
+ "invoicedate"
+ ],
+ "keys": ["invoiceid"]
+ },
+ "public.track": {
+ "columns": [
+ "genreid",
+ "milliseconds",
+ "mediatypeid",
+ "trackid",
+ "composer",
+ "bytes",
+ "name",
+ "albumid",
+ "unitprice"
+ ],
+ "keys": ["trackid"]
+ },
+ "public.mediatype": {
+ "columns": ["mediatypeid", "name"],
+ "keys": ["mediatypeid"]
+ },
+ "public.customer": {
+ "columns": [
+ "country",
+ "firstname",
+ "address",
+ "city",
+ "lastname",
+ "phone",
+ "postalcode",
+ "customerid",
+ "company",
+ "state",
+ "fax",
+ "email",
+ "supportrepid"
+ ],
+ "keys": ["customerid"]
+ },
+ "public.genre": {
+ "columns": ["genreid", "name"],
+ "keys": ["genreid"]
+ },
+ "public.invoiceline": {
+ "columns": [
+ "quantity",
+ "trackid",
+ "invoicelineid",
+ "invoiceid",
+ "unitprice"
+ ],
+ "keys": ["invoicelineid"]
+ },
+ "public.playlist": {
+ "columns": ["playlistid", "name"],
+ "keys": ["playlistid"]
+ },
+ "public.employee": {
+ "columns": [
+ "country",
+ "firstname",
+ "birthdate",
+ "address",
+ "city",
+ "reportsto",
+ "title",
+ "employeeid",
+ "hiredate",
+ "lastname",
+ "phone",
+ "postalcode",
+ "state",
+ "fax",
+ "email"
+ ],
+ "keys": ["employeeid"]
+ },
+ "public.album": {
+ "columns": ["albumid", "artistid", "title"],
+ "keys": ["albumid"]
+ },
+ "public.artist": {
+ "columns": ["name", "artistid"],
+ "keys": ["artistid"]
+ },
+ "public.playlisttrack": {
+ "columns": ["playlistid", "trackid"],
+ "keys": ["playlistid", "trackid"]
+ }
+ }
+ }
+ },
+ "targets": {
+ "target": {
+ "type": "redis",
+ "host": "host.docker.internal",
+ "port": 12002,
+ "password": "test"
+ }
+ },
+ "processors": {
+ "target_data_type": "hash"
+ },
+ "jobs": [
+ {
+ "name": "chinook_customer",
+ "source": {
+ "schema": "public",
+ "table": "Customer"
+ },
+ "output": [
+ {
+ "uses": "redis.write",
+ "with": {
+ "connection": "target",
+ "key": {
+ "expression": "concat(['CustomerId:', CustomerId])",
+ "language": "jmespath"
+ },
+ "expire": 100
+ }
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "targets": {
+ "target": {
+ "type": "redis",
+ "host": "host.docker.internal",
+ "port": 12002,
+ "password": "test"
+ }
+ },
+ "processors": {
+ "target_data_type": "hash"
+ }
+ }
+ ],
+ "title": "Body"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Pipeline updated successfully",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ },
+ "examples": {
+ "dry_run=false": {
+ "summary": "dry_run=false",
+ "value": {
+ "action_id": "1234567890"
+ }
+ },
+ "dry_run=true": {
+ "summary": "dry_run=true",
+ "value": {}
+ }
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "422": {
+ "description": "Configuration validation failed",
+ "content": {
+ "application/json": {
+ "examples": {
+ "Single validation error": {
+ "summary": "Single validation error",
+ "value": {
+ "detail": "Validation failed",
+ "errors": [
+ {
+ "code": "job_validation_error",
+ "message": "Job 'users' table 'dbz.users' was not found in the list of captured source tables.",
+ "details": {
+ "job": "users",
+ "captured_tables": ["customers", "invoices"]
+ }
+ }
+ ]
+ }
+ },
+ "Multiple validation errors": {
+ "summary": "Multiple validation errors",
+ "value": {
+ "detail": "Validation failed",
+ "errors": [
+ {
+ "code": "job_validation_error",
+ "message": "Job name 'user_sync' is not unique",
+ "details": {
+ "job": "user_sync"
+ }
+ },
+ {
+ "code": "validation_error",
+ "message": "Source database 'test1' is not used in any source tables.",
+ "details": {}
+ }
+ ]
+ }
+ }
+ },
+ "schema": {
+ "$ref": "#/components/schemas/ExtendedErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/pipelines/undeploy": {
+ "post": {
+ "tags": ["secure", "deploy"],
+ "summary": "Undeploy Pipelines",
+ "description": "Undeploys a pipeline.",
+ "operationId": "undeploy_pipelines_api_v1_pipelines_undeploy_post",
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ ],
+ "title": "Response Undeploy Pipelines Api V1 Pipelines Undeploy Post"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/pipelines/jobs/functions": {
+ "get": {
+ "tags": ["secure"],
+ "summary": "Pipelines Jobs Functions",
+ "description": "Retrieves a list of all supported JMESPath functions (builtin and custom) that can be used in JMESPath.",
+ "operationId": "pipelines_jobs_functions_api_v1_pipelines_jobs_functions_get",
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelinesFunctionsResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/pipelines/config/templates/ingest/{db_type}": {
+ "get": {
+ "tags": ["secure"],
+ "summary": "Pipelines Config Templates",
+ "description": "Returns the YAML template describing the config for the specified db type.",
+ "operationId": "pipelines_config_templates_api_v1_pipelines_config_templates_ingest__db_type__get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "db_type",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "$ref": "#/components/schemas/DbType"
+ }
+ },
+ {
+ "name": "db_flavor",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/DbFlavor"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Optional database flavor",
+ "title": "Db Flavor"
+ },
+ "description": "Optional database flavor"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/TemplateResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ ],
+ "title": "Response Pipelines Config Templates Api V1 Pipelines Config Templates Ingest Db Type Get",
+ "$ref": "#/components/schemas/TemplateResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "422": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unprocessable Entity"
+ },
+ "501": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Implemented"
+ }
+ }
+ }
+ },
+ "/api/v1/pipelines/jobs/templates/ingest": {
+ "get": {
+ "tags": ["secure"],
+ "summary": "Pipelines Jobs Templates",
+ "description": "Returns the YAML templates describing how to create a job",
+ "operationId": "pipelines_jobs_templates_api_v1_pipelines_jobs_templates_ingest_get",
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/TemplateResponse",
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/TemplateResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ ],
+ "title": "Response Pipelines Jobs Templates Api V1 Pipelines Jobs Templates Ingest Get"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "501": {
+ "description": "Not Implemented",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/pipelines/jobs/dry-run": {
+ "post": {
+ "tags": ["secure"],
+ "summary": "Job Dry Run",
+ "description": "Executes a job using input data in a dry-run mode.",
+ "operationId": "job_dry_run_api_v1_pipelines_jobs_dry_run_post",
+ "requestBody": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/JobDryRunBody"
+ }
+ }
+ },
+ "required": true
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/JobDryRunSuccessResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ ],
+ "title": "Response Job Dry Run Api V1 Pipelines Jobs Dry Run Post"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "422": {
+ "description": "Unprocessable Entity",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/pipelines/strategies": {
+ "get": {
+ "tags": ["secure"],
+ "summary": "Pipelines Strategies",
+ "description": "Retrieves available strategies.",
+ "operationId": "pipelines_strategies_api_v1_pipelines_strategies_get",
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelinesStrategiesResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/pipelines/config/schemas": {
+ "get": {
+ "tags": ["secure"],
+ "summary": "Pipelines Config Schemas",
+ "description": "Retrieves config schema.",
+ "operationId": "pipelines_config_schemas_api_v1_pipelines_config_schemas_get",
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Response Pipelines Config Schemas Api V1 Pipelines Config Schemas Get"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/pipelines/jobs/schemas": {
+ "get": {
+ "tags": ["secure"],
+ "summary": "Pipelines Jobs Schemas",
+ "description": "Retrieves job schemas.",
+ "operationId": "pipelines_jobs_schemas_api_v1_pipelines_jobs_schemas_get",
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Response Pipelines Jobs Schemas Api V1 Pipelines Jobs Schemas Get"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/pipelines/processors": {
+ "put": {
+ "tags": ["secure", "processors"],
+ "summary": "Update Processors",
+ "description": "Updates multiple processor properties in RDI settings.\nThe request body should be a dictionary with property names as keys and new values as values.",
+ "operationId": "update_processors_api_v1_pipelines_processors_put",
+ "requestBody": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Body"
+ },
+ "examples": {
+ "multi_update": {
+ "summary": "Update multiple processor properties",
+ "description": "Example of updating read_batch_size, write_batch_size, and target_data_type.",
+ "value": {
+ "read_batch_size": 1000,
+ "write_batch_size": 500,
+ "target_data_type": "hash"
+ }
+ }
+ }
+ }
+ },
+ "required": true
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ ],
+ "title": "Response Update Processors Api V1 Pipelines Processors Put"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/pipelines/processors/{prop}": {
+ "put": {
+ "tags": ["secure", "processors"],
+ "summary": "Upsert Processors Property",
+ "description": "Upserts a single processor property by property name.\nThe request body should contain a single 'value' field with the new value for the property.\nThe property name is specified in the URL path.",
+ "operationId": "upsert_processors_property_api_v1_pipelines_processors__prop__put",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "prop",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "$ref": "#/components/schemas/ProcessorPropertyName"
+ }
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/ProcessorProperty"
+ },
+ "examples": {
+ "read_batch_size": {
+ "summary": "Update read_batch_size",
+ "description": "Example of updating the read_batch_size property.",
+ "value": {
+ "value": 1000
+ }
+ },
+ "target_data_type": {
+ "summary": "Update target_data_type",
+ "description": "Example of updating the target_data_type property.",
+ "value": {
+ "value": "hash"
+ }
+ },
+ "write_batch_size": {
+ "summary": "Update write_batch_size",
+ "description": "Example of updating the write_batch_size property.",
+ "value": {
+ "value": 500
+ }
+ }
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ },
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error"
+ }
+ ],
+ "title": "Response Upsert Processors Property Api V1 Pipelines Processors Prop Put"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/secrets/{secret_name}": {
+ "put": {
+ "tags": ["secure", "secrets"],
+ "summary": "Set Multi Key Secret",
+ "description": "Stores multiple key-value pairs under a single secret name in the key store.\nNOTE: {secret_name} is legacy and is not used - calculated automatically\n\nArgs:\n secret_name (str): The secret name (legacy, not used - kept for backward compatibility).\n secret_keys (SecretKeyModel): The secret key/value pairs to set.\n\nReturns:\n OperationResponse: The operation response.",
+ "operationId": "set_multi_key_secret_api_v1_secrets__secret_name__put",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "secret_name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "title": "Secret Name"
+ }
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/SecretKeyModel"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Secrets were set successfully",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/OperationResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/secrets": {
+ "put": {
+ "tags": ["secure", "secrets"],
+ "summary": "Set Secrets",
+ "description": "Sets multiple secret values in the key store for multiple secret names.\n\nArgs:\n secrets (SecretsModel): Collection of secrets to store.\n\nReturns:\n OperationResponse: The operation response.",
+ "operationId": "set_secrets_api_v1_secrets_put",
+ "requestBody": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/SecretsModel"
+ }
+ }
+ },
+ "required": true
+ },
+ "responses": {
+ "200": {
+ "description": "Secrets were set successfully",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/OperationResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/monitoring/statistics": {
+ "get": {
+ "tags": ["secure"],
+ "summary": "Statistics",
+ "description": "Retrieves RDI statistics.",
+ "operationId": "statistics_api_v1_monitoring_statistics_get",
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/StatisticsResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/status": {
+ "get": {
+ "tags": ["secure"],
+ "summary": "Get Status",
+ "description": "Retrieves RDI status.",
+ "operationId": "get_status_api_v1_status_get",
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/StatusResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v1/trace/start": {
+ "post": {
+ "tags": ["secure"],
+ "summary": "Start Trace",
+ "description": "Starts a trace session for troubleshooting data transformation.",
+ "operationId": "start_trace_api_v1_trace_start_post",
+ "requestBody": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/TraceRequest"
+ }
+ }
+ },
+ "required": true
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/ActionIdResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ]
+ }
+ },
+ "/api/v2/pipelines": {
+ "get": {
+ "tags": ["v2", "pipelines"],
+ "summary": "Get Pipelines",
+ "description": "Gets all pipelines.",
+ "operationId": "get_pipelines_api_v2_pipelines_get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "array",
+ "items": {
+ "$ref": "#/components/schemas/PipelineResponse"
+ },
+ "title": "Response Get Pipelines Api V2 Pipelines Get"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ }
+ }
+ },
+ "post": {
+ "tags": ["v2", "pipelines"],
+ "summary": "Create Pipeline",
+ "description": "Creates a new pipeline.",
+ "operationId": "create_pipeline_api_v2_pipelines_post",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "dry_run",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "anyOf": [
+ {
+ "type": "boolean"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Only validate the provided configuration without making changes",
+ "default": false,
+ "title": "Dry Run"
+ },
+ "description": "Only validate the provided configuration without making changes"
+ },
+ {
+ "name": "validate_tables",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "type": "boolean",
+ "description": "Validate against source and target databases via collector API",
+ "default": true,
+ "title": "Validate Tables"
+ },
+ "description": "Validate against source and target databases via collector API"
+ },
+ {
+ "name": "validate_cdc",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "type": "boolean",
+ "description": "Validate CDC configuration via collector API",
+ "default": false,
+ "title": "Validate Cdc"
+ },
+ "description": "Validate CDC configuration via collector API"
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineCreateRequest",
+ "description": "Pipeline details"
+ }
+ }
+ }
+ },
+ "responses": {
+ "201": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "409": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Conflict"
+ },
+ "422": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unprocessable Entity"
+ },
+ "503": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Service Unavailable"
+ }
+ }
+ }
+ },
+ "/api/v2/pipelines/{name}": {
+ "get": {
+ "tags": ["v2", "pipelines"],
+ "summary": "Get Pipeline",
+ "description": "Gets a pipeline by name.",
+ "operationId": "get_pipeline_api_v2_pipelines__name__get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ }
+ }
+ },
+ "put": {
+ "tags": ["v2", "pipelines"],
+ "summary": "Update Pipeline",
+ "description": "Updates a pipeline completely, or creates it if it doesn't exist.\nRetries 409 conflicts received from the K8s API server.",
+ "operationId": "update_pipeline_api_v2_pipelines__name__put",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ },
+ {
+ "name": "dry_run",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "anyOf": [
+ {
+ "type": "boolean"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Only validate the provided configuration without making changes",
+ "default": false,
+ "title": "Dry Run"
+ },
+ "description": "Only validate the provided configuration without making changes"
+ },
+ {
+ "name": "validate_tables",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "type": "boolean",
+ "description": "Validate against source and target databases via collector API",
+ "default": true,
+ "title": "Validate Tables"
+ },
+ "description": "Validate against source and target databases via collector API"
+ },
+ {
+ "name": "validate_cdc",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "type": "boolean",
+ "description": "Validate CDC configuration via collector API",
+ "default": false,
+ "title": "Validate Cdc"
+ },
+ "description": "Validate CDC configuration via collector API"
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineUpdateRequest",
+ "description": "Pipeline details"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "422": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unprocessable Entity"
+ },
+ "503": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Service Unavailable"
+ }
+ }
+ },
+ "patch": {
+ "tags": ["v2", "pipelines"],
+ "summary": "Patch Pipeline",
+ "description": "Updates a pipeline partially (only updates specified fields), with retries on 409 conflicts received from the K8s API server.",
+ "operationId": "patch_pipeline_api_v2_pipelines__name__patch",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ },
+ {
+ "name": "dry_run",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "anyOf": [
+ {
+ "type": "boolean"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Only validate the provided configuration without making changes",
+ "default": false,
+ "title": "Dry Run"
+ },
+ "description": "Only validate the provided configuration without making changes"
+ },
+ {
+ "name": "validate_tables",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "type": "boolean",
+ "description": "Validate against source and target databases via collector API",
+ "default": true,
+ "title": "Validate Tables"
+ },
+ "description": "Validate against source and target databases via collector API"
+ },
+ {
+ "name": "validate_cdc",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "type": "boolean",
+ "description": "Validate CDC configuration via collector API",
+ "default": false,
+ "title": "Validate Cdc"
+ },
+ "description": "Validate CDC configuration via collector API"
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelinePatchRequest",
+ "description": "Partial pipeline details"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ },
+ "422": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unprocessable Entity"
+ },
+ "503": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Service Unavailable"
+ }
+ }
+ },
+ "delete": {
+ "tags": ["v2", "pipelines"],
+ "summary": "Delete Pipeline",
+ "description": "Replaces a pipeline with an empty inactive pipeline and resets it.\n\nThis is done to ensure that no artifacts (offsets, schema, data streams etc.) are left behind,\nas this may cause errors and unexpected behavior when trying to re-create the pipeline.",
+ "operationId": "delete_pipeline_api_v2_pipelines__name__delete",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ }
+ ],
+ "responses": {
+ "204": {
+ "description": "Successful Response"
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ }
+ }
+ }
+ },
+ "/api/v2/pipelines/{name}/status": {
+ "get": {
+ "tags": ["v2", "pipelines"],
+ "summary": "Get Pipeline Status",
+ "description": "Gets the status of a pipeline by name.",
+ "operationId": "get_pipeline_status_api_v2_pipelines__name__status_get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineStatusResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ }
+ }
+ }
+ },
+ "/api/v2/pipelines/{name}/start": {
+ "post": {
+ "tags": ["v2", "pipelines"],
+ "summary": "Start Pipeline",
+ "description": "Starts a pipeline, retrying on 409 conflicts received from the K8s API server.",
+ "operationId": "start_pipeline_api_v2_pipelines__name__start_post",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ }
+ }
+ }
+ },
+ "/api/v2/pipelines/{name}/stop": {
+ "post": {
+ "tags": ["v2", "pipelines"],
+ "summary": "Stop Pipeline",
+ "description": "Stops a pipeline, retrying on 409 conflicts received from the K8s API server.",
+ "operationId": "stop_pipeline_api_v2_pipelines__name__stop_post",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ }
+ }
+ }
+ },
+ "/api/v2/pipelines/{name}/reset": {
+ "post": {
+ "tags": ["v2", "pipelines"],
+ "summary": "Reset Pipeline",
+ "description": "Resets a pipeline, retrying on 409 conflicts received from the K8s API server.",
+ "operationId": "reset_pipeline_api_v2_pipelines__name__reset_post",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ }
+ }
+ }
+ },
+ "/api/v2/pipelines/{name}/metric-collections": {
+ "get": {
+ "tags": ["v2", "pipelines", "metric-collections"],
+ "summary": "Get Metric Collections",
+ "description": "Gets all metric collections for a pipeline.",
+ "operationId": "get_metric_collections_api_v2_pipelines__name__metric_collections_get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "array",
+ "items": {
+ "$ref": "#/components/schemas/PipelineMetricCollectionResponse"
+ },
+ "title": "Response Get Metric Collections Api V2 Pipelines Name Metric Collections Get"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ }
+ }
+ }
+ },
+ "/api/v2/pipelines/{name}/metric-collections/{collection_name}": {
+ "get": {
+ "tags": ["v2", "pipelines", "metric-collections"],
+ "summary": "Get Metric Collection",
+ "description": "Gets a metric collection for a pipeline.",
+ "operationId": "get_metric_collection_api_v2_pipelines__name__metric_collections__collection_name__get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ },
+ {
+ "name": "collection_name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Metric collection name",
+ "title": "Collection Name"
+ },
+ "description": "Metric collection name"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineMetricCollectionResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ }
+ }
+ }
+ },
+ "/api/v2/pipelines/{name}/dlqs": {
+ "get": {
+ "tags": ["v2", "dlq"],
+ "summary": "Get all DLQ streams with counts",
+ "description": "Returns all tables that have DLQ records with their total counts. Table names are returned in the format: source_name.schema_name.table_name (for PostgreSQL, Oracle, SQL Server, Spanner) or source_name.database_name.table_name (for MySQL, MariaDB, MongoDB).",
+ "operationId": "get_all_dlqs_api_v2_pipelines__name__dlqs_get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/DlqListResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ },
+ "500": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Internal Server Error"
+ }
+ }
+ }
+ },
+ "/api/v2/pipelines/{name}/dlqs/{full_table_name}/records": {
+ "get": {
+ "tags": ["v2", "dlq"],
+ "summary": "Get DLQ records for a specific stream",
+ "description": "Returns the DLQ records for a specific table with pagination. The stream name must be in the format: source_name.schema_name.table_name (e.g., 'postgres.public.users') or source_name.database_name.table_name (e.g., 'mysql.mydb.users').",
+ "operationId": "get_dlq_records_api_v2_pipelines__name__dlqs__full_table_name__records_get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ },
+ {
+ "name": "full_table_name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Full stream name in format: source_name.schema_name.table_name (e.g., 'postgres.public.users') or source_name.database_name.table_name (e.g., 'mysql.mydb.users')",
+ "title": "Full Table Name"
+ },
+ "description": "Full stream name in format: source_name.schema_name.table_name (e.g., 'postgres.public.users') or source_name.database_name.table_name (e.g., 'mysql.mydb.users')"
+ },
+ {
+ "name": "limit",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "type": "integer",
+ "maximum": 1000,
+ "minimum": 1,
+ "description": "Number of records to return",
+ "default": 20,
+ "title": "Limit"
+ },
+ "description": "Number of records to return"
+ },
+ {
+ "name": "offset",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "type": "integer",
+ "minimum": 0,
+ "description": "Starting position for pagination",
+ "default": 0,
+ "title": "Offset"
+ },
+ "description": "Starting position for pagination"
+ },
+ {
+ "name": "sort_order",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "$ref": "#/components/schemas/SortOrder",
+ "description": "Sort order: 'asc' (oldest first) or 'desc' (newest first)",
+ "default": "desc"
+ },
+ "description": "Sort order: 'asc' (oldest first) or 'desc' (newest first)"
+ },
+ {
+ "name": "fields",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Comma-separated list of fields to include in projection. Allowed fields: __dlq_stream_entry_id, db, opcode, reason_rejected, removed_fields, schema, server_name, source, source_type, table, timestamp",
+ "title": "Fields"
+ },
+ "description": "Comma-separated list of fields to include in projection. Allowed fields: __dlq_stream_entry_id, db, opcode, reason_rejected, removed_fields, schema, server_name, source, source_type, table, timestamp"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/DlqRecordsResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "400": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Bad Request"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ },
+ "500": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Internal Server Error"
+ }
+ }
+ }
+ },
+ "/api/v2/pipelines/{name}/dlqs/{full_table_name}": {
+ "get": {
+ "tags": ["v2", "dlq"],
+ "summary": "Get DLQ info for a specific stream",
+ "description": "Returns the DLQ count for a specific stream. The stream name must be in the format: source_name.schema_name.table_name (e.g., 'postgres.public.users') or source_name.database_name.table_name (e.g., 'mysql.mydb.users').",
+ "operationId": "get_dlq_by_table_name_api_v2_pipelines__name__dlqs__full_table_name__get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ },
+ {
+ "name": "full_table_name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Full stream name in format: source_name.schema_name.table_name (e.g., 'postgres.public.users') or source_name.database_name.table_name (e.g., 'mysql.mydb.users')",
+ "title": "Full Table Name"
+ },
+ "description": "Full stream name in format: source_name.schema_name.table_name (e.g., 'postgres.public.users') or source_name.database_name.table_name (e.g., 'mysql.mydb.users')"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/DlqTableResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ },
+ "500": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Internal Server Error"
+ }
+ }
+ }
+ },
+ "/api/v2/pipelines/{name}/secrets": {
+ "get": {
+ "tags": ["v2", "pipelines", "secrets"],
+ "summary": "Get Pipeline Secrets",
+ "description": "Lists all pipeline secrets.",
+ "operationId": "get_pipeline_secrets_api_v2_pipelines__name__secrets_get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "array",
+ "items": {
+ "$ref": "#/components/schemas/PipelineSecretResponse"
+ },
+ "title": "Response Get Pipeline Secrets Api V2 Pipelines Name Secrets Get"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ }
+ }
+ },
+ "post": {
+ "tags": ["v2", "pipelines", "secrets"],
+ "summary": "Create Secret",
+ "description": "Creates a new pipeline secret.",
+ "operationId": "create_secret_api_v2_pipelines__name__secrets_post",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineSecretCreateRequest",
+ "description": "Pipeline secret details"
+ }
+ }
+ }
+ },
+ "responses": {
+ "201": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineSecretResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ },
+ "409": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Conflict"
+ }
+ }
+ }
+ },
+ "/api/v2/pipelines/{name}/secrets/{key}": {
+ "get": {
+ "tags": ["v2", "pipelines", "secrets"],
+ "summary": "Get Secret",
+ "description": "Gets a pipeline secret by key.",
+ "operationId": "get_secret_api_v2_pipelines__name__secrets__key__get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ },
+ {
+ "name": "key",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline secret key",
+ "title": "Key"
+ },
+ "description": "Pipeline secret key"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineSecretResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ }
+ }
+ },
+ "put": {
+ "tags": ["v2", "pipelines", "secrets"],
+ "summary": "Update Secret",
+ "description": "Updates a pipeline secret, or creates it if it does not exist.",
+ "operationId": "update_secret_api_v2_pipelines__name__secrets__key__put",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ },
+ {
+ "name": "key",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline secret key",
+ "title": "Key"
+ },
+ "description": "Pipeline secret key"
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineSecretUpdateRequest",
+ "description": "Pipeline secret details"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineSecretResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ },
+ "422": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unprocessable Entity"
+ }
+ }
+ },
+ "delete": {
+ "tags": ["v2", "pipelines", "secrets"],
+ "summary": "Delete Secret",
+ "description": "Deletes a pipeline secret by key.",
+ "operationId": "delete_secret_api_v2_pipelines__name__secrets__key__delete",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ },
+ {
+ "name": "key",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline secret key",
+ "title": "Key"
+ },
+ "description": "Pipeline secret key"
+ }
+ ],
+ "responses": {
+ "204": {
+ "description": "Successful Response"
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ }
+ }
+ }
+ },
+ "/api/v2/pipelines/{name}/source-schemas/{source_name}": {
+ "get": {
+ "tags": ["v2", "pipelines"],
+ "summary": "Get Pipeline Source Schemas",
+ "description": "Lists source schemas/databases, tables, and columns for a pipeline source.",
+ "operationId": "get_pipeline_source_schemas_api_v2_pipelines__name__source_schemas__source_name__get",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ },
+ {
+ "name": "source_name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline source name",
+ "title": "Source Name"
+ },
+ "description": "Pipeline source name"
+ },
+ {
+ "name": "schemas",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Comma-separated list of schema/database names to filter on",
+ "title": "Schemas"
+ },
+ "description": "Comma-separated list of schema/database names to filter on"
+ },
+ {
+ "name": "tables",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Comma-separated list of table names to filter on",
+ "title": "Tables"
+ },
+ "description": "Comma-separated list of table names to filter on"
+ },
+ {
+ "name": "details",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "$ref": "#/components/schemas/SourceSchemaDetails",
+ "description": "The level of detail to return: schemas, tables, or columns",
+ "default": "schemas"
+ },
+ "description": "The level of detail to return: schemas, tables, or columns"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/PipelineSourceSchemaResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ },
+ "422": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unprocessable Entity"
+ }
+ }
+ }
+ },
+ "/api/v2/pipelines/{name}/flush-target/{target_name}": {
+ "post": {
+ "tags": ["v2", "pipelines", "targets"],
+ "summary": "Flush Target",
+ "description": "Flushes the target Redis database.\n\nSends a FLUSHALL command to the target Redis database through the collector API.",
+ "operationId": "flush_target_api_v2_pipelines__name__flush_target__target_name__post",
+ "security": [
+ {
+ "JWTBearer": []
+ }
+ ],
+ "parameters": [
+ {
+ "name": "name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Pipeline name",
+ "title": "Name"
+ },
+ "description": "Pipeline name"
+ },
+ {
+ "name": "target_name",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string",
+ "description": "Target name",
+ "title": "Target Name"
+ },
+ "description": "Target name"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "FLUSHALL completed successfully",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/FlushResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Unauthorized"
+ },
+ "403": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Forbidden"
+ },
+ "400": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Bad Request"
+ },
+ "404": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Not Found"
+ },
+ "502": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Bad Gateway"
+ },
+ "503": {
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse"
+ }
+ }
+ },
+ "description": "Service Unavailable"
+ }
+ }
+ }
+ },
+ "/api/v2/info": {
+ "get": {
+ "tags": ["v2", "info"],
+ "summary": "Info",
+ "description": "Gets the API version and the operator leader-election mode.",
+ "operationId": "info_api_v2_info_get",
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/InfoResponse"
+ }
+ }
+ }
+ }
+ },
+ "security": []
+ }
+ }
+ },
+ "components": {
+ "schemas": {
+ "ActionIdResponse": {
+ "properties": {
+ "action_id": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Action Id",
+ "examples": ["1234567890"]
+ }
+ },
+ "type": "object",
+ "title": "ActionIdResponse",
+ "description": "Response model containing an action ID."
+ },
+ "ActionResponse": {
+ "properties": {
+ "action_id": {
+ "type": "string",
+ "title": "Action ID",
+ "examples": ["1715254593439-0"]
+ },
+ "status": {
+ "$ref": "#/components/schemas/TaskStatus",
+ "title": "Action status",
+ "examples": ["completed", "failed"]
+ },
+ "output": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/SourcesOutput"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Action output"
+ },
+ "error": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__sources__Error"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Error object",
+ "examples": [
+ {
+ "message": "Unknown error occurred"
+ }
+ ]
+ }
+ },
+ "type": "object",
+ "required": ["action_id", "status"],
+ "title": "ActionResponse"
+ },
+ "Client": {
+ "properties": {
+ "id": {
+ "type": "string",
+ "title": "Id",
+ "examples": ["975820001001"]
+ },
+ "addr": {
+ "type": "string",
+ "title": "Addr",
+ "examples": ["172.17.0.1:56982"]
+ },
+ "age_sec": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "$ref": "#/components/schemas/NotApplicable"
+ }
+ ],
+ "title": "Age Sec",
+ "examples": [1111, "N/A"]
+ },
+ "idle_sec": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "$ref": "#/components/schemas/NotApplicable"
+ }
+ ],
+ "title": "Idle Sec",
+ "examples": [1111, "N/A"]
+ },
+ "user": {
+ "type": "string",
+ "title": "User",
+ "examples": ["default"]
+ }
+ },
+ "type": "object",
+ "required": ["id", "addr", "age_sec", "idle_sec", "user"],
+ "title": "Client"
+ },
+ "ComponentType": {
+ "type": "string",
+ "enum": [
+ "debezium-collector",
+ "collector-api",
+ "stream-processor",
+ "metrics-exporter",
+ "flink-processor-jobmanager",
+ "flink-processor-taskmanager",
+ "flink-processor",
+ "riotx-collector",
+ "unknown"
+ ],
+ "title": "ComponentType",
+ "description": "Component types for RDI pipeline components."
+ },
+ "ConnectionStatus": {
+ "properties": {
+ "status": {
+ "$ref": "#/components/schemas/Result",
+ "examples": ["success", "failed"]
+ },
+ "error": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__sources__Error"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ }
+ },
+ "type": "object",
+ "required": ["status"],
+ "title": "ConnectionStatus"
+ },
+ "ConnectionStatusSingleConnection": {
+ "properties": {
+ "connected": {
+ "type": "boolean",
+ "title": "Connected"
+ },
+ "error": {
+ "type": "string",
+ "title": "Error"
+ },
+ "invalid_properties": {
+ "anyOf": [
+ {
+ "additionalProperties": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Invalid properties",
+ "description": "Map of invalid property names to their specific error messages",
+ "examples": [
+ {
+ "database.hostname": "Unable to connect: Communications link failure"
+ }
+ ]
+ }
+ },
+ "type": "object",
+ "required": ["connected", "error"],
+ "title": "ConnectionStatusSingleConnection"
+ },
+ "CredentialsInfo": {
+ "properties": {
+ "username": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Username",
+ "description": "RDI database user",
+ "examples": ["admin"]
+ },
+ "password": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Password",
+ "description": "RDI database password",
+ "examples": ["secret-password"]
+ }
+ },
+ "type": "object",
+ "title": "CredentialsInfo",
+ "description": "Credentials information"
+ },
+ "DataStreams": {
+ "properties": {
+ "totals": {
+ "$ref": "#/components/schemas/DataStreamsStatisticsTotal"
+ },
+ "streams": {
+ "additionalProperties": {
+ "$ref": "#/components/schemas/DataStreamsStatistics"
+ },
+ "type": "object",
+ "title": "Streams"
+ }
+ },
+ "type": "object",
+ "required": ["totals", "streams"],
+ "title": "DataStreams"
+ },
+ "DataStreamsStatistics": {
+ "properties": {
+ "total": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Total",
+ "default": 0,
+ "examples": [0]
+ },
+ "pending": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Pending",
+ "default": 0,
+ "examples": [0]
+ },
+ "inserted": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Inserted",
+ "default": 0,
+ "examples": [0]
+ },
+ "updated": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Updated",
+ "default": 0,
+ "examples": [0]
+ },
+ "deleted": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Deleted",
+ "default": 0,
+ "examples": [0]
+ },
+ "filtered": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Filtered",
+ "default": 0,
+ "examples": [0]
+ },
+ "rejected": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Rejected",
+ "default": 0,
+ "examples": [0]
+ },
+ "deduplicated": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Deduplicated",
+ "default": 0,
+ "examples": [0]
+ },
+ "last_arrival": {
+ "type": "string",
+ "title": "Last Arrival"
+ }
+ },
+ "type": "object",
+ "required": ["last_arrival"],
+ "title": "DataStreamsStatistics"
+ },
+ "DataStreamsStatisticsTotal": {
+ "properties": {
+ "total": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Total",
+ "default": 0,
+ "examples": [0]
+ },
+ "pending": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Pending",
+ "default": 0,
+ "examples": [0]
+ },
+ "inserted": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Inserted",
+ "default": 0,
+ "examples": [0]
+ },
+ "updated": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Updated",
+ "default": 0,
+ "examples": [0]
+ },
+ "deleted": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Deleted",
+ "default": 0,
+ "examples": [0]
+ },
+ "filtered": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Filtered",
+ "default": 0,
+ "examples": [0]
+ },
+ "rejected": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Rejected",
+ "default": 0,
+ "examples": [0]
+ },
+ "deduplicated": {
+ "anyOf": [
+ {
+ "type": "integer",
+ "minimum": 0.0
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Deduplicated",
+ "default": 0,
+ "examples": [0]
+ }
+ },
+ "type": "object",
+ "title": "DataStreamsStatisticsTotal"
+ },
+ "DatabaseResponse": {
+ "properties": {
+ "databases": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array",
+ "title": "Databases",
+ "examples": ["database1", "database2"]
+ }
+ },
+ "type": "object",
+ "required": ["databases"],
+ "title": "DatabaseResponse",
+ "description": "Database response class."
+ },
+ "DbFlavor": {
+ "type": "string",
+ "enum": [
+ "mongodb-atlas",
+ "mongodb-replica-set",
+ "mongodb-sharded-cluster"
+ ],
+ "title": "DbFlavor",
+ "description": "Defines supported database flavors."
+ },
+ "DbType": {
+ "type": "string",
+ "enum": [
+ "cassandra",
+ "mariadb",
+ "mongodb",
+ "mysql",
+ "oracle",
+ "postgresql",
+ "snowflake",
+ "sqlserver",
+ "spanner",
+ "redis"
+ ],
+ "title": "DbType",
+ "description": "Defines supported databases."
+ },
+ "DlqListResponse": {
+ "additionalProperties": {
+ "$ref": "#/components/schemas/DlqTableInfo"
+ },
+ "type": "object",
+ "title": "DlqListResponse",
+ "description": "Response model for listing all DLQ tables with their counts.\n\nReturns a dictionary mapping table names to their DLQ info.\nExample: {\"source.schema.table1\": {\"total_count\": 100}, ...}"
+ },
+ "DlqRecordsResponse": {
+ "properties": {
+ "records": {
+ "items": {
+ "additionalProperties": true,
+ "type": "object"
+ },
+ "type": "array",
+ "title": "Records",
+ "description": "List of DLQ records"
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "title": "DlqRecordsResponse",
+ "description": "Response model for DLQ records."
+ },
+ "DlqTableInfo": {
+ "properties": {
+ "total_count": {
+ "type": "integer",
+ "minimum": 0.0,
+ "title": "Total Count",
+ "description": "Total number of records in the DLQ stream",
+ "examples": [1500]
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "required": ["total_count"],
+ "title": "DlqTableInfo",
+ "description": "Model representing DLQ info for a single table."
+ },
+ "DlqTableResponse": {
+ "additionalProperties": {
+ "$ref": "#/components/schemas/DlqTableInfo"
+ },
+ "type": "object",
+ "title": "DlqTableResponse",
+ "description": "Response model for a single DLQ table info.\n\nReturns a dictionary with single table name mapped to its DLQ info.\nExample: {\"source.schema.table\": {\"total_count\": 100}}"
+ },
+ "Entity": {
+ "properties": {
+ "status": {
+ "$ref": "#/components/schemas/RdiStatus",
+ "examples": ["ready"]
+ },
+ "connected": {
+ "anyOf": [
+ {
+ "type": "boolean"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Connected",
+ "examples": [true]
+ },
+ "version": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Version",
+ "examples": ["1.2.3", "1.2.0b17"]
+ },
+ "error": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Error",
+ "examples": ["Error message"]
+ }
+ },
+ "type": "object",
+ "required": ["status"],
+ "title": "Entity"
+ },
+ "ErrorCode": {
+ "type": "string",
+ "enum": [
+ "validation_error",
+ "job_validation_error",
+ "request_validation_error",
+ "cdc_validation_error",
+ "pipeline_error",
+ "pipeline_pending",
+ "pipeline_component_error",
+ "operation_error"
+ ],
+ "title": "ErrorCode",
+ "description": "Error codes"
+ },
+ "ExtendedErrorResponse": {
+ "properties": {
+ "detail": {
+ "type": "string",
+ "title": "Detail",
+ "examples": ["Detailed error message"]
+ },
+ "errors": {
+ "anyOf": [
+ {
+ "items": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__Error"
+ },
+ "type": "array"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Errors"
+ }
+ },
+ "type": "object",
+ "required": ["detail"],
+ "title": "ExtendedErrorResponse",
+ "description": "Model representing an API error response with an additional list of errors."
+ },
+ "FlushResponse": {
+ "properties": {
+ "success": {
+ "type": "boolean",
+ "title": "Success",
+ "description": "Whether the flush operation was successful"
+ },
+ "message": {
+ "type": "string",
+ "title": "Message",
+ "description": "Human-readable result message"
+ }
+ },
+ "type": "object",
+ "required": ["success", "message"],
+ "title": "FlushResponse",
+ "description": "Response model for the target flush operation."
+ },
+ "HTTPValidationError": {
+ "properties": {
+ "detail": {
+ "items": {
+ "$ref": "#/components/schemas/ValidationError"
+ },
+ "type": "array",
+ "title": "Detail"
+ }
+ },
+ "type": "object",
+ "title": "HTTPValidationError"
+ },
+ "InfoResponse": {
+ "properties": {
+ "version": {
+ "type": "string",
+ "title": "Version"
+ },
+ "leader_election_mode": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/LeaderElectionMode"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ }
+ },
+ "type": "object",
+ "required": ["version"],
+ "title": "InfoResponse",
+ "description": "Response model containing information about the API and operator."
+ },
+ "Job": {
+ "properties": {
+ "source": {
+ "$ref": "#/components/schemas/Source"
+ },
+ "transform": {
+ "anyOf": [
+ {
+ "items": {
+ "$ref": "#/components/schemas/Transform"
+ },
+ "type": "array"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Transform"
+ },
+ "output": {
+ "items": {
+ "$ref": "#/components/schemas/Output"
+ },
+ "type": "array",
+ "title": "Output"
+ },
+ "name": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Name",
+ "examples": ["Alex"]
+ }
+ },
+ "type": "object",
+ "required": ["source", "output"],
+ "title": "Job"
+ },
+ "JobDryRunBody": {
+ "properties": {
+ "job": {
+ "$ref": "#/components/schemas/Job",
+ "title": "Job settings to dry run with."
+ },
+ "input_data": {
+ "items": {
+ "additionalProperties": true,
+ "type": "object"
+ },
+ "type": "array",
+ "title": "Input Data",
+ "examples": [
+ [
+ {
+ "COUNTRY": "IL",
+ "FNAME": "John",
+ "LAST_NAME": "Lennon"
+ },
+ {
+ "COUNTRY": "US",
+ "FNAME": "John",
+ "LAST_NAME": "Doe"
+ }
+ ]
+ ]
+ }
+ },
+ "type": "object",
+ "required": ["job", "input_data"],
+ "title": "JobDryRunBody"
+ },
+ "JobDryRunSuccessResponse": {
+ "properties": {
+ "transformation": {
+ "items": {
+ "additionalProperties": true,
+ "type": "object"
+ },
+ "type": "array",
+ "title": "Transformation",
+ "examples": [
+ [
+ {
+ "COUNTRY": "IL",
+ "FIRST_NAME": "John",
+ "LAST_NAME": "Lennon"
+ }
+ ]
+ ]
+ },
+ "output": {
+ "items": {},
+ "type": "array",
+ "title": "Output"
+ }
+ },
+ "type": "object",
+ "required": ["transformation", "output"],
+ "title": "JobDryRunSuccessResponse"
+ },
+ "LeaderElectionMode": {
+ "type": "string",
+ "enum": ["disabled", "leader", "follower"],
+ "title": "LeaderElectionMode",
+ "description": "Operator leader-election mode."
+ },
+ "MetadataResponse": {
+ "properties": {
+ "tables": {
+ "additionalProperties": {
+ "$ref": "#/components/schemas/Table"
+ },
+ "type": "object",
+ "title": "Tables",
+ "examples": [
+ {
+ "table1": {
+ "cdc_ready": true,
+ "columns": {}
+ }
+ }
+ ]
+ }
+ },
+ "type": "object",
+ "required": ["tables"],
+ "title": "MetadataResponse",
+ "description": "Metadata response class."
+ },
+ "NotApplicable": {
+ "type": "string",
+ "title": "NotApplicable",
+ "default": "N/A"
+ },
+ "OperationResponse": {
+ "properties": {
+ "status": {
+ "type": "boolean",
+ "title": "Status"
+ }
+ },
+ "type": "object",
+ "required": ["status"],
+ "title": "OperationResponse",
+ "description": "Represents the response of an operation."
+ },
+ "Output": {
+ "properties": {
+ "uses": {
+ "type": "string",
+ "title": "Uses",
+ "examples": ["redis.write"]
+ },
+ "with": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/WithInput"
+ },
+ {
+ "$ref": "#/components/schemas/WithFields"
+ },
+ {
+ "$ref": "#/components/schemas/WithNest"
+ },
+ {
+ "$ref": "#/components/schemas/WithConnection"
+ }
+ ],
+ "title": "With"
+ }
+ },
+ "type": "object",
+ "required": ["uses", "with"],
+ "title": "Output"
+ },
+ "Pipeline": {
+ "properties": {
+ "status": {
+ "$ref": "#/components/schemas/RdiStatus",
+ "examples": ["ready"]
+ },
+ "state": {
+ "$ref": "#/components/schemas/State",
+ "examples": ["cdc"]
+ },
+ "last_error": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Last Error",
+ "examples": ["Error message"]
+ },
+ "tasks": {
+ "items": {
+ "$ref": "#/components/schemas/Task"
+ },
+ "type": "array",
+ "title": "Tasks"
+ }
+ },
+ "type": "object",
+ "required": ["status", "state", "tasks"],
+ "title": "Pipeline"
+ },
+ "PipelineComponentResponse": {
+ "properties": {
+ "name": {
+ "type": "string",
+ "title": "Name"
+ },
+ "type": {
+ "$ref": "#/components/schemas/ComponentType"
+ },
+ "version": {
+ "type": "string",
+ "title": "Version"
+ },
+ "status": {
+ "$ref": "#/components/schemas/Status",
+ "default": "unknown"
+ },
+ "replicas": {
+ "type": "integer",
+ "title": "Replicas",
+ "default": 0
+ },
+ "errors": {
+ "anyOf": [
+ {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Errors"
+ },
+ "metric_collections": {
+ "anyOf": [
+ {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Metric Collections"
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "required": ["name", "type", "version"],
+ "title": "PipelineComponentResponse",
+ "description": "Response model containing the details of a pipeline component."
+ },
+ "PipelineCreateRequest": {
+ "properties": {
+ "name": {
+ "type": "string",
+ "title": "Name",
+ "default": "default"
+ },
+ "active": {
+ "type": "boolean",
+ "title": "Active",
+ "default": true
+ },
+ "config": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Config"
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "title": "PipelineCreateRequest",
+ "description": "Request model for creating a new pipeline."
+ },
+ "PipelineFunction": {
+ "properties": {
+ "summary": {
+ "type": "string",
+ "title": "Summary",
+ "examples": ["Decodes a base64(RFC 4648) encoded string"]
+ },
+ "arguments": {
+ "items": {
+ "$ref": "#/components/schemas/PipelineFunctionArgument"
+ },
+ "type": "array",
+ "title": "Arguments"
+ }
+ },
+ "type": "object",
+ "required": ["summary", "arguments"],
+ "title": "PipelineFunction"
+ },
+ "PipelineFunctionArgument": {
+ "properties": {
+ "name": {
+ "type": "string",
+ "title": "Name"
+ },
+ "type": {
+ "type": "string",
+ "enum": [
+ "number",
+ "string",
+ "boolean",
+ "object",
+ "array",
+ "null",
+ "any"
+ ],
+ "title": "Type",
+ "default": "string",
+ "examples": ["string"]
+ },
+ "display_text": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Display Text"
+ },
+ "optional": {
+ "type": "boolean",
+ "title": "Optional",
+ "default": false
+ }
+ },
+ "type": "object",
+ "required": ["name"],
+ "title": "PipelineFunctionArgument"
+ },
+ "PipelineMetricCollectionResponse": {
+ "properties": {
+ "name": {
+ "type": "string",
+ "title": "Name"
+ },
+ "component": {
+ "type": "string",
+ "title": "Component"
+ },
+ "metrics": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Metrics"
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "required": ["name", "component", "metrics"],
+ "title": "PipelineMetricCollectionResponse",
+ "description": "Response model containing the details of a metric collection."
+ },
+ "PipelinePatchRequest": {
+ "properties": {
+ "active": {
+ "anyOf": [
+ {
+ "type": "boolean"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Active"
+ },
+ "config": {
+ "anyOf": [
+ {
+ "additionalProperties": true,
+ "type": "object"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Config"
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "title": "PipelinePatchRequest",
+ "description": "Request model for updating a pipeline partially."
+ },
+ "PipelineResponse": {
+ "properties": {
+ "name": {
+ "type": "string",
+ "title": "Name"
+ },
+ "active": {
+ "type": "boolean",
+ "title": "Active"
+ },
+ "config": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Config"
+ },
+ "status": {
+ "$ref": "#/components/schemas/Status",
+ "default": "unknown"
+ },
+ "status_changed_at": {
+ "anyOf": [
+ {
+ "type": "string",
+ "format": "date-time"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Status Changed At"
+ },
+ "errors": {
+ "items": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__Error"
+ },
+ "type": "array",
+ "title": "Errors"
+ },
+ "components": {
+ "items": {
+ "$ref": "#/components/schemas/PipelineComponentResponse"
+ },
+ "type": "array",
+ "title": "Components"
+ },
+ "current": {
+ "type": "boolean",
+ "title": "Current",
+ "default": false
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "required": ["name", "active", "config"],
+ "title": "PipelineResponse",
+ "description": "Response model containing the details of a pipeline."
+ },
+ "PipelineSecretCreateRequest": {
+ "properties": {
+ "key": {
+ "type": "string",
+ "title": "Key"
+ },
+ "value": {
+ "type": "string",
+ "title": "Value"
+ },
+ "type": {
+ "type": "string",
+ "enum": ["simple", "file", "binary_file"],
+ "title": "Type",
+ "default": "simple"
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "required": ["key", "value"],
+ "title": "PipelineSecretCreateRequest",
+ "description": "Request model for creating a new pipeline secret."
+ },
+ "PipelineSecretResponse": {
+ "properties": {
+ "key": {
+ "type": "string",
+ "title": "Key"
+ },
+ "type": {
+ "type": "string",
+ "enum": ["simple", "file", "binary_file"],
+ "title": "Type"
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "required": ["key", "type"],
+ "title": "PipelineSecretResponse",
+ "description": "Response model for a pipeline secret."
+ },
+ "PipelineSecretUpdateRequest": {
+ "properties": {
+ "value": {
+ "type": "string",
+ "title": "Value"
+ },
+ "type": {
+ "type": "string",
+ "enum": ["simple", "file", "binary_file"],
+ "title": "Type",
+ "default": "simple"
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "required": ["value"],
+ "title": "PipelineSecretUpdateRequest",
+ "description": "Request model for updating an existing pipeline secret."
+ },
+ "PipelineSourceSchemaResponse": {
+ "properties": {
+ "schemas": {
+ "additionalProperties": {
+ "$ref": "#/components/schemas/SourceSchemaResponse"
+ },
+ "type": "object",
+ "title": "Schemas",
+ "examples": [
+ {
+ "inventory": {
+ "tables": {
+ "addresses": {
+ "cdc_ready": true,
+ "columns": {
+ "zip": {
+ "primary_key": false,
+ "size": 255,
+ "type": "varchar",
+ "unique_constraint": false
+ }
+ }
+ }
+ }
+ }
+ }
+ ]
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "title": "PipelineSourceSchemaResponse",
+ "description": "Nested source metadata response for a pipeline source."
+ },
+ "PipelineStatusResponse": {
+ "properties": {
+ "status": {
+ "$ref": "#/components/schemas/Status",
+ "default": "unknown"
+ },
+ "status_changed_at": {
+ "anyOf": [
+ {
+ "type": "string",
+ "format": "date-time"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Status Changed At"
+ },
+ "errors": {
+ "items": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__Error"
+ },
+ "type": "array",
+ "title": "Errors"
+ },
+ "components": {
+ "items": {
+ "$ref": "#/components/schemas/PipelineComponentResponse"
+ },
+ "type": "array",
+ "title": "Components"
+ },
+ "current": {
+ "type": "boolean",
+ "title": "Current",
+ "default": false
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "title": "PipelineStatusResponse",
+ "description": "Response model containing the details of a pipeline status."
+ },
+ "PipelineUpdateRequest": {
+ "properties": {
+ "active": {
+ "type": "boolean",
+ "title": "Active",
+ "default": true
+ },
+ "config": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Config"
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "title": "PipelineUpdateRequest",
+ "description": "Request model for updating a pipeline completely."
+ },
+ "PipelinesFunctionsResponse": {
+ "additionalProperties": {
+ "$ref": "#/components/schemas/PipelineFunction"
+ },
+ "type": "object",
+ "title": "PipelinesFunctionsResponse"
+ },
+ "PipelinesStrategiesResponse": {
+ "properties": {
+ "strategies": {
+ "items": {
+ "$ref": "#/components/schemas/StrategyModel"
+ },
+ "type": "array",
+ "title": "Strategies"
+ }
+ },
+ "type": "object",
+ "required": ["strategies"],
+ "title": "PipelinesStrategiesResponse"
+ },
+ "ProcessingPerformance": {
+ "properties": {
+ "total_batches": {
+ "type": "integer",
+ "minimum": 0.0,
+ "title": "Total Batches",
+ "examples": [100]
+ },
+ "batch_size_avg": {
+ "type": "number",
+ "minimum": 0.0,
+ "title": "Batch Size Avg",
+ "examples": [50.2]
+ },
+ "read_time_avg": {
+ "type": "number",
+ "minimum": 0.0,
+ "title": "Read Time Avg",
+ "examples": [10.5]
+ },
+ "transform_time_avg": {
+ "type": "number",
+ "minimum": 0.0,
+ "title": "Transform Time Avg",
+ "examples": [2.3]
+ },
+ "write_time_avg": {
+ "type": "number",
+ "minimum": 0.0,
+ "title": "Write Time Avg",
+ "examples": [4.4]
+ },
+ "process_time_avg": {
+ "type": "number",
+ "minimum": 0.0,
+ "title": "Process Time Avg",
+ "examples": [20.3]
+ },
+ "ack_time_avg": {
+ "type": "number",
+ "minimum": 0.0,
+ "title": "Ack Time Avg",
+ "examples": [5.2]
+ },
+ "total_time_avg": {
+ "type": "number",
+ "minimum": 0.0,
+ "title": "Total Time Avg",
+ "examples": [35.5]
+ },
+ "rec_per_sec_avg": {
+ "type": "number",
+ "minimum": 0.0,
+ "title": "Rec Per Sec Avg",
+ "examples": [100.1]
+ }
+ },
+ "type": "object",
+ "required": [
+ "total_batches",
+ "batch_size_avg",
+ "read_time_avg",
+ "transform_time_avg",
+ "write_time_avg",
+ "process_time_avg",
+ "ack_time_avg",
+ "total_time_avg",
+ "rec_per_sec_avg"
+ ],
+ "title": "ProcessingPerformance"
+ },
+ "ProcessorProperty": {
+ "properties": {
+ "value": {
+ "title": "Value",
+ "examples": ["hash"]
+ }
+ },
+ "type": "object",
+ "required": ["value"],
+ "title": "ProcessorProperty"
+ },
+ "ProcessorPropertyName": {
+ "type": "string",
+ "enum": [
+ "on_failed_retry_interval",
+ "read_batch_size",
+ "dedup",
+ "dedup_max_size",
+ "dedup_strategy",
+ "duration",
+ "write_batch_size",
+ "error_handling",
+ "dlq_max_messages",
+ "target_data_type",
+ "json_update_strategy",
+ "initial_sync_processes",
+ "idle_sleep_time_ms",
+ "idle_streams_check_interval_ms",
+ "busy_streams_check_interval_ms",
+ "wait_enabled",
+ "wait_timeout",
+ "retry_on_replica_failure"
+ ],
+ "title": "ProcessorPropertyName"
+ },
+ "ProcessorState": {
+ "type": "string",
+ "enum": ["unknown", "running", "idling", "stopped"],
+ "title": "ProcessorState",
+ "description": "Enum for the state of the stream processor."
+ },
+ "RdiPipelineStatus": {
+ "properties": {
+ "rdi_version": {
+ "type": "string",
+ "title": "Rdi Version",
+ "examples": ["N/A"]
+ },
+ "address": {
+ "type": "string",
+ "title": "Address",
+ "examples": ["172.17.0.2:12006"]
+ },
+ "run_status": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ProcessorState"
+ },
+ {
+ "$ref": "#/components/schemas/RdiStatus"
+ }
+ ],
+ "title": "Run Status",
+ "examples": ["stopped"]
+ },
+ "sync_mode": {
+ "type": "string",
+ "title": "Sync Mode",
+ "examples": ["N/A"]
+ }
+ },
+ "type": "object",
+ "required": ["rdi_version", "address", "run_status", "sync_mode"],
+ "title": "RdiPipelineStatus"
+ },
+ "RdiStatus": {
+ "type": "string",
+ "enum": ["stopped", "ready", "not-ready"],
+ "title": "RdiStatus",
+ "description": "RDI component status."
+ },
+ "Result": {
+ "type": "string",
+ "enum": ["success", "failed"],
+ "title": "Result",
+ "description": "Represents the result of an operation."
+ },
+ "SchemaResponse": {
+ "properties": {
+ "schemas": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array",
+ "title": "Schemas",
+ "examples": ["schema1", "schema2"]
+ }
+ },
+ "type": "object",
+ "required": ["schemas"],
+ "title": "SchemaResponse",
+ "description": "Schema response class."
+ },
+ "SecretKeyModel": {
+ "additionalProperties": {
+ "type": "string"
+ },
+ "type": "object",
+ "title": "SecretKeyModel",
+ "examples": [
+ {
+ "TARGET_DB_PASSWORD": "password",
+ "TARGET_DB_USERNAME": "admin"
+ }
+ ]
+ },
+ "SecretName": {
+ "type": "string",
+ "enum": [
+ "source-db",
+ "target-db",
+ "rdi-sys-config",
+ "rdi-db-ssl",
+ "source-db-ssl",
+ "target-db-ssl"
+ ],
+ "title": "SecretName",
+ "description": "Defines supported secret names."
+ },
+ "SecretsModel": {
+ "additionalProperties": {
+ "$ref": "#/components/schemas/SecretKeyModel"
+ },
+ "propertyNames": {
+ "$ref": "#/components/schemas/SecretName"
+ },
+ "type": "object",
+ "title": "SecretsModel",
+ "examples": [
+ {
+ "target-db": {
+ "TARGET_DB_PASSWORD": "password",
+ "TARGET_DB_USERNAME": "admin"
+ }
+ }
+ ]
+ },
+ "SnapshotStatus": {
+ "type": "string",
+ "enum": ["running", "failed", "paused", "completed", "N/A"],
+ "title": "SnapshotStatus"
+ },
+ "SortOrder": {
+ "type": "string",
+ "enum": ["asc", "desc"],
+ "title": "SortOrder",
+ "description": "Sort order for DLQ records."
+ },
+ "Source": {
+ "properties": {
+ "schema": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Schema",
+ "examples": ["public"]
+ },
+ "table": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Table",
+ "examples": ["employee"]
+ },
+ "row_format": {
+ "anyOf": [
+ {
+ "type": "string",
+ "enum": ["full", "partial"]
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Row Format",
+ "examples": ["full"]
+ }
+ },
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Source"
+ },
+ "SourceColumnResponse": {
+ "properties": {
+ "type": {
+ "type": "string",
+ "title": "Type",
+ "examples": ["varchar"]
+ },
+ "size": {
+ "anyOf": [
+ {
+ "type": "integer"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Size",
+ "examples": [255]
+ },
+ "primary_key": {
+ "type": "boolean",
+ "title": "Primary Key",
+ "default": false,
+ "examples": [false]
+ },
+ "unique_constraint": {
+ "type": "boolean",
+ "title": "Unique Constraint",
+ "default": false,
+ "examples": [false]
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "required": ["type"],
+ "title": "SourceColumnResponse",
+ "description": "Metadata for a single source column."
+ },
+ "SourceSchemaDetails": {
+ "type": "string",
+ "enum": ["schemas", "tables", "columns"],
+ "title": "SourceSchemaDetails",
+ "description": "Detail level for source metadata responses."
+ },
+ "SourceSchemaResponse": {
+ "properties": {
+ "tables": {
+ "anyOf": [
+ {
+ "additionalProperties": {
+ "$ref": "#/components/schemas/SourceTableResponse"
+ },
+ "type": "object"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Tables",
+ "examples": [
+ {
+ "users": {
+ "cdc_ready": true,
+ "columns": {
+ "id": {
+ "primary_key": true,
+ "size": 32,
+ "type": "int4",
+ "unique_constraint": true
+ }
+ }
+ }
+ }
+ ]
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "title": "SourceSchemaResponse",
+ "description": "Metadata for a single source schema or database."
+ },
+ "SourceTableResponse": {
+ "properties": {
+ "cdc_ready": {
+ "anyOf": [
+ {
+ "type": "boolean"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Cdc Ready",
+ "examples": [true]
+ },
+ "columns": {
+ "anyOf": [
+ {
+ "additionalProperties": {
+ "$ref": "#/components/schemas/SourceColumnResponse"
+ },
+ "type": "object"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Columns",
+ "examples": [
+ {
+ "id": {
+ "primary_key": true,
+ "size": 32,
+ "type": "int4",
+ "unique_constraint": true
+ }
+ }
+ ]
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "title": "SourceTableResponse",
+ "description": "Metadata for a single source table."
+ },
+ "SourcesOutput": {
+ "properties": {
+ "sources": {
+ "additionalProperties": {
+ "$ref": "#/components/schemas/ConnectionStatus"
+ },
+ "type": "object",
+ "title": "Sources"
+ }
+ },
+ "type": "object",
+ "required": ["sources"],
+ "title": "SourcesOutput"
+ },
+ "State": {
+ "type": "string",
+ "enum": ["cdc", "initial-sync", "not-running"],
+ "title": "State",
+ "description": "Represents the state of a pipeline."
+ },
+ "StatisticsConnection": {
+ "properties": {
+ "key": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Key"
+ },
+ "cert": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Cert"
+ },
+ "cacert": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Cacert"
+ },
+ "key_password": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Key Password"
+ },
+ "type": {
+ "$ref": "#/components/schemas/DbType",
+ "default": "redis",
+ "examples": ["redis"]
+ },
+ "host": {
+ "type": "string",
+ "title": "Host",
+ "default": "localhost",
+ "examples": ["localhost"]
+ },
+ "port": {
+ "type": "integer",
+ "exclusiveMinimum": 0.0,
+ "title": "Port",
+ "description": "Positive int or \"${ENV_VAR_NAME}\" placeholder",
+ "default": 5432,
+ "examples": [5432, "${DB_PORT}"]
+ },
+ "database": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Database",
+ "examples": ["rdi"]
+ },
+ "user": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "User",
+ "examples": ["rdi"]
+ },
+ "password": {
+ "type": "string",
+ "title": "Password",
+ "examples": ["********"]
+ },
+ "status": {
+ "type": "string",
+ "title": "Status"
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "required": ["password", "status"],
+ "title": "StatisticsConnection"
+ },
+ "StatisticsResponse": {
+ "properties": {
+ "connections": {
+ "additionalProperties": {
+ "$ref": "#/components/schemas/StatisticsConnection"
+ },
+ "type": "object",
+ "title": "Connections"
+ },
+ "data_streams": {
+ "$ref": "#/components/schemas/DataStreams"
+ },
+ "processing_performance": {
+ "$ref": "#/components/schemas/ProcessingPerformance"
+ },
+ "rdi_pipeline_status": {
+ "$ref": "#/components/schemas/RdiPipelineStatus"
+ },
+ "clients": {
+ "additionalProperties": {
+ "$ref": "#/components/schemas/Client"
+ },
+ "type": "object",
+ "title": "Clients"
+ },
+ "offsets": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Offsets"
+ },
+ "snapshot_status": {
+ "$ref": "#/components/schemas/SnapshotStatus",
+ "examples": ["running"]
+ }
+ },
+ "type": "object",
+ "required": [
+ "connections",
+ "data_streams",
+ "processing_performance",
+ "rdi_pipeline_status",
+ "clients",
+ "offsets",
+ "snapshot_status"
+ ],
+ "title": "StatisticsResponse"
+ },
+ "Status": {
+ "type": "string",
+ "enum": [
+ "started",
+ "stopped",
+ "error",
+ "creating",
+ "updating",
+ "deleting",
+ "starting",
+ "stopping",
+ "resetting",
+ "pending",
+ "unknown"
+ ],
+ "title": "Status",
+ "description": "Pipeline status."
+ },
+ "StatusResponse": {
+ "properties": {
+ "components": {
+ "additionalProperties": {
+ "$ref": "#/components/schemas/Entity"
+ },
+ "type": "object",
+ "title": "Components"
+ },
+ "pipelines": {
+ "additionalProperties": {
+ "$ref": "#/components/schemas/Pipeline"
+ },
+ "type": "object",
+ "title": "Pipelines"
+ }
+ },
+ "type": "object",
+ "required": ["components", "pipelines"],
+ "title": "StatusResponse"
+ },
+ "Strategy": {
+ "type": "string",
+ "enum": ["ingest"],
+ "title": "Strategy",
+ "description": "Defines available strategies."
+ },
+ "StrategyModel": {
+ "properties": {
+ "strategy": {
+ "$ref": "#/components/schemas/Strategy",
+ "examples": ["ingest"]
+ },
+ "databases": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array",
+ "title": "Databases",
+ "examples": ["mongodb", "mysql", "oracle", "sqlserver"]
+ }
+ },
+ "type": "object",
+ "required": ["strategy", "databases"],
+ "title": "StrategyModel"
+ },
+ "Table": {
+ "properties": {
+ "cdc_ready": {
+ "type": "boolean",
+ "title": "Cdc Ready",
+ "examples": [true]
+ },
+ "columns": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Columns"
+ }
+ },
+ "type": "object",
+ "required": ["cdc_ready", "columns"],
+ "title": "Table"
+ },
+ "TableResponse": {
+ "properties": {
+ "tables": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array",
+ "title": "Tables",
+ "examples": ["table1", "table2"]
+ }
+ },
+ "type": "object",
+ "required": ["tables"],
+ "title": "TableResponse",
+ "description": "Table response class."
+ },
+ "TargetsOutput": {
+ "properties": {
+ "targets": {
+ "additionalProperties": {
+ "$ref": "#/components/schemas/ConnectionStatus"
+ },
+ "type": "object",
+ "title": "Targets"
+ }
+ },
+ "type": "object",
+ "required": ["targets"],
+ "title": "TargetsOutput"
+ },
+ "Task": {
+ "properties": {
+ "name": {
+ "type": "string",
+ "title": "Name"
+ },
+ "status": {
+ "$ref": "#/components/schemas/TaskStatus"
+ },
+ "error": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Error",
+ "examples": ["Error message"]
+ },
+ "created_at": {
+ "type": "string",
+ "pattern": "^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}$",
+ "title": "Created At",
+ "examples": ["2024-03-03T10:23:12"]
+ }
+ },
+ "type": "object",
+ "required": ["name", "status", "created_at"],
+ "title": "Task"
+ },
+ "TaskStatus": {
+ "type": "string",
+ "enum": ["pending", "processing", "failed", "completed"],
+ "title": "TaskStatus",
+ "description": "Defines task statuses for Operator"
+ },
+ "TemplateResponse": {
+ "properties": {
+ "template": {
+ "type": "string",
+ "title": "Template",
+ "examples": ["template1"]
+ }
+ },
+ "type": "object",
+ "required": ["template"],
+ "title": "TemplateResponse"
+ },
+ "TokenInfo": {
+ "properties": {
+ "access_token": {
+ "type": "string",
+ "title": "Access Token"
+ },
+ "token_type": {
+ "type": "string",
+ "title": "Token Type"
+ }
+ },
+ "type": "object",
+ "required": ["access_token", "token_type"],
+ "title": "TokenInfo",
+ "description": "Token information"
+ },
+ "TraceRequest": {
+ "properties": {
+ "max_change_records": {
+ "type": "integer",
+ "title": "The maximum number of traced change records.",
+ "default": 10,
+ "examples": [10]
+ },
+ "rejected_only": {
+ "type": "boolean",
+ "title": "Indicating whether to trace only rejected change records.",
+ "default": false,
+ "examples": [false]
+ },
+ "timeout": {
+ "type": "integer",
+ "title": "The maximum duration of the trace in seconds.",
+ "default": 20,
+ "examples": [20]
+ }
+ },
+ "type": "object",
+ "title": "TraceRequest",
+ "description": "Request model for the trace endpoint."
+ },
+ "Transform": {
+ "properties": {
+ "uses": {
+ "type": "string",
+ "title": "Uses",
+ "examples": ["add_field"]
+ },
+ "with": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/WithInput"
+ },
+ {
+ "$ref": "#/components/schemas/WithFields"
+ },
+ {
+ "$ref": "#/components/schemas/WithNest"
+ },
+ {
+ "$ref": "#/components/schemas/WithConnection"
+ }
+ ],
+ "title": "With"
+ }
+ },
+ "type": "object",
+ "required": ["uses", "with"],
+ "title": "Transform"
+ },
+ "ValidationError": {
+ "properties": {
+ "loc": {
+ "items": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "integer"
+ }
+ ]
+ },
+ "type": "array",
+ "title": "Location"
+ },
+ "msg": {
+ "type": "string",
+ "title": "Message"
+ },
+ "type": {
+ "type": "string",
+ "title": "Error Type"
+ }
+ },
+ "type": "object",
+ "required": ["loc", "msg", "type"],
+ "title": "ValidationError"
+ },
+ "WithConnection": {
+ "properties": {
+ "connection": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Connection",
+ "examples": ["target"]
+ },
+ "data_type": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Data Type",
+ "examples": ["hash"]
+ },
+ "args": {
+ "anyOf": [
+ {
+ "additionalProperties": true,
+ "type": "object"
+ },
+ {
+ "items": {},
+ "type": "array"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Args",
+ "examples": [
+ {
+ "member": "value"
+ },
+ ["expr1"]
+ ]
+ },
+ "key": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/WithInput"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "cmd": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Cmd",
+ "description": "Redis command to execute, e.g. HGETALL.",
+ "examples": ["HGETALL"]
+ },
+ "language": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Language",
+ "examples": ["jmespath", "sql"]
+ },
+ "field": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Field",
+ "description": "Target field name for the lookup result."
+ }
+ },
+ "type": "object",
+ "title": "WithConnection"
+ },
+ "WithFieldRef": {
+ "properties": {
+ "field": {
+ "type": "string",
+ "title": "Field",
+ "description": "Field name to reference or remove.",
+ "examples": ["branch"]
+ }
+ },
+ "type": "object",
+ "required": ["field"],
+ "title": "WithFieldRef"
+ },
+ "WithFields": {
+ "properties": {
+ "fields": {
+ "items": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/WithInput"
+ },
+ {
+ "$ref": "#/components/schemas/WithFieldRef"
+ }
+ ]
+ },
+ "type": "array",
+ "title": "Fields"
+ }
+ },
+ "type": "object",
+ "required": ["fields"],
+ "title": "WithFields"
+ },
+ "WithInput": {
+ "properties": {
+ "expression": {
+ "type": "string",
+ "title": "Expression",
+ "examples": [
+ "FNAME || ' ' || LAST_NAME",
+ "concat([fname, ' ' , last_name])"
+ ]
+ },
+ "field": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Field",
+ "description": "This field it available only for a subset of with expressions e.g. `add_field`, `remove_field`.",
+ "examples": ["FullName"]
+ },
+ "language": {
+ "type": "string",
+ "title": "Language",
+ "examples": ["sql", "jmespath"]
+ }
+ },
+ "type": "object",
+ "required": ["expression", "language"],
+ "title": "WithInput"
+ },
+ "WithNest": {
+ "properties": {
+ "nest": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Nest",
+ "description": "Nesting configuration for hierarchical data structures."
+ },
+ "on_update": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "On Update",
+ "examples": ["merge", "replace"]
+ },
+ "data_type": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Data Type",
+ "examples": ["json", "hash"]
+ },
+ "connection": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Connection",
+ "examples": ["target"]
+ }
+ },
+ "type": "object",
+ "required": ["nest"],
+ "title": "WithNest"
+ },
+ "redis_di_api__models__errors__Error": {
+ "properties": {
+ "code": {
+ "$ref": "#/components/schemas/ErrorCode"
+ },
+ "message": {
+ "type": "string",
+ "title": "Message"
+ },
+ "remediation": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Remediation"
+ },
+ "details": {
+ "additionalProperties": true,
+ "type": "object",
+ "title": "Details"
+ }
+ },
+ "type": "object",
+ "required": ["code", "message"],
+ "title": "Error",
+ "description": "Generic error model"
+ },
+ "redis_di_api__models__errors__ErrorResponse": {
+ "properties": {
+ "detail": {
+ "type": "string",
+ "title": "Detail",
+ "examples": ["Detailed error message"]
+ },
+ "errors": {
+ "anyOf": [
+ {
+ "items": {
+ "$ref": "#/components/schemas/redis_di_api__models__errors__Error"
+ },
+ "type": "array"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Errors"
+ }
+ },
+ "type": "object",
+ "required": ["detail"],
+ "title": "ErrorResponse",
+ "description": "Generic error response model"
+ },
+ "redis_di_api__v1__modules__shared__response__Error": {
+ "properties": {
+ "status": {
+ "type": "string",
+ "const": "error",
+ "title": "The status of the run.",
+ "default": "error",
+ "examples": ["error"]
+ },
+ "error": {
+ "type": "string",
+ "title": "The error message.",
+ "examples": ["Job is malformed"]
+ }
+ },
+ "type": "object",
+ "required": ["error"],
+ "title": "Error",
+ "description": "Model representing an error status."
+ },
+ "redis_di_api__v1__modules__shared__response__ErrorResponse": {
+ "properties": {
+ "detail": {
+ "type": "string",
+ "title": "Detail",
+ "examples": ["Detailed error message"]
+ }
+ },
+ "type": "object",
+ "required": ["detail"],
+ "title": "ErrorResponse",
+ "description": "Model representing an API error response."
+ },
+ "redis_di_api__v1__modules__shared__sources__Error": {
+ "properties": {
+ "message": {
+ "type": "string",
+ "title": "Error message",
+ "examples": [
+ "Failed to establish connection to the PostgreSQL database. Invalid credentials"
+ ]
+ }
+ },
+ "type": "object",
+ "required": ["message"],
+ "title": "Error"
+ }
+ },
+ "securitySchemes": {
+ "JWTBearer": {
+ "type": "http",
+ "scheme": "bearer"
+ },
+ "HTTPBearer": {
+ "type": "http",
+ "scheme": "bearer",
+ "bearerFormat": "JWT"
+ }
+ }
+ },
+ "servers": [
+ {
+ "url": "/",
+ "description": "Default server"
+ }
+ ]
+}
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/_index.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/_index.md
new file mode 100644
index 0000000000..76ffee3928
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/_index.md
@@ -0,0 +1,127 @@
+---
+Title: CLI reference
+alwaysopen: false
+categories:
+ - docs
+ - integrate
+ - rs
+ - rdi
+description: Reference for the RDI CLI commands
+group: di
+hideListLinks: false
+linkTitle: CLI commands
+summary:
+ Redis Data Integration keeps Redis in sync with the primary database in near
+ real time.
+type: integration
+weight: 60
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/'
+---
+
+`redis-di` is the command line tool that manages Redis Data Integration (RDI).
+It is a thin client over the RDI REST API, so it works the same way for all
+installation types: [VM]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-vm" >}}),
+[Kubernetes]({{< relref "/integrate/redis-data-integration/1.19.1/installation/install-k8s" >}}), and Redis Cloud.
+Use it to deploy pipelines, manage secrets, inspect status and metrics, and read rejected records.
+
+{{< note >}}RDI 1.19.0 introduced the current API-based CLI. If you are moving from an earlier RDI version,
+see [Compare the previous and current RDI CLI]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/previous-cli-comparison" >}})
+for the connection, context, command, and output changes.
+{{< /note >}}
+
+## Connecting to the API
+
+Most commands connect to the RDI API, which you specify with the `--api-url` option (or the
+`RDI_API_URL` environment variable). Because the API is served over HTTPS, you can also supply
+`--cacert` to trust a private or self-signed certificate, or `--insecure` to skip TLS verification.
+
+The CLI supports three authentication modes, selected by the credentials you provide:
+
+- **User authentication** (JWT): when you set a `--user`, the CLI logs in with that user and a
+ password from `--password`, the `RDI_PASSWORD` environment variable, or an interactive prompt.
+ This is the usual mode for VM and Kubernetes installations.
+- **Redis Cloud authentication**: when you set an `--account-key`, the CLI authenticates to the
+ Redis Cloud API gateway with that account key and a user key from `--user-key`, the `RDI_USER_KEY`
+ environment variable, or an interactive prompt. This is the mode for RDI running in Redis Cloud.
+- **No authentication**: when you set neither a user nor an account key, the CLI connects without
+ authenticating, which is the mode to use when authentication is disabled in the API.
+
+Setting both `--user` and `--account-key` is an error, as is setting both `--cacert` and `--insecure`.
+Passwords and user keys are secrets and are never stored on disk.
+
+## Contexts
+
+Instead of passing the connection options on every command, you can save them in a _context_.
+Contexts are stored in a `~/.redis-di` file that holds a map of named contexts and the active one,
+similar to a `kubeconfig` file. Each context sets an `api-url`, an optional `user` or `account-key`,
+and either a `cacert` or `insecure: true`. Secrets (the password and user key) are never stored, so
+you still supply them per session.
+
+```yaml
+# ~/.redis-di
+current-context: prod
+contexts:
+ prod:
+ api-url: https://rdi.example.com
+ user: default
+ cacert: /etc/rdi/ingress-ca.crt
+ dev:
+ api-url: https://localhost:8443
+ insecure: true
+```
+
+Use the [`set-context`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-context" >}})
+and [`use-context`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-use-context" >}})
+commands to create and select contexts rather than editing the file by hand.
+
+## Commands
+
+Pipeline-scoped commands take the pipeline name as an optional positional argument that defaults to
+`default`, for example `redis-di start [pipeline]`. Sub-resource commands (for a secret, DLQ, or job)
+take their own key or name as the positional argument and target the pipeline with the `-p` / `--pipeline`
+option, which also defaults to `default`.
+
+The commands group as follows:
+
+- **Information**: [`info`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-info" >}}).
+- **Pipelines**: [`list`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list" >}}),
+ [`get`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get" >}}),
+ [`describe`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe" >}}) (alias `status`),
+ [`deploy`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-deploy" >}}) (alias `set`),
+ [`delete`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete" >}}),
+ [`start`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-start" >}}),
+ [`stop`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-stop" >}}), and
+ [`reset`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset" >}}).
+- **Secrets**: [`list-secrets`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-secrets" >}}),
+ [`get-secret`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-secret" >}}),
+ [`describe-secret`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-secret" >}}),
+ [`set-secret`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-secret" >}}), and
+ [`delete-secret`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete-secret" >}}).
+- **Dead-letter queues**: [`list-dlqs`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlqs" >}}),
+ [`get-dlq`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-dlq" >}}), and
+ [`list-dlq-records`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlq-records" >}}) (alias `get-rejected`).
+- **Jobs**: [`list-jobs`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-jobs" >}}),
+ [`get-job`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-job" >}}), and
+ [`describe-job`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-job" >}}).
+- **Metric collections**: [`list-metric-collections`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-metric-collections" >}}) and
+ [`get-metric-collection`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-metric-collection" >}}).
+- **Scaffolding**: [`scaffold`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-scaffold" >}})
+- **Contexts**: [`list-contexts`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-contexts" >}}),
+ [`describe-context`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-context" >}}),
+ [`set-context`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-context" >}}),
+ [`use-context`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-use-context" >}}), and
+ [`delete-context`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete-context" >}}).
+
+On VM installations, the CLI also exposes the
+[`configure-rdi`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-configure-rdi" >}}) and
+[`dump-support-package`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-dump-support-package" >}})
+administration commands.
+
+See the [`redis-di`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di" >}}) page for the
+global options that apply to every command.
+
+## Output formats
+
+The `list` and `get` commands print an aligned, column-based table by default. Pass `-o` / `--output`
+with `json` or `yaml` to emit the underlying data instead, which is useful for scripting and for tools
+such as `jq`. The `describe` commands always print a human-readable, sectioned layout.
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/previous-cli-comparison.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/previous-cli-comparison.md
new file mode 100644
index 0000000000..1cdf49a395
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/previous-cli-comparison.md
@@ -0,0 +1,165 @@
+---
+Title: Compare the previous and current RDI CLI
+linkTitle: Previous CLI comparison
+description: Compare the RDI CLI introduced in version 1.19.0 with the previous CLI
+weight: 5
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/previous-cli-comparison/'
+---
+
+RDI 1.19.0 replaces the previous Python-based `redis-di` CLI with a Go-based CLI.
+The executable name stays the same, and several command names are unchanged, but the
+current CLI is not a drop-in replacement for the previous connection settings or contexts.
+
+The most important difference is what the CLI connects to. The previous CLI connected
+directly to the RDI database and the Kubernetes API. The current CLI is a thin client over
+the RDI REST API.
+
+## Compare the CLIs
+
+| Area | Previous CLI (before RDI 1.19.0) | Current CLI (RDI 1.19.0 and later) |
+| :--- | :------------------------------- | :---------------------------------- |
+| Implementation | Python application | Self-contained Go binary |
+| Installation types | VM installations | VM, Kubernetes, and Redis Cloud installations |
+| Connection | RDI database and Kubernetes API | RDI REST API |
+| Pipeline scope | The `default` pipeline | Multiple named pipelines; defaults to `default` |
+| Connection options | `--rdi-host`, `--rdi-port`, `--rdi-user`, `--rdi-password`, and RDI database TLS options | `--api-url`, `--user`, `--password`, and API TLS options; Redis Cloud also supports `--account-key` and `--user-key` |
+| Contexts | A list of RDI database connections in `~/.redis-di`, with an `is_active` field on each entry | A map of API connections in `~/.redis-di`, with one `current-context` |
+| Output | Human-readable tables | Compact tables and sectioned descriptions; `list` and `get` commands also support JSON and YAML |
+| Secrets | `set-secret` on VM installations; `rdi-secret.sh` for Kubernetes | Create, inspect, update, and delete operations through `redis-di` on every installation type |
+
+## Update the connection options
+
+Replace the RDI database address with the RDI API URL. For example, a previous CLI
+invocation such as:
+
+```bash
+redis-di status \
+ --rdi-host \
+ --rdi-port \
+ --rdi-user \
+ --rdi-password
+```
+
+becomes:
+
+```bash
+redis-di describe \
+ --api-url https:// \
+ --user \
+ --password
+```
+
+The current CLI does not accept the previous RDI database connection options on
+API-based commands. Update scripts and environment variables as follows:
+
+| Previous setting | Current setting |
+| :--------------- | :-------------- |
+| `--rdi-host` and `--rdi-port` | `--api-url` or `RDI_API_URL` |
+| `--rdi-user` or `RDI_REDIS_USERNAME` | `--user` or `RDI_USER` |
+| `--rdi-password` or `RDI_REDIS_PASSWORD` | `--password` or `RDI_PASSWORD` |
+| `--rdi-cacert` | `--cacert` or `RDI_CACERT` |
+| `--rdi-key`, `--rdi-cert`, `--rdi-key-password` | No equivalent; the CLI authenticates to the API instead of the RDI database |
+| `--rdi-namespace` or `RDI_NAMESPACE` | No equivalent; pipeline operations go through the API |
+
+For Redis Cloud, use `--account-key` with `--user-key` instead of `--user` with
+`--password`. See the [CLI reference overview]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli#connecting-to-the-api" >}})
+for all authentication modes.
+
+## Recreate contexts
+
+Both CLIs use `~/.redis-di`, but the file formats and stored connection details are
+different. Previous contexts contain RDI database and Kubernetes connection details and
+cannot be used as API contexts.
+
+The standard VM [`upgrade.sh`]({{< relref "/integrate/redis-data-integration/1.19.1/installation/upgrade#upgrading-a-vm-installation" >}})
+flow preserves the previous context for the VM administration commands and creates a
+default API context. If you replace the CLI independently and still have a previous-format
+`~/.redis-di` file, back it up before creating current contexts:
+
+```bash
+mv ~/.redis-di ~/.redis-di.pre-1.19
+
+redis-di set-context \
+ --api-url https:// \
+ --user
+redis-di use-context
+```
+
+The context commands also changed meaning:
+
+| Task | Previous CLI | Current CLI |
+| :--- | :----------- | :---------- |
+| Create a context | `redis-di add-context [connection options]` | `redis-di set-context [connection options]` |
+| Select the active context | `redis-di set-context ` | `redis-di use-context ` |
+| Update a context | Recreate the context | `redis-di set-context [options to update]` |
+| Remove all contexts | `redis-di delete-all-contexts` | Delete contexts individually with `redis-di delete-context ` |
+
+Passwords and Redis Cloud user keys are not saved in current contexts. Supply them with an
+environment variable, a command option, or the interactive prompt.
+
+## Update commands
+
+Pipeline lifecycle commands keep their previous names. They now take an optional pipeline
+name that defaults to `default`:
+
+```bash
+redis-di deploy [pipeline] --dir
+redis-di start [pipeline]
+redis-di stop [pipeline]
+redis-di reset [pipeline]
+```
+
+Other common tasks changed as follows:
+
+| Task | Previous CLI | Current CLI |
+| :--- | :----------- | :---------- |
+| Inspect pipeline status | `redis-di status` | `redis-di describe [pipeline]`; `status` remains an alias |
+| Continuously refresh status | `redis-di status --live` | `watch -n 1 redis-di describe [pipeline]` |
+| List pipelines | Not available | `redis-di list` |
+| Get a pipeline | Not available | `redis-di get [pipeline]` |
+| Delete a pipeline | Not available | `redis-di delete [pipeline]` |
+| Inspect rejected records | `redis-di get-rejected [options]` | `redis-di list-dlqs`, then `redis-di list-dlq-records `; `get-rejected` remains an alias |
+| Inspect jobs | `redis-di list-jobs` and `redis-di describe-job ` | The same commands, with `--pipeline ` for a non-default pipeline |
+| Manage secrets | `redis-di set-secret ` or `rdi-secret.sh` | `list-secrets`, `get-secret`, `describe-secret`, `set-secret`, and `delete-secret` |
+| Install or upgrade RDI on a VM | `redis-di install` or `redis-di upgrade` | Run `install.sh` or `upgrade.sh` from the VM installation package |
+| Install or upgrade RDI on Kubernetes | Not available | Use the RDI Helm chart |
+| Trace pipeline records | `redis-di trace` | Removed |
+| Create the RDI database | `redis-di create` | Removed; use the installation workflow |
+
+On VM installations, `configure-rdi` and `dump-support-package` remain available through
+`redis-di`. They are not available with a standalone CLI or on Kubernetes, Redis Cloud,
+macOS, or Windows.
+
+The current `describe` output is sourced from the API. It does not include the connected
+Redis client list or Debezium offsets that the previous `status` command read directly.
+The `--live`, `--page-number`, `--page-size`, and `--ingested-only` status options were
+removed.
+
+## Update scripts that read CLI output
+
+The current CLI uses a compact, kubectl-style table for list operations and a sectioned,
+human-readable layout for `describe` operations. Scripts that parse the previous bordered
+tables must be updated.
+
+For list and get operations, prefer machine-readable output instead of parsing a table:
+
+```bash
+redis-di list -o json
+redis-di get -o yaml
+redis-di list-secrets -o json
+```
+
+Command results are written to standard output and diagnostics are written to standard
+error, so you can redirect or pipe results without including log messages.
+
+## Verify the migration
+
+After updating your contexts and commands:
+
+1. Run `redis-di --version` and verify that it reports version 1.19.0 or later.
+1. Run `redis-di info` to verify the API connection and RDI version.
+1. Run `redis-di list` to verify that the expected pipelines are visible.
+1. Run `redis-di describe [pipeline]` to verify pipeline status and metrics.
+1. Run `redis-di help ` to review changed options before updating automation.
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-completion.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-completion.md
new file mode 100644
index 0000000000..fc059b0285
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-completion.md
@@ -0,0 +1,28 @@
+---
+Title: redis-di completion
+linkTitle: redis-di completion
+description: Generates a shell autocompletion script
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-completion/'
+---
+
+Generates an autocompletion script for `redis-di` for the specified shell. Supported shells are
+`bash`, `zsh`, `fish`, and `powershell`.
+
+## Usage
+
+```
+redis-di completion [bash|zsh|fish|powershell]
+```
+
+Run `redis-di completion --help` for the per-shell installation instructions.
+
+## Example
+
+To load completions into the current `bash` session:
+
+```bash
+source <(redis-di completion bash)
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-configure-rdi.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-configure-rdi.md
new file mode 100644
index 0000000000..a3d9561364
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-configure-rdi.md
@@ -0,0 +1,34 @@
+---
+Title: redis-di configure-rdi
+linkTitle: redis-di configure-rdi
+description: Configures the RDI database connection
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-configure-rdi/'
+---
+
+Configures the connection credentials for the RDI database. This is an administration command that is
+available only on VM installations, where `redis-di` forwards it to the bundled `rdi-admin` tool.
+
+## Usage
+
+```
+redis-di configure-rdi [OPTIONS]
+```
+
+## Options
+
+| Option | Description |
+| :------------------- | :--------------------------------------------------------------------------------------- |
+| `-l`, `--log-level` | Log level: `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL` (default `INFO`). |
+| `--rdi-namespace` | RDI Kubernetes namespace (default `rdi`). |
+| `--rdi-host` | Host or IP of the RDI database (required). |
+| `--rdi-port` | Port of the RDI database, `1`–`65535` (required). |
+| `--rdi-user` | RDI database username. |
+| `--rdi-password` | RDI database password. |
+| `--rdi-key` | Private key file to authenticate with. |
+| `--rdi-cert` | Client certificate file to authenticate with. |
+| `--rdi-cacert` | CA certificate file to verify with. |
+| `--rdi-key-password` | Password for unlocking an encrypted private key. |
+| `--rdi-log-level` | Log level for the RDI components. |
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete-context.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete-context.md
new file mode 100644
index 0000000000..9ef0c29026
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete-context.md
@@ -0,0 +1,33 @@
+---
+Title: redis-di delete-context
+linkTitle: redis-di delete-context
+description: Deletes a context
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete-context/'
+---
+
+Deletes a context from the `~/.redis-di` context file. Because this is destructive, the command asks
+for confirmation unless you pass `--force`.
+
+## Usage
+
+```
+redis-di delete-context [flags]
+```
+
+## Options
+
+| Option | Description |
+| :-------- | :---------------------------- |
+| `--force` | Skip the confirmation prompt. |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di delete-context dev --force
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete-secret.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete-secret.md
new file mode 100644
index 0000000000..5d42ddd7af
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete-secret.md
@@ -0,0 +1,36 @@
+---
+Title: redis-di delete-secret
+linkTitle: redis-di delete-secret
+description: Deletes a secret of a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete-secret/'
+---
+
+Deletes a secret of a pipeline. Because this is destructive, the command asks for confirmation unless
+you pass `--force`.
+
+## Usage
+
+```
+redis-di delete-secret [flags]
+```
+
+## Options
+
+| Option | Description |
+| :----------------- | :-------------------------------------------------------------------------------- |
+| `-p`, `--pipeline` | Pipeline to target (default `default`). |
+| `--force` | Skip the confirmation prompt. |
+| `--wait` | Wait for the pipeline to reach the expected state (default `true`). |
+| `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di delete-secret SOURCE_DB_CACERT --force
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete.md
new file mode 100644
index 0000000000..171a9c3484
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete.md
@@ -0,0 +1,37 @@
+---
+Title: redis-di delete
+linkTitle: redis-di delete
+description: Deletes a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete/'
+---
+
+Deletes a pipeline. Because this is destructive, the command asks for confirmation unless you pass
+`--force`.
+
+## Usage
+
+```
+redis-di delete [pipeline] [flags]
+```
+
+The pipeline name is an optional argument that defaults to `default`.
+
+## Options
+
+| Option | Description |
+| :---------- | :-------------------------------------------------------------------------------- |
+| `--force` | Skip the confirmation prompt. |
+| `--wait` | Wait for the pipeline to reach the expected state (default `true`). |
+| `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di delete my-pipeline --force
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-deploy.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-deploy.md
new file mode 100644
index 0000000000..a4f25ba54a
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-deploy.md
@@ -0,0 +1,47 @@
+---
+Title: redis-di deploy
+linkTitle: redis-di deploy
+description: Deploys a pipeline with the specified configuration
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-deploy/'
+---
+
+Deploys a pipeline, creating it or updating it from the configuration in the `--dir` directory. The
+API validates the configuration and rejects an invalid one. By default, the command starts the
+pipeline after deploying and waits for it to reach the expected state. `set` is an alias for this
+command.
+
+## Usage
+
+```
+redis-di deploy [pipeline] [flags]
+```
+
+The pipeline name is an optional argument that defaults to `default`.
+
+## Options
+
+| Option | Description |
+| :------------------ | :----------------------------------------------------------------------------------- |
+| `--dir` | Directory containing the pipeline configuration (default `.`). |
+| `--dry-run` | Validate the configuration without deploying. |
+| `--validate-tables` | Validate the configuration against the source and target databases (default `true`). |
+| `--validate-cdc` | Validate the source database CDC configuration. |
+| `--start` | Start the pipeline after deploying (default `true`). |
+| `--wait` | Wait for the pipeline to reach the expected state (default `true`). |
+| `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+# Deploy the configuration in the current directory
+redis-di deploy
+
+# Validate a configuration folder without deploying it
+redis-di deploy --dir /opt/rdi/config --dry-run
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-context.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-context.md
new file mode 100644
index 0000000000..ef95432152
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-context.md
@@ -0,0 +1,30 @@
+---
+Title: redis-di describe-context
+linkTitle: redis-di describe-context
+description: Describes a context
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-context/'
+---
+
+Describes a single context from the `~/.redis-di` context file, showing its API connection details.
+See the [CLI reference overview]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli#contexts" >}})
+for more about contexts.
+
+## Usage
+
+```
+redis-di describe-context [flags]
+```
+
+## Options
+
+This command takes only the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di describe-context prod
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-job.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-job.md
new file mode 100644
index 0000000000..60949951a1
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-job.md
@@ -0,0 +1,33 @@
+---
+Title: redis-di describe-job
+linkTitle: redis-di describe-job
+description: Describes a job of a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-job/'
+---
+
+Describes a single job of a pipeline, printing its source properties followed by tables that
+summarize its transformations and outputs.
+
+## Usage
+
+```
+redis-di describe-job [flags]
+```
+
+## Options
+
+| Option | Description |
+| :----------------- | :-------------------------------------- |
+| `-p`, `--pipeline` | Pipeline to target (default `default`). |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di describe-job customers_hash_job
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-secret.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-secret.md
new file mode 100644
index 0000000000..b6c85cd13e
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-secret.md
@@ -0,0 +1,33 @@
+---
+Title: redis-di describe-secret
+linkTitle: redis-di describe-secret
+description: Describes a secret of a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-secret/'
+---
+
+Describes a single secret of a pipeline. The API never returns secret values, so the output shows
+only the key and whether it is set, not the stored value.
+
+## Usage
+
+```
+redis-di describe-secret [flags]
+```
+
+## Options
+
+| Option | Description |
+| :----------------- | :-------------------------------------- |
+| `-p`, `--pipeline` | Pipeline to target (default `default`). |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di describe-secret TARGET_DB_PASSWORD
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe.md
new file mode 100644
index 0000000000..2978f68d6e
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe.md
@@ -0,0 +1,79 @@
+---
+Title: redis-di describe
+linkTitle: redis-di describe
+description: Describes a pipeline with its status
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe/'
+---
+
+Describes a pipeline, combining its configuration with its runtime status, components, errors, and
+metrics in a human-readable, sectioned layout. `status` is an alias for this command.
+
+The RDI version is not shown here, because it is a property of the API connection rather than of the
+pipeline; use [`info`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-info" >}})
+to see it.
+
+## Usage
+
+```
+redis-di describe [pipeline] [flags]
+```
+
+The pipeline name is an optional argument that defaults to `default`.
+
+## Options
+
+This command takes only the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di describe
+redis-di status my-pipeline
+```
+
+To watch the status update live, pair the command with `watch`:
+
+```bash
+watch -n 1 redis-di describe
+```
+
+The output has a section for each part of the pipeline, for example:
+
+```
+Name: default
+Active: yes
+Status: started
+Current: yes
+
+Sources:
+ Name Type Db Type Connection Sync Mode Connected
+ ---- ---- ------- ---------- --------- ---------
+ mysql cdc mysql ${HOST_IP}:13000 streaming yes
+
+Targets:
+ Name Db Type Connection Connected
+ ---- ------- ---------- ---------
+ target redis ${HOST_IP}:12000 yes
+
+Jobs:
+ Name Source Transformations Outputs Connections
+ ---- ------ --------------- ------- -----------
+ address_job inventory.addresses 1 1 target
+ customers_hash_job inventory.customers 0 1 target
+
+Components:
+ Name Type Version Status
+ ---- ---- ------- ------
+ collector-api collector-api 0.0.0 started
+ collector-source debezium-collector ... started
+ processor processor 0.0.0 started
+
+Statistics:
+ Name Total Pending Inserted Updated Deleted Filtered Rejected Deduplicated Last Arrival
+ ---- ----- ------- -------- ------- ------- -------- -------- ------------ ------------
+ {rdi}:inventory.customers 4 0 4 0 0 0 0 0 2026-06-18T13:42:44Z
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-dump-support-package.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-dump-support-package.md
new file mode 100644
index 0000000000..5d7f4238c1
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-dump-support-package.md
@@ -0,0 +1,38 @@
+---
+Title: redis-di dump-support-package
+linkTitle: redis-di dump-support-package
+description: Dumps the RDI support package
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-dump-support-package/'
+---
+
+Dumps a comprehensive set of RDI forensics data that you can send to Redis support (see
+[Dump support package]({{< relref "/integrate/redis-data-integration/1.19.1/troubleshooting#dump-support-package" >}})).
+This is an administration command that is available only on VM installations, where `redis-di`
+forwards it to the bundled `rdi-admin` tool.
+
+## Usage
+
+```
+redis-di dump-support-package [OPTIONS]
+```
+
+## Options
+
+| Option | Description |
+| :------------------- | :--------------------------------------------------------------------------------------- |
+| `-l`, `--log-level` | Log level: `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL` (default `INFO`). |
+| `--rdi-namespace` | RDI Kubernetes namespace (default `rdi`). |
+| `--rdi-host` | Host or IP of the RDI database (required). |
+| `--rdi-port` | Port of the RDI database, `1`–`65535` (required). |
+| `--rdi-user` | RDI database username. |
+| `--rdi-password` | RDI database password. |
+| `--rdi-key` | Private key file to authenticate with. |
+| `--rdi-cert` | Client certificate file to authenticate with. |
+| `--rdi-cacert` | CA certificate file to verify with. |
+| `--rdi-key-password` | Password for unlocking an encrypted private key. |
+| `--dir` | Directory where the support file is generated (default `.`). |
+| `--dump-rejected` | Dump rejected records. |
+| `--log-days` | Number of days to look back for log files (default `2`). |
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-dlq.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-dlq.md
new file mode 100644
index 0000000000..4f2639b692
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-dlq.md
@@ -0,0 +1,34 @@
+---
+Title: redis-di get-dlq
+linkTitle: redis-di get-dlq
+description: Gets a dead-letter queue of a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-dlq/'
+---
+
+Gets a single dead-letter queue (DLQ) of a pipeline and prints it in the compact `list-dlqs` table
+format.
+
+## Usage
+
+```
+redis-di get-dlq [flags]
+```
+
+## Options
+
+| Option | Description |
+| :----------------- | :--------------------------------------------------- |
+| `-p`, `--pipeline` | Pipeline to target (default `default`). |
+| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di get-dlq inventory.customers
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-job.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-job.md
new file mode 100644
index 0000000000..8cfb2ce46b
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-job.md
@@ -0,0 +1,35 @@
+---
+Title: redis-di get-job
+linkTitle: redis-di get-job
+description: Gets a job of a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-job/'
+---
+
+Gets a single job of a pipeline and prints it in the compact `list-jobs` table format. Use
+[`describe-job`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-job" >}})
+for the full job view with its transformations and outputs.
+
+## Usage
+
+```
+redis-di get-job [flags]
+```
+
+## Options
+
+| Option | Description |
+| :----------------- | :--------------------------------------------------- |
+| `-p`, `--pipeline` | Pipeline to target (default `default`). |
+| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di get-job customers_hash_job
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-metric-collection.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-metric-collection.md
new file mode 100644
index 0000000000..b8eb730754
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-metric-collection.md
@@ -0,0 +1,36 @@
+---
+Title: redis-di get-metric-collection
+linkTitle: redis-di get-metric-collection
+description: Gets a metric collection of a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-metric-collection/'
+---
+
+Gets a single metric collection of a pipeline, returning its raw metric data. This command is most
+useful with `-o json` or `-o yaml` for scripting and for tools such as `jq`. Use
+[`list-metric-collections`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-metric-collections" >}})
+to see the available collections.
+
+## Usage
+
+```
+redis-di get-metric-collection [flags]
+```
+
+## Options
+
+| Option | Description |
+| :----------------- | :--------------------------------------------------- |
+| `-p`, `--pipeline` | Pipeline to target (default `default`). |
+| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di get-metric-collection processor -o json
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-secret.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-secret.md
new file mode 100644
index 0000000000..51fffde3a1
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-secret.md
@@ -0,0 +1,35 @@
+---
+Title: redis-di get-secret
+linkTitle: redis-di get-secret
+description: Gets a secret of a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-secret/'
+---
+
+Gets a single secret of a pipeline and prints it in the compact `list-secrets` table format. The API
+never returns secret values, so the output shows only the key and whether it is set, not the stored
+value.
+
+## Usage
+
+```
+redis-di get-secret [flags]
+```
+
+## Options
+
+| Option | Description |
+| :----------------- | :--------------------------------------------------- |
+| `-p`, `--pipeline` | Pipeline to target (default `default`). |
+| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di get-secret SOURCE_DB_USERNAME
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get.md
new file mode 100644
index 0000000000..6db6d73f34
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get.md
@@ -0,0 +1,37 @@
+---
+Title: redis-di get
+linkTitle: redis-di get
+description: Gets a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get/'
+---
+
+Gets a single pipeline and prints it in the compact `list` table format. Use
+[`describe`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe" >}})
+for the full pipeline view with its status and metrics.
+
+## Usage
+
+```
+redis-di get [pipeline] [flags]
+```
+
+The pipeline name is an optional argument that defaults to `default`.
+
+## Options
+
+| Option | Description |
+| :--------------- | :--------------------------------------------------- |
+| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di get
+redis-di get my-pipeline -o yaml
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-info.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-info.md
new file mode 100644
index 0000000000..4ddc2dd762
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-info.md
@@ -0,0 +1,22 @@
+---
+Title: redis-di info
+linkTitle: redis-di info
+description: Displays information about the RDI deployment
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-info/'
+---
+
+Displays information about the RDI deployment, such as the RDI version reported by the API.
+
+## Usage
+
+```
+redis-di info [flags]
+```
+
+## Options
+
+This command takes only the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-contexts.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-contexts.md
new file mode 100644
index 0000000000..b02192764a
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-contexts.md
@@ -0,0 +1,30 @@
+---
+Title: redis-di list-contexts
+linkTitle: redis-di list-contexts
+description: Lists all contexts
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-contexts/'
+---
+
+Lists all contexts from the `~/.redis-di` context file and indicates which one is active. See the
+[CLI reference overview]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli#contexts" >}})
+for more about contexts.
+
+## Usage
+
+```
+redis-di list-contexts [flags]
+```
+
+## Options
+
+This command takes only the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di list-contexts
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlq-records.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlq-records.md
new file mode 100644
index 0000000000..f5574a7f1b
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlq-records.md
@@ -0,0 +1,54 @@
+---
+Title: redis-di list-dlq-records
+linkTitle: redis-di list-dlq-records
+description: Lists the rejected records of a dead-letter queue
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlq-records/'
+---
+
+Lists the rejected records of a single dead-letter queue (DLQ), taking the queue name as an argument
+and paging with `--limit`, `--offset`, and `--sort-order`. The operation code is shown by name
+(create, update, delete, read). `get-rejected` is an alias for this command.
+
+Use [`list-dlqs`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlqs" >}})
+to see all the pipeline's dead-letter queues and their record counts.
+
+## Usage
+
+```
+redis-di list-dlq-records [flags]
+```
+
+## Options
+
+| Option | Description |
+| :----------------- | :------------------------------------------------------------------ |
+| `-p`, `--pipeline` | Pipeline to target (default `default`). |
+| `--limit` | Maximum number of records to return (default `20`). |
+| `--offset` | Number of records to skip (default `0`). |
+| `--sort-order` | Sort order: `asc` (oldest first) or `desc` (newest first, default). |
+| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. |
+
+The following options are kept for backward compatibility with the `get-rejected` command and are
+deprecated:
+
+| Deprecated option | Use instead |
+| :---------------- | :-------------------------------- |
+| `--dlq-name` | Pass the DLQ name as an argument. |
+| `--max-records` | `--limit` |
+| `--oldest` | `--sort-order asc` |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+# Newest 20 rejected records of a queue
+redis-di list-dlq-records inventory.customers
+
+# Oldest 100 records, as JSON
+redis-di list-dlq-records inventory.customers --limit 100 --sort-order asc -o json
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlqs.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlqs.md
new file mode 100644
index 0000000000..d4a26d0caa
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlqs.md
@@ -0,0 +1,36 @@
+---
+Title: redis-di list-dlqs
+linkTitle: redis-di list-dlqs
+description: Lists the dead-letter queues of a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlqs/'
+---
+
+Lists the dead-letter queues (DLQs) of a pipeline with their record counts. A DLQ holds the records
+that RDI rejected. Use
+[`list-dlq-records`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlq-records" >}})
+to read the records of a single queue.
+
+## Usage
+
+```
+redis-di list-dlqs [flags]
+```
+
+## Options
+
+| Option | Description |
+| :----------------- | :--------------------------------------------------- |
+| `-p`, `--pipeline` | Pipeline to target (default `default`). |
+| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di list-dlqs
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-jobs.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-jobs.md
new file mode 100644
index 0000000000..7f589e7900
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-jobs.md
@@ -0,0 +1,36 @@
+---
+Title: redis-di list-jobs
+linkTitle: redis-di list-jobs
+description: Lists the jobs of a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-jobs/'
+---
+
+Lists the jobs of a pipeline, one row per job with its source, its transformation and output counts,
+and the target connections of its outputs. Use
+[`describe-job`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-job" >}})
+for the full view of a single job.
+
+## Usage
+
+```
+redis-di list-jobs [flags]
+```
+
+## Options
+
+| Option | Description |
+| :----------------- | :--------------------------------------------------- |
+| `-p`, `--pipeline` | Pipeline to target (default `default`). |
+| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di list-jobs
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-metric-collections.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-metric-collections.md
new file mode 100644
index 0000000000..9ea64a781d
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-metric-collections.md
@@ -0,0 +1,36 @@
+---
+Title: redis-di list-metric-collections
+linkTitle: redis-di list-metric-collections
+description: Lists the metric collections of a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-metric-collections/'
+---
+
+Lists the metric collections of a pipeline. Metric collections hold the raw component metrics that
+the [`describe`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe" >}})
+command summarizes in its Statistics and Performance sections. This command is most useful with
+`-o json` or `-o yaml` for scripting and for tools such as `jq`.
+
+## Usage
+
+```
+redis-di list-metric-collections [flags]
+```
+
+## Options
+
+| Option | Description |
+| :----------------- | :--------------------------------------------------- |
+| `-p`, `--pipeline` | Pipeline to target (default `default`). |
+| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di list-metric-collections
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-secrets.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-secrets.md
new file mode 100644
index 0000000000..9f6b5aa0b6
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-secrets.md
@@ -0,0 +1,35 @@
+---
+Title: redis-di list-secrets
+linkTitle: redis-di list-secrets
+description: Lists the secrets of a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-secrets/'
+---
+
+Lists the secrets of a pipeline. The API never returns secret values, so the output shows only the
+secret keys and whether each one is set, not the stored values.
+
+## Usage
+
+```
+redis-di list-secrets [flags]
+```
+
+## Options
+
+| Option | Description |
+| :----------------- | :--------------------------------------------------- |
+| `-p`, `--pipeline` | Pipeline to target (default `default`). |
+| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di list-secrets
+redis-di list-secrets -p my-pipeline
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list.md
new file mode 100644
index 0000000000..59cd28d0a7
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list.md
@@ -0,0 +1,33 @@
+---
+Title: redis-di list
+linkTitle: redis-di list
+description: Lists all pipelines
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list/'
+---
+
+Lists all pipelines with their status in a compact table.
+
+## Usage
+
+```
+redis-di list [flags]
+```
+
+## Options
+
+| Option | Description |
+| :--------------- | :--------------------------------------------------- |
+| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di list
+redis-di list -o json
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset.md
new file mode 100644
index 0000000000..28bc27442e
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset.md
@@ -0,0 +1,37 @@
+---
+Title: redis-di reset
+linkTitle: redis-di reset
+description: Resets a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset/'
+---
+
+Resets a pipeline into initial full-sync mode, so it reloads a snapshot of the source data before
+resuming change data capture. By default, the command waits for the pipeline to reach a terminal
+state before returning.
+
+## Usage
+
+```
+redis-di reset [pipeline] [flags]
+```
+
+The pipeline name is an optional argument that defaults to `default`.
+
+## Options
+
+| Option | Description |
+| :---------- | :-------------------------------------------------------------------------------- |
+| `--wait` | Wait for the pipeline to reach the expected state (default `true`). |
+| `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di reset
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-scaffold.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-scaffold.md
new file mode 100644
index 0000000000..e6e878ccd9
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-scaffold.md
@@ -0,0 +1,41 @@
+---
+Title: redis-di scaffold
+linkTitle: redis-di scaffold
+description: Generates pipeline configuration files
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-scaffold/'
+---
+
+Generates a starter pipeline configuration for the given source database type. With `--dir`, the
+command writes a `config.yaml` file into that directory, prompting before overwriting an existing
+file unless `--force` is set. Without `--dir`, it prints the configuration to standard output.
+
+## Usage
+
+```
+redis-di scaffold [flags]
+```
+
+## Options
+
+| Option | Description |
+| :------------ | :------------------------------------------------------------------------------------------------------------------------------ |
+| `--db-type` | Source database type (required): `mariadb`, `mongodb`, `mysql`, `oracle`, `postgresql`, `snowflake`, `sqlserver`, or `spanner`. |
+| `--db-flavor` | Source database flavor: `mongodb-atlas`, `mongodb-replica-set`, or `mongodb-sharded-cluster`. |
+| `--dir` | Directory to write `config.yaml` to; prints to standard output when omitted. |
+| `--force` | Skip the confirmation prompt when overwriting an existing file. |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+# Print a PostgreSQL configuration to stdout
+redis-di scaffold --db-type postgresql
+
+# Write a MySQL configuration into a directory
+redis-di scaffold --db-type mysql --dir /opt/rdi/config
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-context.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-context.md
new file mode 100644
index 0000000000..29d097b0e9
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-context.md
@@ -0,0 +1,49 @@
+---
+Title: redis-di set-context
+linkTitle: redis-di set-context
+description: Creates or updates a context
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-context/'
+---
+
+Creates or updates a context in the `~/.redis-di` context file. A context stores an API connection
+so you don't have to pass the connection options on every command. `set-context` merges only the
+options you give on the command line, preserving the rest, and does not change which context is
+active; use [`use-context`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-use-context" >}})
+for that. See the
+[CLI reference overview]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli#contexts" >}})
+for more about contexts.
+
+Secrets (the password and the Redis Cloud user key) are never stored in a context, so `set-context`
+rejects `--password` and `--user-key`.
+
+## Usage
+
+```
+redis-di set-context [flags]
+```
+
+The connection is set from the global options `--api-url`, `--user`, `--account-key`, `--cacert`,
+and `--insecure`.
+
+## Options
+
+| Option | Description |
+| :-------------------- | :-------------------------------------------------------------------------------------------- |
+| `--unset-user` | Clear the stored user, so the context authenticates without a user. |
+| `--unset-account-key` | Clear the stored account key, so the context authenticates without a Redis Cloud account key. |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+# Create or update a context for a VM or Kubernetes installation
+redis-di set-context prod --api-url https://rdi.example.com --user default --cacert /etc/rdi/ingress-ca.crt
+
+# Create or update a context that skips TLS verification
+redis-di set-context dev --api-url https://localhost:8443 --insecure
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-secret.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-secret.md
new file mode 100644
index 0000000000..a6bdc3758c
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-secret.md
@@ -0,0 +1,51 @@
+---
+Title: redis-di set-secret
+linkTitle: redis-di set-secret
+description: Creates or updates a secret of a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-secret/'
+---
+
+Creates or updates a secret of a pipeline. Secrets hold the credentials and certificates that the
+pipeline uses to connect to the source and target databases (see
+[Set secrets]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/deploy#set-secrets" >}})
+for the list of secret names). You can then refer to a secret in the `config.yaml` file with the
+syntax `${SECRET_NAME}`.
+
+The secret value comes from the `[value]` argument, the `--file` option, or the `--literal` option.
+If you provide none of these on an interactive terminal, the command prompts for the value without
+echoing it.
+
+## Usage
+
+```
+redis-di set-secret [value] [flags]
+```
+
+## Options
+
+| Option | Description |
+| :----------------- | :-------------------------------------------------------------------------------- |
+| `-p`, `--pipeline` | Pipeline to target (default `default`). |
+| `--file` | Read the secret value from the file at this path. |
+| `--literal` | Use this literal string as the secret value. |
+| `--wait` | Wait for the pipeline to reach the expected state (default `true`). |
+| `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+# Value from an argument
+redis-di set-secret SOURCE_DB_USERNAME myuser
+
+# Value from a file (for example, a certificate)
+redis-di set-secret SOURCE_DB_CACERT --file /path/to/myca.crt
+
+# Value read from an interactive prompt
+redis-di set-secret SOURCE_DB_PASSWORD
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-start.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-start.md
new file mode 100644
index 0000000000..6203e338cd
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-start.md
@@ -0,0 +1,37 @@
+---
+Title: redis-di start
+linkTitle: redis-di start
+description: Starts a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-start/'
+---
+
+Starts a pipeline. By default, the command waits for the pipeline to reach the `started` state before
+returning.
+
+## Usage
+
+```
+redis-di start [pipeline] [flags]
+```
+
+The pipeline name is an optional argument that defaults to `default`.
+
+## Options
+
+| Option | Description |
+| :---------- | :-------------------------------------------------------------------------------- |
+| `--wait` | Wait for the pipeline to reach the expected state (default `true`). |
+| `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di start
+redis-di start my-pipeline --wait=false
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-stop.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-stop.md
new file mode 100644
index 0000000000..1a91973c79
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-stop.md
@@ -0,0 +1,36 @@
+---
+Title: redis-di stop
+linkTitle: redis-di stop
+description: Stops a pipeline
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-stop/'
+---
+
+Stops a pipeline. By default, the command waits for the pipeline to reach the `stopped` state before
+returning.
+
+## Usage
+
+```
+redis-di stop [pipeline] [flags]
+```
+
+The pipeline name is an optional argument that defaults to `default`.
+
+## Options
+
+| Option | Description |
+| :---------- | :-------------------------------------------------------------------------------- |
+| `--wait` | Wait for the pipeline to reach the expected state (default `true`). |
+| `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). |
+
+This command also accepts the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di stop
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-use-context.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-use-context.md
new file mode 100644
index 0000000000..2a50789ac1
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-use-context.md
@@ -0,0 +1,30 @@
+---
+Title: redis-di use-context
+linkTitle: redis-di use-context
+description: Sets a context to be the active one
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-use-context/'
+---
+
+Sets a context in the `~/.redis-di` context file to be the active one, so its connection details are
+used by subsequent commands. Create or update a context with
+[`set-context`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-context" >}}).
+
+## Usage
+
+```
+redis-di use-context [flags]
+```
+
+## Options
+
+This command takes only the
+[global options]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di#global-options" >}}).
+
+## Example
+
+```bash
+redis-di use-context prod
+```
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di.md b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di.md
new file mode 100644
index 0000000000..727154e489
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/cli/redis-di.md
@@ -0,0 +1,86 @@
+---
+Title: redis-di
+linkTitle: redis-di
+description: Command line tool to manage Redis Data Integration
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/cli/redis-di/'
+---
+
+`redis-di` is the command line tool that manages Redis Data Integration (RDI). It is a thin client
+over the RDI API and works the same way for VM, Kubernetes, and Redis Cloud installations. See the
+[CLI reference overview]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli" >}}) for an
+introduction to connecting, authentication, and contexts.
+
+## Usage
+
+```
+redis-di [command]
+```
+
+Run `redis-di help` (or `redis-di --help`) to list every command, and `redis-di help `
+(or `redis-di --help`) to print the usage, flags, and arguments for a single command.
+
+## Global options
+
+These options apply to every command. Each one can also be set through an `RDI_`-prefixed environment
+variable, for example `RDI_API_URL`, `RDI_USER`, or `RDI_PASSWORD`. Setting a secret such as the
+password through an environment variable keeps it out of your shell history.
+
+| Option | Environment variable | Description |
+| :---------------- | :------------------- | :-------------------------------------------------------------------------------------------------------------- |
+| `--api-url` | `RDI_API_URL` | RDI API base URL. |
+| `--user` | `RDI_USER` | User for API (JWT) authentication. |
+| `--password` | `RDI_PASSWORD` | Password for API (JWT) authentication. Prompted for if a user is set and no password is supplied. |
+| `--account-key` | `RDI_ACCOUNT_KEY` | Redis Cloud account key for API authentication. |
+| `--user-key` | `RDI_USER_KEY` | Redis Cloud user key for API authentication. Prompted for if an account key is set and no user key is supplied. |
+| `--cacert` | `RDI_CACERT` | CA certificate that verifies the API ingress. |
+| `--insecure` | `RDI_INSECURE` | Skip TLS verification of the API ingress (insecure). Mutually exclusive with `--cacert`. |
+| `--context` | `RDI_CONTEXT` | Context to use instead of the active one. |
+| `--log-level` | `RDI_LOG_LEVEL` | Log level: `TRACE`, `DEBUG`, `INFO`, `WARNING`, or `ERROR` (default `INFO`). |
+| `-v`, `--verbose` | | Enable verbose logging, equivalent to `--log-level DEBUG`. |
+| `--version` | | Print the version and build metadata and exit. |
+| `-h`, `--help` | | Print help for the CLI or a command. |
+
+{{< note >}}Setting both `--user` and `--account-key` is an error, because they select mutually exclusive
+authentication modes. Setting both `--cacert` and `--insecure` is also an error.{{< /note >}}
+
+## Commands
+
+| Command | Description |
+| :----------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------- |
+| [`info`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-info" >}}) | Displays information about the RDI deployment |
+| [`list`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list" >}}) | Lists all pipelines |
+| [`get`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get" >}}) | Gets a pipeline |
+| [`describe`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe" >}}) | Describes a pipeline with its status (alias `status`) |
+| [`deploy`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-deploy" >}}) | Deploys a pipeline with the specified configuration (alias `set`) |
+| [`delete`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete" >}}) | Deletes a pipeline |
+| [`start`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-start" >}}) | Starts a pipeline |
+| [`stop`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-stop" >}}) | Stops a pipeline |
+| [`reset`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset" >}}) | Resets a pipeline |
+| [`list-secrets`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-secrets" >}}) | Lists the secrets of a pipeline |
+| [`get-secret`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-secret" >}}) | Gets a secret of a pipeline |
+| [`describe-secret`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-secret" >}}) | Describes a secret of a pipeline |
+| [`set-secret`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-secret" >}}) | Creates or updates a secret of a pipeline |
+| [`delete-secret`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete-secret" >}}) | Deletes a secret of a pipeline |
+| [`list-dlqs`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlqs" >}}) | Lists the dead-letter queues of a pipeline |
+| [`get-dlq`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-dlq" >}}) | Gets a dead-letter queue of a pipeline |
+| [`list-dlq-records`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-dlq-records" >}}) | Lists the rejected records of a dead-letter queue (alias `get-rejected`) |
+| [`list-jobs`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-jobs" >}}) | Lists the jobs of a pipeline |
+| [`get-job`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-job" >}}) | Gets a job of a pipeline |
+| [`describe-job`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-job" >}}) | Describes a job of a pipeline |
+| [`list-metric-collections`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-metric-collections" >}}) | Lists the metric collections of a pipeline |
+| [`get-metric-collection`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-get-metric-collection" >}}) | Gets a metric collection of a pipeline |
+| [`scaffold`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-scaffold" >}}) | Generates pipeline configuration files |
+| [`list-contexts`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-list-contexts" >}}) | Lists all contexts |
+| [`describe-context`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-describe-context" >}}) | Describes a context |
+| [`set-context`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-set-context" >}}) | Creates or updates a context |
+| [`use-context`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-use-context" >}}) | Sets a context to be the active one |
+| [`delete-context`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-delete-context" >}}) | Deletes a context |
+| [`completion`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-completion" >}}) | Generates a shell autocompletion script |
+
+On VM installations, the CLI also exposes the
+[`configure-rdi`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-configure-rdi" >}}),
+[`dump-support-package`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-dump-support-package" >}}),
+and `admin` administration commands.
diff --git a/content/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference.md b/content/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference.md
new file mode 100644
index 0000000000..e2190ab93b
--- /dev/null
+++ b/content/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference.md
@@ -0,0 +1,1090 @@
+---
+Title: Redis Data Integration configuration file
+linkTitle: RDI configuration file
+description: Redis Data Integration configuration file reference
+weight: 10
+alwaysopen: false
+categories: ["redis-di"]
+url: '/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference/'
+---
+
+Configuration file for Redis Data Integration (RDI) source collectors and target connections.
+
+
+**Properties**
+
+|Name|Type|Description|Required|
+|----|----|-----------|--------|
+|[**sources**](#sources).keys`.keys`
+
+
+John 30