diff --git a/content/operate/rc/rdi/_index.md b/content/operate/rc/rdi/_index.md index 9ecae41741..ba1cd6efab 100644 --- a/content/operate/rc/rdi/_index.md +++ b/content/operate/rc/rdi/_index.md @@ -41,14 +41,12 @@ which presents the considerations in a straightforward question-and-answer forma ```decision-tree ``` - - ### When to use RDI RDI is a good fit when: - You want your app/micro-services to read from Redis to scale reads at speed. -- You want to transfer data to Redis from a *single* source database. +- You want to transfer data to Redis from one or more source databases. - You must use a slow database as the system of record for the app. - The app must always *write* its data to the slow database. - Your app can tolerate *eventual* consistency of data in the Redis cache. @@ -87,11 +85,14 @@ RDI is not a good fit when: ways to replicate data across Redis databases such as replicaOf). - Your target Redis database is configured with Active-Active topology. Active-Active is not supported as an RDI Cloud target database. - Your database administrator has rejected RDI's requirements for the source database. - ## Data pipeline architecture -An RDI data pipeline sits between your source database and your target Redis database. Initially, the pipeline reads all of the data and imports it into the target database during the *initial sync* phase. After this initial sync is complete, the data pipeline enters the *streaming* phase, where changes are captured as they happen. Changes in the source database are added to the target within a few seconds of capture. The data pipeline translates relational database rows to Redis hashes or JSON documents. +An RDI data pipeline connects one or more source databases to one target Redis database. Sources can use the same or different supported database types. Each source has its own collector, connectivity, credentials, and selection of tables and columns. All sources share the pipeline's processor and target. + +Each source first imports its selected data during the *initial sync* phase, then captures changes during the *streaming* phase. The pipeline transforms the captured records and writes them to Redis. You can monitor and manage each source from the pipeline dashboard. + +RDI Cloud uses the Flink processor for all pipelines. For more info on how RDI works, see [RDI Architecture]({{}}). @@ -99,7 +100,7 @@ For more info on how RDI works, see [RDI Architecture]({{}}) to learn how to share your source database credentials and TLS certificates with Redis Cloud. -Connections to the source database use Java Database Connectivity (JDBC) through [AWS PrivateLink](https://aws.amazon.com/privatelink/), ensuring that the data pipeline is only exposed to the specific database endpoint. See [Set up connectivity]({{}}) to learn how to connect your PrivateLink to the Redis Cloud VPC. +Configure connectivity separately for each source. A source can use a public endpoint or [AWS PrivateLink](https://aws.amazon.com/privatelink/), subject to the source-specific requirements in [Prerequisites](#prerequisites). See [Set up connectivity]({{}}) to learn how to connect your PrivateLink to the Redis Cloud VPC. RDI encrypts all network connections with TLS. The pipeline will process data from the source database in-memory and write it to the target database using a TLS connection. There are no external connections to your data pipeline except from Redis Cloud management services. @@ -108,7 +109,7 @@ RDI encrypts all network connections with TLS. The pipeline will process data fr Before you can create a data pipeline, you must have: - A [Redis Cloud Pro database]({{< relref "/operate/rc/databases/create-database/create-pro-database-new" >}}) hosted on Amazon Web Services (AWS). This will be the target database. -- One supported source database that is publicly accessible or hosted on an AWS EC2 instance, AWS RDS, or AWS Aurora: +- One or more supported source databases that are publicly accessible or hosted on an AWS EC2 instance, AWS RDS, or AWS Aurora: | Database | Versions | AWS RDS Versions | |:---|:---|:---| @@ -132,7 +133,7 @@ Please be aware of the following limitations: - The target database can use TLS, but can not use mutual TLS. - If your source database is not publicly accessible, or if it is a MongoDB Atlas or Snowflake database, it must be hosted on AWS. - You must use a [custom encryption key on AWS](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html) to create the instance hosting the database. -- One source database can only be synced to one target database. +- Each pipeline has one target database shared by all of its sources. - If the source database is not publicly accessible, you must be able to set up AWS PrivateLink to connect your source database to your target database. RDI only works with AWS PrivateLink and not VPC Peering or other private connectivity options. - Mutual TLS is not supported for AWS RDS and AWS Aurora source databases. {{< /note >}} @@ -144,13 +145,17 @@ To get started fast with RDI on Redis Cloud, see the [RDI Cloud quick start]({{< To create a new data pipeline, you need to: 1. [Create a Data Integration workspace]({{}}) for your Pro subscription. -1. [Prepare your source database]({{}}) and any associated credentials. +1. [Prepare each source database]({{}}) and any associated credentials. 1. [Define the source connection and data pipeline]({{}}) by selecting which tables to sync. Once your data pipeline is defined, you can [view and edit]({{}}) it. For complete production setups, including SQL Server failover handling, see [Production use cases]({{}}). +## Billing and common questions + +See the [RDI Cloud FAQ]({{< relref "/operate/rc/rdi/faq" >}}) for billing examples, reset and flush behavior, and working with multiple sources. + ## Maintenance windows RDI Cloud maintenance follows the same subscription-wide maintenance window as your Redis Cloud Pro subscription. During a maintenance window, your data pipeline may experience brief interruptions as Redis applies updates. diff --git a/content/operate/rc/rdi/create-workspace.md b/content/operate/rc/rdi/create-workspace.md index 37c59e12e8..5185def009 100644 --- a/content/operate/rc/rdi/create-workspace.md +++ b/content/operate/rc/rdi/create-workspace.md @@ -21,6 +21,8 @@ Before you can create your first Data Integration pipeline for a Redis Cloud sub {{< embed-md "rc-rdi-create-rdi-workspace.md" >}} +{{The Add pipeline control is available while the workspace is being created.}} + ## View workspace status and details You can view your workspace in one of the following ways: @@ -28,11 +30,11 @@ You can view your workspace in one of the following ways: - From the Redis Cloud console, go to the **Data integration** page, or - From your subscription, select the **Data Integration** tab -{{The workspace section of the Data Integration tab for a database}} +{{Workspace pipeline list showing a streaming pipeline and its Sources column.}} -There, you'll see your workspace, as well as any pipeline that you've created in it. To see your workspace details, including the deployment CIDR and region information, select **More actions > Workspace details**. +There, you'll see your workspace and its pipelines. The **Sources** column lists each pipeline's source names and database types. A source with unfinished configuration is marked **Pending setup**. Open its draft to continue setup. To see your workspace details, including the deployment CIDR and region information, select **Workspace actions > Workspace details**. -{{The delete workspace button.}} +{{Workspace actions menu with Workspace details and Delete workspace.}} ## Delete workspace @@ -40,6 +42,6 @@ There, you'll see your workspace, as well as any pipeline that you've created in Make sure to [delete your data pipeline]({{}}) before deleting your workspace. {{< /warning >}} -To delete your workspace, select **More actions > Delete workspace** from your workspace. +To delete your workspace, select **Workspace actions > Delete workspace** from your workspace. -{{The delete workspace button.}} \ No newline at end of file +{{Workspace actions menu with Workspace details and Delete workspace.}} diff --git a/content/operate/rc/rdi/define.md b/content/operate/rc/rdi/define.md index 102d2714dd..fd8584882f 100644 --- a/content/operate/rc/rdi/define.md +++ b/content/operate/rc/rdi/define.md @@ -8,188 +8,164 @@ categories: - docs - operate - rc -description: Define the source connection and data pipeline. +description: Connect one or more source databases to a Redis Cloud target and define your data pipeline. hideListLinks: true weight: 4 --- -After you have [prepared your source database]({{}}) and connection information, and [created the workspace for your pipeline]({{}}), you can set up your new pipeline. +After you have [prepared each source database]({{}}) and [created a workspace]({{}}), you can create a pipeline. One pipeline can ingest data from several source databases into one Redis target. -In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Integration** tab. You'll see your subscription's workspace. Select **Add pipeline** to add a pipeline to the workspace. +In the [Redis Cloud console](https://cloud.redis.io/), open your target database's **Data Integration** tab and select **Add pipeline**. You can also open the workspace from the **Data Integration** page or your subscription's **Data Integration** tab. To continue an existing draft, open its actions menu and select **Resume pipeline setup**. -You can also go to the **Pipelines** page from the left-hand menu and select **Add pipeline** from your workspace. Or, you can go to the **Data Integration** tab of your subscription and select **Add pipeline** from your workspace. +{{The Add pipeline control is available while the workspace is being created.}} -{{The workspace section of the Data Integration tab for a database. Select Add pipeline to add a pipeline.}} +To create a pipeline: -If you've started to create a pipeline, you'll see a draft pipeline. Select **More actions > Resume pipeline setup** to continue with pipeline setup. +1. [**Settings**](#settings): Select the shared target, default data structure, and processor properties. +1. [**Add sources**](#pipeline-setup): Select source types and give each source a unique name. +1. [**Configure source**](#source-configuration): Configure and test connectivity and credentials for each source. +1. [**Select data**](#dataset): Select schemas, tables, and columns for each source. +1. [**Add transformations**](#transformations): Add jobs and assign them to sources. +1. [**Review & deploy**](#review-and-deploy): Review all sources and deploy the pipeline. -{{The workspace section of the Data Integration tab for a database with a draft pipeline. Select Resume pipeline setup to continue.}} +## Settings -Creating a pipeline is split into the following steps: +1. Select the target from the **Target database** list. All sources write to this database. -1. [**Pipeline setup**](#pipeline-setup): Defines the source database type and the target database. -2. [**Source configuration**](#source-configuration): Defines and tests the source database connectivity and credentials. -3. [**Dataset**](#dataset): Defines the data to import to your Redis database. -4. [**Transformations**](#transformations): Defines how records are stored in Redis. -5. [**Review & deploy**](#review-and-deploy): Shows your pipeline details and deploys it. + {{The target database list in pipeline Settings.}} -## Pipeline setup +1. Select **Hash** or **JSON** as the **Default data structure**. Transformation jobs can override how individual records are written. +1. If needed, configure **Processor properties**. These apply to the whole pipeline, not to an individual source. See the [processor configuration reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors-data-processing-configuration" >}}). -In the **Pipeline setup** step: + {{The processor advanced properties editor with key and value fields.}} -1. Select your source database type. The following database types are supported: - - MySQL - - mariaDB - - Oracle - - SQL Server - - PostgreSQL - - Snowflake _(Preview)_ - {{The select source database type list and source name field.}} +1. Select **Continue**. -1. Enter a name for your source database in the **Source name** field. This is a display label for the source database on Redis Cloud. It does not affect the pipeline's configuration. -1. Select the target Redis Cloud database from the **Target database** list. +RDI Cloud uses the Flink processor. - {{The target database list.}} +## Add sources {#pipeline-setup} -Select **Continue to source** to move to the **Source configuration** step. +1. Select a source database type: **MySQL**, **MariaDB**, **Oracle**, **SQL Server**, **PostgreSQL**, **MongoDB**, or **Snowflake** (Preview). +1. Enter a unique **Source name**, for example `inventory-mysql`. Use up to 22 characters: lowercase letters, numbers, and hyphens. Start with a lowercase letter and end with a letter or number. The names `rdi`, `source`, `target`, and `api` cannot be used for sources. +1. To include another source, select **Add source** in the **Sources** list and repeat these steps. You can combine different database types or add several sources of the same type. +1. Select **Continue**. -{{The Continue to source button.}} +The source name identifies the source in the pipeline configuration and transformation jobs; it is not just a display label. These naming rules were introduced with RDI 2.0.0. Sources that existed before the upgrade retain their names, even if they do not meet these rules. -## Source configuration +Select a source in the **Sources** list to configure its connection and dataset. Each source has separate progress indicators for its details, configuration, and data selection. -During the **Source configuration** step, you'll share the connectivity information and credentials you created when you prepared your source database. +## Configure source {#source-configuration} -This step is separated into three expandable sections: +Repeat the following configuration for every source. The **Configure source** step has three expandable sections: -1. [Source connectivity](#source-connectivity) defines the connectivity method for your source database. -2. [Secrets](#secrets) defines the authentication needed for your source database. -3. [Source configuration](#source-configuration-section) defines the source-specific settings for your database connection. +1. [Source connectivity](#source-connectivity) +1. [Secrets](#secrets) +1. [Source configuration](#source-configuration-section) -When you've completed all three sections, select **Test source** to test Redis Cloud's connection with the source database. After the test completes, select **Continue to dataset** to move to the [**Dataset**](#dataset) step. - -{{The Continue to dataset button.}} +Complete the sections and select **Test source**. Correct any reported errors before continuing. Select each other source and test its configuration, then select **Continue** to open **Select data**. ### Source connectivity -Select whether you want your pipeline to connect to your database using **AWS Private Link** or using the **Public endpoint**. +Choose **AWS Private Link** or **Public Endpoint** for the selected source, according to its [connectivity requirements]({{}}). -- If your pipeline uses AWS PrivateLink, enter the **Private Link service name** of the [PrivateLink connected to your source database]({{< relref "/operate/rc/rdi/setup#set-up-connectivity" >}}). +- For **AWS Private Link**, enter the **Private Link service name** from your [endpoint service]({{< relref "/operate/rc/rdi/setup#set-up-connectivity" >}}). Select **Connect to Private Link** and wait for connectivity to complete. If the connection fails, check the service name and its allowed principal. - {{The Source database connectivity section for PrivateLink connection.}} + {{AWS Private Link connectivity with the service name and Connect to Private Link control.}} - Select **Connect to Private Link** to test your Private Link connectivity. This will take a few minutes. - - If Redis Cloud can't find your PrivateLink connection, make sure that the PrivateLink service name is correct and that Redis Cloud is listed as an Allowed Principal for your VPC. See [Set up connectivity]({{}}) for more info. +- For **Public Endpoint**, enter the source IP address or hostname. Add the Redis Cloud outbound IP address shown in the console to your source database's allowlist. -- If your pipeline uses the public endpoint, enter the source IP address or hostname in the **Source IP address / Hostname** field. + {{Public endpoint connectivity with source hostname and Redis Cloud outbound IP addresses.}} - {{The Source database connectivity section for Public endpoint connection.}} +Configure connectivity for each source separately. Sources in the same pipeline can use different connectivity methods. ### Secrets -{{The Secrets section.}} +Enter the Amazon Resource Name (ARN) of the selected source's [database credentials secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) in **Credentials Secret ARN**. + +{{The Credentials Secret ARN field, transit security options, and Validate control.}} + +For Snowflake, select the authentication method matching your secret: -Enter the ARN of your [database credentials secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) in the **Credentials secret ARN** field. +- **Password**: The credentials secret contains `username` and `password`. +- **Key-pair**: The credentials secret contains `username`; any `password` is ignored. Provide the private key secret ARN. The private key is stored in plain text PEM format. -For Snowflake source databases, select the authentication method that matches the [credentials secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) you created: +Under **Transit security**, select the mode required by your source: -- **Password**: The credentials secret must contain both `username` and `password`. -- **Key-pair**: The credentials secret must contain the `username` (if a `password` is present, it is ignored). Enter the ARN of your [client key secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) in the **Private Key Secret ARN** field. This secret must contain the private key in plain text PEM format. +- **TLS** (Transport Layer Security): Provide the CA certificate secret ARN when your source requires it. -Under **Transit security**: + {{TLS transit security with the CA Certificate Secret ARN field.}} -- If your database requires TLS, select **TLS**. Enter the ARN of your [CA certificate secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) in the **CA Certificate Secret ARN** field. - {{The Source database connectivity section, with TLS selected and the CA Cert Secret ARN field.}} -- If your database requires mTLS, select **mTLS**. - {{The Source database connectivity section, with mTLS selected and the Client Certificate Secret ARN and Client Key Secret ARN fields.}} +- **mTLS** (mutual TLS): Provide the CA certificate, client certificate, and client private key secret ARNs. Also provide the client key passphrase secret ARN if the key is encrypted. - Enter the following secrets in the fields: - - **CA Certificate Secret ARN**: [CA certificate secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) - - **Client Certificate Secret ARN**: [Client certificate secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) - - **Client Private Key Secret ARN**: [Client key secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) - - **Password secret ARN for secret store** (_Optional_): [Client key passphrase secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) + {{mTLS transit security with certificate, private key, and optional password secret ARN fields.}} -Select **Validate** to check that Redis Cloud can access your secrets. +Select **Validate** to check access to the selected source's secrets. Repeat this for each source. The AWS secret contents and permissions are described in [Share source database credentials]({{< relref "/operate/rc/rdi/setup#share-source-database-credentials" >}}). ### Source configuration {#source-configuration-section} -In this section, you'll enter your database details. This depends on your database type, and can include: +Enter the selected source's database settings. The fields depend on the database type and can include: -- **Port**: The database's port -- **Database(s)**: Your database's name, or the root database *(PostgreSQL, Oracle only)*, or a comma-separated list of one or more databases you want to connect to *(SQL Server and MongoDB only)* -- **Database Server ID**: Unique ID for the replication client. Enter a number that is not used by any existing replication clients *(mySQL and mariaDB only)* -- **PDB**: Name of the Oracle pluggable database *(Oracle only)* -- **Connection string**: The connection information for your database *(MongoDB only)* +- **Port**: The source database port. +- **Database(s)**: The database name, or a comma-separated list for source types that support multiple databases. +- **Database Server ID**: A replication client ID for MySQL or MariaDB. Use a different ID for each collector connecting to the same database server. +- **PDB**: The Oracle pluggable database. +- **Connection string**: The MongoDB connection information. -Under **Collector properties**, Select **Edit advanced properties** to configure additional optional properties for your pipeline. +{{Source-specific database, port, and collector properties.}} -{{The collector properties section.}} +Use **Collector properties** for additional source and sink settings. These settings apply to the selected source. See the [collector source properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsource-advanced-source-settings" >}}) and [collector sink properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsink-rdi-collector-stream-writer-configuration" >}}). -{{The advanced properties section.}} +{{The Edit advanced properties control under Collector properties.}} -You can add collector source properties in the **Collector source properties** section and collector sink properties in the **Collector sink properties** section. See the RDI configuration file reference for all available [collector source properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsource" >}}) and [collector sink properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsink" >}}). Select **Save properties** to return to Source configuration. +{{The advanced properties dialog with separate collector source and sink properties.}} -## Dataset +## Select data {#dataset} -In this step, you'll select the data that you want to import and synchronize with your primary database. +Select each source in the **Sources** list and choose the data to ingest from that source. {{< warning >}} -Do not write data directly to the target database outside of RDI. Writing to the target database from other sources can cause transformation failures and data inconsistencies. If you need to reset the pipeline and resync from the source, any data written to the target outside of RDI will be lost. +Do not write data directly to keys managed by RDI. Changes from another application can cause transformation failures or data inconsistencies, and RDI can overwrite them. A pipeline reset does not flush the target database. **Flush target database** is a separate action that deletes all target data, including data written outside RDI. See [Data recovery]({{< relref "/operate/rc/rdi/faq#data-recovery" >}}). {{< /warning >}} -  - -{{< note >}} -This step may change depending on your source database. -{{< /note >}} - -{{The dataset step.}} -1. In the **Schemas** section, select the schema(s) you want to migrate to the target database from the list. +1. Select a schema in **Schemas** to see its tables. +1. Select the tables to ingest in **Tables**. - {{The dataset step with a schema selected.}} + {{Selecting a schema shows its tables for ingestion.}} -1. When you select a schema, you will see its tables in the **Tables** section. Redis Cloud will automatically select all tables for import. You can de-select any columns you do not wish to import to your Redis database. +1. Select a table to see its **Columns**, then select the columns to ingest. -1. Select a table to view its columns in the **Columns** section. You can de-select any columns you do not wish to import. + {{Selecting a table shows its columns and the columns selected for ingestion.}} - {{The columns section, with a few columns selected from one table}} +1. If a table has no unique key, use the key control next to the column that identifies each record. Review any missing-key warnings before continuing. - If any tables are missing a unique key, a warning will appear in the **Data modeling** section. Select **Show affected** to filter the **Tables** section to the tables without a unique key. - - {{The dataset step filtered to show tables that are missing a unique key.}} + {{The missing unique key warning with the affected table and its columns.}} - For these tables, select the key icon next to the column that defines a unique key. + {{The key control beside the column used to identify a record.}} - {{The unique key icon.}} +1. Repeat for the other sources, then select **Continue**. -Select **Continue to transformations** to move to the **Transformations** step. +The available schema, table, and column controls depend on the source type. Each source keeps its own selection, including when two sources contain tables with the same name. -{{The Continue to dataset button.}} +## Add transformations {#transformations} -## Transformations +Transformation jobs are optional. Without a matching job, RDI writes records using the pipeline's default data structure. -In this step, you'll choose how the pipeline will store your data in Redis. +1. Select **Upload jobs** to upload the [transformation job files]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) needed for your selected tables. +1. Check the **Source name** assignment for each job. In a multi-source pipeline, `source.server_name` identifies the source the job reads from. Select the source name you chose during setup. +1. Review each job's validation status and correct errors. -{{The Transformations step.}} + {{Example transformation jobs with their source assignments and validation status.}} -1. Select how your records will be stored in Redis. You can choose **Hash** or **JSON**. +1. Select **Continue to review & deploy**. -1. Under **Transformation jobs**, you can supply one or more [transformation job files]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) that specify how you want to transform the captured data before writing it to the target. Select **Upload jobs** to upload your job files. When you upload job files, Redis Cloud will validate the job files to check for errors. +For the Flink processor, source matchers can use lists or entries prefixed with `regex:` to select several tables. Jobs must not overlap on the same table. See [Transformation examples]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}). -1. Select **Edit advanced properties** to add any processor properties to control how the data is processed. +## Review and deploy {#review-and-deploy} - {{The Advanced Processor properties.}} +Review every source's connection, selected data, and transformation jobs, together with the shared target and settings. Use the source actions to return to a section that needs changes. - See the [RDI configuration file reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors" >}}) for all available processor properties. - -## Review and deploy - -Review the tables you selected in the **Review and deploy** step. If everything looks correct, select **Deploy pipeline** to start ingesting data from your source database. +Select **Deploy pipeline** to start the pipeline. Each source performs its initial snapshot and then captures ongoing changes. Snapshot duration depends on the amount of selected data and source performance. {{The Deploy pipeline button.}} -At this point, the data pipeline will ingest data from the source database to your target Redis database. This process will take time, especially if you have a lot of records in your source database. - -After this initial sync is complete, the data pipeline enters the *change streaming* phase, where changes are captured as they happen. Changes in the source database are added to the target within a few seconds of capture. - -You can view the status of your data pipeline in the **Data pipeline** tab of your database. See [View and edit data pipeline]({{}}) to learn more. +Open the pipeline's [Dashboard and Metrics tabs]({{}}) to follow progress for each source. diff --git a/content/operate/rc/rdi/faq.md b/content/operate/rc/rdi/faq.md new file mode 100644 index 0000000000..ada3e8513a --- /dev/null +++ b/content/operate/rc/rdi/faq.md @@ -0,0 +1,111 @@ +--- +Title: RDI Cloud FAQ +alwaysopen: false +categories: +- docs +- operate +- rc +description: Answers to common questions about RDI on Redis Cloud. +hideListLinks: true +weight: 6 +--- + +## Data recovery + +### What happens when I reset the pipeline? {#reset-pipeline} + +A pipeline reset clears the internal RDI state for all sources, including their saved positions. When the pipeline runs, RDI takes a new snapshot of the selected data from every source, applies the current transformations, and then resumes streaming changes. If you reset a stopped pipeline, it remains stopped until you start it. + +A reset does **not** flush the target Redis database. Records already in the target remain until RDI overwrites or deletes them through normal processing. Keys that are no longer produced by the current dataset or transformations can remain in the target after a reset. For example, changing a transformation's key prefix and resetting creates keys with the new prefix without deleting keys with the old prefix. + +See [Reset data pipeline]({{< relref "/operate/rc/rdi/view-edit#reset-data-pipeline" >}}) for the steps. + +### What happens when I flush the target database? + +**Flush target database** permanently deletes all data from the shared target, including data from every source and data written outside RDI. Stop the pipeline before flushing. The source databases are not changed. + +Flushing does not clear RDI's saved source positions. If you only start the pipeline afterwards, it resumes from those positions. It does not automatically reload records that have not changed in the source. + +See [Flush the target database]({{< relref "/operate/rc/rdi/view-edit#flush-the-target-database" >}}). + +### How do I reload data after a flush? {#reload-after-flush} + +1. Wait for the flush to finish. +1. [Reset the pipeline]({{< relref "/operate/rc/rdi/view-edit#reset-data-pipeline" >}}) while it is stopped, and wait for the reset to finish. +1. [Start the pipeline]({{< relref "/operate/rc/rdi/view-edit#stop-and-restart-data-pipeline" >}}) to take new snapshots of the selected data from all sources. +1. Check each source's initial sync progress and record counts on the **Dashboard** and **Metrics** tabs. Wait for initial sync to finish before relying on the target as a complete copy of the selected data. + +RDI reloads data available in the source databases using the current dataset and transformation settings. It cannot restore data that existed only in the target. + +## Multiple sources + +### What happens when I reset one source? {#reset-one-source} + +RDI clears the selected source's internal state and takes a new snapshot of its selected data. The whole pipeline, including the other sources and the processor, restarts during this operation. Other sources keep their saved positions and resume streaming without a new snapshot. + +All records already in the target remain, including records from the reset source. The new snapshot can overwrite that source's records. Resetting a source does not selectively delete its target data. + +See [Reset a source]({{< relref "/operate/rc/rdi/view-edit#reset-source" >}}). + +### Can I flush data for a single source? {#flush-one-source} + +Currently, RDI cannot flush target data for a single source. Connect to the target Redis database and selectively delete the records you want to remove. Identify them from your key naming and transformation rules, and check that other sources do not write to the same keys. Do not use **Flush target database** for this purpose: it deletes data for all sources. + +Stop the affected source before deleting its records and allow its pending records to finish processing. Starting the source resumes ingestion, and later changes can recreate records. To reload all its selected data, [reset that source]({{< relref "/operate/rc/rdi/view-edit#reset-source" >}}). + +### Does deleting a source delete its target data? + +No. Deleting a source removes its pipeline configuration and internal RDI state. Records it already wrote to the target remain. You must remove or reassign transformation jobs that refer to the source before deleting it. See [Remove a source]({{< relref "/operate/rc/rdi/view-edit#remove-source" >}}). + +## Upgrades and maintenance + +### What happens during an RDI Cloud upgrade? + +Redis manages RDI upgrades in Redis Cloud. Maintenance follows your Redis Cloud Pro subscription's [maintenance window]({{< relref "/operate/rc/rdi#maintenance-windows" >}}). + +During an upgrade, monitoring may be temporarily unavailable, and ingestion pauses while the pipeline components restart. A streaming pipeline then resumes from its saved state and processes the changes accumulated during the interruption. + +A routine upgrade does not flush the target database or require you to reset the pipeline. Existing target records remain available, but they may temporarily lag behind the source data. + +## Billing + +### How is RDI usage calculated? + +RDI usage is measured in RDI processing units (RPUs). The usage consists of: + +| Component | RPUs | +| :-- | --: | +| Workspace with a deployed pipeline | 1 | +| Each running source collector | 2 | +| Each running processor replica | 2 | + +Usage is measured every minute and billed using the peak usage in each hour. Adding sources or processor replicas increases usage. Initial sync and streaming use the same RPU rate. The price per RPU depends on your region and pricing plan. Contact your Redis account team for the applicable rate. + +For example, one workspace with one source and one processor replica uses `1 + 2 + 2 = 5 RPUs`. With two sources and one processor replica, it uses `1 + 4 + 2 = 7 RPUs`. Adding a second processor replica increases that to `1 + 4 + 4 = 9 RPUs`. + +These examples cover RDI usage. Target Redis database and applicable network charges are separate. + +### How does stopping sources or the pipeline affect billing? {#stopping-and-billing} + +Stopping a source removes its collector usage, but the processor and workspace remain billable. Stopping every source individually still leaves the processor running. Stopping the whole pipeline also stops the processor, but the workspace charge continues while the pipeline remains deployed. + +The following examples assume one workspace, two sources, and one processor replica. Each state must apply for a full billing hour after the stop has completed: + +| State | Workspace RPUs | Collector RPUs | Processor RPUs | Total RPUs | +| :-- | --: | --: | --: | --: | +| Both sources running | 1 | 4 | 2 | 7 | +| One source stopped | 1 | 2 | 2 | 5 | +| Both sources stopped individually; pipeline still running | 1 | 0 | 2 | 3 | +| Whole pipeline stopped | 1 | 0 | 0 | 1 | + +A brief stop might not reduce the charge for that hour. For example, if usage reaches 7 RPUs before you stop one source, the hourly peak remains 7 RPUs. A later full hour with that source stopped is billed at 5 RPUs, assuming the other resources stay the same. + +### Am I charged if no source records change? + +Yes. RDI usage is based on the running collectors and processor replicas, not the number of records processed. An idle streaming pipeline still uses resources. + +### When does RDI billing start and stop? + +Creating a workspace or saving a setup draft does not start RDI usage billing. Billing starts when a pipeline is deployed. Stopping the pipeline reduces usage to the workspace charge. Deleting the deployed pipeline ends new RDI usage when no deployed pipelines remain in the workspace; usage already recorded for the hour can still be billed. + +Delete an unused pipeline and then [delete its workspace]({{< relref "/operate/rc/rdi/create-workspace#delete-workspace" >}}) when you no longer need RDI. This does not delete the target Redis database or stop its separate charges. diff --git a/content/operate/rc/rdi/quick-start.md b/content/operate/rc/rdi/quick-start.md index 944f26b0fb..159e01ea79 100644 --- a/content/operate/rc/rdi/quick-start.md +++ b/content/operate/rc/rdi/quick-start.md @@ -40,22 +40,25 @@ Before you can create your first Data Integration pipeline for a Redis Cloud sub ## Get required ARNs -1. On the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Integration** tab. -1. Select **Add pipeline**. - {{The workspace section of the Data Integration tab for a database. Select Add pipeline to add a pipeline.}} -1. Select **PostgreSQL** as the source database type. - {{The select source database type list.}} -1. Enter a name for your source database in the **Source name** field. This is a display label for the source database on Redis Cloud. It does not affect the pipeline's configuration. -1. Select **Continue to source** to move to the **Source configuration** step. +This example creates one PostgreSQL source. You can [add more sources]({{< relref "/operate/rc/rdi/view-edit#add-source" >}}) after the pipeline is running. - {{The select source database type list.}} -1. Under **Source connectivity**, save the provided ARN. This will be the `redis_privatelink_arn` you will need later. +1. On the [Redis Cloud console](https://cloud.redis.io/), open your target database's **Data Integration** tab and select **Add pipeline**. - {{The setup connectivity section containing the private link ARN.}} + {{The Add pipeline control is available while the workspace is being created.}} -1. Under **Secrets**, save the provided ARN. This will be the `redis_secrets_arn` you will need later. +1. In **Settings**, select your target database and choose **Hash** or **JSON** as the default data structure, then select **Continue**. +1. In **Add sources**, select **PostgreSQL**. +1. Enter a **Source name** of your choice, for example `inventory-postgres`. +1. Select **Continue** to open **Configure source**. +1. Under **Source connectivity**, copy the **Role ARN**. Use it as `redis_privatelink_arn` in the Terraform configuration. - {{The setup connectivity section containing the credentials ARN.}} + {{The source connectivity Role ARN and availability zones.}} + +1. Under **Secrets**, copy the **Role ARN**. Use it as `redis_secrets_arn` in the Terraform configuration. + + {{The Role ARN in the Secrets section.}} + +1. Select **Save & exit** while you create the source resources. ## Create the source database and network resources @@ -71,8 +74,8 @@ Before you can create your first Data Integration pipeline for a Redis Cloud sub - `azs`: The availability zone IDs where your Redis Cloud database is deployed. - `port`: The port number for the new PostgreSQL source database. - `name`: A prefix for all of the created AWS resources. - - `redis_secrets_arn`: The source database credentials and certificates ARN from the Redis Cloud console. - - `redis_privatelink_arn`: The PrivateLink ARN from the Redis Cloud console. + - `redis_secrets_arn`: The role ARN from **Secrets** in the Redis Cloud console. + - `redis_privatelink_arn`: The role ARN from **Source connectivity** in the Redis Cloud console. 1. To view the configuration, run: @@ -108,60 +111,40 @@ If you lose any outputs, run `terraform output` to view them again. ## Resume pipeline setup -1. Return to the [Redis Cloud console](https://cloud.redis.io/). Go to your target database and select the **Data Integration** tab. -1. You'll see a draft pipeline in the workspace you created. Select **More actions > Resume pipeline setup** to continue with pipeline setup. - - {{The workspace section of the Data Integration tab for a database with a draft pipeline. Select Resume pipeline setup to continue.}} - -1. Continue to the **Source configuration** step. - -1. In the **Source connectivity** section, enter the `vpc_endpoint_service_name` output in the **PrivateLink service name** field. - - {{The Source database connectivity section for PrivateLink connection.}} - -1. Select **Connect to Private Link** to test your Private Link connectivity. This will take a few minutes, but you can continue while it's testing. - -1. In the **Secrets** section, enter the `secret_arn` output in the **Credentials secret ARN** field. +1. Return to your database's **Data Integration** tab in the [Redis Cloud console](https://cloud.redis.io/). +1. Open the draft pipeline's actions menu and select **Resume pipeline setup**. +1. Open **Configure source** and select your PostgreSQL source in the **Sources** list. +1. Under **Source connectivity**, enter the Terraform `vpc_endpoint_service_name` output as the **Private Link service name**. +1. Select **Connect to Private Link** and wait for connectivity to complete. - {{The Secrets section.}} + {{AWS Private Link connectivity with the service name and Connect to Private Link control.}} -1. Select **Validate** to check that Redis Cloud can access your secrets. +1. Under **Secrets**, enter the `secret_arn` output as **Credentials Secret ARN**. +1. Select **Validate** to check access to the secret. -1. In the **Source configuration** section, enter the terraform outputs in the following fields. - - **Database**: `database` - - **Port**: `port` + {{The Credentials Secret ARN field, transit security options, and Validate control.}} -1. Select **Test source** to test Redis Cloud's connection with the source database. After the test completes, select **Continue to dataset**. +1. Under **Source configuration**, enter the `database` and `port` Terraform outputs in the matching fields. +1. Select **Test source** and correct any validation errors, then select **Continue**. +1. In **Select data**, select the schemas, tables, and columns to ingest. Review the selected key for each table. - {{The Continue to dataset button.}} + {{Selecting a schema shows its tables for ingestion.}} -1. In the **Schemas** section, select the schema(s) you want to migrate to the target database from the list. + {{Selecting a table shows its columns and the columns selected for ingestion.}} - {{The dataset step with a schema selected.}} - -1. When you select a schema, you will see its tables in the **Tables** section. Redis Cloud will automatically select all tables for import. You can de-select any columns you do not wish to import to your Redis database. - -1. Select a table to view its columns in the **Columns** section. You can de-select any columns you do not wish to import. - - {{The columns section, with a few columns selected from one table}} - -1. Select **Continue to transformations** to move to the **Transformations** step. - - {{The Continue to dataset button.}} - -1. Select how your records will be stored in Redis. You can choose **Hash** or **JSON**. - - {{The Transformations step.}} - -1. Review the tables you selected in the **Review and deploy** step. If everything looks correct, select **Deploy pipeline** to start ingesting data from your source database. +1. Select **Continue** to open **Add transformations**. For this example, you can keep the default mapping without adding jobs. +1. Select **Continue to review & deploy**. +1. Review the source and target, then select **Deploy pipeline**. {{The Deploy pipeline button.}} -At this point, the data pipeline will ingest data from the source database to your target Redis database. This process will take time, especially if you have a lot of records in your source database. +The source first imports its selected data, then captures ongoing changes. Open the pipeline's **Dashboard** or **Metrics** tab to follow its progress. + +The following example shows the metrics for one selected source in a pipeline with multiple sources. -After this initial sync is complete, the data pipeline enters the *change streaming* phase, where changes are captured as they happen. Changes in the source database are added to the target within a few seconds of capture. +{{Metrics for one selected source in a pipeline, including snapshot progress and per-table record counts.}} -You can view the status of your data pipeline in the **Data pipeline** tab of your database. See [View and edit data pipeline]({{}}) to learn more. +See [View and edit data pipeline]({{}}) for source actions, dataset changes, and monitoring. ## Delete sample resources diff --git a/content/operate/rc/rdi/setup.md b/content/operate/rc/rdi/setup.md index a743d92276..5c1a1328b3 100644 --- a/content/operate/rc/rdi/setup.md +++ b/content/operate/rc/rdi/setup.md @@ -15,7 +15,7 @@ weight: 3 ## Prepare source database -Before using the pipeline, you must first prepare your source database to use the Debezium connector for change data capture (CDC). See [Prerequisites]({{}}) to find a list of supported source databases and database versions. +Prepare every source database before adding it to a pipeline. Each source needs its own change data capture (CDC) configuration, connectivity, and credentials. See [Prerequisites]({{}}) for a list of supported source databases and database versions. See [Prepare source databases]({{}}) to find steps for your database type: - [MongoDB Atlas]({{}}) @@ -37,32 +37,24 @@ See the [RDI architecture overview]({{< relref "/integrate/redis-data-integratio Before you can set up your source connectivity and secrets, you need the AWS Account ID for your Redis Cloud cluster so that you can give it access to your connectivity and secrets. -1. On the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Integration** tab. -1. Select **Add pipeline**. - {{The workspace section of the Data Integration tab for a database. Select Add pipeline to add a pipeline.}} -1. Select your source database type. The following database types are supported: - - MySQL - - mariaDB - - Oracle - - SQL Server - - PostgreSQL - - MongoDB - - Snowflake - {{The select source database type list.}} -1. Enter a name for your source database in the **Source name** field. This is a display label for the source database on Redis Cloud. It does not affect the pipeline's configuration. -1. Select **Continue to source** to move to the **Source configuration** step. +1. On the [Redis Cloud console](https://cloud.redis.io/), open your target database's **Data Integration** tab. +1. Select **Add pipeline**, or resume an existing draft. To add a source to a running pipeline, select **Add source** on its **Dashboard**. - {{The select source database type list.}} + {{The Add pipeline control is available while the workspace is being created.}} -1. Under **Source connectivity**, save the provided ARN and extract the AWS account ID for the account associated with your Redis Cloud cluster from it. +1. For a new pipeline, complete **Settings**, including the target database, and select **Continue**. +1. In **Add sources**, select the source type and enter a unique **Source name**. This name identifies the source in the pipeline configuration and transformation jobs. See [Add sources]({{< relref "/operate/rc/rdi/define#pipeline-setup" >}}) for naming rules. +1. Select **Continue** to open **Configure source**. +1. Under **Source connectivity**, copy the **Role ARN** and extract its AWS account ID. - {{The Private Link Role ARN and availability zones.}} + {{The source connectivity Role ARN and availability zones.}} - The AWS account ID is the string of numbers after `arn:aws:iam::` in the ARN. For example, if the ARN is `arn:aws:iam::123456789012:role/redis-data-pipeline`, the AWS account ID is `123456789012`. + The account ID is the number after `arn:aws:iam::`. For example, `arn:aws:iam::123456789012:role/redis-data-pipeline` contains account ID `123456789012`. -1. If your source database is accessible via the public endpoint and you want to use public connectivity for your data pipeline, select **Public endpoint** and save the **Redis Cloud outbound IP address** to add to your source database's allow list. +1. For a source using **Public Endpoint**, also copy the Redis Cloud outbound IP address to add to the source database's allowlist. +1. Select **Save & exit** to return to setup after preparing connectivity and secrets. -Select **Save & exit** to exit pipeline setup. You'll come back here when you [define your source connection and data pipeline]({{}}). +Repeat the preparation for each source. Keep track of which endpoint service and secrets belong to each source; configuring one source does not configure the others. ## Set up AWS Private Link connectivity {#set-up-connectivity} @@ -366,7 +358,7 @@ Copy the connection string from the **Private Endpoint** connection method only. ## Share source database credentials -You need to share your source database credentials and certificates in an Amazon secret with Redis Cloud so that the pipeline can connect to your database. +Share the credentials and certificates for each source through AWS Secrets Manager. Enter the matching secret ARNs when you configure that source in the console. To do this, you need to: 1. [Create an encryption key](#create-encryption-key) using AWS Key Management Service with the right permissions. @@ -493,6 +485,4 @@ In the [AWS Management Console](https://console.aws.amazon.com/), use the **Serv ## Next steps -After you have set up your source database and prepared connectivity and credentials, select **Define source database** to [define your source connection and data pipeline]({{}}). - -{{The define source database button.}} +After you have prepared connectivity and credentials for each source, resume your pipeline draft from the workspace and complete **Configure source**. Continue with [Create data pipeline]({{}}). diff --git a/content/operate/rc/rdi/view-edit.md b/content/operate/rc/rdi/view-edit.md index 944c7d04ed..da4c2c9319 100644 --- a/content/operate/rc/rdi/view-edit.md +++ b/content/operate/rc/rdi/view-edit.md @@ -13,60 +13,98 @@ hideListLinks: true weight: 4 --- -To view or manage your data pipeline, select it from your workspace from the **Data Integration** page or from the **Data Integration** tab in your subscription or database. +To manage a pipeline, select it from your workspace on the **Data Integration** page or from the **Data Integration** tab of your subscription or database. The pipeline page has the following tabs: - [Dashboard](#dashboard) - [Metrics](#metrics) +- [Settings](#settings) - [Configuration](#configuration) - [Dataset](#dataset) - [Transformations](#transformations) -The following sections describe each of these tabs, as well as related diagnostic views and actions. +Use the **Sources** list in **Metrics**, **Configuration**, and **Dataset** to select the source you want to inspect. **Settings** applies to the whole pipeline. **Transformations** lists jobs with their source assignments. ## Dashboard -The **Dashboard** tab shows an overview and high-level statistics for the data pipeline. +The **Dashboard** shows pipeline health, processor and target status, throughput, pending records, record health, and processor load. It also shows the number of sources, selected tables, and transformation jobs. -{{The Dashboard tab for a pipeline.}} +{{Pipeline dashboard with throughput, pending records, record health, and processor load.}} -- **Pipeline health**: The status of the data pipeline. Possible statuses include: - | Status | Description | - |--------|-------------| - | **Streaming** | The data pipeline is capturing new changes from the source database as they happen. Changes in the source database are added to the target database within a few seconds. | - | **Stopped**| The data pipeline has been [stopped](#stop-and-restart-data-pipeline). | - | **Error** | There is an error in the data pipeline. [Reset the pipeline](#reset-data-pipeline) and contact support if the issue persists. | -- **Total ingested**: Total number of records ingested from the source database. -- **Pending**: Total number of records that are being processed and have not been inserted into the target database. -- **Record health**: The number of records that were rejected from the database, and the number of records that were filtered from being inserted into the database. If there are rejected records, select the rejected count to open the [Rejected records](#rejected-records) view. -- **Data latency**: How long it takes for a new record to be ingested from the source database. +Each source has a card showing its database type, name, activity, and status. Expand a card for more details. Open the source's **More actions** menu to view its configuration, edit its dataset or transformations, or manage the source. -### Change target database +{{Three source cards: one PostgreSQL source and two MySQL sources, all streaming.}} + +{{Source actions for configuration, dataset, transformations, stop, reset, and deletion.}} + +A pipeline can be streaming, stopped, or in an error or transitional state. If there is a problem, you can use the source cards and **Metrics** tab to identify the affected source before taking remedial action. Pipeline health and individual source status can differ, for example when one source is stopped. + +### Add a source {#add-source} + +1. On **Dashboard**, select **Add source**. +1. In **Add sources**, select **Add source** in the **Sources** list, choose the new database type, and enter a unique source name. Existing sources remain listed with their connection details read-only. +1. Complete **Configure source**, **Select data**, and **Add transformations** for the new source. See [Create data pipeline]({{< relref "/operate/rc/rdi/define" >}}) for the configuration steps. +1. Review the changes in **Review & deploy** and select **Deploy pipeline**. + +{{Add-source wizard showing existing sources and the Add source control.}} + +Adding a source keeps the existing pipeline's shared target and settings, so the wizard starts at **Add sources**. A source with an incomplete setup appears with **Pending setup** in the workspace. You can resume an existing draft setup at any time to complete it. If you remove a source, wait for the removal process to finish before adding another source. + +When extending an older single-source pipeline, preserve the original source name and its existing job assignments. You do not need to recreate the pipeline to add a source. + +### Stop and start a source {#stop-and-start-source} + +1. On **Dashboard**, open the source card's **More actions** menu. +1. Select **Stop source**, then confirm with **Stop source**. + +Stopping one source pauses data capture from that source without stopping the other sources. Starting it again resumes from its saved position. To resume a source, open its **More actions** menu and select **Start source**, then confirm with **Start source**. A source processes data only while the pipeline is running. See [How does stopping sources affect billing?]({{< relref "/operate/rc/rdi/faq#stopping-and-billing" >}}). + +### Reset a source {#reset-source} -You can change the target database for your pipeline if the new target database is in the same subscription of the current target database. To do this: +Resetting one source starts a new snapshot and reprocesses its selected data. -1. From the **Dashboard** tab of your pipeline, select **More Actions > Change pipeline target** next to the target database of the pipeline. +1. On **Dashboard**, open the source card's **More actions** menu. +1. Select **Reset source**. +1. Review the effect and confirm with **Reset source**. -{{The More Actions menu for the target database in the Pipeline's dashboard tab.}} +{{Reset source confirmation explaining that other sources retain their data and the pipeline temporarily stops.}} -2. Select the new target database from the list. +The reset clears that source's internal RDI streams, offsets, schema history, rejected records, and processing counters. The whole pipeline and all its sources restart during the reset. Other sources keep their saved positions and resume streaming. All records already in the shared target database remain, including those from the reset source. The new snapshot can overwrite records for that source. See [What happens when I reset one source?]({{< relref "/operate/rc/rdi/faq#reset-one-source" >}}). -{{The Change pipeline target database menu.}} +### Remove a source {#remove-source} -3. Select **Change target** to continue. +Before deleting a source, remove or reassign transformation jobs that refer to it and apply the changes. RDI rejects deletion while a transformation job still refers to the source. -Changing the target database restarts the pipeline, but it does not automatically re-ingest all data into the new target database. If you want the existing data re-ingested after the change, manually [reset the pipeline](#reset-data-pipeline). +1. On **Dashboard**, open the source card's **More actions** menu. +1. Select **Delete source**. +1. Review the confirmation and select **Delete source**. + +{{Delete source confirmation for the selected PostgreSQL source.}} + +Deleting a source removes its data selection and internal RDI state, including streams, offsets, schema history, rejected records, and processing counters. Records already written to the target Redis database remain there. Other sources retain their data. The whole pipeline stops while RDI cleans up the removed source and starts again afterwards. + +Source deletion cannot be undone. To remove its records from the target, see [Can I flush data for a single source?]({{< relref "/operate/rc/rdi/faq#flush-one-source" >}}). + +### Change target database + +You can change the target to another database in the same subscription. + +1. On **Dashboard**, open the target card's **More actions** menu and select **Change pipeline target**. +1. Select the new target database. +1. Confirm with **Change target**. + +All sources share the new target. Changing the target restarts the pipeline but does not automatically re-ingest existing records. [Reset the pipeline](#reset-data-pipeline) if you need to copy the existing source data to the new target. ## Metrics -The **Metrics** tab shows the following metrics for each data stream: +Select a source in the **Sources** list to see its connection status, snapshot progress, queue usage, errors, and table-level record counts. Available collector diagnostics depend on the source type. Expand **Diagnostics** to see additional metrics for the selected source. -{{The Metrics tab for a pipeline.}} +{{Metrics for the selected PostgreSQL source, including collector diagnostics and table-level counts.}} | Metric | Description | |--------|-------------| -| **Name** | Name of the data stream. Each stream corresponds to a table from the source database. | +| **Source table** | Name of the data stream. Each stream corresponds to a table from the source database. | | **Total** | Total number of records that arrived from the source table. | | **Pending** | Number of records from the source table that are waiting to be processed. | | **Inserted** | Number of new records from the source table that have been written to the target database. | @@ -77,9 +115,7 @@ The **Metrics** tab shows the following metrics for each data stream: ### View metrics endpoints -You can use [Prometheus and Grafana]({{< relref "/integrate/prometheus-with-redis-cloud/" >}}) to track and display metrics for the data pipeline. - -To view the metrics endpoints for the source collector and pipeline processor, select **More actions**, and then **Show metrics**. You can add these endpoints as Prometheus targets to start tracking your RDI metrics. +On **Metrics**, select **Connect to Prometheus** to view the available collector and processor endpoints. Add the required endpoints to your Prometheus configuration. Select the relevant source when inspecting per-source metrics. Prometheus endpoints are exposed on Redis Cloud's internal network. To access this network, enable [VPC peering]({{< relref "/operate/rc/security/vpc-peering" >}}) or [AWS Transit Gateway]({{< relref "/operate/rc/security/aws-transit-gateway" >}}). See [Prometheus and Grafana with Redis Cloud]({{< relref "/integrate/prometheus-with-redis-cloud/" >}}) for more information. @@ -89,6 +125,7 @@ VPC peering and AWS Transit Gateway are the tested and supported methods for acc For more information about available RDI metrics, see [Observability]({{< relref "/integrate/redis-data-integration/observability" >}}). + ## Rejected records The **Rejected records** view shows records that RDI sent to the dead letter queue (DLQ) because processing failed. Open it from the rejected count on the **Dashboard** tab or from a table-level rejected count on the **Metrics** tab. @@ -108,118 +145,104 @@ the corresponding DLQ stream directly. For more information about why records are rejected and how RDI stores them, see [Rejected records]({{< relref "/integrate/redis-data-integration/data-pipelines/rejected-records" >}}). -## Configuration - -The **Configuration** tab shows your source connectivity, secrets, and collector configuration properties. -{{The Configuration tab for a pipeline.}} +## Settings -If you accidentally overwrite the access for your load balancer or secrets, you can find the required ARNs in the **Source connectivity** or **Secrets** section of this tab. +The **Settings** tab contains the default data structure (**Hash** or **JSON**) and advanced processor properties. These settings apply to all sources in the pipeline. -### Edit collector properties +{{Pipeline-wide Settings tab with default data structure and processor properties.}} -From the **Configuration** tab of your pipeline, select **Edit collector properties** to change your collector properties. +Select **Edit** to change these settings, then **Save changes** and **Apply and restart**. RDI Cloud uses the Flink processor. Review [processor properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors-data-processing-configuration" >}}) before changing them. -{{The advanced properties section.}} +{{The processor advanced properties editor with key and value fields.}} -You can add collector source properties in the **Collector source properties** section and collector sink properties in the **Collector sink properties** section. See the RDI configuration file reference for all available [collector source properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsource" >}}) and [collector sink properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsink" >}}). Select **Save properties** to save the collector properties. +## Configuration -## Dataset +Select a source in the **Sources** list to view its connectivity, secret references, and collector configuration. The connection details and secrets of a deployed source are read-only in this view. -The **Dataset** tab shows the data from your source database that is ingested into your target database. +### Edit collector properties -{{The Dataset tab for a pipeline.}} +1. Select the source in **Configuration**. +1. Select **Edit collector properties**. +1. Update the [collector source properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsource-advanced-source-settings" >}}) or [collector sink properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsink-rdi-collector-stream-writer-configuration" >}}) for that source. -### Edit dataset + {{The advanced properties dialog with separate collector source and sink properties.}} -To change the dataset for your data pipeline: +1. Save the properties and review the restart confirmation before applying the changes. -1. From the **Dataset** tab of your pipeline, select **Edit**. +## Dataset - {{The Edit button.}} +Select a source in the **Sources** list to view the schemas, tables, columns, and keys selected for ingestion from that source. -1. In the **Schemas** section, select the schema(s) you want to migrate to the target database from the list. +{{Dataset tab with a source selector and selected schema, tables, and columns.}} - {{The dataset step with a schema selected.}} +### Edit dataset -1. When you select a schema, you will see its tables in the **Tables** section. Redis Cloud will automatically select all tables for import. You can de-select any columns you do not wish to import to your Redis database. +1. Select the source, then select **Edit**. -1. Select a table to view its columns in the **Columns** section. You can de-select any columns you do not wish to import. + {{The Edit button.}} - {{The columns section, with a few columns selected from one table}} +1. Select the schemas, tables, and columns to ingest. Review the record key for each selected table and correct any missing-key warnings. - If any tables are missing a unique key, a warning will appear in the **Data modeling** section. Select **Show affected** to filter the **Tables** section to the tables without a unique key. - - {{The dataset step filtered to show tables that are missing a unique key.}} + {{Selecting a schema shows its tables for ingestion.}} - For these tables, select the key icon next to the column that defines a unique key. + {{Selecting a table shows its columns and the columns selected for ingestion.}} - {{The unique key icon.}} + {{The missing unique key warning with the affected table and its columns.}} -1. Select **Save changes** to save your changes. + {{The key control beside the column used to identify a record.}} -1. Redis Cloud will display a warning that the data pipeline will restart. Select **Apply and restart** to proceed. +1. Select **Save changes**. +1. Review the restart warning and select **Apply and restart**. -At this point, the data pipeline will apply the changes. The data pipeline will re-ingest data from the source database to the target database. After this initial sync is complete, the data pipeline enters the *change streaming* phase, where changes are captured as they happen. +Dataset changes belong to the selected source. Applying changes restarts the pipeline and can temporarily interrupt processing. Use [Reset a source](#reset-source) when you need a new snapshot of that source. ## Transformations -The **Transformations** tab shows the default data structure for records in your target database and any transformation jobs or processor properties you've set for your pipeline. +The **Transformations** tab lists the pipeline's jobs, their source assignments, matching databases, schemas and tables, and validation status. -{{The Transformations tab for a pipeline.}} +{{Transformation jobs assigned to three different sources, with verified status.}} ### Edit transformations -To edit any of the information in the **Transformations** tab: - -1. From the **Transformations** tab of your pipeline, select **Edit**. - -1. Select how your records will be stored in Redis. You can choose **Hash** or **JSON**. - -1. Under **Transformation jobs**, you can supply one or more [transformation job files]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) that specify how you want to transform the captured data before writing it to the target. Select **Upload jobs** to upload your job files. When you upload job files, Redis Cloud will validate the job files to check for errors. +1. Select **Edit** on **Transformations**. +1. Add, upload, or edit the [transformation jobs]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) needed for your tables. +1. Select the **Source name** for each job. +1. Resolve job validation errors and select **Save changes**. +1. Review the restart warning and select **Apply and restart**. -1. Select **Edit advanced properties** to add any processor properties to control how the data is processed. - - {{The Advanced Processor properties.}} - - See the [RDI configuration file reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors" >}}) for all available processor properties. - -1. Select **Save changes**. - -1. Redis Cloud will display a warning that the data pipeline will restart. Select **Apply and restart** to proceed. - -At this point, the data pipeline will apply the changes. The data pipeline will re-ingest data from the source database to the target database. After this initial sync is complete, the data pipeline enters the *change streaming* phase, where changes are captured as they happen. +The Flink processor accepts lists and `regex:` entries in source selection patterns, but rejects jobs where more than one job can apply to a given table. The default data structure and processor properties are in [Settings](#settings). ## Reset data pipeline -Resetting the data pipeline creates a new baseline snapshot from the current state of your source database, and re-processes the data from the source database to the target Redis database. You may want to reset the pipeline if the source and target databases were disconnected or you made large changes to the data pipeline. +Resetting the whole pipeline clears its internal RDI state for all sources. A running pipeline restarts and takes a new snapshot for every source. A stopped pipeline remains stopped until you [start it](#stop-and-restart-data-pipeline). RDI reprocesses the selected data using the current transformations. Reset alone does not delete records from the target database. See [What happens when I reset the pipeline?]({{< relref "/operate/rc/rdi/faq#reset-pipeline" >}}). -To reset the data pipeline and restart the ingest process: +1. Open **Pipeline actions** and select **Reset pipeline**. +1. Review the confirmation and select **Reset data pipeline**. -1. From your pipeline, select **More actions**, and then **Reset pipeline**. - -1. If you want to flush the database, check **Flush target database**. - -1. Select **Reset data pipeline**. - -At this point, the data pipeline will re-ingest data from the source database to your target Redis database. +To re-run the snapshot for just one source while preserving other sources' internal data, see [Reset a source](#reset-source). ## Stop and restart data pipeline -To stop the data pipeline from synchronizing new data: +1. Open **Pipeline actions** and select **Stop pipeline**. +1. Confirm with **Stop pipeline**. -1. From your pipeline, select **More actions**, and then **Stop pipeline**. +Stopping the pipeline pauses processing for all sources. To resume, open **Pipeline actions**, select **Start pipeline**, and confirm with **Start pipeline**. Use the [source actions](#stop-and-start-source) to control an individual source separately. Stopping all sources individually leaves the processor running. See [Billing]({{< relref "/operate/rc/rdi/faq#billing" >}}) for the difference in charges. -1. Select **Stop data pipeline** to confirm. +## Flush the target database -Stopping the data pipeline will suspend data processing. To restart the pipeline from the **Data pipeline** tab, select **More actions**, and then **Start pipeline**. +Flushing permanently deletes **all data** from the target database, including records from every source and data written outside RDI. It does not reset source positions or automatically reload the data. -## Delete pipeline +1. [Stop the pipeline](#stop-and-restart-data-pipeline) and wait until it is stopped. +1. Open **Pipeline actions** and select **Flush target database**. This action is disabled while the pipeline is running. +1. Check the target database and confirm with **Flush target database**. -To delete the data pipeline: +To refill the target, follow [How do I reload data after a flush?]({{< relref "/operate/rc/rdi/faq#reload-after-flush" >}}). -1. From your pipeline, select **More actions**, and then **Delete pipeline**. +## Delete pipeline -1. Select **Delete data pipeline** to confirm. +1. Return to the workspace list on the **Data Integration** page or your database's **Data Integration** tab. +1. Open the pipeline's actions menu and select **Delete pipeline**. +1. Review and confirm the deletion. -Deleted data pipelines cannot be recovered. You may also want to [delete your workspace]({{}}). +Deleted pipelines cannot be recovered. If the pipeline has an associated setup draft, deletion also removes that draft. You can then [delete the workspace]({{}}) when it is no longer needed. diff --git a/static/images/rc/rdi/rdi-2-add-sources.png b/static/images/rc/rdi/rdi-2-add-sources.png new file mode 100644 index 0000000000..071bf5a9ee Binary files /dev/null and b/static/images/rc/rdi/rdi-2-add-sources.png differ diff --git a/static/images/rc/rdi/rdi-2-dashboard.png b/static/images/rc/rdi/rdi-2-dashboard.png new file mode 100644 index 0000000000..f1aa0932b6 Binary files /dev/null and b/static/images/rc/rdi/rdi-2-dashboard.png differ diff --git a/static/images/rc/rdi/rdi-2-dataset.png b/static/images/rc/rdi/rdi-2-dataset.png new file mode 100644 index 0000000000..b82317ab62 Binary files /dev/null and b/static/images/rc/rdi/rdi-2-dataset.png differ diff --git a/static/images/rc/rdi/rdi-2-delete-source.png b/static/images/rc/rdi/rdi-2-delete-source.png new file mode 100644 index 0000000000..58014c81ed Binary files /dev/null and b/static/images/rc/rdi/rdi-2-delete-source.png differ diff --git a/static/images/rc/rdi/rdi-2-metrics.png b/static/images/rc/rdi/rdi-2-metrics.png new file mode 100644 index 0000000000..0d73a90035 Binary files /dev/null and b/static/images/rc/rdi/rdi-2-metrics.png differ diff --git a/static/images/rc/rdi/rdi-2-reset-source.png b/static/images/rc/rdi/rdi-2-reset-source.png new file mode 100644 index 0000000000..47d771c672 Binary files /dev/null and b/static/images/rc/rdi/rdi-2-reset-source.png differ diff --git a/static/images/rc/rdi/rdi-2-settings.png b/static/images/rc/rdi/rdi-2-settings.png new file mode 100644 index 0000000000..ef3d29b910 Binary files /dev/null and b/static/images/rc/rdi/rdi-2-settings.png differ diff --git a/static/images/rc/rdi/rdi-2-source-actions.png b/static/images/rc/rdi/rdi-2-source-actions.png new file mode 100644 index 0000000000..ae11c76e67 Binary files /dev/null and b/static/images/rc/rdi/rdi-2-source-actions.png differ diff --git a/static/images/rc/rdi/rdi-2-source-configuration.png b/static/images/rc/rdi/rdi-2-source-configuration.png new file mode 100644 index 0000000000..618b17eedc Binary files /dev/null and b/static/images/rc/rdi/rdi-2-source-configuration.png differ diff --git a/static/images/rc/rdi/rdi-2-sources.png b/static/images/rc/rdi/rdi-2-sources.png new file mode 100644 index 0000000000..6c9778a465 Binary files /dev/null and b/static/images/rc/rdi/rdi-2-sources.png differ diff --git a/static/images/rc/rdi/rdi-2-transformation-jobs.png b/static/images/rc/rdi/rdi-2-transformation-jobs.png new file mode 100644 index 0000000000..5ad41dc97d Binary files /dev/null and b/static/images/rc/rdi/rdi-2-transformation-jobs.png differ diff --git a/static/images/rc/rdi/rdi-2-workspace-actions.png b/static/images/rc/rdi/rdi-2-workspace-actions.png new file mode 100644 index 0000000000..487d086cd7 Binary files /dev/null and b/static/images/rc/rdi/rdi-2-workspace-actions.png differ diff --git a/static/images/rc/rdi/rdi-2-workspace-sources.png b/static/images/rc/rdi/rdi-2-workspace-sources.png new file mode 100644 index 0000000000..2010f9e095 Binary files /dev/null and b/static/images/rc/rdi/rdi-2-workspace-sources.png differ