diff --git a/content/operate/rc/rdi/create-workspace.md b/content/operate/rc/rdi/create-workspace.md index 5185def009..fba75d1c24 100644 --- a/content/operate/rc/rdi/create-workspace.md +++ b/content/operate/rc/rdi/create-workspace.md @@ -21,7 +21,8 @@ Before you can create your first Data Integration pipeline for a Redis Cloud sub {{< embed-md "rc-rdi-create-rdi-workspace.md" >}} -{{The Add pipeline control is available while the workspace is being created.}} +![The Add pipeline control is available while the workspace is being created.](/images/rc/rdi/rdi-workspace-add-pipeline.png) +{width="80%"} ## View workspace status and details @@ -30,11 +31,13 @@ You can view your workspace in one of the following ways: - From the Redis Cloud console, go to the **Data integration** page, or - From your subscription, select the **Data Integration** tab -{{Workspace pipeline list showing a streaming pipeline and its Sources column.}} +![Workspace pipeline list showing a streaming pipeline and its Sources column.](/images/rc/rdi/rdi-2-workspace-sources.png) +{width="100%"} There, you'll see your workspace and its pipelines. The **Sources** column lists each pipeline's source names and database types. A source with unfinished configuration is marked **Pending setup**. Open its draft to continue setup. To see your workspace details, including the deployment CIDR and region information, select **Workspace actions > Workspace details**. -{{Workspace actions menu with Workspace details and Delete workspace.}} +![Workspace actions menu with Workspace details and Delete workspace.](/images/rc/rdi/rdi-2-workspace-actions.png) +{width="240px"} ## Delete workspace @@ -44,4 +47,5 @@ Make sure to [delete your data pipeline]({{ Delete workspace** from your workspace. -{{Workspace actions menu with Workspace details and Delete workspace.}} +![Workspace actions menu with Workspace details and Delete workspace.](/images/rc/rdi/rdi-2-workspace-actions.png) +{width="240px"} diff --git a/content/operate/rc/rdi/define.md b/content/operate/rc/rdi/define.md index fd8584882f..8bb99d62b1 100644 --- a/content/operate/rc/rdi/define.md +++ b/content/operate/rc/rdi/define.md @@ -17,7 +17,8 @@ After you have [prepared each source database]({{}} +![The Add pipeline control is available while the workspace is being created.](/images/rc/rdi/rdi-workspace-add-pipeline.png) +{width="80%"} To create a pipeline: @@ -32,12 +33,14 @@ To create a pipeline: 1. Select the target from the **Target database** list. All sources write to this database. - {{The target database list in pipeline Settings.}} + ![The target database list in pipeline Settings.](/images/rc/rdi/rdi-choose-target.png) + {width="80%"} 1. Select **Hash** or **JSON** as the **Default data structure**. Transformation jobs can override how individual records are written. 1. If needed, configure **Processor properties**. These apply to the whole pipeline, not to an individual source. See the [processor configuration reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors-data-processing-configuration" >}}). - {{The processor advanced properties editor with key and value fields.}} + ![The processor advanced properties editor with key and value fields.](/images/rc/rdi/rdi-processor-advanced-properties.png) + {width="80%"} 1. Select **Continue**. @@ -70,11 +73,13 @@ Choose **AWS Private Link** or **Public Endpoint** for the selected source, acco - For **AWS Private Link**, enter the **Private Link service name** from your [endpoint service]({{< relref "/operate/rc/rdi/setup#set-up-connectivity" >}}). Select **Connect to Private Link** and wait for connectivity to complete. If the connection fails, check the service name and its allowed principal. - {{AWS Private Link connectivity with the service name and Connect to Private Link control.}} + ![AWS Private Link connectivity with the service name and Connect to Private Link control.](/images/rc/rdi/rdi-source-configuration-source-connectivity-privatelink.png) + {width="80%"} - For **Public Endpoint**, enter the source IP address or hostname. Add the Redis Cloud outbound IP address shown in the console to your source database's allowlist. - {{Public endpoint connectivity with source hostname and Redis Cloud outbound IP addresses.}} + ![Public endpoint connectivity with source hostname and Redis Cloud outbound IP addresses.](/images/rc/rdi/rdi-source-configuration-source-connectivity-public.png) + {width="80%"} Configure connectivity for each source separately. Sources in the same pipeline can use different connectivity methods. @@ -82,7 +87,8 @@ Configure connectivity for each source separately. Sources in the same pipeline Enter the Amazon Resource Name (ARN) of the selected source's [database credentials secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) in **Credentials Secret ARN**. -{{The Credentials Secret ARN field, transit security options, and Validate control.}} +![The Credentials Secret ARN field, transit security options, and Validate control.](/images/rc/rdi/rdi-source-configuration-secrets.png) +{width="80%"} For Snowflake, select the authentication method matching your secret: @@ -93,11 +99,13 @@ Under **Transit security**, select the mode required by your source: - **TLS** (Transport Layer Security): Provide the CA certificate secret ARN when your source requires it. - {{TLS transit security with the CA Certificate Secret ARN field.}} + ![TLS transit security with the CA Certificate Secret ARN field.](/images/rc/rdi/rdi-define-tls.png) + {width="80%"} - **mTLS** (mutual TLS): Provide the CA certificate, client certificate, and client private key secret ARNs. Also provide the client key passphrase secret ARN if the key is encrypted. - {{mTLS transit security with certificate, private key, and optional password secret ARN fields.}} + ![mTLS transit security with certificate, private key, and optional password secret ARN fields.](/images/rc/rdi/rdi-define-mtls.png) + {width="80%"} Select **Validate** to check access to the selected source's secrets. Repeat this for each source. The AWS secret contents and permissions are described in [Share source database credentials]({{< relref "/operate/rc/rdi/setup#share-source-database-credentials" >}}). @@ -111,13 +119,16 @@ Enter the selected source's database settings. The fields depend on the database - **PDB**: The Oracle pluggable database. - **Connection string**: The MongoDB connection information. -{{Source-specific database, port, and collector properties.}} +![Source-specific database, port, and collector properties.](/images/rc/rdi/rdi-2-source-configuration.png) +{width="80%"} Use **Collector properties** for additional source and sink settings. These settings apply to the selected source. See the [collector source properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsource-advanced-source-settings" >}}) and [collector sink properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsink-rdi-collector-stream-writer-configuration" >}}). -{{The Edit advanced properties control under Collector properties.}} +![The Edit advanced properties control under Collector properties.](/images/rc/rdi/rdi-source-configuration-collector-properties.png) +{width="500px"} -{{The advanced properties dialog with separate collector source and sink properties.}} +![The advanced properties dialog with separate collector source and sink properties.](/images/rc/rdi/rdi-advanced-properties.png) +{width="80%"} ## Select data {#dataset} @@ -130,17 +141,21 @@ Do not write data directly to keys managed by RDI. Changes from another applicat 1. Select a schema in **Schemas** to see its tables. 1. Select the tables to ingest in **Tables**. - {{Selecting a schema shows its tables for ingestion.}} + ![Selecting a schema shows its tables for ingestion.](/images/rc/rdi/rdi-dataset-schema-selected.png) + {width="75%"} 1. Select a table to see its **Columns**, then select the columns to ingest. - {{Selecting a table shows its columns and the columns selected for ingestion.}} + ![Selecting a table shows its columns and the columns selected for ingestion.](/images/rc/rdi/rdi-select-columns.png) + {width="75%"} 1. If a table has no unique key, use the key control next to the column that identifies each record. Review any missing-key warnings before continuing. - {{The missing unique key warning with the affected table and its columns.}} + ![The missing unique key warning with the affected table and its columns.](/images/rc/rdi/rdi-dataset-missing-unique-key.png) + {width="75%"} - {{The key control beside the column used to identify a record.}} + ![The key control beside the column used to identify a record.](/images/rc/rdi/rdi-unique-key-selected.png) + {width="500px"} 1. Repeat for the other sources, then select **Continue**. @@ -154,7 +169,8 @@ Transformation jobs are optional. Without a matching job, RDI writes records usi 1. Check the **Source name** assignment for each job. In a multi-source pipeline, `source.server_name` identifies the source the job reads from. Select the source name you chose during setup. 1. Review each job's validation status and correct errors. - {{Example transformation jobs with their source assignments and validation status.}} + ![Example transformation jobs with their source assignments and validation status.](/images/rc/rdi/rdi-2-transformation-jobs.png) + {width="100%"} 1. Select **Continue to review & deploy**. @@ -166,6 +182,7 @@ Review every source's connection, selected data, and transformation jobs, togeth Select **Deploy pipeline** to start the pipeline. Each source performs its initial snapshot and then captures ongoing changes. Snapshot duration depends on the amount of selected data and source performance. -{{The Deploy pipeline button.}} +![The Deploy pipeline button.](/images/rc/rdi/rdi-confirm-deploy.png) +{width="175px"} Open the pipeline's [Dashboard and Metrics tabs]({{}}) to follow progress for each source. diff --git a/content/operate/rc/rdi/quick-start.md b/content/operate/rc/rdi/quick-start.md index 159e01ea79..7ea512b851 100644 --- a/content/operate/rc/rdi/quick-start.md +++ b/content/operate/rc/rdi/quick-start.md @@ -44,7 +44,8 @@ This example creates one PostgreSQL source. You can [add more sources]({{< relre 1. On the [Redis Cloud console](https://cloud.redis.io/), open your target database's **Data Integration** tab and select **Add pipeline**. - {{The Add pipeline control is available while the workspace is being created.}} + ![The Add pipeline control is available while the workspace is being created.](/images/rc/rdi/rdi-workspace-add-pipeline.png) + {width="80%"} 1. In **Settings**, select your target database and choose **Hash** or **JSON** as the default data structure, then select **Continue**. 1. In **Add sources**, select **PostgreSQL**. @@ -52,11 +53,13 @@ This example creates one PostgreSQL source. You can [add more sources]({{< relre 1. Select **Continue** to open **Configure source**. 1. Under **Source connectivity**, copy the **Role ARN**. Use it as `redis_privatelink_arn` in the Terraform configuration. - {{The source connectivity Role ARN and availability zones.}} + ![The source connectivity Role ARN and availability zones.](/images/rc/rdi/rdi-setup-connectivity-arn.png) + {width="80%"} 1. Under **Secrets**, copy the **Role ARN**. Use it as `redis_secrets_arn` in the Terraform configuration. - {{The Role ARN in the Secrets section.}} + ![The Role ARN in the Secrets section.](/images/rc/rdi/rdi-credentials-arn.png) + {width="80%"} 1. Select **Save & exit** while you create the source resources. @@ -117,32 +120,38 @@ If you lose any outputs, run `terraform output` to view them again. 1. Under **Source connectivity**, enter the Terraform `vpc_endpoint_service_name` output as the **Private Link service name**. 1. Select **Connect to Private Link** and wait for connectivity to complete. - {{AWS Private Link connectivity with the service name and Connect to Private Link control.}} + ![AWS Private Link connectivity with the service name and Connect to Private Link control.](/images/rc/rdi/rdi-source-configuration-source-connectivity-privatelink.png) + {width="80%"} 1. Under **Secrets**, enter the `secret_arn` output as **Credentials Secret ARN**. 1. Select **Validate** to check access to the secret. - {{The Credentials Secret ARN field, transit security options, and Validate control.}} + ![The Credentials Secret ARN field, transit security options, and Validate control.](/images/rc/rdi/rdi-source-configuration-secrets.png) + {width="80%"} 1. Under **Source configuration**, enter the `database` and `port` Terraform outputs in the matching fields. 1. Select **Test source** and correct any validation errors, then select **Continue**. 1. In **Select data**, select the schemas, tables, and columns to ingest. Review the selected key for each table. - {{Selecting a schema shows its tables for ingestion.}} + ![Selecting a schema shows its tables for ingestion.](/images/rc/rdi/rdi-dataset-schema-selected.png) + {width="75%"} - {{Selecting a table shows its columns and the columns selected for ingestion.}} + ![Selecting a table shows its columns and the columns selected for ingestion.](/images/rc/rdi/rdi-select-columns.png) + {width="75%"} 1. Select **Continue** to open **Add transformations**. For this example, you can keep the default mapping without adding jobs. 1. Select **Continue to review & deploy**. 1. Review the source and target, then select **Deploy pipeline**. - {{The Deploy pipeline button.}} + ![The Deploy pipeline button.](/images/rc/rdi/rdi-confirm-deploy.png) + {width="175px"} The source first imports its selected data, then captures ongoing changes. Open the pipeline's **Dashboard** or **Metrics** tab to follow its progress. The following example shows the metrics for one selected source in a pipeline with multiple sources. -{{Metrics for one selected source in a pipeline, including snapshot progress and per-table record counts.}} +![Metrics for one selected source in a pipeline, including snapshot progress and per-table record counts.](/images/rc/rdi/rdi-2-metrics.png) +{width="80%"} See [View and edit data pipeline]({{}}) for source actions, dataset changes, and monitoring. diff --git a/content/operate/rc/rdi/setup.md b/content/operate/rc/rdi/setup.md index 56a5712b0d..3bd9ba554c 100644 --- a/content/operate/rc/rdi/setup.md +++ b/content/operate/rc/rdi/setup.md @@ -143,14 +143,16 @@ Before you can set up your source connectivity and secrets, you need the AWS Acc 1. On the [Redis Cloud console](https://cloud.redis.io/), open your target database's **Data Integration** tab. 1. Select **Add pipeline**, or resume an existing draft. To add a source to a running pipeline, select **Add source** on its **Dashboard**. - {{The Add pipeline control is available while the workspace is being created.}} + ![The Add pipeline control is available while the workspace is being created.](/images/rc/rdi/rdi-workspace-add-pipeline.png) + {width="80%"} 1. For a new pipeline, complete **Settings**, including the target database, and select **Continue**. 1. In **Add sources**, select the source type and enter a unique **Source name**. This name identifies the source in the pipeline configuration and transformation jobs. See [Add sources]({{< relref "/operate/rc/rdi/define#pipeline-setup" >}}) for naming rules. 1. Select **Continue** to open **Configure source**. 1. Under **Source connectivity**, copy the **Role ARN** and extract its AWS account ID. - {{The source connectivity Role ARN and availability zones.}} + ![The source connectivity Role ARN and availability zones.](/images/rc/rdi/rdi-setup-connectivity-arn.png) + {width="80%"} The account ID is the number after `arn:aws:iam::`. For example, `arn:aws:iam::123456789012:role/redis-data-pipeline` contains account ID `123456789012`. @@ -171,11 +173,13 @@ The following diagrams show the network setup for the different database setups: - Database hosted on an AWS EC2 instance: - {{The network setup for a database hosted on an AWS EC2 instance.}} + ![The network setup for a database hosted on an AWS EC2 instance.](/images/rc/rdi/rdi-setup-diagram-ec2.png) + {width="80%"} - Database hosted on AWS RDS or AWS Aurora: - {{The network setup for a database hosted on AWS RDS or AWS Aurora.}} + ![The network setup for a database hosted on AWS RDS or AWS Aurora.](/images/rc/rdi/rdi-setup-diagram-aurora.png) + {width="80%"} Select the steps for your database setup. diff --git a/content/operate/rc/rdi/view-edit.md b/content/operate/rc/rdi/view-edit.md index e2acadbeed..54d0a95748 100644 --- a/content/operate/rc/rdi/view-edit.md +++ b/content/operate/rc/rdi/view-edit.md @@ -30,13 +30,16 @@ Use the **Sources** list in **Metrics**, **Configuration**, and **Dataset** to s The **Dashboard** shows pipeline health, processor and target status, throughput, pending records, record health, and processor load. It also shows the number of sources, selected tables, and transformation jobs. -{{Pipeline dashboard with throughput, pending records, record health, and processor load.}} +![Pipeline dashboard with throughput, pending records, record health, and processor load.](/images/rc/rdi/rdi-2-dashboard.png) +{width="100%"} Each source has a card showing its database type, name, activity, and status. Expand a card for more details. Open the source's **More actions** menu to view its configuration, edit its dataset or transformations, or manage the source. -{{Three source cards: one PostgreSQL source and two MySQL sources, all streaming.}} +![Three source cards: one PostgreSQL source and two MySQL sources, all streaming.](/images/rc/rdi/rdi-2-sources.png) +{width="460px"} -{{Source actions for configuration, dataset, transformations, stop, reset, and deletion.}} +![Source actions for configuration, dataset, transformations, stop, reset, and deletion.](/images/rc/rdi/rdi-2-source-actions.png) +{width="350px"} A pipeline can be streaming, stopped, or in an error or transitional state. If there is a problem, you can use the source cards and **Metrics** tab to identify the affected source before taking remedial action. Pipeline health and individual source status can differ, for example when one source is stopped. @@ -47,7 +50,8 @@ A pipeline can be streaming, stopped, or in an error or transitional state. If t 1. Complete **Configure source**, **Select data**, and **Add transformations** for the new source. See [Create data pipeline]({{< relref "/operate/rc/rdi/define" >}}) for the configuration steps. 1. Review the changes in **Review & deploy** and select **Deploy pipeline**. -{{Add-source wizard showing existing sources and the Add source control.}} +![Add-source wizard showing existing sources and the Add source control.](/images/rc/rdi/rdi-2-add-sources.png) +{width="100%"} Adding a source keeps the existing pipeline's shared target and settings, so the wizard starts at **Add sources**. A source with an incomplete setup appears with **Pending setup** in the workspace. You can resume an existing draft setup at any time to complete it. If you remove a source, wait for the removal process to finish before adding another source. @@ -68,7 +72,8 @@ Resetting one source starts a new snapshot and reprocesses its selected data. 1. Select **Reset source**. 1. Review the effect and confirm with **Reset source**. -{{Reset source confirmation explaining that other sources retain their data and the pipeline temporarily stops.}} +![Reset source confirmation explaining that other sources retain their data and the pipeline temporarily stops.](/images/rc/rdi/rdi-2-reset-source.png) +{width="600px"} The reset clears that source's internal RDI streams, offsets, schema history, rejected records, and processing counters. The whole pipeline and all its sources restart during the reset. Other sources keep their saved positions and resume streaming. All records already in the shared target database remain, including those from the reset source. The new snapshot can overwrite records for that source. See [What happens when I reset one source?]({{< relref "/operate/rc/rdi/faq#reset-one-source" >}}). @@ -80,7 +85,8 @@ Before deleting a source, remove or reassign transformation jobs that refer to i 1. Select **Delete source**. 1. Review the confirmation and select **Delete source**. -{{Delete source confirmation for the selected PostgreSQL source.}} +![Delete source confirmation for the selected PostgreSQL source.](/images/rc/rdi/rdi-2-delete-source.png) +{width="600px"} Deleting a source removes its data selection and internal RDI state, including streams, offsets, schema history, rejected records, and processing counters. Records already written to the target Redis database remain there. Other sources retain their data. The whole pipeline stops while RDI cleans up the removed source and starts again afterwards. @@ -100,7 +106,8 @@ All sources share the new target. Changing the target restarts the pipeline but Select a source in the **Sources** list to see its connection status, snapshot progress, queue usage, errors, and table-level record counts. Available collector diagnostics depend on the source type. Expand **Diagnostics** to see additional metrics for the selected source. -{{Metrics for the selected PostgreSQL source, including collector diagnostics and table-level counts.}} +![Metrics for the selected PostgreSQL source, including collector diagnostics and table-level counts.](/images/rc/rdi/rdi-2-metrics.png) +{width="100%"} | Metric | Description | |--------|-------------| @@ -150,11 +157,13 @@ For more information about why records are rejected and how RDI stores them, see The **Settings** tab contains the default data structure (**Hash** or **JSON**) and advanced processor properties. These settings apply to all sources in the pipeline. -{{Pipeline-wide Settings tab with default data structure and processor properties.}} +![Pipeline-wide Settings tab with default data structure and processor properties.](/images/rc/rdi/rdi-2-settings.png) +{width="100%"} Select **Edit** to change these settings, then **Save changes** and **Apply and restart**. RDI Cloud uses the Flink processor. Review [processor properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors-data-processing-configuration" >}}) before changing them. -{{The processor advanced properties editor with key and value fields.}} +![The processor advanced properties editor with key and value fields.](/images/rc/rdi/rdi-processor-advanced-properties.png) +{width="80%"} ## Configuration @@ -166,7 +175,8 @@ Select a source in the **Sources** list to view its connectivity, secret referen 1. Select **Edit collector properties**. 1. Update the [collector source properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsource-advanced-source-settings" >}}) or [collector sink properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsink-rdi-collector-stream-writer-configuration" >}}) for that source. - {{The advanced properties dialog with separate collector source and sink properties.}} + ![The advanced properties dialog with separate collector source and sink properties.](/images/rc/rdi/rdi-advanced-properties.png) + {width="80%"} 1. Save the properties and review the restart confirmation before applying the changes. @@ -174,23 +184,29 @@ Select a source in the **Sources** list to view its connectivity, secret referen Select a source in the **Sources** list to view the schemas, tables, columns, and keys selected for ingestion from that source. -{{Dataset tab with a source selector and selected schema, tables, and columns.}} +![Dataset tab with a source selector and selected schema, tables, and columns.](/images/rc/rdi/rdi-2-dataset.png) +{width="100%"} ### Edit dataset 1. Select the source, then select **Edit**. - {{The Edit button.}} + ![The Edit button.](/images/rc/rdi/rdi-view-edit-button.png) + {width="100px"} 1. Select the schemas, tables, and columns to ingest. Review the record key for each selected table and correct any missing-key warnings. - {{Selecting a schema shows its tables for ingestion.}} + ![Selecting a schema shows its tables for ingestion.](/images/rc/rdi/rdi-dataset-schema-selected.png) + {width="75%"} - {{Selecting a table shows its columns and the columns selected for ingestion.}} + ![Selecting a table shows its columns and the columns selected for ingestion.](/images/rc/rdi/rdi-select-columns.png) + {width="75%"} - {{The missing unique key warning with the affected table and its columns.}} + ![The missing unique key warning with the affected table and its columns.](/images/rc/rdi/rdi-dataset-missing-unique-key.png) + {width="75%"} - {{The key control beside the column used to identify a record.}} + ![The key control beside the column used to identify a record.](/images/rc/rdi/rdi-unique-key-selected.png) + {width="500px"} 1. Select **Save changes**. 1. Review the restart warning and select **Apply and restart**. @@ -201,7 +217,8 @@ Dataset changes belong to the selected source. Applying changes restarts the pip The **Transformations** tab lists the pipeline's jobs, their source assignments, matching databases, schemas and tables, and validation status. -{{Transformation jobs assigned to three different sources, with verified status.}} +![Transformation jobs assigned to three different sources, with verified status.](/images/rc/rdi/rdi-2-transformation-jobs.png) +{width="100%"} ### Edit transformations