Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions content/embeds/rc-rdi-create-rdi-workspace.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
To create a Data Integration workspace for an existing [Pro subscription]({{< relref "/operate/rc/databases/create-database/create-pro-database-new" >}}):
To create a Data Integration workspace for an existing [Pro subscription](/content/operate/rc/databases/create-database/create-pro-database-new.md):

1. From the Redis Cloud console, select **Data Integration** from the left-hand menu. If you don't have any workspaces yet, select **Create workspace** to go to the **Create workspace** page.

Expand All @@ -8,7 +8,7 @@ To create a Data Integration workspace for an existing [Pro subscription]({{< re

{{<image filename="images/rc/rdi/rdi-new-workspace-button.png" alt="The new workspace button." width=150px >}}

You can also go to the **Data Integration** tab from your subscription or database page and select **Create workspace** to go to the **Create workspace** page for your subscription.
You can also go to the **Data Integration** tab from your subscription or database page and select **Create workspace** to go to the **Create workspace** page for your subscription.

{{<image filename="images/rc/rdi/rdi-create-workspace-button.png" alt="The create workspace button." width=200px >}}

Expand All @@ -33,4 +33,4 @@ To create a Data Integration workspace for an existing [Pro subscription]({{< re

{{<image filename="images/rc/rdi/rdi-create-workspace-button.png" alt="The create workspace button." width=200px >}}

Your workspace will be created in the background. You can select **Create pipeline** to [create your pipeline]({{<relref "/operate/rc/rdi/define">}}) while the workspace is provisioning, or you can select **Create pipeline later** to go back to the Redis Cloud console.
Your workspace will be created in the background. You can select **Create pipeline** to [create your pipeline](/content/operate/rc/rdi/define.md) while the workspace is provisioning, or you can select **Create pipeline later** to go back to the Redis Cloud console.
2 changes: 1 addition & 1 deletion content/embeds/rc-rdi-secrets-permissions.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,4 @@
}
```

After you store this secret, you can view and copy the [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_iam-permissions.html#iam-resources) of your secret on the secret details page. Save the secret ARN to use when you [define your source database]({{<relref "/operate/rc/rdi/define">}}).
After you store this secret, you can view and copy the [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_iam-permissions.html#iam-resources) of your secret on the secret details page. Save the secret ARN to use when you [define your source database](/content/operate/rc/rdi/define.md).
14 changes: 7 additions & 7 deletions content/embeds/rdi-db-reqs.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@
* If you are deploying RDI for a production environment then secure this database with a password
and TLS.
* Set the database's
[eviction policy]({{< relref "/operate/rs/databases/memory-performance/eviction-policy" >}}) to `noeviction`. Note that you can't set this using
[`rladmin`]({{< relref "/operate/rs/references/cli-utilities/rladmin" >}}),
[eviction policy](/content/operate/rs/databases/memory-performance/eviction-policy.md) to `noeviction`. Note that you can't set this using
[`rladmin`](/content/operate/rs/references/cli-utilities/rladmin/_index.md),
so you must either do it using the admin UI or with the following
[REST API]({{< relref "/operate/rs/references/rest-api" >}})
[REST API](/content/operate/rs/references/rest-api/_index.md)
command:

```bash
Expand All @@ -17,11 +17,11 @@
-X PUT https://<CLUSTER_FQDN>:9443/v1/bdbs/<BDB_UID>
```
* Set the database's
[data persistence]({{< relref "/operate/rs/databases/configure/database-persistence" >}})
[data persistence](/content/operate/rs/databases/configure/database-persistence.md)
to AOF - fsync every 1 sec. Note that you can't set this using
[`rladmin`]({{< relref "/operate/rs/references/cli-utilities/rladmin" >}}),
[`rladmin`](/content/operate/rs/references/cli-utilities/rladmin/_index.md),
so you must either do it using the admin UI or with the following
[REST API]({{< relref "/operate/rs/references/rest-api" >}})
[REST API](/content/operate/rs/references/rest-api/_index.md)
commands:

```bash
Expand All @@ -34,7 +34,7 @@
-H "Content-Type: application/json" \
-X PUT https://<CLUSTER_FQDN>:9443/v1/bdbs/<BDB_UID>
```
If you don't have permissions to use AOF persistence, please check the [Using RDI without persistence]({{< relref "/integrate/redis-data-integration/faq#can-i-use-rdi-without-persistence-enabled" >}}) section in the FAQ.
If you don't have permissions to use AOF persistence, please check the [Using RDI without persistence](/content/integrate/redis-data-integration/faq.md#can-i-use-rdi-without-persistence-enabled) section in the FAQ.

* **Ensure that the RDI database is not clustered.** RDI will not work correctly if the
RDI database is clustered (but note that the target database *can* be clustered without
Expand Down
8 changes: 4 additions & 4 deletions content/embeds/rdi-when-to-use.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,15 @@ RDI is a good fit when:
- You want a self-managed solution or AWS based solution.
- The source data changes frequently in small increments.
- The source database has no more than 20K changes per second.
- RDI throughput during [full sync]({{< relref "/integrate/redis-data-integration/data-pipelines#pipeline-lifecycle" >}})
- RDI throughput during [full sync](/content/integrate/redis-data-integration/data-pipelines/_index.md#pipeline-lifecycle)
stays below 60K records per second, assuming an average record size of 1KB and a pipeline without transformations.
- RDI throughput during [CDC]({{< relref "/integrate/redis-data-integration/data-pipelines#pipeline-lifecycle" >}})
- RDI throughput during [CDC](/content/integrate/redis-data-integration/data-pipelines/_index.md#pipeline-lifecycle)
stays below 20K records per second, assuming an average record size of 1KB and a pipeline without transformations.
- The total data size is no larger than 200GB, so a full sync completes in under an hour without exceeding the throughput
limits above. RDI can ingest larger datasets, but it will take longer than an hour.
- You don鈥檛 need to perform join operations on the data from several tables
into a [nested Redis JSON object]({{< relref "/integrate/redis-data-integration/data-pipelines/data-denormalization#joining-one-to-many-relationships" >}}).
- RDI supports the [data transformations]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) you need for your app.
into a [nested Redis JSON object](/content/integrate/redis-data-integration/data-pipelines/data-denormalization.md#joining-one-to-many-relationships).
- RDI supports the [data transformations](/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md) you need for your app.
- Your data caching needs are too complex or demanding to implement and maintain yourself.
- Your database administrator has reviewed RDI's requirements for the source database and
confirmed that they are acceptable.
55 changes: 27 additions & 28 deletions content/operate/rc/rdi/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@ weight: 38
tocEmbedHeaders: true
---

Redis Cloud now supports [Redis Data Integration (RDI)]({{<relref "integrate/redis-data-integration">}}), a fast and simple way to bring your data into Redis from other types of primary databases.
Redis Cloud now supports [Redis Data Integration (RDI)](/content/integrate/redis-data-integration/_index.md), a fast and simple way to bring your data into Redis from other types of primary databases.

A relational database usually handles queries much more slowly than a Redis database. If your application uses a relational database and makes many more reads than writes (which is the typical case) then you can improve performance by using Redis as a cache to handle the read queries quickly. Redis Cloud uses [ingest]({{<relref "/integrate/redis-data-integration/">}}) to help you offload all read queries from the application database to Redis automatically.
A relational database usually handles queries much more slowly than a Redis database. If your application uses a relational database and makes many more reads than writes (which is the typical case) then you can improve performance by using Redis as a cache to handle the read queries quickly. Redis Cloud uses [ingest](/content/integrate/redis-data-integration/_index.md) to help you offload all read queries from the application database to Redis automatically.

Using a data pipeline lets you have a cache that is always ready for queries. RDI Data pipelines ensure that any changes made to your primary database are captured in your Redis cache within a few seconds, preventing cache misses and stale data within the cache.

Expand All @@ -35,7 +35,7 @@ apps with a rapidly-growing number of users; the performance of the main databas
but it will soon struggle to handle the increasing demand without a cache.

Use the information in the sections below to determine whether RDI is a good fit for your architecture. See also the
[decision tree for using RDI]({{<relref "/integrate/redis-data-integration/when-to-use#decision-tree-for-using-rdi">}})
[decision tree for using RDI](/content/integrate/redis-data-integration/when-to-use.md#decision-tree-for-using-rdi)
which presents the considerations in a straightforward question-and-answer format.

```decision-tree
Expand All @@ -53,21 +53,21 @@ Each source first imports its selected data during the *initial sync* phase, the

RDI Cloud uses the Flink processor for all pipelines.

For more info on how RDI works, see [RDI Architecture]({{<relref "/integrate/redis-data-integration/architecture">}}).
For more info on how RDI works, see [RDI Architecture](/content/integrate/redis-data-integration/architecture/_index.md).

### Pipeline security

Data pipelines are set up to ensure a high level of data security. Source database credentials and TLS secrets are stored in AWS secret manager and shared using the Kubernetes CSI driver for secrets. See [Share source database credentials]({{<relref "/operate/rc/rdi/setup#share-source-database-credentials">}}) to learn how to share your source database credentials and TLS certificates with Redis Cloud.
Data pipelines are set up to ensure a high level of data security. Source database credentials and TLS secrets are stored in AWS secret manager and shared using the Kubernetes CSI driver for secrets. See [Share source database credentials](/content/operate/rc/rdi/setup.md#share-source-database-credentials) to learn how to share your source database credentials and TLS certificates with Redis Cloud.

Configure connectivity separately for each source. A source can use a public endpoint or [AWS PrivateLink](https://aws.amazon.com/privatelink/), subject to the source-specific requirements in [Prerequisites](#prerequisites). See [Set up connectivity]({{<relref "/operate/rc/rdi/setup#set-up-connectivity">}}) to learn how to connect your PrivateLink to the Redis Cloud VPC.
Configure connectivity separately for each source. A source can use a public endpoint or [AWS PrivateLink](https://aws.amazon.com/privatelink/), subject to the source-specific requirements in [Prerequisites](#prerequisites). See [Set up connectivity](/content/operate/rc/rdi/setup.md#set-up-connectivity) to learn how to connect your PrivateLink to the Redis Cloud VPC.

RDI encrypts all network connections with TLS. The pipeline will process data from the source database in-memory and write it to the target database using a TLS connection. There are no external connections to your data pipeline except from Redis Cloud management services.

## Prerequisites

Before you can create a data pipeline, you must have:

- A [Redis Cloud Pro database]({{< relref "/operate/rc/databases/create-database/create-pro-database-new" >}}) hosted on Amazon Web Services (AWS). This will be the target database.
- A [Redis Cloud Pro database](/content/operate/rc/databases/create-database/create-pro-database-new.md) hosted on Amazon Web Services (AWS). This will be the target database.
- One or more supported source databases that are publicly accessible or hosted on an AWS EC2 instance, AWS RDS, or AWS Aurora:

| Database | Versions | AWS RDS Versions |
Expand All @@ -84,40 +84,39 @@ Before you can create a data pipeline, you must have:
| Snowflake | - | - |


{{< note >}}
Please be aware of the following limitations:

- The target database must be a Redis Cloud Pro database hosted on Amazon Web Services (AWS). Redis Cloud Essentials databases and databases hosted on Google Cloud do not support Data Integration.
- The target database must use [high availability]({{< relref "/operate/rc/databases/configuration/high-availability" >}}). It can use either single-zone or multi-zone high availability.
- The target database can use TLS, but can not use mutual TLS.
- The target database can't use Active-Active topology.
- If your source database is not publicly accessible, or if it is a MongoDB Atlas or Snowflake database, it must be hosted on AWS.
- You must use a [custom encryption key on AWS](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html) to create the instance hosting the database.
- Each pipeline has one target database shared by all of its sources.
- If the source database is not publicly accessible, you must be able to set up AWS PrivateLink to connect your source database to your target database. RDI only works with AWS PrivateLink and not VPC Peering or other private connectivity options.
- Mutual TLS is not supported for AWS RDS and AWS Aurora source databases.
{{< /note >}}
> [!NOTE]
> Please be aware of the following limitations:
>
> - The target database must be a Redis Cloud Pro database hosted on Amazon Web Services (AWS). Redis Cloud Essentials databases and databases hosted on Google Cloud do not support Data Integration.
> - The target database must use [high availability](/content/operate/rc/databases/configuration/high-availability.md). It can use either single-zone or multi-zone high availability.
> - The target database can use TLS, but can not use mutual TLS.
> - The target database can't use Active-Active topology.
> - If your source database is not publicly accessible, or if it is a MongoDB Atlas or Snowflake database, it must be hosted on AWS.
> - You must use a [custom encryption key on AWS](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html) to create the instance hosting the database.
> - Each pipeline has one target database shared by all of its sources.
> - If the source database is not publicly accessible, you must be able to set up AWS PrivateLink to connect your source database to your target database. RDI only works with AWS PrivateLink and not VPC Peering or other private connectivity options.
> - Mutual TLS is not supported for AWS RDS and AWS Aurora source databases.

## Get started

To get started fast with RDI on Redis Cloud, see the [RDI Cloud quick start]({{<relref "operate/rc/rdi/quick-start">}}) to create a data pipeline between a PostgreSQL source database and a Redis Cloud target database.
To get started fast with RDI on Redis Cloud, see the [RDI Cloud quick start](/content/operate/rc/rdi/quick-start.md) to create a data pipeline between a PostgreSQL source database and a Redis Cloud target database.

To create a new data pipeline, you need to:

1. [Create a Data Integration workspace]({{<relref "/operate/rc/rdi/create-workspace">}}) for your Pro subscription.
1. [Prepare each source database]({{<relref "/operate/rc/rdi/setup">}}) and any associated credentials.
1. [Define the source connection and data pipeline]({{<relref "/operate/rc/rdi/define">}}) by selecting which tables to sync.
1. [Create a Data Integration workspace](/content/operate/rc/rdi/create-workspace.md) for your Pro subscription.
1. [Prepare each source database](/content/operate/rc/rdi/setup.md) and any associated credentials.
1. [Define the source connection and data pipeline](/content/operate/rc/rdi/define.md) by selecting which tables to sync.

Once your data pipeline is defined, you can [view and edit]({{<relref "/operate/rc/rdi/view-edit">}}) it.
Once your data pipeline is defined, you can [view and edit](/content/operate/rc/rdi/view-edit.md) it.

For complete production setups, including SQL Server failover handling, see [Production use cases]({{<relref "/operate/rc/rdi/use-cases">}}).
For complete production setups, including SQL Server failover handling, see [Production use cases](/content/operate/rc/rdi/use-cases/_index.md).

## Billing and common questions

See the [RDI Cloud FAQ]({{< relref "/operate/rc/rdi/faq" >}}) for billing examples, reset and flush behavior, and working with multiple sources.
See the [RDI Cloud FAQ](/content/operate/rc/rdi/faq.md) for billing examples, reset and flush behavior, and working with multiple sources.

## Maintenance windows

RDI Cloud maintenance follows the same subscription-wide maintenance window as your Redis Cloud Pro subscription. During a maintenance window, your data pipeline may experience brief interruptions as Redis applies updates.

To control when maintenance occurs, [set a manual maintenance window]({{< relref "/operate/rc/subscriptions/maintenance/set-maintenance-windows" >}}) for your Redis Cloud Pro subscription. Any maintenance window you configure applies to both your databases and your RDI data pipeline.
To control when maintenance occurs, [set a manual maintenance window](/content/operate/rc/subscriptions/maintenance/set-maintenance-windows.md) for your Redis Cloud Pro subscription. Any maintenance window you configure applies to both your databases and your RDI data pipeline.
5 changes: 2 additions & 3 deletions content/operate/rc/rdi/create-workspace.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,8 @@ There, you'll see your workspace and its pipelines. The **Sources** column lists

## Delete workspace

{{< warning >}}
Make sure to [delete your data pipeline]({{<relref "/operate/rc/rdi/view-edit#delete-pipeline">}}) before deleting your workspace.
{{< /warning >}}
> [!WARNING]
> Make sure to [delete your data pipeline](/content/operate/rc/rdi/view-edit.md#delete-pipeline) before deleting your workspace.

To delete your workspace, select **Workspace actions > Delete workspace** from your workspace.

Expand Down
Loading
Loading