From fa635c7fb6fc7f71209c14a0e23250832f1b4ded Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Sun, 18 Oct 2026 04:23:51 +0300 Subject: [PATCH 1/4] RDSC-6239: Correct database PrivateLink setup instructions --- content/operate/rc/rdi/setup.md | 52 +++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 22 deletions(-) diff --git a/content/operate/rc/rdi/setup.md b/content/operate/rc/rdi/setup.md index 7de00a64c0..5203fec6fe 100644 --- a/content/operate/rc/rdi/setup.md +++ b/content/operate/rc/rdi/setup.md @@ -15,6 +15,8 @@ weight: 3 ## Prepare source database +Create the target Redis Cloud Pro database before starting the pipeline. See the [quickstart prerequisites](/content/operate/rc/rdi/quick-start.md#prerequisites). + Prepare every source database before adding it to a pipeline. Each source needs its own change data capture (CDC) configuration, connectivity, and credentials. See [Prerequisites](/content/operate/rc/rdi/_index.md#prerequisites) for a list of supported source databases and database versions. See [Prepare source databases](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md) to find steps for your database type: @@ -188,7 +190,7 @@ Select the steps for your database setup. To set up PrivateLink for a database hosted on an EC2 instance: -1. [Create a network load balancer](#create-network-load-balancer-ec2) that will route incoming HTTP requests to your database. +1. [Create a network load balancer](#create-network-load-balancer-ec2) that will route incoming TCP connections to your database. 1. [Create an endpoint service](#create-endpoint-service-ec2) through AWS PrivateLink. ### Create network load balancer {#create-network-load-balancer-ec2} @@ -199,29 +201,32 @@ In the [AWS Management Console](https://console.aws.amazon.com/), use the **Serv - **Scheme**: Select **Internal**. - **Load balancer IP address type**: Select **IPv4**. 1. In **Network mapping**, select the VPC and availability zone associated with your source database. -1. In **Security groups**, select the security group associated with your source database, or another security group that allows traffic from PrivateLink and allows traffic to the database. +1. Create a dedicated security group in the source database VPC for the network load balancer. Leave its inbound rules empty for this PrivateLink connection. Allow outbound TCP traffic to the database security group on the database port (`5432` for PostgreSQL unless you configured another port). +1. In the load balancer settings, select this group under **Security groups**. 1. In **Listeners and routing**: - 1. Select **Create target group** to [create a target group](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-target-group.html) with the following settings: + 1. Select **Create target group** to [create a target group](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-target-group.html) with the following settings: 1. In **Specify group details**: - **Target type**: Select **Instances**. - - **Protocol : Port**: Select **TCP**, and then enter the port number where your database is exposed. + - **Protocol : Port**: Select **TCP**, and then enter the database port (`5432` for PostgreSQL unless you configured another port). - The **IP address type** and **VPC** should be selected already and match the VPC you selected earlier. + 1. In **Health checks**, select **TCP** and use the traffic port. 1. In **Register targets**, select the EC2 instance that runs your source database, enter the port, and select **Include as pending below**. Then, select **Create target group** to create your target group. Return to **Listeners and routing** in the Network Load Balancer setup. 1. Set the following **Listener** properties: - **Protocol**: Select **TCP**. - - **Port**: Enter your source database's port. + - **Port**: Enter your source database's port (`5432` for PostgreSQL unless you configured another port), matching the target group. - **Default action**: Select the target group you created in the previous step. 1. Review the network load balancer settings, and then select **Create load balancer** to continue. 1. After the network load balancer is active, select **Security**. - If you selected the same security group as your source database, you must not enforce security group rules on PrivateLink traffic. Select **Edit** and then deselect **Enforce inbound rules on PrivateLink traffic**, and then select **Save changes**. + Select **Edit**, deselect **Enforce inbound rules on PrivateLink traffic**, and select **Save changes**. This allows accepted PrivateLink connections through the load balancer without an inbound CIDR rule. Configure allowed principals and connection acceptance in the endpoint service steps that follow. -1. Select the security group ID to open the Security group settings. +1. Open the security group attached to your source database. 1. Select **Edit inbound rules**, then **Add rule** to add a rule with the following settings: - - **Type**: Select **HTTP**. - - **Source**: Select **Anywhere - IPv4**. + - **Type**: Select **Custom TCP**. + - **Port range**: Enter the database port (`5432` for PostgreSQL unless you configured another port). + - **Source**: Select the network load balancer security group ID. This rule allows both database connections and the TCP health checks configured on the same port. Select **Save rules** to save your changes. -1. Select **Actions** > **Edit Load Balancer Attributes**. +1. Return to the network load balancer and select **Actions** > **Edit Load Balancer Attributes**. - Under **Load balancer targets selection policy** select **Enable cross-zone load balancing**. Click the **Save Changes** button. @@ -266,36 +271,39 @@ In the [AWS Management Console](https://console.aws.amazon.com/), use the **Serv - **Scheme**: Select **Internal**. - **Load balancer IP address type**: Select **IPv4**. 1. In **Network mapping**, select the VPC and availability zone associated with your source database. -1. In **Security groups**, select the security group associated with your source database, or another security group that allows traffic from PrivateLink and allows traffic to the database. +1. Create a dedicated security group in the source database VPC for the network load balancer. Leave its inbound rules empty for this PrivateLink connection. Allow outbound TCP traffic to the database security group on the database port (`5432` for PostgreSQL unless you configured another port). +1. In the load balancer settings, select this group under **Security groups**. 1. In **Listeners and routing**: - 1. Select **Create target group** to [create a target group](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-target-group.html) with the following settings: + 1. Select **Create target group** to [create a target group](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-target-group.html) with the following settings: 1. In **Specify group details**: - **Target type**: Select **IP Addresses**. - - **Protocol : Port**: Select **TCP**, and then enter the port number where your database is exposed. + - **Protocol : Port**: Select **TCP**, and then enter the database port (`5432` for PostgreSQL unless you configured another port). - The **IP address type** and **VPC** should be selected already and match the VPC you selected earlier. - 1. In **Register targets**, enter the static IP address of your database, enter the port, and select **Include as pending below**. Then, select **Create target group** to create your target group. Return to **Listeners and routing** in the Network Load Balancer setup. + 1. In **Health checks**, select **TCP** and use the traffic port. + 1. In **Register targets**, enter the currently resolved private IPv4 address of your database, enter the port, and select **Include as pending below**. Then, select **Create target group** to create your target group. Return to **Listeners and routing** in the Network Load Balancer setup. - To get the static IP address of your database, run the following command on an EC2 instance in the same VPC as the database: + To get the current private IPv4 address of your database, run the following command on an EC2 instance in the same VPC as the database: ```sh $ nslookup ``` - Replace `` with the endpoint of your RDS or Aurora database. + Replace `` with the writer endpoint of your RDS or Aurora database. The address can change after failover or maintenance. [Configure the existing failover Lambda](#setup-lambda-function) to update the target group when the writer address changes. 1. Set the following **Listener** properties: - **Protocol**: Select **TCP**. - - **Port**: Enter your source database's port. + - **Port**: Enter your source database's port (`5432` for PostgreSQL unless you configured another port), matching the target group. - **Default action**: Select the target group you created in the previous step. 1. Review the network load balancer settings, and then select **Create load balancer** to continue. 1. After the network load balancer is active, select **Security**. - If you selected the same security group as your source database, you must not enforce security group rules on PrivateLink traffic. Select **Edit** and then deselect **Enforce inbound rules on PrivateLink traffic**, and then select **Save changes**. + Select **Edit**, deselect **Enforce inbound rules on PrivateLink traffic**, and select **Save changes**. This allows accepted PrivateLink connections through the load balancer without an inbound CIDR rule. Configure allowed principals and connection acceptance in the endpoint service steps that follow. -1. Select the security group ID to open the Security group settings. +1. Open the security group attached to your source database. 1. Select **Edit inbound rules**, then **Add rule** to add a rule with the following settings: - - **Type**: Select **HTTP**. - - **Source**: Select **Anywhere - IPv4**. + - **Type**: Select **Custom TCP**. + - **Port range**: Enter the database port (`5432` for PostgreSQL unless you configured another port). + - **Source**: Select the network load balancer security group ID. This rule allows both database connections and the TCP health checks configured on the same port. Select **Save rules** to save your changes. -1. Select **Actions** > **Edit Load Balancer Attributes**. +1. Return to the network load balancer and select **Actions** > **Edit Load Balancer Attributes**. - Under **Load balancer targets selection policy** select **Enable cross-zone load balancing**. Click the **Save Changes** button. From b3eac292043a71d0a648941888fe631713a4b529 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Mon, 19 Oct 2026 20:49:52 +0300 Subject: [PATCH 2/4] RDSC-6239: Show direct PrivateLink connectivity and Lambda updates --- content/operate/rc/rdi/rds-proxy.md | 5 ++ content/operate/rc/rdi/setup.md | 32 +++++---- .../rdi/rdi-setup-diagram-aurora-lambda.svg | 71 +++++++++++++++++++ 3 files changed, 94 insertions(+), 14 deletions(-) create mode 100644 static/images/rc/rdi/rdi-setup-diagram-aurora-lambda.svg diff --git a/content/operate/rc/rdi/rds-proxy.md b/content/operate/rc/rdi/rds-proxy.md index cfa0a29389..9d276c04e6 100644 --- a/content/operate/rc/rdi/rds-proxy.md +++ b/content/operate/rc/rdi/rds-proxy.md @@ -29,6 +29,11 @@ RDS Proxy is a fully managed, highly available database proxy for Amazon RDS. Wh - **Added complexity**: RDS Proxy adds an additional layer between RDI and your database. - **Lambda provides better failover**: The Lambda function approach handles failover scenarios more efficiently. +The diagram shows the RDS Proxy alternative. It does not apply to PostgreSQL sources. + +![RDI connects through PrivateLink and a network load balancer to RDS Proxy, which forwards connections to the RDS or Aurora database.](/images/rc/rdi/rdi-setup-diagram-aurora.png) +{width="100%"} + If you still need to use RDS Proxy, follow the instructions below. ## Prerequisites diff --git a/content/operate/rc/rdi/setup.md b/content/operate/rc/rdi/setup.md index 5203fec6fe..fb623d7e34 100644 --- a/content/operate/rc/rdi/setup.md +++ b/content/operate/rc/rdi/setup.md @@ -169,7 +169,7 @@ Repeat the preparation for each source. Keep track of which endpoint service and If your source database is not accessible via a public endpoint, you need to set up an endpoint service through AWS PrivateLink to be able to connect to it. For how traffic flows over PrivateLink, how to connect to an on-premises database over AWS Direct Connect, and how to keep the connection available during failover, see the [AWS PrivateLink reference](/content/operate/rc/rdi/networking/aws-privatelink.md). -The following diagrams show the network setup for the different database setups: +Choose the diagram and instructions for your source database: - Database hosted on an AWS EC2 instance: @@ -178,8 +178,10 @@ The following diagrams show the network setup for the different database setups: - Database hosted on AWS RDS or AWS Aurora: - ![The network setup for a database hosted on AWS RDS or AWS Aurora.](/images/rc/rdi/rdi-setup-diagram-aurora.png) - {width="80%"} + ![RDI connects to the RDS or Aurora writer through PrivateLink and an internal network load balancer. Lambda updates the target IP address after an RDS event.](/images/rc/rdi/rdi-setup-diagram-aurora-lambda.svg) + {width="100%"} + + Solid arrows show database connections. Dashed arrows show how Lambda updates the target group when the database address changes. Lambda does not carry database traffic. Select the steps for your database setup. @@ -201,7 +203,9 @@ In the [AWS Management Console](https://console.aws.amazon.com/), use the **Serv - **Scheme**: Select **Internal**. - **Load balancer IP address type**: Select **IPv4**. 1. In **Network mapping**, select the VPC and availability zone associated with your source database. -1. Create a dedicated security group in the source database VPC for the network load balancer. Leave its inbound rules empty for this PrivateLink connection. Allow outbound TCP traffic to the database security group on the database port (`5432` for PostgreSQL unless you configured another port). +1. Create a security group for the network load balancer in the source database VPC: + - **Inbound rules**: Leave empty. You will disable inbound rule checks for PrivateLink traffic in a later step. + - **Outbound rules**: Allow TCP to the database security group on the database port (`5432` for PostgreSQL unless you configured another port). 1. In the load balancer settings, select this group under **Security groups**. 1. In **Listeners and routing**: 1. Select **Create target group** to [create a target group](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-target-group.html) with the following settings: @@ -252,16 +256,14 @@ For more details on AWS PrivateLink, see [Share your services through AWS Privat --tab-sep-- -To set up PrivateLink for a database hosted on AWS RDS or AWS Aurora: - -To connect to your RDS or Aurora database, we recommend using a Lambda function approach. This provides a reliable and secure connection method for all database types. +Connect the network load balancer (NLB) directly to your RDS or Aurora database. The database address can change after failover or maintenance. We recommend using the Lambda function to keep the NLB target group up to date. -1. [Create a network load balancer](#create-network-load-balancer-rds) that will route incoming requests to your database. +1. [Create a network load balancer](#create-network-load-balancer-rds) that forwards TCP connections to your database. 1. [Create an endpoint service](#create-endpoint-service-rds) through AWS PrivateLink. -1. [Set up Lambda function connectivity](#setup-lambda-function) to route requests to your database. +1. [Set up the Lambda function](#setup-lambda-function) to update the target group when the database address changes. > [!NOTE] -> If you have specific requirements that necessitate using RDS Proxy instead of the recommended Lambda function approach, see the [RDS Proxy setup guide](/content/operate/rc/rdi/rds-proxy.md). Note that RDS Proxy is not recommended and does not work with PostgreSQL. +> RDS Proxy is not recommended for RDI and does not support PostgreSQL logical replication. If you need this alternative for a compatible database, see the [RDS Proxy setup guide](/content/operate/rc/rdi/rds-proxy.md). ### Create network load balancer {#create-network-load-balancer-rds} @@ -271,7 +273,9 @@ In the [AWS Management Console](https://console.aws.amazon.com/), use the **Serv - **Scheme**: Select **Internal**. - **Load balancer IP address type**: Select **IPv4**. 1. In **Network mapping**, select the VPC and availability zone associated with your source database. -1. Create a dedicated security group in the source database VPC for the network load balancer. Leave its inbound rules empty for this PrivateLink connection. Allow outbound TCP traffic to the database security group on the database port (`5432` for PostgreSQL unless you configured another port). +1. Create a security group for the network load balancer in the source database VPC: + - **Inbound rules**: Leave empty. You will disable inbound rule checks for PrivateLink traffic in a later step. + - **Outbound rules**: Allow TCP to the database security group on the database port (`5432` for PostgreSQL unless you configured another port). 1. In the load balancer settings, select this group under **Security groups**. 1. In **Listeners and routing**: 1. Select **Create target group** to [create a target group](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-target-group.html) with the following settings: @@ -327,12 +331,12 @@ After you create the endpoint service, you need to add Redis Cloud as an Allowed For more details on AWS PrivateLink, see [Share your services through AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html). -### Set up Lambda function connectivity {#setup-lambda-function} +### Set up Lambda to update database addresses {#setup-lambda-function} > [!NOTE] -> Setting up the Lambda function is optional but recommended for production environments. The Lambda function provides automatic failover handling and a more robust connection to your RDS or Aurora database. +> The Lambda function is optional but recommended for production. Without it, you must update the target group yourself when the database address changes. -The Lambda function monitors RDS failover events and automatically updates the NLB Target Group to point to the new primary instance's IP address. This ensures RDI reconnects automatically after a failover. +RDS sends event notifications through Amazon Simple Notification Service (SNS). SNS invokes the Lambda function. Lambda looks up the current writer address using the Domain Name System (DNS) and updates the NLB target group. RDI can reconnect after the new database target is ready. #### Option 1: Use the Redis Terraform module diff --git a/static/images/rc/rdi/rdi-setup-diagram-aurora-lambda.svg b/static/images/rc/rdi/rdi-setup-diagram-aurora-lambda.svg new file mode 100644 index 0000000000..2bef6e34be --- /dev/null +++ b/static/images/rc/rdi/rdi-setup-diagram-aurora-lambda.svg @@ -0,0 +1,71 @@ + +RDI connects directly to RDS or Aurora through AWS PrivateLink +RDI connects through a VPC endpoint, an endpoint service, and an internal network load balancer to the database writer. The target group stores the writer IP address. On an RDS event, SNS invokes Lambda. Lambda resolves the writer endpoint with DNS and updates the target group. Lambda does not carry database traffic. The load balancer and database have separate security groups. SNS and Lambda are shown outside the customer VPC. + + + + + + +RDI with RDS or Aurora: direct connection and Lambda updates + +Customer AWS account + +Redis Cloud VPC + +Customer VPC + + +NLB security group +Database security group + +RDI + +VPC +endpoint + +Endpoint +service + +Internal network +load balancer + +IP target group +Writer IP + +RDS / Aurora +Database writer + + + + + +PrivateLink +PostgreSQL example: TCP 5432 for the listener, target and health check + +Amazon SNS +1. Invoke Lambda + +Lambda function +Update addresses + +DNS lookup +2. Resolve writer + +RDS event + + + +3. Update target IP +Lambda keeps the target group pointed at the current writer. +Database connections go through the NLB, not through Lambda. + +Database connection + +Failover update + From c13720f48e0e57b928183734ef2fedc17b0f316c Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Mon, 19 Oct 2026 20:57:56 +0300 Subject: [PATCH 3/4] RDSC-6239: Match PrivateLink diagram to Redis visual style --- .../rdi/rdi-setup-diagram-aurora-lambda.svg | 162 +++++++++++------- 1 file changed, 104 insertions(+), 58 deletions(-) diff --git a/static/images/rc/rdi/rdi-setup-diagram-aurora-lambda.svg b/static/images/rc/rdi/rdi-setup-diagram-aurora-lambda.svg index 2bef6e34be..9635a9a9bb 100644 --- a/static/images/rc/rdi/rdi-setup-diagram-aurora-lambda.svg +++ b/static/images/rc/rdi/rdi-setup-diagram-aurora-lambda.svg @@ -1,71 +1,117 @@ - + RDI connects directly to RDS or Aurora through AWS PrivateLink RDI connects through a VPC endpoint, an endpoint service, and an internal network load balancer to the database writer. The target group stores the writer IP address. On an RDS event, SNS invokes Lambda. Lambda resolves the writer endpoint with DNS and updates the target group. Lambda does not carry database traffic. The load balancer and database have separate security groups. SNS and Lambda are shown outside the customer VPC. - - + + - + RDI with RDS or Aurora: direct connection and Lambda updates - + Customer AWS account - + Redis Cloud VPC - + Customer VPC - - -NLB security group -Database security group - -RDI - -VPC -endpoint - -Endpoint -service - -Internal network -load balancer - -IP target group -Writer IP - -RDS / Aurora -Database writer - - - - - + + +NLB security group +Database security group + +RDI + +VPC +endpoint + +Endpoint +service + +Internal network +load balancer + +IP target group +Writer IP + +RDS / Aurora +Database writer + + + + + PrivateLink PostgreSQL example: TCP 5432 for the listener, target and health check - -Amazon SNS -1. Invoke Lambda - -Lambda function -Update addresses - -DNS lookup -2. Resolve writer - -RDS event - - - -3. Update target IP -Lambda keeps the target group pointed at the current writer. -Database connections go through the NLB, not through Lambda. - -Database connection - -Failover update + +Amazon SNS +1. Invoke Lambda + +Lambda function +Update addresses + +DNS lookup +2. Resolve writer + +RDS event + + + +3. Update target IP +Lambda keeps the target group pointed at the current writer. +Database connections go through the NLB, not through Lambda. + +Database connection + +Failover update + + From f9d3c3e3087b757e758104978d92b1777f13e5d4 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Mon, 19 Oct 2026 21:19:38 +0300 Subject: [PATCH 4/4] RDSC-6239: Use original Redis icons in PrivateLink diagram --- .../rdi/rdi-setup-diagram-aurora-lambda.svg | 85 ++++++------------- 1 file changed, 26 insertions(+), 59 deletions(-) diff --git a/static/images/rc/rdi/rdi-setup-diagram-aurora-lambda.svg b/static/images/rc/rdi/rdi-setup-diagram-aurora-lambda.svg index 9635a9a9bb..74a01831ce 100644 --- a/static/images/rc/rdi/rdi-setup-diagram-aurora-lambda.svg +++ b/static/images/rc/rdi/rdi-setup-diagram-aurora-lambda.svg @@ -9,8 +9,6 @@ @@ -28,17 +26,17 @@ text { font-family: ui-monospace, SFMono-Regular, Consolas, "Liberation Mono", m RDI -VPC -endpoint +VPC +endpoint -Endpoint -service +Endpoint +service -Internal network -load balancer +Internal network +load balancer -IP target group -Writer IP +IP target group +Writer IP RDS / Aurora Database writer @@ -50,14 +48,14 @@ text { font-family: ui-monospace, SFMono-Regular, Consolas, "Liberation Mono", m PrivateLink PostgreSQL example: TCP 5432 for the listener, target and health check -Amazon SNS -1. Invoke Lambda +Amazon SNS +1. Invoke Lambda -Lambda function -Update addresses +Lambda function +Update addresses -DNS lookup -2. Resolve writer +DNS lookup +2. Resolve writer RDS event @@ -71,47 +69,16 @@ text { font-family: ui-monospace, SFMono-Regular, Consolas, "Liberation Mono", m Failover update - + +