From ae52ab4b8578db746603f9b52f535800af632f38 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Wed, 23 Sep 2026 10:59:35 +0300 Subject: [PATCH 01/11] Document Cloud RDI processor scaling --- content/operate/rc/rdi/_index.md | 3 + content/operate/rc/rdi/faq.md | 33 +++++++++ content/operate/rc/rdi/scale-processor.md | 86 +++++++++++++++++++++++ content/operate/rc/rdi/view-edit.md | 2 +- 4 files changed, 123 insertions(+), 1 deletion(-) create mode 100644 content/operate/rc/rdi/scale-processor.md diff --git a/content/operate/rc/rdi/_index.md b/content/operate/rc/rdi/_index.md index ba1cd6efab..21a1db613c 100644 --- a/content/operate/rc/rdi/_index.md +++ b/content/operate/rc/rdi/_index.md @@ -152,6 +152,9 @@ Once your data pipeline is defined, you can [view and edit]({{}}). +To change the processing capacity of a Flink pipeline, see [Scale a data +pipeline processor]({{< relref "/operate/rc/rdi/scale-processor" >}}). + ## Billing and common questions See the [RDI Cloud FAQ]({{< relref "/operate/rc/rdi/faq" >}}) for billing examples, reset and flush behavior, and working with multiple sources. diff --git a/content/operate/rc/rdi/faq.md b/content/operate/rc/rdi/faq.md index ada3e8513a..1a6e954d55 100644 --- a/content/operate/rc/rdi/faq.md +++ b/content/operate/rc/rdi/faq.md @@ -109,3 +109,36 @@ Yes. RDI usage is based on the running collectors and processor replicas, not th Creating a workspace or saving a setup draft does not start RDI usage billing. Billing starts when a pipeline is deployed. Stopping the pipeline reduces usage to the workspace charge. Deleting the deployed pipeline ends new RDI usage when no deployed pipelines remain in the workspace; usage already recorded for the hour can still be billed. Delete an unused pipeline and then [delete its workspace]({{< relref "/operate/rc/rdi/create-workspace#delete-workspace" >}}) when you no longer need RDI. This does not delete the target Redis database or stop its separate charges. + +## Processor scaling + +### Does Cloud RDI automatically scale the Flink processor? + +No. Cloud RDI does not automatically add or remove TaskManagers based on +processor load, pending records, throughput, or backpressure. Set the desired +number of TaskManagers with +`advanced.resources.taskManager.replicas`. See [Scale a data pipeline +processor]({{< relref "/operate/rc/rdi/scale-processor" >}}). + +### How do I increase processing capacity for a pipeline? + +For a Flink pipeline, edit its **Settings** and set +`advanced.resources.taskManager.replicas` to the needed number. Save the +change, then apply and restart the pipeline. The setting does not apply to +Classic pipelines. + +### Can I see that the processor has scaled in the console? + +The pipeline configuration shows the requested number of TaskManagers. The +Cloud RDI console does not currently show the ready replica count or a scale +event. Its Metrics tab shows data-stream record counts and pending records, but +not the replica count. + +For the ready replica count, use the RDI API pipeline-status response and +inspect the `flink-processor` component's `replicas` value. See [Confirm the +applied capacity]({{< relref "/operate/rc/rdi/scale-processor" >}}). + +### Can I use billing to confirm a scaling change? + +No. Billing is not a real-time deployment-status signal. Use the pipeline +configuration, processor metrics, or the RDI API status instead. diff --git a/content/operate/rc/rdi/scale-processor.md b/content/operate/rc/rdi/scale-processor.md new file mode 100644 index 0000000000..04297979a9 --- /dev/null +++ b/content/operate/rc/rdi/scale-processor.md @@ -0,0 +1,86 @@ +--- +Title: Scale a data pipeline processor +aliases: + - /operate/rc/databases/rdi/scale-processor/ + - /operate/rc/databases/rdi/scale-processor +alwaysopen: false +categories: +- docs +- operate +- rc +description: Change the processing capacity of a Redis Cloud data pipeline. +hideListLinks: true +weight: 5 +--- + +Cloud RDI pipelines that use the Flink processor can run more than one +TaskManager. Adding TaskManagers increases the processing capacity available to +the pipeline. + +Cloud RDI does not automatically add or remove TaskManagers based on CPU use, +pending records, throughput, or backpressure. Set the number of TaskManagers +when you need more processing capacity. + +{{< note >}} +This setting applies only to Flink pipelines. Classic pipelines do not use +Flink TaskManagers. +{{< /note >}} + +## Set the number of TaskManagers + +1. From the Cloud RDI **Pipelines** list, select the pipeline. +1. Select the **Settings** tab and select **Edit**. +1. Add or update the `advanced.resources.taskManager.replicas` property with + the number of TaskManagers you need. The value must be a whole number of at + least `1`. +1. Select **Save changes**, then select **Apply and restart** to apply the + change. + +When you set this property, Cloud RDI uses that number of TaskManagers. If you +do not set it, Cloud RDI calculates the number from the pipeline parallelism +and the configured TaskManager slots. This is a configuration-time calculation, +not reactive autoscaling while the pipeline is running. + +For API-based configuration, the same property is represented as follows: + +```yaml +processors: + advanced: + resources: + taskManager: + replicas: 3 +``` + +## Decide when to scale + +Use the pipeline **Metrics** tab to decide whether the processor needs more +capacity. For example, a growing pending-record count or lower-than-required +throughput can show that the pipeline needs investigation. Check the source and +target systems too: increasing TaskManagers does not remove a bottleneck +outside the processor. + +## Confirm the applied capacity + +The console shows the TaskManager value saved in the pipeline configuration. +It does not currently show the number of ready TaskManagers or a scale event. +The console metrics show data-stream record counts and pending records, but +they do not include a TaskManager replica-count metric. + +If your organization collects Prometheus metrics for RDI, use them to +investigate processor behavior. They are not a replacement for the status API +when you need the ready replica count. + +If you use the RDI API, get the [pipeline status]({{< relref +"/integrate/redis-data-integration/reference/api-reference" >}}) and inspect +the `flink-processor` entry in `components`. Its `replicas` value is the number +of ready processor replicas. For example: + +```json +{ + "name": "flink-processor", + "replicas": 3 +} +``` + +Billing is not a real-time way to confirm that a scaling change has completed. +Use the saved configuration, processor metrics, or the status API instead. diff --git a/content/operate/rc/rdi/view-edit.md b/content/operate/rc/rdi/view-edit.md index da4c2c9319..e39f205a39 100644 --- a/content/operate/rc/rdi/view-edit.md +++ b/content/operate/rc/rdi/view-edit.md @@ -152,7 +152,7 @@ The **Settings** tab contains the default data structure (**Hash** or **JSON**) {{Pipeline-wide Settings tab with default data structure and processor properties.}} -Select **Edit** to change these settings, then **Save changes** and **Apply and restart**. RDI Cloud uses the Flink processor. Review [processor properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors-data-processing-configuration" >}}) before changing them. +Select **Edit** to change these settings, then **Save changes** and **Apply and restart**. RDI Cloud uses the Flink processor. Review [processor properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors-data-processing-configuration" >}}) before changing them. To change processor capacity, see [Scale a data pipeline processor]({{< relref "/operate/rc/rdi/scale-processor" >}}). {{The processor advanced properties editor with key and value fields.}} From d81fdeb472ea21e0788e9010b655e30dc0bfe288 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Fri, 25 Sep 2026 16:31:45 +0300 Subject: [PATCH 02/11] Document Cloud RDI dashboard replica counts --- content/operate/rc/rdi/faq.md | 15 +++++++++------ content/operate/rc/rdi/scale-processor.md | 14 +++++++++----- 2 files changed, 18 insertions(+), 11 deletions(-) diff --git a/content/operate/rc/rdi/faq.md b/content/operate/rc/rdi/faq.md index 1a6e954d55..5dc54903b2 100644 --- a/content/operate/rc/rdi/faq.md +++ b/content/operate/rc/rdi/faq.md @@ -129,10 +129,13 @@ Classic pipelines. ### Can I see that the processor has scaled in the console? -The pipeline configuration shows the requested number of TaskManagers. The -Cloud RDI console does not currently show the ready replica count or a scale -event. Its Metrics tab shows data-stream record counts and pending records, but -not the replica count. +Yes. The Dashboard shows the processor replica count next to the processor +status. It shows the running count and configured count in the format +`running / configured replicas`. For example, `3 / 3 replicas` means three +TaskManagers are running and the pipeline is configured for three. + +The Metrics tab shows data-stream record counts and pending records, but not +the replica count. For the ready replica count, use the RDI API pipeline-status response and inspect the `flink-processor` component's `replicas` value. See [Confirm the @@ -140,5 +143,5 @@ applied capacity]({{< relref "/operate/rc/rdi/scale-processor" >}}). ### Can I use billing to confirm a scaling change? -No. Billing is not a real-time deployment-status signal. Use the pipeline -configuration, processor metrics, or the RDI API status instead. +No. Billing is not a real-time deployment-status signal. Use the Dashboard or +the RDI API status instead. diff --git a/content/operate/rc/rdi/scale-processor.md b/content/operate/rc/rdi/scale-processor.md index 04297979a9..7a7713e54c 100644 --- a/content/operate/rc/rdi/scale-processor.md +++ b/content/operate/rc/rdi/scale-processor.md @@ -61,10 +61,14 @@ outside the processor. ## Confirm the applied capacity -The console shows the TaskManager value saved in the pipeline configuration. -It does not currently show the number of ready TaskManagers or a scale event. -The console metrics show data-stream record counts and pending records, but -they do not include a TaskManager replica-count metric. +The **Dashboard** shows the processor replica count next to the processor +status. It shows the running count and configured count in the format +`running / configured replicas`. For example, `3 / 3 replicas` means three +TaskManagers are running and the pipeline is configured for three. If only one +count is available, the Dashboard identifies it as either running or configured. + +The console metrics show data-stream record counts and pending records. They do +not include a TaskManager replica-count metric. If your organization collects Prometheus metrics for RDI, use them to investigate processor behavior. They are not a replacement for the status API @@ -83,4 +87,4 @@ of ready processor replicas. For example: ``` Billing is not a real-time way to confirm that a scaling change has completed. -Use the saved configuration, processor metrics, or the status API instead. +Use the Dashboard or the status API instead. From 1cd97247a588754dbe678f63022eabb791a3c500 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Fri, 25 Sep 2026 21:15:04 +0300 Subject: [PATCH 03/11] Document Cloud RDI scaling signals --- content/operate/rc/rdi/scale-processor.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/content/operate/rc/rdi/scale-processor.md b/content/operate/rc/rdi/scale-processor.md index 7a7713e54c..fc9628c00a 100644 --- a/content/operate/rc/rdi/scale-processor.md +++ b/content/operate/rc/rdi/scale-processor.md @@ -53,11 +53,20 @@ processors: ## Decide when to scale -Use the pipeline **Metrics** tab to decide whether the processor needs more -capacity. For example, a growing pending-record count or lower-than-required -throughput can show that the pipeline needs investigation. Check the source and -target systems too: increasing TaskManagers does not remove a bottleneck -outside the processor. +Use the **Dashboard** to decide whether the processor needs more capacity: + +- Check **Throughput** and **Pending**. A growing pending-record count or + lower-than-required throughput can show that the processor needs + investigation. +- Check **Processor load**. A processor that remains **Busy** while pending + records grow can need more capacity. An **Idle** processor is waiting for + changes from the sources. +- Check **RDI database load**. If the RDI database remains in an out-of-memory + state, the processor cannot process data fast enough. Increasing the number + of TaskManagers can help. + +Check the source and target systems too. Increasing TaskManagers does not +remove a bottleneck outside the processor. ## Confirm the applied capacity From 60fa637eb805d7d535425c3cf4eaa1ac309e3d2a Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Fri, 25 Sep 2026 22:09:34 +0300 Subject: [PATCH 04/11] Document Cloud RDI capacity tuning --- content/operate/rc/rdi/faq.md | 11 ++- content/operate/rc/rdi/scale-processor.md | 104 ++++++++++++++++++++-- 2 files changed, 103 insertions(+), 12 deletions(-) diff --git a/content/operate/rc/rdi/faq.md b/content/operate/rc/rdi/faq.md index 5dc54903b2..fdd9564dd1 100644 --- a/content/operate/rc/rdi/faq.md +++ b/content/operate/rc/rdi/faq.md @@ -122,10 +122,15 @@ processor]({{< relref "/operate/rc/rdi/scale-processor" >}}). ### How do I increase processing capacity for a pipeline? -For a Flink pipeline, edit its **Settings** and set +First identify whether the collector, RDI database, Flink processor, or target +database limits the pipeline. You can increase collector ingestion capacity +with source-specific advanced collector properties. You can increase the RDI +database throughput in the database **Performance** settings. For a Flink +processor bottleneck, edit **Settings** and set `advanced.resources.taskManager.replicas` to the needed number. Save the -change, then apply and restart the pipeline. The setting does not apply to -Classic pipelines. +change, then apply and restart the pipeline. The TaskManager setting does not +apply to Classic pipelines. See [Scale a data pipeline processor]({{< relref +"/operate/rc/rdi/scale-processor" >}}) for signals and starting values. ### Can I see that the processor has scaled in the console? diff --git a/content/operate/rc/rdi/scale-processor.md b/content/operate/rc/rdi/scale-processor.md index fc9628c00a..6fbdbfc746 100644 --- a/content/operate/rc/rdi/scale-processor.md +++ b/content/operate/rc/rdi/scale-processor.md @@ -13,9 +13,9 @@ hideListLinks: true weight: 5 --- -Cloud RDI pipelines that use the Flink processor can run more than one -TaskManager. Adding TaskManagers increases the processing capacity available to -the pipeline. +Cloud RDI pipeline capacity can be limited by the collector, the RDI database, +the Flink processor, or the target database. Identify the bottleneck before you +change a setting. Cloud RDI does not automatically add or remove TaskManagers based on CPU use, pending records, throughput, or backpressure. Set the number of TaskManagers @@ -26,6 +26,90 @@ This setting applies only to Flink pipelines. Classic pipelines do not use Flink TaskManagers. {{< /note >}} +## Increase collector ingestion capacity + +Tune collector properties when the collector cannot ingest data from the +source quickly enough. These properties are specific to each source. In the +Cloud console, select the source in **Configuration**, then select **Edit +collector properties**. + +The console exposes these commonly tuned Debezium properties when they apply to +the selected source: + +| Collector section | Property | Effect | +| --- | --- | --- | +| Source | `snapshot.max.threads` | Increases the threads used for the initial snapshot. | +| Source | `snapshot.fetch.size` | Increases the rows fetched in each snapshot batch. | +| Source | `max.batch.size` | Increases the records processed in each batch. | +| Source | `max.queue.size` | Increases the records buffered in collector memory. | +| Source | `poll.interval.ms` | Reduces the wait before the collector polls for CDC changes. | +| Source | `record.processing.threads` | Increases the threads that process captured records. | +| Sink | `redis.batch.size` | Increases the records written to the RDI database in each batch. | +| Sink | `redis.flush.interval.ms` | Reduces the maximum wait before the collector flushes a batch. | + +Use only properties shown for the selected source. The available properties can +vary by source database. + +The following are starting points for a collector with eight CPU cores. Test +them with a representative workload before you use them in production. + +### Prioritize snapshot throughput + +Use this profile when completing the initial snapshot is more important than +CDC latency. + +| Collector section | Property | Value | +| --- | --- | --- | +| Source | `snapshot.max.threads` | `2` | +| Source | `snapshot.fetch.size` | `40000` | +| Source | `max.batch.size` | `16000` | +| Source | `max.queue.size` | `92000` | +| Source | `poll.interval.ms` | `10` | +| Source | `record.processing.threads` | `6` | +| Sink | `redis.batch.size` | `10000` | +| Sink | `redis.flush.interval.ms` | `1` | + +This profile assigns two snapshot threads and six record-processing threads. +The queue needs enough collector memory to buffer 92,000 records. + +### Prioritize CDC latency + +Use this profile when CDC latency is more important than snapshot throughput. + +| Collector section | Property | Value | +| --- | --- | --- | +| Source | `snapshot.max.threads` | `1` | +| Source | `snapshot.fetch.size` | `40000` | +| Source | `max.batch.size` | `32000` | +| Source | `max.queue.size` | `140000` | +| Source | `poll.interval.ms` | `1` | +| Source | `record.processing.threads` | `8` | +| Sink | `redis.batch.size` | `12000` | +| Sink | `redis.flush.interval.ms` | `1` | + +This profile uses frequent polling to reduce CDC latency. The 140,000-record +queue needs more collector memory. Monitor the collector memory use and lower +the queue size if memory pressure occurs. + +## Increase RDI database throughput + +Each pipeline has an RDI database named `rdi-pipeline-bdb-`. Its +default throughput is 25,000 operations per second (ops/sec). You can increase +the throughput to 50,000, 75,000, or 100,000 ops/sec. + +1. In the Cloud console, open the RDI database for the pipeline. +1. Edit the database **Performance** settings. +1. Increase **Throughput** to the next value. +1. Check the pipeline Dashboard before making another increase. + +Increase throughput gradually. Stop increasing it when the pipeline no longer +improves. + +Increase RDI database throughput when the database frequently reports an +out-of-memory (OOM) state and the processor is not continuously **Busy**. This +can mean that the database cannot accept or serve records fast enough for the +pipeline. + ## Set the number of TaskManagers 1. From the Cloud RDI **Pipelines** list, select the pipeline. @@ -61,12 +145,14 @@ Use the **Dashboard** to decide whether the processor needs more capacity: - Check **Processor load**. A processor that remains **Busy** while pending records grow can need more capacity. An **Idle** processor is waiting for changes from the sources. -- Check **RDI database load**. If the RDI database remains in an out-of-memory - state, the processor cannot process data fast enough. Increasing the number - of TaskManagers can help. - -Check the source and target systems too. Increasing TaskManagers does not -remove a bottleneck outside the processor. +- Check **RDI database load**. If the RDI database remains full or frequently + reports an out-of-memory (OOM) state while the processor remains **Busy**, + the processor might not read records as fast as collectors write them. + Increasing the number of TaskManagers can help. + +Check the source and target systems too. For example, a processor can be +**Idle** because the target database is overloaded. Increasing TaskManagers +does not remove a bottleneck outside the processor. ## Confirm the applied capacity From 510a241cd6922b7f635af081e63a3fce07cf35fe Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Fri, 25 Sep 2026 22:13:41 +0300 Subject: [PATCH 05/11] Clarify Cloud RDI processor model --- content/operate/rc/rdi/_index.md | 4 ++-- content/operate/rc/rdi/faq.md | 16 ++++++++-------- content/operate/rc/rdi/scale-processor.md | 17 ++++++----------- content/operate/rc/rdi/view-edit.md | 2 +- 4 files changed, 17 insertions(+), 22 deletions(-) diff --git a/content/operate/rc/rdi/_index.md b/content/operate/rc/rdi/_index.md index 21a1db613c..abf6e7eb75 100644 --- a/content/operate/rc/rdi/_index.md +++ b/content/operate/rc/rdi/_index.md @@ -152,8 +152,8 @@ Once your data pipeline is defined, you can [view and edit]({{}}). -To change the processing capacity of a Flink pipeline, see [Scale a data -pipeline processor]({{< relref "/operate/rc/rdi/scale-processor" >}}). +To change processing capacity, see [Increase Cloud RDI pipeline capacity]({{< +relref "/operate/rc/rdi/scale-processor" >}}). ## Billing and common questions diff --git a/content/operate/rc/rdi/faq.md b/content/operate/rc/rdi/faq.md index fdd9564dd1..132720806c 100644 --- a/content/operate/rc/rdi/faq.md +++ b/content/operate/rc/rdi/faq.md @@ -112,25 +112,25 @@ Delete an unused pipeline and then [delete its workspace]({{< relref "/operate/r ## Processor scaling -### Does Cloud RDI automatically scale the Flink processor? +### Does Cloud RDI automatically scale the processor? No. Cloud RDI does not automatically add or remove TaskManagers based on processor load, pending records, throughput, or backpressure. Set the desired number of TaskManagers with -`advanced.resources.taskManager.replicas`. See [Scale a data pipeline -processor]({{< relref "/operate/rc/rdi/scale-processor" >}}). +`advanced.resources.taskManager.replicas`. See [Increase Cloud RDI pipeline +capacity]({{< relref "/operate/rc/rdi/scale-processor" >}}). ### How do I increase processing capacity for a pipeline? First identify whether the collector, RDI database, Flink processor, or target database limits the pipeline. You can increase collector ingestion capacity with source-specific advanced collector properties. You can increase the RDI -database throughput in the database **Performance** settings. For a Flink -processor bottleneck, edit **Settings** and set +database throughput in the database **Performance** settings. For a processor +bottleneck, edit **Settings** and set `advanced.resources.taskManager.replicas` to the needed number. Save the -change, then apply and restart the pipeline. The TaskManager setting does not -apply to Classic pipelines. See [Scale a data pipeline processor]({{< relref -"/operate/rc/rdi/scale-processor" >}}) for signals and starting values. +change, then apply and restart the pipeline. See [Increase Cloud RDI pipeline +capacity]({{< relref "/operate/rc/rdi/scale-processor" >}}) for signals and +starting values. ### Can I see that the processor has scaled in the console? diff --git a/content/operate/rc/rdi/scale-processor.md b/content/operate/rc/rdi/scale-processor.md index 6fbdbfc746..1d0c3105a7 100644 --- a/content/operate/rc/rdi/scale-processor.md +++ b/content/operate/rc/rdi/scale-processor.md @@ -1,5 +1,5 @@ --- -Title: Scale a data pipeline processor +Title: Increase Cloud RDI pipeline capacity aliases: - /operate/rc/databases/rdi/scale-processor/ - /operate/rc/databases/rdi/scale-processor @@ -8,24 +8,19 @@ categories: - docs - operate - rc -description: Change the processing capacity of a Redis Cloud data pipeline. +description: Increase collector, RDI database, and processor capacity for a Cloud RDI pipeline. hideListLinks: true weight: 5 --- -Cloud RDI pipeline capacity can be limited by the collector, the RDI database, -the Flink processor, or the target database. Identify the bottleneck before you -change a setting. +Every Cloud RDI pipeline uses the Flink processor. Pipeline capacity can be +limited by the collector, the RDI database, the processor, or the target +database. Identify the bottleneck before you change a setting. Cloud RDI does not automatically add or remove TaskManagers based on CPU use, pending records, throughput, or backpressure. Set the number of TaskManagers when you need more processing capacity. -{{< note >}} -This setting applies only to Flink pipelines. Classic pipelines do not use -Flink TaskManagers. -{{< /note >}} - ## Increase collector ingestion capacity Tune collector properties when the collector cannot ingest data from the @@ -110,7 +105,7 @@ out-of-memory (OOM) state and the processor is not continuously **Busy**. This can mean that the database cannot accept or serve records fast enough for the pipeline. -## Set the number of TaskManagers +## Increase processor capacity 1. From the Cloud RDI **Pipelines** list, select the pipeline. 1. Select the **Settings** tab and select **Edit**. diff --git a/content/operate/rc/rdi/view-edit.md b/content/operate/rc/rdi/view-edit.md index e39f205a39..1ba3f78be6 100644 --- a/content/operate/rc/rdi/view-edit.md +++ b/content/operate/rc/rdi/view-edit.md @@ -152,7 +152,7 @@ The **Settings** tab contains the default data structure (**Hash** or **JSON**) {{Pipeline-wide Settings tab with default data structure and processor properties.}} -Select **Edit** to change these settings, then **Save changes** and **Apply and restart**. RDI Cloud uses the Flink processor. Review [processor properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors-data-processing-configuration" >}}) before changing them. To change processor capacity, see [Scale a data pipeline processor]({{< relref "/operate/rc/rdi/scale-processor" >}}). +Select **Edit** to change these settings, then **Save changes** and **Apply and restart**. Every Cloud RDI pipeline uses the Flink processor. Review [processor properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors-data-processing-configuration" >}}) before changing them. To change processor capacity, see [Increase Cloud RDI pipeline capacity]({{< relref "/operate/rc/rdi/scale-processor" >}}). {{The processor advanced properties editor with key and value fields.}} From 88c75fcdcd006e299d6e7d4a067e1bbffceacc03 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Fri, 25 Sep 2026 22:18:08 +0300 Subject: [PATCH 06/11] Clarify replica confirmation guidance --- content/operate/rc/rdi/faq.md | 2 +- content/operate/rc/rdi/scale-processor.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/content/operate/rc/rdi/faq.md b/content/operate/rc/rdi/faq.md index 132720806c..1ea2030fdc 100644 --- a/content/operate/rc/rdi/faq.md +++ b/content/operate/rc/rdi/faq.md @@ -142,7 +142,7 @@ TaskManagers are running and the pipeline is configured for three. The Metrics tab shows data-stream record counts and pending records, but not the replica count. -For the ready replica count, use the RDI API pipeline-status response and +For programmatic confirmation, use the RDI API pipeline-status response and inspect the `flink-processor` component's `replicas` value. See [Confirm the applied capacity]({{< relref "/operate/rc/rdi/scale-processor" >}}). diff --git a/content/operate/rc/rdi/scale-processor.md b/content/operate/rc/rdi/scale-processor.md index 1d0c3105a7..5ee05f1218 100644 --- a/content/operate/rc/rdi/scale-processor.md +++ b/content/operate/rc/rdi/scale-processor.md @@ -161,10 +161,10 @@ The console metrics show data-stream record counts and pending records. They do not include a TaskManager replica-count metric. If your organization collects Prometheus metrics for RDI, use them to -investigate processor behavior. They are not a replacement for the status API -when you need the ready replica count. +investigate processor behavior. They do not provide a TaskManager replica-count +metric. -If you use the RDI API, get the [pipeline status]({{< relref +For programmatic confirmation, get the [pipeline status]({{< relref "/integrate/redis-data-integration/reference/api-reference" >}}) and inspect the `flink-processor` entry in `components`. Its `replicas` value is the number of ready processor replicas. For example: From 1f2cabb7eb83161dcee11e42ac28879157ef68d1 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Fri, 25 Sep 2026 22:20:20 +0300 Subject: [PATCH 07/11] Correct Cloud collector capacity guidance --- content/operate/rc/rdi/faq.md | 2 +- content/operate/rc/rdi/scale-processor.md | 49 +++++------------------ 2 files changed, 10 insertions(+), 41 deletions(-) diff --git a/content/operate/rc/rdi/faq.md b/content/operate/rc/rdi/faq.md index 1ea2030fdc..709052d576 100644 --- a/content/operate/rc/rdi/faq.md +++ b/content/operate/rc/rdi/faq.md @@ -130,7 +130,7 @@ bottleneck, edit **Settings** and set `advanced.resources.taskManager.replicas` to the needed number. Save the change, then apply and restart the pipeline. See [Increase Cloud RDI pipeline capacity]({{< relref "/operate/rc/rdi/scale-processor" >}}) for signals and -starting values. +tuning guidance. ### Can I see that the processor has scaled in the console? diff --git a/content/operate/rc/rdi/scale-processor.md b/content/operate/rc/rdi/scale-processor.md index 5ee05f1218..450ad445a5 100644 --- a/content/operate/rc/rdi/scale-processor.md +++ b/content/operate/rc/rdi/scale-processor.md @@ -45,46 +45,15 @@ the selected source: Use only properties shown for the selected source. The available properties can vary by source database. -The following are starting points for a collector with eight CPU cores. Test -them with a representative workload before you use them in production. - -### Prioritize snapshot throughput - -Use this profile when completing the initial snapshot is more important than -CDC latency. - -| Collector section | Property | Value | -| --- | --- | --- | -| Source | `snapshot.max.threads` | `2` | -| Source | `snapshot.fetch.size` | `40000` | -| Source | `max.batch.size` | `16000` | -| Source | `max.queue.size` | `92000` | -| Source | `poll.interval.ms` | `10` | -| Source | `record.processing.threads` | `6` | -| Sink | `redis.batch.size` | `10000` | -| Sink | `redis.flush.interval.ms` | `1` | - -This profile assigns two snapshot threads and six record-processing threads. -The queue needs enough collector memory to buffer 92,000 records. - -### Prioritize CDC latency - -Use this profile when CDC latency is more important than snapshot throughput. - -| Collector section | Property | Value | -| --- | --- | --- | -| Source | `snapshot.max.threads` | `1` | -| Source | `snapshot.fetch.size` | `40000` | -| Source | `max.batch.size` | `32000` | -| Source | `max.queue.size` | `140000` | -| Source | `poll.interval.ms` | `1` | -| Source | `record.processing.threads` | `8` | -| Sink | `redis.batch.size` | `12000` | -| Sink | `redis.flush.interval.ms` | `1` | - -This profile uses frequent polling to reduce CDC latency. The 140,000-record -queue needs more collector memory. Monitor the collector memory use and lower -the queue size if memory pressure occurs. +Cloud RDI collectors have two CPUs and 8 GB of RAM. You cannot select a +different collector size. Test each change with a representative workload +before you use it in production. + +Larger batches and queues use more collector memory. More snapshot and +record-processing threads share the same two CPUs. A shorter poll interval can +reduce CDC latency but can increase load on the source database. Change one +property at a time and use the Dashboard to compare throughput, pending +records, and processor load. ## Increase RDI database throughput From ae3c03c1f6e3d1531e71540a6cb98bdc90c824d7 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Fri, 25 Sep 2026 22:21:35 +0300 Subject: [PATCH 08/11] Use consistent collector resource numerals --- content/operate/rc/rdi/scale-processor.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/operate/rc/rdi/scale-processor.md b/content/operate/rc/rdi/scale-processor.md index 450ad445a5..08c1cdc2a8 100644 --- a/content/operate/rc/rdi/scale-processor.md +++ b/content/operate/rc/rdi/scale-processor.md @@ -45,12 +45,12 @@ the selected source: Use only properties shown for the selected source. The available properties can vary by source database. -Cloud RDI collectors have two CPUs and 8 GB of RAM. You cannot select a +Cloud RDI collectors have 2 CPUs and 8 GB of RAM. You cannot select a different collector size. Test each change with a representative workload before you use it in production. Larger batches and queues use more collector memory. More snapshot and -record-processing threads share the same two CPUs. A shorter poll interval can +record-processing threads share the same 2 CPUs. A shorter poll interval can reduce CDC latency but can increase load on the source database. Change one property at a time and use the Dashboard to compare throughput, pending records, and processor load. From 26dba2c08468d6eb7bc47b51847e6505619c0ceb Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Fri, 25 Sep 2026 23:27:21 +0300 Subject: [PATCH 09/11] Document Cloud RDI capacity limits --- content/operate/rc/rdi/faq.md | 29 +++++++++++++++++++++++ content/operate/rc/rdi/scale-processor.md | 9 +++++-- 2 files changed, 36 insertions(+), 2 deletions(-) diff --git a/content/operate/rc/rdi/faq.md b/content/operate/rc/rdi/faq.md index 709052d576..454aaaabe5 100644 --- a/content/operate/rc/rdi/faq.md +++ b/content/operate/rc/rdi/faq.md @@ -57,6 +57,35 @@ Stop the affected source before deleting its records and allow its pending recor No. Deleting a source removes its pipeline configuration and internal RDI state. Records it already wrote to the target remain. You must remove or reassign transformation jobs that refer to the source before deleting it. See [Remove a source]({{< relref "/operate/rc/rdi/view-edit#remove-source" >}}). +## Capacity and network planning + +### How many sources and processor replicas can a pipeline use? + +Cloud RDI runs one Flink processor for each pipeline. A pipeline supports up to +10 sources and up to 10 TaskManagers. Each source uses a collector, and each +TaskManager is a processor replica. + +The source and TaskManager limits do not guarantee that an existing workspace +has enough network capacity to run them all. Plan the workspace CIDR before you +scale near these limits. See [Increase Cloud RDI pipeline capacity]({{< relref +"/operate/rc/rdi/scale-processor" >}}). + +### What CIDR range should I use for Cloud RDI? + +For a new three-Availability-Zone workspace that you plan to scale to 10 +sources and 10 TaskManagers, use a Classless Inter-Domain Routing (CIDR) range +of at least `/21`. A `/22` can lack the network capacity to start all +collectors and TaskManagers at this scale, even when it reports free individual +IP addresses. + +Cloud RDI needs complete IP address blocks in each Availability Zone to start +new nodes. Node replacements and maintenance can temporarily need additional +capacity. A `/21` is planning guidance, not a guarantee for every workload. + +You cannot enlarge an existing workspace CIDR. Choose a range that leaves +capacity for planned growth, replacement nodes, and maintenance, and is +compatible with your selected connectivity method. + ## Upgrades and maintenance ### What happens during an RDI Cloud upgrade? diff --git a/content/operate/rc/rdi/scale-processor.md b/content/operate/rc/rdi/scale-processor.md index 08c1cdc2a8..8c8d8e6499 100644 --- a/content/operate/rc/rdi/scale-processor.md +++ b/content/operate/rc/rdi/scale-processor.md @@ -21,6 +21,11 @@ Cloud RDI does not automatically add or remove TaskManagers based on CPU use, pending records, throughput, or backpressure. Set the number of TaskManagers when you need more processing capacity. +A pipeline supports up to 10 sources and up to 10 TaskManagers. Before you +scale near these limits, make sure the workspace has enough network capacity. +See [Capacity and network planning]({{< relref +"/operate/rc/rdi/faq#capacity-and-network-planning" >}}). + ## Increase collector ingestion capacity Tune collector properties when the collector cannot ingest data from the @@ -79,8 +84,8 @@ pipeline. 1. From the Cloud RDI **Pipelines** list, select the pipeline. 1. Select the **Settings** tab and select **Edit**. 1. Add or update the `advanced.resources.taskManager.replicas` property with - the number of TaskManagers you need. The value must be a whole number of at - least `1`. + the number of TaskManagers you need. The value must be a whole number from + `1` through `10`. 1. Select **Save changes**, then select **Apply and restart** to apply the change. From 04fed1e48fe88a1d49131bf7ae221d8b92ac0cad Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Fri, 25 Sep 2026 23:30:30 +0300 Subject: [PATCH 10/11] Clarify default Cloud RDI CIDR capacity --- content/operate/rc/rdi/faq.md | 9 ++++----- content/operate/rc/rdi/scale-processor.md | 7 ++++--- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/content/operate/rc/rdi/faq.md b/content/operate/rc/rdi/faq.md index 454aaaabe5..9879f1c265 100644 --- a/content/operate/rc/rdi/faq.md +++ b/content/operate/rc/rdi/faq.md @@ -72,11 +72,10 @@ scale near these limits. See [Increase Cloud RDI pipeline capacity]({{< relref ### What CIDR range should I use for Cloud RDI? -For a new three-Availability-Zone workspace that you plan to scale to 10 -sources and 10 TaskManagers, use a Classless Inter-Domain Routing (CIDR) range -of at least `/21`. A `/22` can lack the network capacity to start all -collectors and TaskManagers at this scale, even when it reports free individual -IP addresses. +Cloud RDI suggests a Classless Inter-Domain Routing (CIDR) range of `/22` by +default. Use it for a pipeline with up to 5 sources and 5 TaskManagers. If you +plan to use more sources or TaskManagers, choose at least a `/21` CIDR range +when you create the workspace. Cloud RDI needs complete IP address blocks in each Availability Zone to start new nodes. Node replacements and maintenance can temporarily need additional diff --git a/content/operate/rc/rdi/scale-processor.md b/content/operate/rc/rdi/scale-processor.md index 8c8d8e6499..ad4fc70398 100644 --- a/content/operate/rc/rdi/scale-processor.md +++ b/content/operate/rc/rdi/scale-processor.md @@ -21,9 +21,10 @@ Cloud RDI does not automatically add or remove TaskManagers based on CPU use, pending records, throughput, or backpressure. Set the number of TaskManagers when you need more processing capacity. -A pipeline supports up to 10 sources and up to 10 TaskManagers. Before you -scale near these limits, make sure the workspace has enough network capacity. -See [Capacity and network planning]({{< relref +A pipeline supports up to 10 sources and up to 10 TaskManagers. The default +Classless Inter-Domain Routing (CIDR) range is `/22` for pipelines with up to 5 +sources and 5 TaskManagers. Before you scale beyond that, choose a larger CIDR +when you create the workspace. See [Capacity and network planning]({{< relref "/operate/rc/rdi/faq#capacity-and-network-planning" >}}). ## Increase collector ingestion capacity From 3d057148a98cc2b866a3154a4932992d6b37e8b7 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Fri, 25 Sep 2026 23:34:13 +0300 Subject: [PATCH 11/11] Rename Cloud RDI scaling guide --- content/operate/rc/rdi/_index.md | 4 ++-- content/operate/rc/rdi/faq.md | 15 +++++++-------- .../rdi/{scale-processor.md => scale-pipeline.md} | 4 +++- content/operate/rc/rdi/view-edit.md | 2 +- 4 files changed, 13 insertions(+), 12 deletions(-) rename content/operate/rc/rdi/{scale-processor.md => scale-pipeline.md} (98%) diff --git a/content/operate/rc/rdi/_index.md b/content/operate/rc/rdi/_index.md index abf6e7eb75..88a0a2612c 100644 --- a/content/operate/rc/rdi/_index.md +++ b/content/operate/rc/rdi/_index.md @@ -152,8 +152,8 @@ Once your data pipeline is defined, you can [view and edit]({{}}). -To change processing capacity, see [Increase Cloud RDI pipeline capacity]({{< -relref "/operate/rc/rdi/scale-processor" >}}). +To change processing capacity, see [Scale a Cloud RDI pipeline]({{< +relref "/operate/rc/rdi/scale-pipeline" >}}). ## Billing and common questions diff --git a/content/operate/rc/rdi/faq.md b/content/operate/rc/rdi/faq.md index 9879f1c265..668ad7324f 100644 --- a/content/operate/rc/rdi/faq.md +++ b/content/operate/rc/rdi/faq.md @@ -67,8 +67,8 @@ TaskManager is a processor replica. The source and TaskManager limits do not guarantee that an existing workspace has enough network capacity to run them all. Plan the workspace CIDR before you -scale near these limits. See [Increase Cloud RDI pipeline capacity]({{< relref -"/operate/rc/rdi/scale-processor" >}}). +scale near these limits. See [Scale a Cloud RDI pipeline]({{< relref +"/operate/rc/rdi/scale-pipeline" >}}). ### What CIDR range should I use for Cloud RDI? @@ -145,8 +145,8 @@ Delete an unused pipeline and then [delete its workspace]({{< relref "/operate/r No. Cloud RDI does not automatically add or remove TaskManagers based on processor load, pending records, throughput, or backpressure. Set the desired number of TaskManagers with -`advanced.resources.taskManager.replicas`. See [Increase Cloud RDI pipeline -capacity]({{< relref "/operate/rc/rdi/scale-processor" >}}). +`advanced.resources.taskManager.replicas`. See [Scale a Cloud RDI pipeline]({{< +relref "/operate/rc/rdi/scale-pipeline" >}}). ### How do I increase processing capacity for a pipeline? @@ -156,9 +156,8 @@ with source-specific advanced collector properties. You can increase the RDI database throughput in the database **Performance** settings. For a processor bottleneck, edit **Settings** and set `advanced.resources.taskManager.replicas` to the needed number. Save the -change, then apply and restart the pipeline. See [Increase Cloud RDI pipeline -capacity]({{< relref "/operate/rc/rdi/scale-processor" >}}) for signals and -tuning guidance. +change, then apply and restart the pipeline. See [Scale a Cloud RDI pipeline]({{< +relref "/operate/rc/rdi/scale-pipeline" >}}) for signals and tuning guidance. ### Can I see that the processor has scaled in the console? @@ -172,7 +171,7 @@ the replica count. For programmatic confirmation, use the RDI API pipeline-status response and inspect the `flink-processor` component's `replicas` value. See [Confirm the -applied capacity]({{< relref "/operate/rc/rdi/scale-processor" >}}). +applied capacity]({{< relref "/operate/rc/rdi/scale-pipeline" >}}). ### Can I use billing to confirm a scaling change? diff --git a/content/operate/rc/rdi/scale-processor.md b/content/operate/rc/rdi/scale-pipeline.md similarity index 98% rename from content/operate/rc/rdi/scale-processor.md rename to content/operate/rc/rdi/scale-pipeline.md index ad4fc70398..cd4a16ee79 100644 --- a/content/operate/rc/rdi/scale-processor.md +++ b/content/operate/rc/rdi/scale-pipeline.md @@ -1,6 +1,8 @@ --- -Title: Increase Cloud RDI pipeline capacity +Title: Scale a Cloud RDI pipeline aliases: + - /operate/rc/rdi/scale-processor/ + - /operate/rc/rdi/scale-processor - /operate/rc/databases/rdi/scale-processor/ - /operate/rc/databases/rdi/scale-processor alwaysopen: false diff --git a/content/operate/rc/rdi/view-edit.md b/content/operate/rc/rdi/view-edit.md index 1ba3f78be6..015552e970 100644 --- a/content/operate/rc/rdi/view-edit.md +++ b/content/operate/rc/rdi/view-edit.md @@ -152,7 +152,7 @@ The **Settings** tab contains the default data structure (**Hash** or **JSON**) {{Pipeline-wide Settings tab with default data structure and processor properties.}} -Select **Edit** to change these settings, then **Save changes** and **Apply and restart**. Every Cloud RDI pipeline uses the Flink processor. Review [processor properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors-data-processing-configuration" >}}) before changing them. To change processor capacity, see [Increase Cloud RDI pipeline capacity]({{< relref "/operate/rc/rdi/scale-processor" >}}). +Select **Edit** to change these settings, then **Save changes** and **Apply and restart**. Every Cloud RDI pipeline uses the Flink processor. Review [processor properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors-data-processing-configuration" >}}) before changing them. To change processor capacity, see [Scale a Cloud RDI pipeline]({{< relref "/operate/rc/rdi/scale-pipeline" >}}). {{The processor advanced properties editor with key and value fields.}}