Document Cloud RDI pipeline capacity

content/operate/rc/rdi/_index.md

@@ -152,6 +152,9 @@ Once your data pipeline is defined, you can [view and edit]({{<relref "/operate/
 
 For complete production setups, including SQL Server failover handling, see [Production use cases]({{<relref "/operate/rc/rdi/use-cases">}}).
 
+To change processing capacity, see [Scale a Cloud RDI pipeline]({{<
+relref "/operate/rc/rdi/scale-pipeline" >}}).
+
 ## Billing and common questions
 
 See the [RDI Cloud FAQ]({{< relref "/operate/rc/rdi/faq" >}}) for billing examples, reset and flush behavior, and working with multiple sources.

content/operate/rc/rdi/faq.md

@@ -57,6 +57,34 @@ Stop the affected source before deleting its records and allow its pending recor
 
 No. Deleting a source removes its pipeline configuration and internal RDI state. Records it already wrote to the target remain. You must remove or reassign transformation jobs that refer to the source before deleting it. See [Remove a source]({{< relref "/operate/rc/rdi/view-edit#remove-source" >}}).
 
+## Capacity and network planning
+
+### How many sources and processor replicas can a pipeline use?
+
+Cloud RDI runs one Flink processor for each pipeline. A pipeline supports up to
+10 sources and up to 10 TaskManagers. Each source uses a collector, and each
+TaskManager is a processor replica.
+
+The source and TaskManager limits do not guarantee that an existing workspace
+has enough network capacity to run them all. Plan the workspace CIDR before you
+scale near these limits. See [Scale a Cloud RDI pipeline]({{< relref
+"/operate/rc/rdi/scale-pipeline" >}}).
+
+### What CIDR range should I use for Cloud RDI?
+
+Cloud RDI suggests a Classless Inter-Domain Routing (CIDR) range of `/22` by
+default. Use it for a pipeline with up to 5 sources and 5 TaskManagers. If you
+plan to use more sources or TaskManagers, choose at least a `/21` CIDR range
+when you create the workspace.
+
+Cloud RDI needs complete IP address blocks in each Availability Zone to start
+new nodes. Node replacements and maintenance can temporarily need additional
+capacity. A `/21` is planning guidance, not a guarantee for every workload.
+
+You cannot enlarge an existing workspace CIDR. Choose a range that leaves
+capacity for planned growth, replacement nodes, and maintenance, and is
+compatible with your selected connectivity method.
+
 ## Upgrades and maintenance
 
 ### What happens during an RDI Cloud upgrade?
@@ -109,3 +137,43 @@ Yes. RDI usage is based on the running collectors and processor replicas, not th
 Creating a workspace or saving a setup draft does not start RDI usage billing. Billing starts when a pipeline is deployed. Stopping the pipeline reduces usage to the workspace charge. Deleting the deployed pipeline ends new RDI usage when no deployed pipelines remain in the workspace; usage already recorded for the hour can still be billed.
 
 Delete an unused pipeline and then [delete its workspace]({{< relref "/operate/rc/rdi/create-workspace#delete-workspace" >}}) when you no longer need RDI. This does not delete the target Redis database or stop its separate charges.
+
+## Processor scaling
+
+### Does Cloud RDI automatically scale the processor?
+
+No. Cloud RDI does not automatically add or remove TaskManagers based on
+processor load, pending records, throughput, or backpressure. Set the desired
+number of TaskManagers with
+`advanced.resources.taskManager.replicas`. See [Scale a Cloud RDI pipeline]({{<
+relref "/operate/rc/rdi/scale-pipeline" >}}).
+
+### How do I increase processing capacity for a pipeline?
+
+First identify whether the collector, RDI database, Flink processor, or target
+database limits the pipeline. You can increase collector ingestion capacity
+with source-specific advanced collector properties. You can increase the RDI
+database throughput in the database **Performance** settings. For a processor
+bottleneck, edit **Settings** and set
+`advanced.resources.taskManager.replicas` to the needed number. Save the
+change, then apply and restart the pipeline. See [Scale a Cloud RDI pipeline]({{<
+relref "/operate/rc/rdi/scale-pipeline" >}}) for signals and tuning guidance.
+
+### Can I see that the processor has scaled in the console?
+
+Yes. The Dashboard shows the processor replica count next to the processor
+status. It shows the running count and configured count in the format
+`running / configured replicas`. For example, `3 / 3 replicas` means three
+TaskManagers are running and the pipeline is configured for three.
+
+The Metrics tab shows data-stream record counts and pending records, but not
+the replica count.
+
+For programmatic confirmation, use the RDI API pipeline-status response and
+inspect the `flink-processor` component's `replicas` value. See [Confirm the
+applied capacity]({{< relref "/operate/rc/rdi/scale-pipeline" >}}).
+
+### Can I use billing to confirm a scaling change?
+
+No. Billing is not a real-time deployment-status signal. Use the Dashboard or
+the RDI API status instead.

content/operate/rc/rdi/scale-pipeline.md

@@ -0,0 +1,157 @@
+---
+Title: Scale a Cloud RDI pipeline
+aliases:
+    - /operate/rc/rdi/scale-processor/
+    - /operate/rc/rdi/scale-processor
+    - /operate/rc/databases/rdi/scale-processor/
+    - /operate/rc/databases/rdi/scale-processor
+alwaysopen: false
+categories:
+- docs
+- operate
+- rc
+description: Increase collector, RDI database, and processor capacity for a Cloud RDI pipeline.
+hideListLinks: true
+weight: 5
+---
+
+Every Cloud RDI pipeline uses the Flink processor. Pipeline capacity can be
+limited by the collector, the RDI database, the processor, or the target
+database. Identify the bottleneck before you change a setting.
+
+Cloud RDI does not automatically add or remove TaskManagers based on CPU use,
+pending records, throughput, or backpressure. Set the number of TaskManagers
+when you need more processing capacity.
+
+A pipeline supports up to 10 sources and up to 10 TaskManagers. The default
+Classless Inter-Domain Routing (CIDR) range is `/22` for pipelines with up to 5
+sources and 5 TaskManagers. Before you scale beyond that, choose a larger CIDR
+when you create the workspace. See [Capacity and network planning]({{< relref
+"/operate/rc/rdi/faq#capacity-and-network-planning" >}}).
+
+## Increase collector ingestion capacity
+
+Tune collector properties when the collector cannot ingest data from the
+source quickly enough. These properties are specific to each source. In the
+Cloud console, select the source in **Configuration**, then select **Edit
+collector properties**.
+
+The console exposes these commonly tuned Debezium properties when they apply to
+the selected source:
+
+| Collector section | Property | Effect |
+| --- | --- | --- |
+| Source | `snapshot.max.threads` | Increases the threads used for the initial snapshot. |
+| Source | `snapshot.fetch.size` | Increases the rows fetched in each snapshot batch. |
+| Source | `max.batch.size` | Increases the records processed in each batch. |
+| Source | `max.queue.size` | Increases the records buffered in collector memory. |
+| Source | `poll.interval.ms` | Reduces the wait before the collector polls for CDC changes. |
+| Source | `record.processing.threads` | Increases the threads that process captured records. |
+| Sink | `redis.batch.size` | Increases the records written to the RDI database in each batch. |
+| Sink | `redis.flush.interval.ms` | Reduces the maximum wait before the collector flushes a batch. |
+
+Use only properties shown for the selected source. The available properties can
+vary by source database.
+
+Cloud RDI collectors have 2 CPUs and 8 GB of RAM. You cannot select a
+different collector size. Test each change with a representative workload
+before you use it in production.
+
+Larger batches and queues use more collector memory. More snapshot and
+record-processing threads share the same 2 CPUs. A shorter poll interval can
+reduce CDC latency but can increase load on the source database. Change one
+property at a time and use the Dashboard to compare throughput, pending
+records, and processor load.
+
+## Increase RDI database throughput
+
+Each pipeline has an RDI database named `rdi-pipeline-bdb-<target_db_id>`. Its
+default throughput is 25,000 operations per second (ops/sec). You can increase
+the throughput to 50,000, 75,000, or 100,000 ops/sec.
+
+1. In the Cloud console, open the RDI database for the pipeline.
+1. Edit the database **Performance** settings.
+1. Increase **Throughput** to the next value.
+1. Check the pipeline Dashboard before making another increase.
+
+Increase throughput gradually. Stop increasing it when the pipeline no longer
+improves.
+
+Increase RDI database throughput when the database frequently reports an
+out-of-memory (OOM) state and the processor is not continuously **Busy**. This
+can mean that the database cannot accept or serve records fast enough for the
+pipeline.
+
+## Increase processor capacity
+
+1. From the Cloud RDI **Pipelines** list, select the pipeline.
+1. Select the **Settings** tab and select **Edit**.
+1. Add or update the `advanced.resources.taskManager.replicas` property with
+   the number of TaskManagers you need. The value must be a whole number from
+   `1` through `10`.
+1. Select **Save changes**, then select **Apply and restart** to apply the
+   change.
+
+When you set this property, Cloud RDI uses that number of TaskManagers. If you
+do not set it, Cloud RDI calculates the number from the pipeline parallelism
+and the configured TaskManager slots. This is a configuration-time calculation,
+not reactive autoscaling while the pipeline is running.
+
+For API-based configuration, the same property is represented as follows:
+
+```yaml
+processors:
+  advanced:
+    resources:
+      taskManager:
+        replicas: 3
+```
+
+## Decide when to scale
+
+Use the **Dashboard** to decide whether the processor needs more capacity:
+
+- Check **Throughput** and **Pending**. A growing pending-record count or
+  lower-than-required throughput can show that the processor needs
+  investigation.
+- Check **Processor load**. A processor that remains **Busy** while pending
+  records grow can need more capacity. An **Idle** processor is waiting for
+  changes from the sources.
+- Check **RDI database load**. If the RDI database remains full or frequently
+  reports an out-of-memory (OOM) state while the processor remains **Busy**,
+  the processor might not read records as fast as collectors write them.
+  Increasing the number of TaskManagers can help.
+
+Check the source and target systems too. For example, a processor can be
+**Idle** because the target database is overloaded. Increasing TaskManagers
+does not remove a bottleneck outside the processor.
+
+## Confirm the applied capacity
+
+The **Dashboard** shows the processor replica count next to the processor
+status. It shows the running count and configured count in the format
+`running / configured replicas`. For example, `3 / 3 replicas` means three
+TaskManagers are running and the pipeline is configured for three. If only one
+count is available, the Dashboard identifies it as either running or configured.
+
+The console metrics show data-stream record counts and pending records. They do
+not include a TaskManager replica-count metric.
+
+If your organization collects Prometheus metrics for RDI, use them to
+investigate processor behavior. They do not provide a TaskManager replica-count
+metric.
+
+For programmatic confirmation, get the [pipeline status]({{< relref
+"/integrate/redis-data-integration/reference/api-reference" >}}) and inspect
+the `flink-processor` entry in `components`. Its `replicas` value is the number
+of ready processor replicas. For example:
+
+```json
+{
+  "name": "flink-processor",
+  "replicas": 3
+}
+```
+
+Billing is not a real-time way to confirm that a scaling change has completed.
+Use the Dashboard or the status API instead.

content/operate/rc/rdi/view-edit.md

@@ -152,7 +152,7 @@ The **Settings** tab contains the default data structure (**Hash** or **JSON**)
 
 {{<image filename="images/rc/rdi/rdi-2-settings.png" alt="Pipeline-wide Settings tab with default data structure and processor properties." width=100% >}}
 
-Select **Edit** to change these settings, then **Save changes** and **Apply and restart**. RDI Cloud uses the Flink processor. Review [processor properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors-data-processing-configuration" >}}) before changing them.
+Select **Edit** to change these settings, then **Save changes** and **Apply and restart**. Every Cloud RDI pipeline uses the Flink processor. Review [processor properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors-data-processing-configuration" >}}) before changing them. To change processor capacity, see [Scale a Cloud RDI pipeline]({{< relref "/operate/rc/rdi/scale-pipeline" >}}).
 
 {{<image filename="images/rc/rdi/rdi-processor-advanced-properties.png" alt="The processor advanced properties editor with key and value fields." width=80% >}}