Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions assets/css/index.css
Original file line number Diff line number Diff line change
Expand Up @@ -1863,3 +1863,49 @@ a[href*="#no-click"], img[src*="#no-click"] {
white-space: nowrap;
border: 0;
}

/* Qualified chat widget accommodations. The widget (production only,
injected via GTM, owned by web/marketing) replaces root scrolling with a
<body> scroller: it measures the root scrollbar width N, then applies
html { width: calc(100dvw - N); margin-right: N; overflow: hidden } and
body { width: calc(100% + N) }, expecting body's scrollbar to land where
the root scrollbar was. When N > 0 (classic, always-visible scrollbars:
Windows, or macOS with a mouse), body overshoots html's box by N and
html's own overflow: hidden clips body's scrollbar out of view — the page
scrollbar disappears the moment the widget loads. The widget marks <html>
with q-docked-target while it is active and q-docked while the chat panel
is open. */

/* While the chat is closed, neutralize the width/margin compensation on
both elements so body fills the window exactly and its scrollbar sits at
the window edge, unclipped. While the panel is docked open (q-docked) the
widget's own geometry must win, so none of this applies then. */
html[q-docked-target]:not([q-docked]) {
width: 100dvw !important;
min-width: 100dvw !important;
max-width: 100dvw !important;
margin-right: 0 !important;
}

html[q-docked-target]:not([q-docked]) body {
width: 100% !important;
max-width: 100% !important;
}

/* The html { scrollbar-gutter: stable } rule set earlier in this file would
re-reserve a gutter inside the widget's overflow-hidden html box and push
body's scrollbar inboard by another scrollbar-width. Scoped to the widget
being active so normal pages keep the stable gutter. */
html[q-docked-target] {
scrollbar-gutter: auto !important;
}

/* While the chat is closed, the widget's invisible #q-messenger-frame
iframe overlays page content near the right edge with pointer events
enabled, stealing clicks and hovers from whatever is underneath —
including the page scrollbar, since Firefox hit-tests page content before
non-root scrollbars. Let events pass through; q-docked restores normal
pointer events for the open panel. */
html:not([q-docked]) #q-messenger-frame {
pointer-events: none !important;
}
1 change: 1 addition & 0 deletions content/develop/ai/ecosystem-integrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ Redis integrates with a wide range of AI frameworks, platforms, and tools to enh

## Additional integrations

- [**AISIX AI Gateway & Redis**](https://docs.api7.ai/ai-gateway/traffic-controls/caching): Use Redis to share response cache entries and coordinate [rate-limit counters](https://docs.api7.ai/ai-gateway/traffic-controls/rate-limits) across AISIX gateway instances.
- [**Microsoft Semantic Kernel**](https://learn.microsoft.com/en-us/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/redis-connector?pivots=programming-language-csharp): Use Redis as a vector store connector with Microsoft's Semantic Kernel framework.
- [**DocArray**](https://docs.docarray.org/user_guide/storing/index_redis/): Leverage Redis as a document store and vector database with Jina AI's DocArray.
- [**Redis Cloud on Vercel**](https://redis.io/blog/redis-cloud-now-available-on-vercel-marketplace/): Deploy and manage Redis databases directly from your Vercel dashboard with the Redis Cloud integration. Refer to the [setup guide]({{< relref "/operate/rc/cloud-integrations/vercel" >}}) for more details.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,26 +16,42 @@ type: integration
weight: 35
---

RDI ships with two stream processor implementations. The default *classic*
RDI ships with two stream processor implementations. The *classic*
processor is implemented in Python. The *Flink* processor is built on top of
[Apache Flink](https://flink.apache.org/). Both run on VM and Kubernetes
installations. The Flink processor can achieve much higher throughput
during snapshots, scales horizontally by changing the number of TaskManager replicas,
and uses Flink checkpointing for fault tolerance. See [Stream processor implementations]({{< relref "/integrate/redis-data-integration/architecture#stream-processor-implementations" >}})
for an overview.

The classic processor is the default in RDI 1.19.0. The Flink processor is the
default starting with RDI 2.0.0.

This page describes how to migrate an existing pipeline from the classic
processor to the Flink processor. The steps are the same on VMs and Kubernetes,
except for the optional Helm-level tuning in [Step 1](#step-1-configure-the-flink-processor-at-the-helm-chart-level-kubernetes),
which applies to Kubernetes only.

## Before you migrate

This procedure migrates the pipeline processor on RDI 1.19.0. It does not
upgrade RDI.

Before you start, save your existing configuration and jobs. Wait for the
Comment thread
ZdravkoDonev-redis marked this conversation as resolved.
initial snapshot to finish. Interrupting it causes the snapshot to restart
from the beginning.

{{< warning >}}
Switching processors with records still in the RDI input streams can leave
records unprocessed. Stop the collector and let the classic processor empty
the streams before switching.
{{< /warning >}}

Confirm that your pipeline is compatible with the Flink processor:

- `JSON.MERGE` semantics differ from the classic processor's Lua-based merge
when null values are involved (see
[`use_native_json_merge`]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors" >}})).
[`use_native_json_merge`]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors-data-processing-configuration" >}})).
The Flink processor always uses the native `JSON.MERGE` command when the
target database supports it.
- Ensure your Kubernetes cluster or VM has enough capacity for the Flink JobManager
Expand All @@ -46,7 +62,7 @@ Confirm that your pipeline is compatible with the Flink processor:
## Step 1: Configure the Flink processor at the Helm chart level (Kubernetes)

This step applies to **Kubernetes** installations only. On VM installations,
skip it and enable the Flink processor per pipeline in step 2.
continue with [Step 2](#step-2-disable-source-collection).

The Flink processor is always available — no opt-in is required at the Helm
chart level. The defaults are sized for typical workloads, so you can skip
Expand All @@ -55,27 +71,137 @@ TaskManager defaults, add an `operator.dataPlane.flinkProcessor` block to
your `rdi-values.yaml` file and run `helm upgrade` as described in
[Configure the Flink processor]({{< relref "/integrate/redis-data-integration/installation/install-k8s#configure-the-flink-processor" >}}).
Existing pipelines continue to run on the classic processor until you switch
them in step 2.
them in [Step 4](#step-4-switch-processors-and-resume-collection).

For VM installations, skip this step. You can configure per-pipeline Flink
resources in step 4.
resources in [Step 6](#step-6-tune-the-flink-processor-optional).

## Step 2: Disable source collection

In your existing `config.yaml`, add `active: false` under the source and set
`processors.type` to `classic`. Use your existing source name and preserve
all other source, target, processor, and job settings. This example shows
only the fields to change:

```yaml
sources:
<existing-source-name>:
active: false
processors:
type: classic
```

Deploy the complete configuration directory, including the existing jobs:

```bash
redis-di deploy --dir <pipeline-config-directory>
```

Wait for the deployment to finish and the source collector to stop. Keep
the pipeline active so the classic processor can process the remaining input records.
Do not use `redis-di stop` for this step, because it also stops the processor.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Do not use `redis-di stop` for this step, because it also stops the processor.


Applications can continue writing to the source database while collection
is disabled. Make sure the database change log retains the whole paused
interval. When the collector restarts, it resumes from the saved source
position and processes those changes.
Comment on lines +104 to +107

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Applications can continue writing to the source database while collection
is disabled. Make sure the database change log retains the whole paused
interval. When the collector restarts, it resumes from the saved source
position and processes those changes.
Applications can continue writing to the source database while collection
is disabled. When the collector restarts, it resumes from the saved source
position and processes those changes.


## Step 3: Wait for the input streams to empty

After the collector has stopped, wait for every input stream to have a length
of `0` in three complete checks, five seconds apart. An error, missing
statistics, or an unexpectedly empty stream list does not count as `0`.
Do not include DLQ streams. Use any of the following methods.

### Check with `redis-di`

Run:

```bash
redis-di describe default

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
redis-di describe default
redis-di describe

```

## Step 2: Switch the pipeline to the Flink processor
In the **Statistics** table, the **Pending** value for each classic processor
stream is its current length. Confirm that every input stream is listed and
that the values agree with the Redis command checks below. This **Pending**
value is different from consumer-group pending entries. `XPENDING` or group
lag of `0` alone does not prove that a stream is empty.
Comment on lines +124 to +128

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
In the **Statistics** table, the **Pending** value for each classic processor
stream is its current length. Confirm that every input stream is listed and
that the values agree with the Redis command checks below. This **Pending**
value is different from consumer-group pending entries. `XPENDING` or group
lag of `0` alone does not prove that a stream is empty.
In the **Statistics** table, the **Pending** value for each classic processor
stream is its current length. Confirm that every input stream is listed.


In the pipeline's `config.yaml`, set
### Check with Redis commands

Connect an authenticated Redis client to the **RDI database** that stores
the pipeline's input streams, not the target database. Find the input stream
keys with
[`SCAN`]({{< relref "/commands/scan" >}}):

```text
SCAN 0 MATCH data:{rdi}:* COUNT 1000 TYPE stream
```

If the returned cursor is not `0`, pass it to the next command:

```text
SCAN <returned-cursor> MATCH data:{rdi}:* COUNT 1000 TYPE stream
```

Repeat with each new cursor until the returned cursor is `0`, even if an
intermediate result contains no keys.

For every input stream returned, run
[`XLEN`]({{< relref "/commands/xlen" >}}):

```text
XLEN <input-stream-key>
```

Run a complete `SCAN` and all `XLEN` commands in each of the three checks.

### Check with Redis Insight

1. Connect Redis Insight to the RDI database and open **Browse**.
1. Filter by the pipeline's input stream pattern. For the default pipeline,
use `data:{rdi}:*`. Confirm that all input streams are listed.
1. Open each stream, select **Stream Data**, and use the refresh button.
Confirm that **Entries** is `0`.
1. Repeat the complete inventory and entry check three times, five seconds
apart.

You can also open the built-in **CLI** and run the `SCAN` and `XLEN` commands
shown above. The Browser and CLI results must contain the same streams and
lengths.

If records remain, keep the classic processor running and resolve its
processing errors before continuing. Do not delete stream entries, reset the
pipeline, or move consumer-group positions to make the count reach `0`.
Comment on lines +159 to +175

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Check with Redis Insight
1. Connect Redis Insight to the RDI database and open **Browse**.
1. Filter by the pipeline's input stream pattern. For the default pipeline,
use `data:{rdi}:*`. Confirm that all input streams are listed.
1. Open each stream, select **Stream Data**, and use the refresh button.
Confirm that **Entries** is `0`.
1. Repeat the complete inventory and entry check three times, five seconds
apart.
You can also open the built-in **CLI** and run the `SCAN` and `XLEN` commands
shown above. The Browser and CLI results must contain the same streams and
lengths.
If records remain, keep the classic processor running and resolve its
processing errors before continuing. Do not delete stream entries, reset the
pipeline, or move consumer-group positions to make the count reach `0`.


## Step 4: Switch processors and resume collection

After the drain check passes, remove the source's `active: false` setting
from the existing `config.yaml` and set
[`processors.type`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}})
to `flink`:

```yaml
processors:
type: flink
...
```

Then redeploy the pipeline. The operator stops the classic processor pods
and starts the Flink JobManager and TaskManager workloads for the pipeline.
RDI 1.19.0 requires this setting because its default processor is `classic`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
RDI 1.19.0 requires this setting because its default processor is `classic`.
RDI 1.19.x requires this setting because its default processor is `classic`. On RDI 2.0.0 the default processor is `flink`, so this setting is no longer required.


Keep the remaining configuration and jobs, then redeploy the complete
configuration directory:

```bash
redis-di deploy --dir <pipeline-config-directory>
```

Wait for the classic processor to terminate and the Flink JobManager and
TaskManager workloads to become healthy. Confirm that collection resumes
from the saved source position and changes committed during the pause reach
the target. Verify new inserts, updates, and deletes. The processor migration
is complete after these checks pass.

## Step 3: Adapt deprecated and classic-only properties
## Step 5: Adapt deprecated and classic-only properties

Some `processors` properties are no-ops, classic-only, or have moved to
`processors.advanced` for the Flink processor. The following table lists the
Expand All @@ -97,7 +223,7 @@ and the Flink processor silently ignores classic-only top-level properties, so k
both top-level properties and their `processors.advanced` equivalents lets
you switch back without further edits.

## Step 4: Tune the Flink processor (optional)
## Step 6: Tune the Flink processor (optional)

Fine-tune the Flink processor through the `processors.advanced` section.
For example:
Expand All @@ -121,10 +247,10 @@ processors:
```

See the
[`processors.advanced` reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors" >}})
[`processors.advanced` reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processorsadvanced-advanced-configuration" >}})
for the full set of available properties.

## Step 5: Update observability
## Step 7: Update observability

The Flink processor exposes Prometheus metrics directly
from the Flink JobManager and TaskManager pods.
Expand All @@ -135,6 +261,6 @@ for the `ServiceMonitor` configuration and the available metrics.
## Rolling back

To revert a pipeline to the classic processor, set `processors.type` back to
`classic` (or remove the property) and redeploy the pipeline. The
`processors.advanced` section is silently ignored by the classic processor,
so you don't need to remove it before switching back.
`classic` and redeploy the pipeline. This setting is required on RDI 2.0.0,
where the default is `flink`. The classic processor silently ignores
`processors.advanced`, so you don't need to remove it before switching back.
28 changes: 25 additions & 3 deletions content/integrate/redis-data-integration/installation/upgrade.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,28 @@ type: integration
weight: 30
---

{{< note >}}
Comment thread
ZdravkoDonev-redis marked this conversation as resolved.
Before upgrading to RDI 2.0.0, review the
[processor default change](#upgrading-to-rdi-200).
{{< /note >}}

## Upgrading to RDI 2.0.0

RDI 2.0.0 changes the default processor from `classic` to `flink`. This default
applies when the pipeline's `config.yaml` omits `processors.type`.

For an existing classic pipeline, choose one of these options before upgrading:

- To migrate to Flink, first follow
[Migrate from the classic processor to the Flink processor]({{< relref "/integrate/redis-data-integration/installation/migration-classic-to-flink" >}})
on RDI 1.19.0. This stops the collector and drains the input streams before
switching processors. Then upgrade RDI.
- To keep the classic processor, set `processors.type: classic` in the
pipeline's `config.yaml` and deploy it before upgrading.

If your pipeline already uses `processors.type: flink`, no processor change
is needed. Continue with the upgrade instructions for your installation.

## Upgrading a VM installation

Follow the steps below to upgrade an existing
Expand Down Expand Up @@ -181,11 +203,11 @@ The
fully supported on both VM and Kubernetes installations after upgrading to
RDI 1.19.0. Once the upgrade completes, it is always available —
no opt-in is required, and the defaults are sized for typical workloads.
Upgrading does not change the processor used by existing pipelines, which keep
running on the classic processor until you explicitly switch them by
setting
On RDI 1.19.0, existing classic pipelines keep using that processor until
you switch them by setting
[`processors.type`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}})
to `flink` in their `config.yaml`.
RDI 2.0.0 changes this default; see [Upgrading to RDI 2.0.0](#upgrading-to-rdi-200).

On Kubernetes, to override the Flink processor defaults, add an
`operator.dataPlane.flinkProcessor` block to your `rdi-values.yaml` file as
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,6 @@ This is a maintenance release of Redis for Kubernetes to support Redis Software

### New limitations

- **Upgrading directly from Redis for Kubernetes 7.22.2-43 to 8.0.20-26 is not supported** due to an upgrade path limitation between these builds. Remain on 7.22.2-43 until a later 8.0.20 maintenance release is available.
- **Upgrading directly from Redis for Kubernetes 7.22.2-43 to 8.0.20-26 is not supported in OLM-based deployments** due to an upgrade path limitation between these builds. Upgrade to 8.0.20-27 or later instead.

See [7.22.2 releases]({{<relref "/operate/kubernetes/release-notes/7-22-2-releases/">}}) for information on other known limitations.
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,6 @@ This is a maintenance release to support [Redis Software 8.0.20]({{<relref "/ope

### New limitations

- **Upgrading directly from Redis for Kubernetes 7.22.2-43 to 8.0.20-26 is not supported** due to an upgrade path limitation between these builds. Remain on 7.22.2-43 until a later 8.0.20 maintenance release is available.
- **Upgrading directly from Redis for Kubernetes 7.22.2-43 to 8.0.20-26 is not supported in OLM-based deployments** due to an upgrade path limitation between these builds. Upgrade to 8.0.20-27 or later instead.

See [8.0.20 releases]({{<relref "/operate/kubernetes/release-notes/8-0-20-releases/">}}) for information on other known limitations.
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,25 @@ weight: 95
---
## Requirements

RedisTimeSeries v1.10.24 requires:
RedisTimeSeries v1.10.27 requires:

- Minimum Redis compatibility version (database): 7.2
- Minimum Redis Enterprise Software version (cluster): 7.2.4

## v1.10.27 (Sep 2026)

This is a maintenance release for RedisTimeSeries 1.10.

Update urgency: `SECURITY`: There are security fixes in the release.

Details:

- **Security**
- Prevent heap buffer overflows and invalid memory access when restoring malformed time-series data.

- Bug fixes:
- Fix `TS.INCRBY` replication to preserve the primary’s timestamp when using server-generated timestamps.

## v1.10.24 (May 2026)

This is a maintenance release for RedisTimeSeries 1.10.
Expand Down
Loading
Loading