Skip to content

DOC-7086: Migrate RDI reference/ core + data-transformation/ to render hooks #4058

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andy-stark-redis merged 1 commit into from
Sep 18, 2026
Merged

DOC-7086: Migrate RDI reference/ core + data-transformation/ to render hooks #4058

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions content/integrate/redis-data-integration/1.19.1/reference/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,5 @@ weight: 60
url: '/integrate/redis-data-integration/1.19.1/reference/'
---

For API clients, [RDI API v1 is deprecated as of RDI 1.19.0]({{< relref "/integrate/redis-data-integration/1.19.1/reference/api-migration" >}}).
Use [RDI API v2]({{< relref "/integrate/redis-data-integration/1.19.1/reference/api-reference" >}}) for new integrations and migrate existing clients. API v1 will not be extended with new RDI features and may be removed in a future RDI version.
For API clients, [RDI API v1 is deprecated as of RDI 1.19.0](/content/integrate/redis-data-integration/1.19.1/reference/api-migration.md).
Use [RDI API v2](/content/integrate/redis-data-integration/1.19.1/reference/api-reference.md) for new integrations and migrate existing clients. API v1 will not be extended with new RDI features and may be removed in a future RDI version.
9 changes: 4 additions & 5 deletions content/integrate/redis-data-integration/1.19.1/reference/api-migration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,14 @@ weight: 61
url: '/integrate/redis-data-integration/1.19.1/reference/api-migration/'
---

RDI API v1 is deprecated as of RDI 1.19.0. Existing v1 endpoints remain available for backward compatibility, but Redis recommends moving all integrations to API v2. API v1 will not be extended with new RDI features and may be removed in a future RDI version. See the [RDI API reference]({{< relref "/integrate/redis-data-integration/1.19.1/reference/api-reference" >}}) for the current request and response schemas.
RDI API v1 is deprecated as of RDI 1.19.0. Existing v1 endpoints remain available for backward compatibility, but Redis recommends moving all integrations to API v2. API v1 will not be extended with new RDI features and may be removed in a future RDI version. See the [RDI API reference](/content/integrate/redis-data-integration/1.19.1/reference/api-reference.md) for the current request and response schemas.

## What changes in API v2

API v2 uses the pipeline resource to represent the current state of a pipeline. Operations update that resource and return its current state, so applications no longer need to poll a separate action ID. API v2 scopes related operations under a pipeline name in the request path.

{{< note >}}
RDI 1.19.0 supports only one pipeline, which must be named `default`. Support for other pipeline names will be added in a future version.
{{< /note >}}
> [!NOTE]
> RDI 1.19.0 supports only one pipeline, which must be named `default`. Support for other pipeline names will be added in a future version.

The API version is part of the URL. Update `/api/v1` requests to use `/api/v2` where a corresponding v2 endpoint is available. You should also review the request and response models, because they can differ between versions.

Expand Down Expand Up @@ -51,7 +50,7 @@ The API version is part of the URL. Update `/api/v1` requests to use `/api/v2` w
| `POST /api/v1/pipelines/undeploy` | `DELETE /api/v2/pipelines/{name}` |
| `POST /api/v1/trace/start` | `POST /api/v2/pipelines/{name}/traces` |

API v2 also adds endpoints for DLQ inspection, target flushing, metric collections, and API information. See the [API reference]({{< relref "/integrate/redis-data-integration/1.19.1/reference/api-reference" >}}) for the complete list.
API v2 also adds endpoints for DLQ inspection, target flushing, metric collections, and API information. See the [API reference](/content/integrate/redis-data-integration/1.19.1/reference/api-reference.md) for the complete list.

## v1 endpoints without a v2 equivalent

Expand Down
4 changes: 2 additions & 2 deletions ...egrate/redis-data-integration/1.19.1/reference/data-transformation/add_field.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ Add one field
| **field** | `string` | Field<br/> | yes |
| **expression** | `string` | Expression<br/> | yes |
| **language** | `string` | Language<br/>Enum: `"jmespath"`, `"sql"`<br/> | yes |
| **cache** | `object` | Cache the result of the field expression. See [`cache`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/cache" >}}) for the property list. **Flink processor only.**<br/> | no |
| **cache** | `object` | Cache the result of the field expression. See [`cache`](/content/integrate/redis-data-integration/1.19.1/reference/data-transformation/cache.md) for the property list. **Flink processor only.**<br/> | no |

**Additional Properties:** not allowed

Expand Down Expand Up @@ -91,7 +91,7 @@ Fields
| **field** | `string` | Field<br/> | yes |
| **expression** | `string` | Expression<br/> | yes |
| **language** | `string` | Language<br/>Enum: `"jmespath"`, `"sql"`<br/> | yes |
| **cache** | `object` | Cache the result of the field expression. See [`cache`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/cache" >}}) for the property list. **Flink processor only.**<br/> | no |
| **cache** | `object` | Cache the result of the field expression. See [`cache`](/content/integrate/redis-data-integration/1.19.1/reference/data-transformation/cache.md) for the property list. **Flink processor only.**<br/> | no |

**Item Additional Properties:** not allowed

Expand Down
10 changes: 5 additions & 5 deletions .../integrate/redis-data-integration/1.19.1/reference/data-transformation/cache.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,10 @@ fields referenced by the expression, not from the full record.
The `cache:` block can be added to the following transformations and
output expressions:

- The expression in [`add_field`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/add_field" >}}) (single-field and per-item form).
- The expression in [`filter`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/filter" >}}).
- The expression in [`map`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/map" >}}).
- The argument expressions in [`redis.lookup`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/lookup" >}}). The same block accepts a `lookup_cache:` variant that caches the lookup result returned by Redis.
- The expression in [`add_field`](/content/integrate/redis-data-integration/1.19.1/reference/data-transformation/add_field.md) (single-field and per-item form).
- The expression in [`filter`](/content/integrate/redis-data-integration/1.19.1/reference/data-transformation/filter.md).
- The expression in [`map`](/content/integrate/redis-data-integration/1.19.1/reference/data-transformation/map.md).
- The argument expressions in [`redis.lookup`](/content/integrate/redis-data-integration/1.19.1/reference/data-transformation/lookup.md). The same block accepts a `lookup_cache:` variant that caches the lookup result returned by Redis.
- The dynamic `key` and `expire` expressions of the `redis.write` output.

**Flink processor only.** The classic processor silently ignores `cache:` blocks.
Expand Down Expand Up @@ -65,5 +65,5 @@ transform:
```

See
[Caching expression results]({{< relref "/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/caching-expression-results" >}})
[Caching expression results](/content/integrate/redis-data-integration/1.19.1/data-pipelines/transform-examples/caching-expression-results.md)
for additional examples.
2 changes: 1 addition & 1 deletion ...integrate/redis-data-integration/1.19.1/reference/data-transformation/filter.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Filter records
| -------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- |
| **expression** | `string` | Expression<br/> | yes |
| **language** | `string` | Language<br/>Enum: `"jmespath"`, `"sql"`<br/> | yes |
| **cache** | `object` | Cache the result of the filter expression. See [`cache`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/cache" >}}) for the property list. **Flink processor only.**<br/> | no |
| **cache** | `object` | Cache the result of the filter expression. See [`cache`](/content/integrate/redis-data-integration/1.19.1/reference/data-transformation/cache.md) for the property list. **Flink processor only.**<br/> | no |

**Additional Properties:** not allowed

Expand Down
4 changes: 2 additions & 2 deletions ...integrate/redis-data-integration/1.19.1/reference/data-transformation/lookup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ url: '/integrate/redis-data-integration/1.19.1/reference/data-transformation/loo
| [**args**](#args) | `string[]` | Redis command arguments | yes |
| **language** | `string` | Language<br/>Enum: `"jmespath"`, `"sql"`<br/> | yes |
| **field** | `string` | The target field to write the result to<br/> | yes |
| **cache** | `object` | Cache the result of the argument expressions. See [`cache`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/cache" >}}) for the property list. **Flink processor only.**<br/> | no |
| **lookup_cache** | `object` | Cache the lookup results returned by Redis across batches, keyed by the resolved command arguments. Uses the same property list as [`cache`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/cache" >}}). **Flink processor only.**<br/> | no |
| **cache** | `object` | Cache the result of the argument expressions. See [`cache`](/content/integrate/redis-data-integration/1.19.1/reference/data-transformation/cache.md) for the property list. **Flink processor only.**<br/> | no |
| **lookup_cache** | `object` | Cache the lookup results returned by Redis across batches, keyed by the resolved command arguments. Uses the same property list as [`cache`](/content/integrate/redis-data-integration/1.19.1/reference/data-transformation/cache.md). **Flink processor only.**<br/> | no |
| [**batch**](#batch) | `object` | Override the default batching behavior for `redis.lookup` lookups. **Flink processor only.**<br/> | no |

**Additional Properties:** not allowed
Expand Down
2 changes: 1 addition & 1 deletion ...nt/integrate/redis-data-integration/1.19.1/reference/data-transformation/map.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Map a record into a new output based on expressions
| ----------------------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| [**expression**](#expression) | `object`, `string` | Expression<br/> | yes |
| **language** | `string` | Language<br/>Enum: `"jmespath"`, `"sql"`<br/> | yes |
| **cache** | `object` | Cache the result of the map expression. See [`cache`]({{< relref "/integrate/redis-data-integration/1.19.1/reference/data-transformation/cache" >}}) for the property list. **Flink processor only.**<br/> | no |
| **cache** | `object` | Cache the result of the map expression. See [`cache`](/content/integrate/redis-data-integration/1.19.1/reference/data-transformation/cache.md) for the property list. **Flink processor only.**<br/> | no |

**Additional Properties:** not allowed

Expand Down
4 changes: 2 additions & 2 deletions content/integrate/redis-data-integration/reference/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,5 @@ type: integration
weight: 60
---

For API clients, [RDI API v1 is deprecated as of RDI 1.19.0]({{< relref "/integrate/redis-data-integration/reference/api-migration" >}}).
Use [RDI API v2]({{< relref "/integrate/redis-data-integration/reference/api-reference" >}}) for new integrations and migrate existing clients. API v1 will not be extended with new RDI features and may be removed in a future RDI version.
For API clients, [RDI API v1 is deprecated as of RDI 1.19.0](/content/integrate/redis-data-integration/reference/api-migration.md).
Use [RDI API v2](/content/integrate/redis-data-integration/reference/api-reference.md) for new integrations and migrate existing clients. API v1 will not be extended with new RDI features and may be removed in a future RDI version.
13 changes: 6 additions & 7 deletions content/integrate/redis-data-integration/reference/api-migration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,22 +15,21 @@ type: integration
weight: 61
---

RDI API v1 is deprecated as of RDI 1.19.0. Existing v1 endpoints remain available for backward compatibility, but Redis recommends moving all integrations to API v2. API v1 will not be extended with new RDI features and may be removed in a future RDI version. See the [RDI API reference]({{< relref "/integrate/redis-data-integration/reference/api-reference" >}}) for the current request and response schemas.
RDI API v1 is deprecated as of RDI 1.19.0. Existing v1 endpoints remain available for backward compatibility, but Redis recommends moving all integrations to API v2. API v1 will not be extended with new RDI features and may be removed in a future RDI version. See the [RDI API reference](/content/integrate/redis-data-integration/reference/api-reference.md) for the current request and response schemas.

## What changes in API v2

API v2 uses the pipeline resource to represent the current state of a pipeline. Operations update that resource and return its current state, so applications no longer need to poll a separate action ID. API v2 scopes related operations under a pipeline name in the request path.

{{< note >}}
RDI 1.19.0 supports only one pipeline, which must be named `default`. Support for other pipeline names will be added in a future version.
{{< /note >}}
> [!NOTE]
> RDI 1.19.0 supports only one pipeline, which must be named `default`. Support for other pipeline names will be added in a future version.

The API version is part of the URL. Update `/api/v1` requests to use `/api/v2` where a corresponding v2 endpoint is available. You should also review the request and response models, because they can differ between versions.

## API v1 and multiple sources

API v1 handles single-source pipelines only, so
you must use API v2 to manage [a pipeline with several sources]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}). In particular:
you must use API v2 to manage [a pipeline with several sources](/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md). In particular:

- `POST /api/v1/pipelines`, `PATCH /api/v1/pipelines`, and `POST /api/v1/pipelines/sources/dry-run` reject a configuration with more than one source with `422 Unprocessable Content` and `Only a single source per pipeline is supported`.
- `PUT /api/v1/secrets`, `PUT /api/v1/secrets/{secret_name}`, and the v1 source management endpoints only handle a pipeline whose single source still uses the legacy `source` names, which covers a pipeline created before RDI supported multiple sources and one that you create through v1. They reject any other pipeline with `Use API v2 to manage sources and secrets of this pipeline.`
Expand Down Expand Up @@ -62,7 +61,7 @@ you must use API v2 to manage [a pipeline with several sources]({{< relref "/int
| `POST /api/v1/pipelines/undeploy` | `PUT /api/v2/pipelines/{name}` with an empty configuration |
| `POST /api/v1/trace/start` | `POST /api/v2/pipelines/{name}/traces` |

API v2 also adds endpoints for DLQ inspection, target flushing, metric collections, and API information. See the [API reference]({{< relref "/integrate/redis-data-integration/reference/api-reference" >}}) for the complete list.
API v2 also adds endpoints for DLQ inspection, target flushing, metric collections, and API information. See the [API reference](/content/integrate/redis-data-integration/reference/api-reference.md) for the complete list.

## v1 endpoints without a v2 equivalent

Expand Down Expand Up @@ -128,7 +127,7 @@ Note:
2. Add the pipeline name to each v2 request. The only pipeline in 1.19.0 is always named `default`.
3. Check the pipeline response, or call `GET /api/v2/pipelines/{name}/status`, instead of polling an action ID.
4. Use `POST /api/v2/pipelines`, `PUT /api/v2/pipelines/{name}`, or `PATCH /api/v2/pipelines/{name}` to update source, target, processor, and secret-provider settings as needed. When using `PATCH`, omit the configuration sections that you do not want to change.
5. Change secret requests to address a secret by a database-independent key and a `db` parameter. Where a v1 request set `SOURCE_DB_PASSWORD`, a v2 request sets the `PASSWORD` key with `db` naming the source, and `TARGET_DB_PASSWORD` becomes the `PASSWORD` key with `db=target`. See [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}). The environment variable that `config.yaml` references is unchanged for a source that existed before RDI supported multiple sources; see [Existing names are kept after an upgrade]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}).
5. Change secret requests to address a secret by a database-independent key and a `db` parameter. Where a v1 request set `SOURCE_DB_PASSWORD`, a v2 request sets the `PASSWORD` key with `db` naming the source, and `TARGET_DB_PASSWORD` becomes the `PASSWORD` key with `db=target`. See [Set secrets](/content/integrate/redis-data-integration/data-pipelines/deploy.md#set-secrets). The environment variable that `config.yaml` references is unchanged for a source that existed before RDI supported multiple sources; see [Existing names are kept after an upgrade](/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md#existing-names-are-kept-after-an-upgrade).
6. Use `GET /api/v2/pipelines/{name}/metric-collections/{collection_name}` for monitoring and `GET /api/v2/pipelines/{name}/source-schemas/{source_name}` for source metadata.
7. Test creating, updating, validating, starting, stopping, resetting, and deleting a pipeline on a non-production RDI 1.19.0 or later installation before updating production applications.

Expand Down
Loading
Loading