From a252aa427d006b49d62a706aa93ac87cd54c16f7 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Thu, 15 Oct 2026 13:56:00 +0300 Subject: [PATCH 1/5] DOC: clarify PostgreSQL publication setup for RDI --- .../aws-aurora-rds/aws-aur-pgsql.md | 12 +- .../data-pipelines/prepare-dbs/postgresql.md | 122 +++++++++++++++++- 2 files changed, 130 insertions(+), 4 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md index c18fcd7b05..d37ca6c598 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md @@ -150,8 +150,18 @@ to capture changes. Replace `` with the schema name and `` with the username of the Debezium user. 1. - Connect to your database as the `postgres` user and create a publication for the database: + Configure the PostgreSQL publication. For most RDI deployments, follow the + [PostgreSQL publication and replica identity guidance]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql" >}}) + and use `publication.autocreate.mode: filtered` when RDI captures selected tables. + + If you intentionally want to publish every table in the database, first check that + every table has a primary key or another replica identity. Then connect to your database + as the `postgres` user and run: ```sql CREATE PUBLICATION dbz_publication FOR ALL TABLES; ``` + + Do not use this command without checking the replica identity of all tables. A later + `UPDATE` or `DELETE` on a published table without a replica identity can fail in the + application that writes to PostgreSQL. diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md index 55ae12fc33..2e262aed33 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md @@ -39,6 +39,7 @@ checklist as you complete each step. - [ ] [Install the logical decoding output plug-in](#install-the-logical-decoding-output-plug-in) - [ ] [Configure the PostgreSQL server](#configure-the-postgresql-server) - [ ] [Set up permissions](#set-up-permissions) +- [ ] [Check replica identity and choose a publication mode](#check-replica-identity-and-choose-a-publication-mode) - [ ] [Set privileges for Debezium to create PostgreSQL publications with pgoutput](#set-privileges-for-debezium-to-create-postgresql-publications-with-pgoutput) - [ ] [Configure PostgreSQL for replication with the Debezium connector host](#configure-postgresql-for-replication-with-the-debezium-connector-host) ``` @@ -183,6 +184,120 @@ needs to perform replications, but you might also need to grant other permission CREATE ROLE REPLICATION LOGIN; ``` +## Check replica identity and choose a publication mode + +Before you start RDI, check the tables that RDI will capture and choose how PostgreSQL +should publish their changes. + +PostgreSQL needs a [replica identity](https://www.postgresql.org/docs/current/logical-replication-publication.html) +to identify the old row when it publishes an `UPDATE` or `DELETE`. By default, PostgreSQL +uses the table's primary key. A table without a primary key has no usable replica identity +unless you configure one. Inserts do not need a replica identity. + +Run the following query to find ordinary tables that do not have a primary key and use the +default identity, or that explicitly have no identity: + +```sql +SELECT n.nspname AS schema_name, + c.relname AS table_name, + CASE c.relreplident + WHEN 'd' THEN 'default' + WHEN 'n' THEN 'nothing' + WHEN 'f' THEN 'full' + WHEN 'i' THEN 'index' + END AS replica_identity +FROM pg_class AS c +JOIN pg_namespace AS n ON n.oid = c.relnamespace +WHERE c.relkind = 'r' + AND n.nspname NOT IN ('pg_catalog', 'information_schema') + AND ( + c.relreplident = 'n' + OR ( + c.relreplident = 'd' + AND NOT EXISTS ( + SELECT 1 + FROM pg_index AS i + WHERE i.indrelid = c.oid + AND i.indisprimary + ) + ) + ) +ORDER BY n.nspname, c.relname; +``` + +If this query returns a table that will be published, add a primary key where possible. +You can also configure a suitable unique index as the replica identity: + +```sql +ALTER TABLE REPLICA IDENTITY USING INDEX ; +``` + +As a fallback, you can record the full old row in the write-ahead log: + +```sql +ALTER TABLE REPLICA IDENTITY FULL; +``` + +`REPLICA IDENTITY FULL` can increase write-ahead log volume and reduce performance, so use +it only when a primary key or suitable unique index is not possible. The `keys` setting in +an RDI table definition creates a key for the Redis record. It does not create a PostgreSQL +primary key or replica identity. + +### Choose a publication mode + +The `publication.autocreate.mode` setting controls how RDI and Debezium manage the +PostgreSQL publication: + +| Mode | Behavior | Recommended use | +| --- | --- | --- | +| `filtered` | Creates or updates the publication to contain the tables configured for the RDI source. | Recommended when RDI captures selected tables. RDI manages the publication when the pipeline is redeployed with a changed table filter. | +| `disabled` | Does not create or change the publication. The publication must already exist. | Use when a database administrator manages the publication. | +| `all_tables` | Creates a `FOR ALL TABLES` publication when the publication does not exist. If it already exists, RDI uses the existing publication. | Use when all tables in the database should be published, after checking their replica identities. This is the RDI default. | + +The `all_tables` mode can publish changes for tables that are not listed in the RDI +pipeline. If an application later updates or deletes a row in a published table without +a replica identity, PostgreSQL can reject that application write. For this reason, +`filtered` is the recommended mode for a pipeline that captures only selected tables. + +For a customer-managed publication, create an explicit table list and disable automatic +publication management. For example: + +```sql +CREATE PUBLICATION rdi_publication +FOR TABLE public.customers, public.orders; +``` + +Then set the following source properties: + +```yaml +publication.name: rdi_publication +publication.autocreate.mode: disabled +``` + +### PostgreSQL publication error FAQ + +#### I get this PostgreSQL error. What should I do? + +You may see the following error when an application updates a published table without a +replica identity: + +```text +SQLSTATE 55000 +ERROR: cannot update table "example_table" because it does not have a replica identity and publishes updates +HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE. +``` + +Check whether the table should be captured by RDI. If it should, add a primary key or +configure a suitable replica identity. As a fallback, run: + +```sql +ALTER TABLE REPLICA IDENTITY FULL; +``` + +If the table should not be captured, use `publication.autocreate.mode: filtered` or remove +the table from the customer-managed publication. If you changed the RDI table filters or +publication mode, redeploy the pipeline so RDI applies the new configuration. + ## Set privileges for Debezium to create PostgreSQL publications with `pgoutput` The Debezium user needs specific permissions to work with the `pgoutput` plug-in. @@ -241,10 +356,11 @@ the replication group using the following commands: ALTER TABLE table_name OWNER TO replication_group; ``` -You must also set the value of the `publication.autocreate.mode` parameter to `filtered` -to allow Debezium to specify the publication configuration. See the +If you want Debezium to create or update a publication for the tables configured in the RDI +source, set `publication.autocreate.mode` to `filtered`. See the [Debezium documentation for `publication.autocreate.mode`](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-publication-autocreate-mode) -to learn more about this setting. +to learn more about this setting. If a database administrator manages an existing +publication, use `disabled` instead, as described in [Choose a publication mode](#choose-a-publication-mode). ## Configure PostgreSQL for replication with the Debezium connector host From 1aef8074ecaf6370ccd376b192ee338e243b4027 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Thu, 15 Oct 2026 15:11:22 +0300 Subject: [PATCH 2/5] DOC: update versioned RDI PostgreSQL guidance --- .../aws-aurora-rds/aws-aur-pgsql.md | 12 +- .../data-pipelines/prepare-dbs/postgresql.md | 122 +++++++++++++++++- 2 files changed, 130 insertions(+), 4 deletions(-) diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md index 3254e9fd2f..a7df75cc8a 100644 --- a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md +++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md @@ -147,8 +147,18 @@ to capture changes. Replace `` with the schema name and `` with the username of the Debezium user. 1. - Connect to your database as the `postgres` user and create a publication for the database: + Configure the PostgreSQL publication. For most RDI deployments, follow the + [PostgreSQL publication and replica identity guidance]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql" >}}) + and use `publication.autocreate.mode: filtered` when RDI captures selected tables. + + If you intentionally want to publish every table in the database, first check that + every table has a primary key or another replica identity. Then connect to your database + as the `postgres` user and run: ```sql CREATE PUBLICATION dbz_publication FOR ALL TABLES; ``` + + Do not use this command without checking the replica identity of all tables. A later + `UPDATE` or `DELETE` on a published table without a replica identity can fail in the + application that writes to PostgreSQL. diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md index 31ccc7149f..4ce48b8e1e 100644 --- a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md +++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md @@ -36,6 +36,7 @@ checklist as you complete each step. - [ ] [Install the logical decoding output plug-in](#install-the-logical-decoding-output-plug-in) - [ ] [Configure the PostgreSQL server](#configure-the-postgresql-server) - [ ] [Set up permissions](#set-up-permissions) +- [ ] [Check replica identity and choose a publication mode](#check-replica-identity-and-choose-a-publication-mode) - [ ] [Set privileges for Debezium to create PostgreSQL publications with pgoutput](#set-privileges-for-debezium-to-create-postgresql-publications-with-pgoutput) - [ ] [Configure PostgreSQL for replication with the Debezium connector host](#configure-postgresql-for-replication-with-the-debezium-connector-host) ``` @@ -180,6 +181,120 @@ needs to perform replications, but you might also need to grant other permission CREATE ROLE REPLICATION LOGIN; ``` +## Check replica identity and choose a publication mode + +Before you start RDI, check the tables that RDI will capture and choose how PostgreSQL +should publish their changes. + +PostgreSQL needs a [replica identity](https://www.postgresql.org/docs/current/logical-replication-publication.html) +to identify the old row when it publishes an `UPDATE` or `DELETE`. By default, PostgreSQL +uses the table's primary key. A table without a primary key has no usable replica identity +unless you configure one. Inserts do not need a replica identity. + +Run the following query to find ordinary tables that do not have a primary key and use the +default identity, or that explicitly have no identity: + +```sql +SELECT n.nspname AS schema_name, + c.relname AS table_name, + CASE c.relreplident + WHEN 'd' THEN 'default' + WHEN 'n' THEN 'nothing' + WHEN 'f' THEN 'full' + WHEN 'i' THEN 'index' + END AS replica_identity +FROM pg_class AS c +JOIN pg_namespace AS n ON n.oid = c.relnamespace +WHERE c.relkind = 'r' + AND n.nspname NOT IN ('pg_catalog', 'information_schema') + AND ( + c.relreplident = 'n' + OR ( + c.relreplident = 'd' + AND NOT EXISTS ( + SELECT 1 + FROM pg_index AS i + WHERE i.indrelid = c.oid + AND i.indisprimary + ) + ) + ) +ORDER BY n.nspname, c.relname; +``` + +If this query returns a table that will be published, add a primary key where possible. +You can also configure a suitable unique index as the replica identity: + +```sql +ALTER TABLE REPLICA IDENTITY USING INDEX ; +``` + +As a fallback, you can record the full old row in the write-ahead log: + +```sql +ALTER TABLE REPLICA IDENTITY FULL; +``` + +`REPLICA IDENTITY FULL` can increase write-ahead log volume and reduce performance, so use +it only when a primary key or suitable unique index is not possible. The `keys` setting in +an RDI table definition creates a key for the Redis record. It does not create a PostgreSQL +primary key or replica identity. + +### Choose a publication mode + +The `publication.autocreate.mode` setting controls how RDI and Debezium manage the +PostgreSQL publication: + +| Mode | Behavior | Recommended use | +| --- | --- | --- | +| `filtered` | Creates or updates the publication to contain the tables configured for the RDI source. | Recommended when RDI captures selected tables. RDI manages the publication when the pipeline is redeployed with a changed table filter. | +| `disabled` | Does not create or change the publication. The publication must already exist. | Use when a database administrator manages the publication. | +| `all_tables` | Creates a `FOR ALL TABLES` publication when the publication does not exist. If it already exists, RDI uses the existing publication. | Use when all tables in the database should be published, after checking their replica identities. This is the RDI default. | + +The `all_tables` mode can publish changes for tables that are not listed in the RDI +pipeline. If an application later updates or deletes a row in a published table without +a replica identity, PostgreSQL can reject that application write. For this reason, +`filtered` is the recommended mode for a pipeline that captures only selected tables. + +For a customer-managed publication, create an explicit table list and disable automatic +publication management. For example: + +```sql +CREATE PUBLICATION rdi_publication +FOR TABLE public.customers, public.orders; +``` + +Then set the following source properties: + +```yaml +publication.name: rdi_publication +publication.autocreate.mode: disabled +``` + +### PostgreSQL publication error FAQ + +#### I get this PostgreSQL error. What should I do? + +You may see the following error when an application updates a published table without a +replica identity: + +```text +SQLSTATE 55000 +ERROR: cannot update table "example_table" because it does not have a replica identity and publishes updates +HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE. +``` + +Check whether the table should be captured by RDI. If it should, add a primary key or +configure a suitable replica identity. As a fallback, run: + +```sql +ALTER TABLE REPLICA IDENTITY FULL; +``` + +If the table should not be captured, use `publication.autocreate.mode: filtered` or remove +the table from the customer-managed publication. If you changed the RDI table filters or +publication mode, redeploy the pipeline so RDI applies the new configuration. + ## Set privileges for Debezium to create PostgreSQL publications with `pgoutput` The Debezium user needs specific permissions to work with the `pgoutput` plug-in. @@ -238,10 +353,11 @@ the replication group using the following commands: ALTER TABLE table_name OWNER TO replication_group; ``` -You must also set the value of the `publication.autocreate.mode` parameter to `filtered` -to allow Debezium to specify the publication configuration. See the +If you want Debezium to create or update a publication for the tables configured in the RDI +source, set `publication.autocreate.mode` to `filtered`. See the [Debezium documentation for `publication.autocreate.mode`](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-publication-autocreate-mode) -to learn more about this setting. +to learn more about this setting. If a database administrator manages an existing +publication, use `disabled` instead, as described in [Choose a publication mode](#choose-a-publication-mode). ## Configure PostgreSQL for replication with the Debezium connector host From 75545b24c904f4c5ef5d3351b75cdcfc01e3c811 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Thu, 15 Oct 2026 15:21:15 +0300 Subject: [PATCH 3/5] DOC: address PostgreSQL publication review comments --- .../data-pipelines/prepare-dbs/postgresql.md | 20 +++++++++++++------ .../data-pipelines/prepare-dbs/postgresql.md | 20 +++++++++++++------ 2 files changed, 28 insertions(+), 12 deletions(-) diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md index 4ce48b8e1e..8e75a9e2f1 100644 --- a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md +++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md @@ -191,8 +191,10 @@ to identify the old row when it publishes an `UPDATE` or `DELETE`. By default, P uses the table's primary key. A table without a primary key has no usable replica identity unless you configure one. Inserts do not need a replica identity. -Run the following query to find ordinary tables that do not have a primary key and use the -default identity, or that explicitly have no identity: +Run the following query while connected to the database that RDI uses. You do not put the +database name in the query; select that database in your SQL editor or connection settings. +The query checks all non-system schemas. To check only one schema, uncomment the marked line +and replace `` with your schema name. ```sql SELECT n.nspname AS schema_name, @@ -207,6 +209,8 @@ FROM pg_class AS c JOIN pg_namespace AS n ON n.oid = c.relnamespace WHERE c.relkind = 'r' AND n.nspname NOT IN ('pg_catalog', 'information_schema') + -- Optional: uncomment and replace to check one schema only. + -- AND n.nspname = '' AND ( c.relreplident = 'n' OR ( @@ -247,9 +251,9 @@ PostgreSQL publication: | Mode | Behavior | Recommended use | | --- | --- | --- | +| **`all_tables` (default)** | Creates a `FOR ALL TABLES` publication when the publication does not exist. If it already exists, RDI uses the existing publication. | Use when all tables in the database should be published, after checking their replica identities. | | `filtered` | Creates or updates the publication to contain the tables configured for the RDI source. | Recommended when RDI captures selected tables. RDI manages the publication when the pipeline is redeployed with a changed table filter. | | `disabled` | Does not create or change the publication. The publication must already exist. | Use when a database administrator manages the publication. | -| `all_tables` | Creates a `FOR ALL TABLES` publication when the publication does not exist. If it already exists, RDI uses the existing publication. | Use when all tables in the database should be published, after checking their replica identities. This is the RDI default. | The `all_tables` mode can publish changes for tables that are not listed in the RDI pipeline. If an application later updates or deletes a row in a published table without @@ -264,11 +268,15 @@ CREATE PUBLICATION rdi_publication FOR TABLE public.customers, public.orders; ``` -Then set the following source properties: +Then set the following properties under the PostgreSQL source in your RDI `config.yaml`: ```yaml -publication.name: rdi_publication -publication.autocreate.mode: disabled +sources: + : + advanced: + source: + publication.name: rdi_publication + publication.autocreate.mode: disabled ``` ### PostgreSQL publication error FAQ diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md index 2e262aed33..8da8fa6966 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md @@ -194,8 +194,10 @@ to identify the old row when it publishes an `UPDATE` or `DELETE`. By default, P uses the table's primary key. A table without a primary key has no usable replica identity unless you configure one. Inserts do not need a replica identity. -Run the following query to find ordinary tables that do not have a primary key and use the -default identity, or that explicitly have no identity: +Run the following query while connected to the database that RDI uses. You do not put the +database name in the query; select that database in your SQL editor or connection settings. +The query checks all non-system schemas. To check only one schema, uncomment the marked line +and replace `` with your schema name. ```sql SELECT n.nspname AS schema_name, @@ -210,6 +212,8 @@ FROM pg_class AS c JOIN pg_namespace AS n ON n.oid = c.relnamespace WHERE c.relkind = 'r' AND n.nspname NOT IN ('pg_catalog', 'information_schema') + -- Optional: uncomment and replace to check one schema only. + -- AND n.nspname = '' AND ( c.relreplident = 'n' OR ( @@ -250,9 +254,9 @@ PostgreSQL publication: | Mode | Behavior | Recommended use | | --- | --- | --- | +| **`all_tables` (default)** | Creates a `FOR ALL TABLES` publication when the publication does not exist. If it already exists, RDI uses the existing publication. | Use when all tables in the database should be published, after checking their replica identities. | | `filtered` | Creates or updates the publication to contain the tables configured for the RDI source. | Recommended when RDI captures selected tables. RDI manages the publication when the pipeline is redeployed with a changed table filter. | | `disabled` | Does not create or change the publication. The publication must already exist. | Use when a database administrator manages the publication. | -| `all_tables` | Creates a `FOR ALL TABLES` publication when the publication does not exist. If it already exists, RDI uses the existing publication. | Use when all tables in the database should be published, after checking their replica identities. This is the RDI default. | The `all_tables` mode can publish changes for tables that are not listed in the RDI pipeline. If an application later updates or deletes a row in a published table without @@ -267,11 +271,15 @@ CREATE PUBLICATION rdi_publication FOR TABLE public.customers, public.orders; ``` -Then set the following source properties: +Then set the following properties under the PostgreSQL source in your RDI `config.yaml`: ```yaml -publication.name: rdi_publication -publication.autocreate.mode: disabled +sources: + : + advanced: + source: + publication.name: rdi_publication + publication.autocreate.mode: disabled ``` ### PostgreSQL publication error FAQ From d303acb13bf1f5b00f7c098061e4b8077ff6d997 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Sun, 18 Oct 2026 03:56:52 +0300 Subject: [PATCH 4/5] DOC: align Cloud PostgreSQL publications and replica identity guidance --- .../data-pipelines/data-denormalization.md | 13 ++- .../aws-aurora-rds/aws-aur-pgsql.md | 28 ++++--- .../data-pipelines/prepare-dbs/postgresql.md | 82 ++++++++++++++++--- .../data-pipelines/prepare-dbs/supabase.md | 5 +- .../data-pipelines/data-denormalization.md | 13 ++- .../aws-aurora-rds/aws-aur-pgsql.md | 28 ++++--- .../data-pipelines/prepare-dbs/postgresql.md | 82 ++++++++++++++++--- .../data-pipelines/prepare-dbs/supabase.md | 5 +- content/operate/rc/rdi/define.md | 12 +++ content/operate/rc/rdi/quick-start.md | 4 + content/operate/rc/rdi/setup.md | 30 +++++++ 11 files changed, 244 insertions(+), 58 deletions(-) diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization.md index 8fa42654e4..eacf01dd3e 100644 --- a/content/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization.md +++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization.md @@ -217,11 +217,10 @@ There are several important things to note when you use nesting: ALTER TABLE REPLICA IDENTITY FULL; ``` - This configuration affects the information written to the write-ahead log (WAL) and whether it is available - for RDI to capture. By default, PostgreSQL only records - modified fields in the log, which means that it might omit the `parent_key`. This can cause incorrect updates to the - Redis key in the destination database. - See the - [Debezium PostgreSQL Connector Documentation](https://debezium.io/documentation/reference/connectors/postgresql.html#postgresql-replica-identity) - for more information about this. + Nesting needs the child's `parent_key` to identify the parent Redis document, including + when the child is deleted. The default replica identity does not provide the full old + row, so a non-key `parent_key` can be unavailable. `FULL` records all old column values + in the write-ahead log (WAL). It can increase WAL volume and reduce performance. + See [Which PostgreSQL row values are available](/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md#understand-which-row-values-are-available) + for the difference between old row values and unchanged large values. - Prior to RDI v1.12.1, there is a known limitation if you change the foreign key of a child object. In that scenario, the child object will be added to the new parent, but the old parent will not be updated. diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md index a7df75cc8a..55def8274a 100644 --- a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md +++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md @@ -147,18 +147,24 @@ to capture changes. Replace `` with the schema name and `` with the username of the Debezium user. 1. - Configure the PostgreSQL publication. For most RDI deployments, follow the - [PostgreSQL publication and replica identity guidance]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql" >}}) - and use `publication.autocreate.mode: filtered` when RDI captures selected tables. - - If you intentionally want to publish every table in the database, first check that - every table has a primary key or another replica identity. Then connect to your database - as the `postgres` user and run: + Connect to your database as the `postgres` user and create a publication containing + only the tables RDI should capture. First [check their replica identities](/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md#check-replica-identity-and-choose-a-publication-mode). + Replace the example table names with your selected tables: ```sql - CREATE PUBLICATION dbz_publication FOR ALL TABLES; + CREATE PUBLICATION rdi_publication + FOR TABLE public.customers, public.orders; ``` - Do not use this command without checking the replica identity of all tables. A later - `UPDATE` or `DELETE` on a published table without a replica identity can fail in the - application that writes to PostgreSQL. + Set `publication.name` to `rdi_publication` and `publication.autocreate.mode` to + `disabled` under the source's advanced source settings. On Redis Cloud, enter these + values under [**Collector properties**](/content/operate/rc/rdi/setup.md#postgresql-publications). + Update the publication's table list when you change the tables RDI captures. + + Alternatively, let RDI manage the publication with `publication.autocreate.mode: filtered` + after granting the [required ownership and creation permissions](/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md#set-privileges-for-debezium-to-create-postgresql-publications-with-pgoutput). + + Use `FOR ALL TABLES` only when you intend to publish every current and future table. + An application `UPDATE` or `DELETE` on any published table without replica identity can + fail, even if RDI does not capture that table. If you already created an all-tables + publication, follow [Replace an existing all-tables publication](/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md#replace-an-existing-all-tables-publication). diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md index 8e75a9e2f1..9b348db69a 100644 --- a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md +++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md @@ -239,10 +239,27 @@ As a fallback, you can record the full old row in the write-ahead log: ALTER TABLE REPLICA IDENTITY FULL; ``` -`REPLICA IDENTITY FULL` can increase write-ahead log volume and reduce performance, so use -it only when a primary key or suitable unique index is not possible. The `keys` setting in -an RDI table definition creates a key for the Redis record. It does not create a PostgreSQL -primary key or replica identity. +`REPLICA IDENTITY FULL` can increase write-ahead log volume and reduce performance. +Use it when no suitable key is available or when RDI needs the full old row, such as +[denormalizing child tables with nesting](/content/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization.md#using-nesting). +The `keys` setting in an RDI table definition creates a key for the Redis record. +It does not create a PostgreSQL primary key or replica identity. + +### Understand which row values are available + +Replica identity controls the old row values available to logical decoding for updates +and deletes. With the default identity, PostgreSQL uses the primary key; it does not +provide a complete old row. `REPLICA IDENTITY FULL` records all old column values. + +Update events are not generally limited to changed fields. However, PostgreSQL can omit +unchanged large values stored through The Oversized-Attribute Storage Technique (TOAST) +when they are not part of the replica identity. Debezium represents unavailable values +with a placeholder. With `FULL`, these values are available in the event's old and new row +images. See [Debezium's TOAST guidance](https://debezium.io/documentation/reference/3.0/connectors/postgresql.html#postgresql-toasted-values). + +For example, nesting a child record requires its `parent_key` to identify the parent +Redis document, including when the child is deleted. Set `REPLICA IDENTITY FULL` on +PostgreSQL child tables as required by the [nesting instructions](/content/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization.md#using-nesting). ### Choose a publication mode @@ -256,12 +273,17 @@ PostgreSQL publication: | `disabled` | Does not create or change the publication. The publication must already exist. | Use when a database administrator manages the publication. | The `all_tables` mode can publish changes for tables that are not listed in the RDI -pipeline. If an application later updates or deletes a row in a published table without -a replica identity, PostgreSQL can reject that application write. For this reason, +pipeline, including tables created in the future. If an application later updates or +deletes a row in a published table without a replica identity, PostgreSQL can reject +that application write. For this reason, `filtered` is the recommended mode for a pipeline that captures only selected tables. +The `filtered` mode requires permission to create the publication and ownership rights on +the tables it manages. Use a separate publication for each source with different table +filters, so one source does not change another source's publication. + For a customer-managed publication, create an explicit table list and disable automatic -publication management. For example: +publication management. Replace the example table names with the tables you want to capture: ```sql CREATE PUBLICATION rdi_publication @@ -279,6 +301,43 @@ sources: publication.autocreate.mode: disabled ``` +For RDI on Redis Cloud, enter these properties under **Collector properties** in the +source configuration. See [Configure PostgreSQL publications on Redis Cloud](/content/operate/rc/rdi/setup.md#postgresql-publications). +With `disabled`, update the publication's table list yourself when changing the source's +selected tables. Selecting a table in RDI does not add it to a customer-managed publication. + +### Replace an existing all-tables publication + +Changing the RDI table selection does not narrow a `FOR ALL TABLES` publication. +Changing `publication.autocreate.mode` to `filtered` also does not convert it: +PostgreSQL rejects attempts to set, add, or remove individual tables in that publication. + +Have your database administrator plan the change with the owners of every consumer of +the publication: + +1. Check existing publications in the source database: + + ```sql + SELECT pubname, puballtables, pubupdate, pubdelete + FROM pg_publication; + ``` + +1. Identify every consumer of the broad publication and coordinate a maintenance window. +1. Stop the affected RDI sources before replacing the publication. +1. Replace the broad publication with an explicit table-list publication. You can recreate + it under the same name, or create a new publication and update each consumer's publication + name. For RDI, use `publication.autocreate.mode: disabled` for an administrator-managed + publication. On Redis Cloud, contact support if the running source's collector properties + cannot be edited in the console. +1. Remove the old broad publication only after coordinating all consumers. Leaving it in + place still subjects its tables to replica-identity requirements, even if RDI uses a + different publication or is stopped. +1. Resume the affected sources and verify inserts, updates, and deletes reach Redis. + +Keep the replication slots and saved RDI offsets during this change. Do not reset the +pipeline or drop its slot as a publication repair. If other consumers still need the broad +publication, configure replica identity on its affected tables instead of removing it. + ### PostgreSQL publication error FAQ #### I get this PostgreSQL error. What should I do? @@ -299,9 +358,12 @@ configure a suitable replica identity. As a fallback, run: ALTER TABLE REPLICA IDENTITY FULL; ``` -If the table should not be captured, use `publication.autocreate.mode: filtered` or remove -the table from the customer-managed publication. If you changed the RDI table filters or -publication mode, redeploy the pipeline so RDI applies the new configuration. +If the table should not be captured, remove it from an explicit table-list publication or +use `filtered` with the required permissions. For an existing `FOR ALL TABLES` publication, +follow [Replace an existing all-tables publication](#replace-an-existing-all-tables-publication). +Check all publications containing the table: another publication that publishes updates or +deletes can still require replica identity. After changing RDI configuration, redeploy the +pipeline so RDI applies it. ## Set privileges for Debezium to create PostgreSQL publications with `pgoutput` diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/supabase.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/supabase.md index 4e86b37b73..c4bacdd049 100644 --- a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/supabase.md +++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/supabase.md @@ -152,7 +152,10 @@ CREATE PUBLICATION rdi_publication Creating the publication explicitly avoids granting table ownership or broad publication-creation permissions to the RDI role and limits the publication's -table scope. +table scope. Before creating it, [check replica identity](/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md#check-replica-identity-and-choose-a-publication-mode) +for every included table. Updates and deletes on published tables require a usable replica +identity. Update the publication's table list when you add or remove captured tables; +`publication.autocreate.mode: disabled` leaves that list under your control. ## 5. Configure TLS diff --git a/content/integrate/redis-data-integration/data-pipelines/data-denormalization.md b/content/integrate/redis-data-integration/data-pipelines/data-denormalization.md index 6cacf08712..f09e8bf000 100644 --- a/content/integrate/redis-data-integration/data-pipelines/data-denormalization.md +++ b/content/integrate/redis-data-integration/data-pipelines/data-denormalization.md @@ -219,11 +219,10 @@ There are several important things to note when you use nesting: ALTER TABLE REPLICA IDENTITY FULL; ``` - This configuration affects the information written to the write-ahead log (WAL) and whether it is available - for RDI to capture. By default, PostgreSQL only records - modified fields in the log, which means that it might omit the `parent_key`. This can cause incorrect updates to the - Redis key in the destination database. - See the - [Debezium PostgreSQL Connector Documentation](https://debezium.io/documentation/reference/connectors/postgresql.html#postgresql-replica-identity) - for more information about this. + Nesting needs the child's `parent_key` to identify the parent Redis document, including + when the child is deleted. The default replica identity does not provide the full old + row, so a non-key `parent_key` can be unavailable. `FULL` records all old column values + in the write-ahead log (WAL). It can increase WAL volume and reduce performance. + See [Which PostgreSQL row values are available](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#understand-which-row-values-are-available) + for the difference between old row values and unchanged large values. - Prior to RDI v1.12.1, there is a known limitation if you change the foreign key of a child object. In that scenario, the child object will be added to the new parent, but the old parent will not be updated. diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md index d37ca6c598..5ee748a98d 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md @@ -150,18 +150,24 @@ to capture changes. Replace `` with the schema name and `` with the username of the Debezium user. 1. - Configure the PostgreSQL publication. For most RDI deployments, follow the - [PostgreSQL publication and replica identity guidance]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql" >}}) - and use `publication.autocreate.mode: filtered` when RDI captures selected tables. - - If you intentionally want to publish every table in the database, first check that - every table has a primary key or another replica identity. Then connect to your database - as the `postgres` user and run: + Connect to your database as the `postgres` user and create a publication containing + only the tables RDI should capture. First [check their replica identities](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#check-replica-identity-and-choose-a-publication-mode). + Replace the example table names with your selected tables: ```sql - CREATE PUBLICATION dbz_publication FOR ALL TABLES; + CREATE PUBLICATION rdi_publication + FOR TABLE public.customers, public.orders; ``` - Do not use this command without checking the replica identity of all tables. A later - `UPDATE` or `DELETE` on a published table without a replica identity can fail in the - application that writes to PostgreSQL. + Set `publication.name` to `rdi_publication` and `publication.autocreate.mode` to + `disabled` under the source's advanced source settings. On Redis Cloud, enter these + values under [**Collector properties**](/content/operate/rc/rdi/setup.md#postgresql-publications). + Update the publication's table list when you change the tables RDI captures. + + Alternatively, let RDI manage the publication with `publication.autocreate.mode: filtered` + after granting the [required ownership and creation permissions](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#set-privileges-for-debezium-to-create-postgresql-publications-with-pgoutput). + + Use `FOR ALL TABLES` only when you intend to publish every current and future table. + An application `UPDATE` or `DELETE` on any published table without replica identity can + fail, even if RDI does not capture that table. If you already created an all-tables + publication, follow [Replace an existing all-tables publication](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#replace-an-existing-all-tables-publication). diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md index 8da8fa6966..295b93ae4a 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md @@ -242,10 +242,27 @@ As a fallback, you can record the full old row in the write-ahead log: ALTER TABLE REPLICA IDENTITY FULL; ``` -`REPLICA IDENTITY FULL` can increase write-ahead log volume and reduce performance, so use -it only when a primary key or suitable unique index is not possible. The `keys` setting in -an RDI table definition creates a key for the Redis record. It does not create a PostgreSQL -primary key or replica identity. +`REPLICA IDENTITY FULL` can increase write-ahead log volume and reduce performance. +Use it when no suitable key is available or when RDI needs the full old row, such as +[denormalizing child tables with nesting](/content/integrate/redis-data-integration/data-pipelines/data-denormalization.md#using-nesting). +The `keys` setting in an RDI table definition creates a key for the Redis record. +It does not create a PostgreSQL primary key or replica identity. + +### Understand which row values are available + +Replica identity controls the old row values available to logical decoding for updates +and deletes. With the default identity, PostgreSQL uses the primary key; it does not +provide a complete old row. `REPLICA IDENTITY FULL` records all old column values. + +Update events are not generally limited to changed fields. However, PostgreSQL can omit +unchanged large values stored through The Oversized-Attribute Storage Technique (TOAST) +when they are not part of the replica identity. Debezium represents unavailable values +with a placeholder. With `FULL`, these values are available in the event's old and new row +images. See [Debezium's TOAST guidance](https://debezium.io/documentation/reference/3.0/connectors/postgresql.html#postgresql-toasted-values). + +For example, nesting a child record requires its `parent_key` to identify the parent +Redis document, including when the child is deleted. Set `REPLICA IDENTITY FULL` on +PostgreSQL child tables as required by the [nesting instructions](/content/integrate/redis-data-integration/data-pipelines/data-denormalization.md#using-nesting). ### Choose a publication mode @@ -259,12 +276,17 @@ PostgreSQL publication: | `disabled` | Does not create or change the publication. The publication must already exist. | Use when a database administrator manages the publication. | The `all_tables` mode can publish changes for tables that are not listed in the RDI -pipeline. If an application later updates or deletes a row in a published table without -a replica identity, PostgreSQL can reject that application write. For this reason, +pipeline, including tables created in the future. If an application later updates or +deletes a row in a published table without a replica identity, PostgreSQL can reject +that application write. For this reason, `filtered` is the recommended mode for a pipeline that captures only selected tables. +The `filtered` mode requires permission to create the publication and ownership rights on +the tables it manages. Use a separate publication for each source with different table +filters, so one source does not change another source's publication. + For a customer-managed publication, create an explicit table list and disable automatic -publication management. For example: +publication management. Replace the example table names with the tables you want to capture: ```sql CREATE PUBLICATION rdi_publication @@ -282,6 +304,43 @@ sources: publication.autocreate.mode: disabled ``` +For RDI on Redis Cloud, enter these properties under **Collector properties** in the +source configuration. See [Configure PostgreSQL publications on Redis Cloud](/content/operate/rc/rdi/setup.md#postgresql-publications). +With `disabled`, update the publication's table list yourself when changing the source's +selected tables. Selecting a table in RDI does not add it to a customer-managed publication. + +### Replace an existing all-tables publication + +Changing the RDI table selection does not narrow a `FOR ALL TABLES` publication. +Changing `publication.autocreate.mode` to `filtered` also does not convert it: +PostgreSQL rejects attempts to set, add, or remove individual tables in that publication. + +Have your database administrator plan the change with the owners of every consumer of +the publication: + +1. Check existing publications in the source database: + + ```sql + SELECT pubname, puballtables, pubupdate, pubdelete + FROM pg_publication; + ``` + +1. Identify every consumer of the broad publication and coordinate a maintenance window. +1. Stop the affected RDI sources before replacing the publication. +1. Replace the broad publication with an explicit table-list publication. You can recreate + it under the same name, or create a new publication and update each consumer's publication + name. For RDI, use `publication.autocreate.mode: disabled` for an administrator-managed + publication. On Redis Cloud, contact support if the running source's collector properties + cannot be edited in the console. +1. Remove the old broad publication only after coordinating all consumers. Leaving it in + place still subjects its tables to replica-identity requirements, even if RDI uses a + different publication or is stopped. +1. Resume the affected sources and verify inserts, updates, and deletes reach Redis. + +Keep the replication slots and saved RDI offsets during this change. Do not reset the +pipeline or drop its slot as a publication repair. If other consumers still need the broad +publication, configure replica identity on its affected tables instead of removing it. + ### PostgreSQL publication error FAQ #### I get this PostgreSQL error. What should I do? @@ -302,9 +361,12 @@ configure a suitable replica identity. As a fallback, run: ALTER TABLE REPLICA IDENTITY FULL; ``` -If the table should not be captured, use `publication.autocreate.mode: filtered` or remove -the table from the customer-managed publication. If you changed the RDI table filters or -publication mode, redeploy the pipeline so RDI applies the new configuration. +If the table should not be captured, remove it from an explicit table-list publication or +use `filtered` with the required permissions. For an existing `FOR ALL TABLES` publication, +follow [Replace an existing all-tables publication](#replace-an-existing-all-tables-publication). +Check all publications containing the table: another publication that publishes updates or +deletes can still require replica identity. After changing RDI configuration, redeploy the +pipeline so RDI applies it. ## Set privileges for Debezium to create PostgreSQL publications with `pgoutput` diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md index d559b06ab3..65b66ea348 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md @@ -151,7 +151,10 @@ CREATE PUBLICATION rdi_publication Creating the publication explicitly avoids granting table ownership or broad publication-creation permissions to the RDI role and limits the publication's -table scope. +table scope. Before creating it, [check replica identity](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#check-replica-identity-and-choose-a-publication-mode) +for every included table. Updates and deletes on published tables require a usable replica +identity. Update the publication's table list when you add or remove captured tables; +`publication.autocreate.mode: disabled` leaves that list under your control. ## 5. Configure TLS diff --git a/content/operate/rc/rdi/define.md b/content/operate/rc/rdi/define.md index 692bd553f7..e74894a4d8 100644 --- a/content/operate/rc/rdi/define.md +++ b/content/operate/rc/rdi/define.md @@ -130,10 +130,18 @@ Use **Collector properties** for additional source and sink settings. These sett ![The advanced properties dialog with separate collector source and sink properties.](/images/rc/rdi/rdi-advanced-properties.png) {width="80%"} +For PostgreSQL, [configure the publication properties](/content/operate/rc/rdi/setup.md#postgresql-publications) +before deployment. Use `disabled` for an administrator-managed table-list publication, or +`filtered` with the required permissions for RDI to manage it. + ## Select data {#dataset} Select each source in the **Sources** list and choose the data to ingest from that source. +For PostgreSQL, this selection does not narrow an existing `FOR ALL TABLES` publication. +If your administrator manages the publication, update its table list to match the selection. +See [PostgreSQL publications](/content/operate/rc/rdi/setup.md#postgresql-publications). + > [!WARNING] > Do not write data directly to keys managed by RDI. Changes from another application can cause transformation failures or data inconsistencies, and RDI can overwrite them. A pipeline reset does not flush the target database. **Flush target database** is a separate action that deletes all target data, including data written outside RDI. See [Data recovery](/content/operate/rc/rdi/faq.md#data-recovery). @@ -156,6 +164,10 @@ Select each source in the **Sources** list and choose the data to ingest from th ![The key control beside the column used to identify a record.](/images/rc/rdi/rdi-unique-key-selected.png) {width="500px"} + For PostgreSQL, choosing a key here does not create a primary key or replica identity + in the source database. [Configure replica identity](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#check-replica-identity-and-choose-a-publication-mode) + separately for tables that publish updates or deletes. + 1. Repeat for the other sources, then select **Continue**. The available schema, table, and column controls depend on the source type. Each source keeps its own selection, including when two sources contain tables with the same name. diff --git a/content/operate/rc/rdi/quick-start.md b/content/operate/rc/rdi/quick-start.md index 4c578cdc1b..15bf88fc32 100644 --- a/content/operate/rc/rdi/quick-start.md +++ b/content/operate/rc/rdi/quick-start.md @@ -129,6 +129,10 @@ If you lose any outputs, run `terraform output` to view them again. {width="80%"} 1. Under **Source configuration**, enter the `database` and `port` Terraform outputs in the matching fields. +1. Under **Collector properties**, set `publication.autocreate.mode` to `filtered` so RDI + creates a publication for the selected tables. This example uses the `postgres` user, + which has the required permissions. For an administrator-managed publication or an + existing all-tables publication, follow [PostgreSQL publications](/content/operate/rc/rdi/setup.md#postgresql-publications). 1. Select **Test source** and correct any validation errors, then select **Continue**. 1. In **Select data**, select the schemas, tables, and columns to ingest. Review the selected key for each table. diff --git a/content/operate/rc/rdi/setup.md b/content/operate/rc/rdi/setup.md index 7de00a64c0..ae4f7e8b26 100644 --- a/content/operate/rc/rdi/setup.md +++ b/content/operate/rc/rdi/setup.md @@ -33,6 +33,36 @@ See [Prepare source databases](/content/integrate/redis-data-integration/data-pi See the [RDI architecture overview](/content/integrate/redis-data-integration/architecture/_index.md#overview) for more information about CDC. +### PostgreSQL publications {#postgresql-publications} + +For PostgreSQL sources, including databases hosted on EC2, Aurora/RDS, and Supabase, +[check replica identity](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#check-replica-identity-and-choose-a-publication-mode) +before deploying. Selecting tables in the console does not narrow an existing +`FOR ALL TABLES` publication. That publication includes current and future tables, and +updates or deletes on any published table without replica identity can fail in your +application. + +For a publication managed by your database administrator, create an explicit table list +as shown in the [PostgreSQL preparation guide](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#choose-a-publication-mode). +In **Configure source**, under **Collector properties**, open **Edit advanced properties** +and enter these source properties: + +| Property | Value | +|:--|:--| +| `publication.name` | The name of the publication created by your administrator | +| `publication.autocreate.mode` | `disabled` | + +Keep the publication's table list aligned with the source's selected tables. With +`disabled`, RDI does not update the publication when you change the selection. + +To let RDI create and manage a publication for selected tables, use `filtered` instead. +This requires [publication creation and table ownership permissions](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#set-privileges-for-debezium-to-create-postgresql-publications-with-pgoutput). +Use a separate publication for each source with different table selections. + +If a source already uses `FOR ALL TABLES`, follow +[Replace an existing all-tables publication](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#replace-an-existing-all-tables-publication). +Changing the collector property alone does not convert that publication. + ### Supabase {#supabase} You can use a hosted [Supabase](https://supabase.com/) PostgreSQL database as From 01a1a436ad427aa45197cc927c68bee480890617 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Sun, 18 Oct 2026 04:28:53 +0300 Subject: [PATCH 5/5] RDSC-6238: Check usable replica identity indexes in PostgreSQL guidance --- .../data-pipelines/prepare-dbs/postgresql.md | 14 +++++++++++--- .../data-pipelines/prepare-dbs/postgresql.md | 14 +++++++++++--- 2 files changed, 22 insertions(+), 6 deletions(-) diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md index 9b348db69a..4bc8677325 100644 --- a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md +++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md @@ -214,19 +214,27 @@ WHERE c.relkind = 'r' AND ( c.relreplident = 'n' OR ( - c.relreplident = 'd' + c.relreplident IN ('d', 'i') AND NOT EXISTS ( SELECT 1 FROM pg_index AS i WHERE i.indrelid = c.oid - AND i.indisprimary + AND i.indislive + AND i.indisvalid + AND i.indisunique + AND i.indimmediate + AND i.indpred IS NULL + AND CASE WHEN c.relreplident = 'd' + THEN i.indisprimary + ELSE i.indisreplident + END ) ) ) ORDER BY n.nspname, c.relname; ``` -If this query returns a table that will be published, add a primary key where possible. +If this query returns a table that will be published, add a primary key that is not `DEFERRABLE` where possible. A deferrable primary key cannot serve as the replica identity. You can also configure a suitable unique index as the replica identity: ```sql diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md index 295b93ae4a..1ae6637135 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md @@ -217,19 +217,27 @@ WHERE c.relkind = 'r' AND ( c.relreplident = 'n' OR ( - c.relreplident = 'd' + c.relreplident IN ('d', 'i') AND NOT EXISTS ( SELECT 1 FROM pg_index AS i WHERE i.indrelid = c.oid - AND i.indisprimary + AND i.indislive + AND i.indisvalid + AND i.indisunique + AND i.indimmediate + AND i.indpred IS NULL + AND CASE WHEN c.relreplident = 'd' + THEN i.indisprimary + ELSE i.indisreplident + END ) ) ) ORDER BY n.nspname, c.relname; ``` -If this query returns a table that will be published, add a primary key where possible. +If this query returns a table that will be published, add a primary key that is not `DEFERRABLE` where possible. A deferrable primary key cannot serve as the replica identity. You can also configure a suitable unique index as the replica identity: ```sql