diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization.md index 8fa42654e4..eacf01dd3e 100644 --- a/content/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization.md +++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization.md @@ -217,11 +217,10 @@ There are several important things to note when you use nesting: ALTER TABLE REPLICA IDENTITY FULL; ``` - This configuration affects the information written to the write-ahead log (WAL) and whether it is available - for RDI to capture. By default, PostgreSQL only records - modified fields in the log, which means that it might omit the `parent_key`. This can cause incorrect updates to the - Redis key in the destination database. - See the - [Debezium PostgreSQL Connector Documentation](https://debezium.io/documentation/reference/connectors/postgresql.html#postgresql-replica-identity) - for more information about this. + Nesting needs the child's `parent_key` to identify the parent Redis document, including + when the child is deleted. The default replica identity does not provide the full old + row, so a non-key `parent_key` can be unavailable. `FULL` records all old column values + in the write-ahead log (WAL). It can increase WAL volume and reduce performance. + See [Which PostgreSQL row values are available](/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md#understand-which-row-values-are-available) + for the difference between old row values and unchanged large values. - Prior to RDI v1.12.1, there is a known limitation if you change the foreign key of a child object. In that scenario, the child object will be added to the new parent, but the old parent will not be updated. diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md index 3254e9fd2f..55def8274a 100644 --- a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md +++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md @@ -147,8 +147,24 @@ to capture changes. Replace `` with the schema name and `` with the username of the Debezium user. 1. - Connect to your database as the `postgres` user and create a publication for the database: + Connect to your database as the `postgres` user and create a publication containing + only the tables RDI should capture. First [check their replica identities](/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md#check-replica-identity-and-choose-a-publication-mode). + Replace the example table names with your selected tables: ```sql - CREATE PUBLICATION dbz_publication FOR ALL TABLES; + CREATE PUBLICATION rdi_publication + FOR TABLE public.customers, public.orders; ``` + + Set `publication.name` to `rdi_publication` and `publication.autocreate.mode` to + `disabled` under the source's advanced source settings. On Redis Cloud, enter these + values under [**Collector properties**](/content/operate/rc/rdi/setup.md#postgresql-publications). + Update the publication's table list when you change the tables RDI captures. + + Alternatively, let RDI manage the publication with `publication.autocreate.mode: filtered` + after granting the [required ownership and creation permissions](/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md#set-privileges-for-debezium-to-create-postgresql-publications-with-pgoutput). + + Use `FOR ALL TABLES` only when you intend to publish every current and future table. + An application `UPDATE` or `DELETE` on any published table without replica identity can + fail, even if RDI does not capture that table. If you already created an all-tables + publication, follow [Replace an existing all-tables publication](/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md#replace-an-existing-all-tables-publication). diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md index 31ccc7149f..4bc8677325 100644 --- a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md +++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md @@ -36,6 +36,7 @@ checklist as you complete each step. - [ ] [Install the logical decoding output plug-in](#install-the-logical-decoding-output-plug-in) - [ ] [Configure the PostgreSQL server](#configure-the-postgresql-server) - [ ] [Set up permissions](#set-up-permissions) +- [ ] [Check replica identity and choose a publication mode](#check-replica-identity-and-choose-a-publication-mode) - [ ] [Set privileges for Debezium to create PostgreSQL publications with pgoutput](#set-privileges-for-debezium-to-create-postgresql-publications-with-pgoutput) - [ ] [Configure PostgreSQL for replication with the Debezium connector host](#configure-postgresql-for-replication-with-the-debezium-connector-host) ``` @@ -180,6 +181,198 @@ needs to perform replications, but you might also need to grant other permission CREATE ROLE REPLICATION LOGIN; ``` +## Check replica identity and choose a publication mode + +Before you start RDI, check the tables that RDI will capture and choose how PostgreSQL +should publish their changes. + +PostgreSQL needs a [replica identity](https://www.postgresql.org/docs/current/logical-replication-publication.html) +to identify the old row when it publishes an `UPDATE` or `DELETE`. By default, PostgreSQL +uses the table's primary key. A table without a primary key has no usable replica identity +unless you configure one. Inserts do not need a replica identity. + +Run the following query while connected to the database that RDI uses. You do not put the +database name in the query; select that database in your SQL editor or connection settings. +The query checks all non-system schemas. To check only one schema, uncomment the marked line +and replace `` with your schema name. + +```sql +SELECT n.nspname AS schema_name, + c.relname AS table_name, + CASE c.relreplident + WHEN 'd' THEN 'default' + WHEN 'n' THEN 'nothing' + WHEN 'f' THEN 'full' + WHEN 'i' THEN 'index' + END AS replica_identity +FROM pg_class AS c +JOIN pg_namespace AS n ON n.oid = c.relnamespace +WHERE c.relkind = 'r' + AND n.nspname NOT IN ('pg_catalog', 'information_schema') + -- Optional: uncomment and replace to check one schema only. + -- AND n.nspname = '' + AND ( + c.relreplident = 'n' + OR ( + c.relreplident IN ('d', 'i') + AND NOT EXISTS ( + SELECT 1 + FROM pg_index AS i + WHERE i.indrelid = c.oid + AND i.indislive + AND i.indisvalid + AND i.indisunique + AND i.indimmediate + AND i.indpred IS NULL + AND CASE WHEN c.relreplident = 'd' + THEN i.indisprimary + ELSE i.indisreplident + END + ) + ) + ) +ORDER BY n.nspname, c.relname; +``` + +If this query returns a table that will be published, add a primary key that is not `DEFERRABLE` where possible. A deferrable primary key cannot serve as the replica identity. +You can also configure a suitable unique index as the replica identity: + +```sql +ALTER TABLE REPLICA IDENTITY USING INDEX ; +``` + +As a fallback, you can record the full old row in the write-ahead log: + +```sql +ALTER TABLE REPLICA IDENTITY FULL; +``` + +`REPLICA IDENTITY FULL` can increase write-ahead log volume and reduce performance. +Use it when no suitable key is available or when RDI needs the full old row, such as +[denormalizing child tables with nesting](/content/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization.md#using-nesting). +The `keys` setting in an RDI table definition creates a key for the Redis record. +It does not create a PostgreSQL primary key or replica identity. + +### Understand which row values are available + +Replica identity controls the old row values available to logical decoding for updates +and deletes. With the default identity, PostgreSQL uses the primary key; it does not +provide a complete old row. `REPLICA IDENTITY FULL` records all old column values. + +Update events are not generally limited to changed fields. However, PostgreSQL can omit +unchanged large values stored through The Oversized-Attribute Storage Technique (TOAST) +when they are not part of the replica identity. Debezium represents unavailable values +with a placeholder. With `FULL`, these values are available in the event's old and new row +images. See [Debezium's TOAST guidance](https://debezium.io/documentation/reference/3.0/connectors/postgresql.html#postgresql-toasted-values). + +For example, nesting a child record requires its `parent_key` to identify the parent +Redis document, including when the child is deleted. Set `REPLICA IDENTITY FULL` on +PostgreSQL child tables as required by the [nesting instructions](/content/integrate/redis-data-integration/1.19.1/data-pipelines/data-denormalization.md#using-nesting). + +### Choose a publication mode + +The `publication.autocreate.mode` setting controls how RDI and Debezium manage the +PostgreSQL publication: + +| Mode | Behavior | Recommended use | +| --- | --- | --- | +| **`all_tables` (default)** | Creates a `FOR ALL TABLES` publication when the publication does not exist. If it already exists, RDI uses the existing publication. | Use when all tables in the database should be published, after checking their replica identities. | +| `filtered` | Creates or updates the publication to contain the tables configured for the RDI source. | Recommended when RDI captures selected tables. RDI manages the publication when the pipeline is redeployed with a changed table filter. | +| `disabled` | Does not create or change the publication. The publication must already exist. | Use when a database administrator manages the publication. | + +The `all_tables` mode can publish changes for tables that are not listed in the RDI +pipeline, including tables created in the future. If an application later updates or +deletes a row in a published table without a replica identity, PostgreSQL can reject +that application write. For this reason, +`filtered` is the recommended mode for a pipeline that captures only selected tables. + +The `filtered` mode requires permission to create the publication and ownership rights on +the tables it manages. Use a separate publication for each source with different table +filters, so one source does not change another source's publication. + +For a customer-managed publication, create an explicit table list and disable automatic +publication management. Replace the example table names with the tables you want to capture: + +```sql +CREATE PUBLICATION rdi_publication +FOR TABLE public.customers, public.orders; +``` + +Then set the following properties under the PostgreSQL source in your RDI `config.yaml`: + +```yaml +sources: + : + advanced: + source: + publication.name: rdi_publication + publication.autocreate.mode: disabled +``` + +For RDI on Redis Cloud, enter these properties under **Collector properties** in the +source configuration. See [Configure PostgreSQL publications on Redis Cloud](/content/operate/rc/rdi/setup.md#postgresql-publications). +With `disabled`, update the publication's table list yourself when changing the source's +selected tables. Selecting a table in RDI does not add it to a customer-managed publication. + +### Replace an existing all-tables publication + +Changing the RDI table selection does not narrow a `FOR ALL TABLES` publication. +Changing `publication.autocreate.mode` to `filtered` also does not convert it: +PostgreSQL rejects attempts to set, add, or remove individual tables in that publication. + +Have your database administrator plan the change with the owners of every consumer of +the publication: + +1. Check existing publications in the source database: + + ```sql + SELECT pubname, puballtables, pubupdate, pubdelete + FROM pg_publication; + ``` + +1. Identify every consumer of the broad publication and coordinate a maintenance window. +1. Stop the affected RDI sources before replacing the publication. +1. Replace the broad publication with an explicit table-list publication. You can recreate + it under the same name, or create a new publication and update each consumer's publication + name. For RDI, use `publication.autocreate.mode: disabled` for an administrator-managed + publication. On Redis Cloud, contact support if the running source's collector properties + cannot be edited in the console. +1. Remove the old broad publication only after coordinating all consumers. Leaving it in + place still subjects its tables to replica-identity requirements, even if RDI uses a + different publication or is stopped. +1. Resume the affected sources and verify inserts, updates, and deletes reach Redis. + +Keep the replication slots and saved RDI offsets during this change. Do not reset the +pipeline or drop its slot as a publication repair. If other consumers still need the broad +publication, configure replica identity on its affected tables instead of removing it. + +### PostgreSQL publication error FAQ + +#### I get this PostgreSQL error. What should I do? + +You may see the following error when an application updates a published table without a +replica identity: + +```text +SQLSTATE 55000 +ERROR: cannot update table "example_table" because it does not have a replica identity and publishes updates +HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE. +``` + +Check whether the table should be captured by RDI. If it should, add a primary key or +configure a suitable replica identity. As a fallback, run: + +```sql +ALTER TABLE REPLICA IDENTITY FULL; +``` + +If the table should not be captured, remove it from an explicit table-list publication or +use `filtered` with the required permissions. For an existing `FOR ALL TABLES` publication, +follow [Replace an existing all-tables publication](#replace-an-existing-all-tables-publication). +Check all publications containing the table: another publication that publishes updates or +deletes can still require replica identity. After changing RDI configuration, redeploy the +pipeline so RDI applies it. + ## Set privileges for Debezium to create PostgreSQL publications with `pgoutput` The Debezium user needs specific permissions to work with the `pgoutput` plug-in. @@ -238,10 +431,11 @@ the replication group using the following commands: ALTER TABLE table_name OWNER TO replication_group; ``` -You must also set the value of the `publication.autocreate.mode` parameter to `filtered` -to allow Debezium to specify the publication configuration. See the +If you want Debezium to create or update a publication for the tables configured in the RDI +source, set `publication.autocreate.mode` to `filtered`. See the [Debezium documentation for `publication.autocreate.mode`](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-publication-autocreate-mode) -to learn more about this setting. +to learn more about this setting. If a database administrator manages an existing +publication, use `disabled` instead, as described in [Choose a publication mode](#choose-a-publication-mode). ## Configure PostgreSQL for replication with the Debezium connector host diff --git a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/supabase.md b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/supabase.md index 4e86b37b73..c4bacdd049 100644 --- a/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/supabase.md +++ b/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/supabase.md @@ -152,7 +152,10 @@ CREATE PUBLICATION rdi_publication Creating the publication explicitly avoids granting table ownership or broad publication-creation permissions to the RDI role and limits the publication's -table scope. +table scope. Before creating it, [check replica identity](/content/integrate/redis-data-integration/1.19.1/data-pipelines/prepare-dbs/postgresql.md#check-replica-identity-and-choose-a-publication-mode) +for every included table. Updates and deletes on published tables require a usable replica +identity. Update the publication's table list when you add or remove captured tables; +`publication.autocreate.mode: disabled` leaves that list under your control. ## 5. Configure TLS diff --git a/content/integrate/redis-data-integration/data-pipelines/data-denormalization.md b/content/integrate/redis-data-integration/data-pipelines/data-denormalization.md index 6cacf08712..f09e8bf000 100644 --- a/content/integrate/redis-data-integration/data-pipelines/data-denormalization.md +++ b/content/integrate/redis-data-integration/data-pipelines/data-denormalization.md @@ -219,11 +219,10 @@ There are several important things to note when you use nesting: ALTER TABLE REPLICA IDENTITY FULL; ``` - This configuration affects the information written to the write-ahead log (WAL) and whether it is available - for RDI to capture. By default, PostgreSQL only records - modified fields in the log, which means that it might omit the `parent_key`. This can cause incorrect updates to the - Redis key in the destination database. - See the - [Debezium PostgreSQL Connector Documentation](https://debezium.io/documentation/reference/connectors/postgresql.html#postgresql-replica-identity) - for more information about this. + Nesting needs the child's `parent_key` to identify the parent Redis document, including + when the child is deleted. The default replica identity does not provide the full old + row, so a non-key `parent_key` can be unavailable. `FULL` records all old column values + in the write-ahead log (WAL). It can increase WAL volume and reduce performance. + See [Which PostgreSQL row values are available](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#understand-which-row-values-are-available) + for the difference between old row values and unchanged large values. - Prior to RDI v1.12.1, there is a known limitation if you change the foreign key of a child object. In that scenario, the child object will be added to the new parent, but the old parent will not be updated. diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md index c18fcd7b05..5ee748a98d 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md @@ -150,8 +150,24 @@ to capture changes. Replace `` with the schema name and `` with the username of the Debezium user. 1. - Connect to your database as the `postgres` user and create a publication for the database: + Connect to your database as the `postgres` user and create a publication containing + only the tables RDI should capture. First [check their replica identities](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#check-replica-identity-and-choose-a-publication-mode). + Replace the example table names with your selected tables: ```sql - CREATE PUBLICATION dbz_publication FOR ALL TABLES; + CREATE PUBLICATION rdi_publication + FOR TABLE public.customers, public.orders; ``` + + Set `publication.name` to `rdi_publication` and `publication.autocreate.mode` to + `disabled` under the source's advanced source settings. On Redis Cloud, enter these + values under [**Collector properties**](/content/operate/rc/rdi/setup.md#postgresql-publications). + Update the publication's table list when you change the tables RDI captures. + + Alternatively, let RDI manage the publication with `publication.autocreate.mode: filtered` + after granting the [required ownership and creation permissions](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#set-privileges-for-debezium-to-create-postgresql-publications-with-pgoutput). + + Use `FOR ALL TABLES` only when you intend to publish every current and future table. + An application `UPDATE` or `DELETE` on any published table without replica identity can + fail, even if RDI does not capture that table. If you already created an all-tables + publication, follow [Replace an existing all-tables publication](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#replace-an-existing-all-tables-publication). diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md index 55ae12fc33..1ae6637135 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md @@ -39,6 +39,7 @@ checklist as you complete each step. - [ ] [Install the logical decoding output plug-in](#install-the-logical-decoding-output-plug-in) - [ ] [Configure the PostgreSQL server](#configure-the-postgresql-server) - [ ] [Set up permissions](#set-up-permissions) +- [ ] [Check replica identity and choose a publication mode](#check-replica-identity-and-choose-a-publication-mode) - [ ] [Set privileges for Debezium to create PostgreSQL publications with pgoutput](#set-privileges-for-debezium-to-create-postgresql-publications-with-pgoutput) - [ ] [Configure PostgreSQL for replication with the Debezium connector host](#configure-postgresql-for-replication-with-the-debezium-connector-host) ``` @@ -183,6 +184,198 @@ needs to perform replications, but you might also need to grant other permission CREATE ROLE REPLICATION LOGIN; ``` +## Check replica identity and choose a publication mode + +Before you start RDI, check the tables that RDI will capture and choose how PostgreSQL +should publish their changes. + +PostgreSQL needs a [replica identity](https://www.postgresql.org/docs/current/logical-replication-publication.html) +to identify the old row when it publishes an `UPDATE` or `DELETE`. By default, PostgreSQL +uses the table's primary key. A table without a primary key has no usable replica identity +unless you configure one. Inserts do not need a replica identity. + +Run the following query while connected to the database that RDI uses. You do not put the +database name in the query; select that database in your SQL editor or connection settings. +The query checks all non-system schemas. To check only one schema, uncomment the marked line +and replace `` with your schema name. + +```sql +SELECT n.nspname AS schema_name, + c.relname AS table_name, + CASE c.relreplident + WHEN 'd' THEN 'default' + WHEN 'n' THEN 'nothing' + WHEN 'f' THEN 'full' + WHEN 'i' THEN 'index' + END AS replica_identity +FROM pg_class AS c +JOIN pg_namespace AS n ON n.oid = c.relnamespace +WHERE c.relkind = 'r' + AND n.nspname NOT IN ('pg_catalog', 'information_schema') + -- Optional: uncomment and replace to check one schema only. + -- AND n.nspname = '' + AND ( + c.relreplident = 'n' + OR ( + c.relreplident IN ('d', 'i') + AND NOT EXISTS ( + SELECT 1 + FROM pg_index AS i + WHERE i.indrelid = c.oid + AND i.indislive + AND i.indisvalid + AND i.indisunique + AND i.indimmediate + AND i.indpred IS NULL + AND CASE WHEN c.relreplident = 'd' + THEN i.indisprimary + ELSE i.indisreplident + END + ) + ) + ) +ORDER BY n.nspname, c.relname; +``` + +If this query returns a table that will be published, add a primary key that is not `DEFERRABLE` where possible. A deferrable primary key cannot serve as the replica identity. +You can also configure a suitable unique index as the replica identity: + +```sql +ALTER TABLE REPLICA IDENTITY USING INDEX ; +``` + +As a fallback, you can record the full old row in the write-ahead log: + +```sql +ALTER TABLE REPLICA IDENTITY FULL; +``` + +`REPLICA IDENTITY FULL` can increase write-ahead log volume and reduce performance. +Use it when no suitable key is available or when RDI needs the full old row, such as +[denormalizing child tables with nesting](/content/integrate/redis-data-integration/data-pipelines/data-denormalization.md#using-nesting). +The `keys` setting in an RDI table definition creates a key for the Redis record. +It does not create a PostgreSQL primary key or replica identity. + +### Understand which row values are available + +Replica identity controls the old row values available to logical decoding for updates +and deletes. With the default identity, PostgreSQL uses the primary key; it does not +provide a complete old row. `REPLICA IDENTITY FULL` records all old column values. + +Update events are not generally limited to changed fields. However, PostgreSQL can omit +unchanged large values stored through The Oversized-Attribute Storage Technique (TOAST) +when they are not part of the replica identity. Debezium represents unavailable values +with a placeholder. With `FULL`, these values are available in the event's old and new row +images. See [Debezium's TOAST guidance](https://debezium.io/documentation/reference/3.0/connectors/postgresql.html#postgresql-toasted-values). + +For example, nesting a child record requires its `parent_key` to identify the parent +Redis document, including when the child is deleted. Set `REPLICA IDENTITY FULL` on +PostgreSQL child tables as required by the [nesting instructions](/content/integrate/redis-data-integration/data-pipelines/data-denormalization.md#using-nesting). + +### Choose a publication mode + +The `publication.autocreate.mode` setting controls how RDI and Debezium manage the +PostgreSQL publication: + +| Mode | Behavior | Recommended use | +| --- | --- | --- | +| **`all_tables` (default)** | Creates a `FOR ALL TABLES` publication when the publication does not exist. If it already exists, RDI uses the existing publication. | Use when all tables in the database should be published, after checking their replica identities. | +| `filtered` | Creates or updates the publication to contain the tables configured for the RDI source. | Recommended when RDI captures selected tables. RDI manages the publication when the pipeline is redeployed with a changed table filter. | +| `disabled` | Does not create or change the publication. The publication must already exist. | Use when a database administrator manages the publication. | + +The `all_tables` mode can publish changes for tables that are not listed in the RDI +pipeline, including tables created in the future. If an application later updates or +deletes a row in a published table without a replica identity, PostgreSQL can reject +that application write. For this reason, +`filtered` is the recommended mode for a pipeline that captures only selected tables. + +The `filtered` mode requires permission to create the publication and ownership rights on +the tables it manages. Use a separate publication for each source with different table +filters, so one source does not change another source's publication. + +For a customer-managed publication, create an explicit table list and disable automatic +publication management. Replace the example table names with the tables you want to capture: + +```sql +CREATE PUBLICATION rdi_publication +FOR TABLE public.customers, public.orders; +``` + +Then set the following properties under the PostgreSQL source in your RDI `config.yaml`: + +```yaml +sources: + : + advanced: + source: + publication.name: rdi_publication + publication.autocreate.mode: disabled +``` + +For RDI on Redis Cloud, enter these properties under **Collector properties** in the +source configuration. See [Configure PostgreSQL publications on Redis Cloud](/content/operate/rc/rdi/setup.md#postgresql-publications). +With `disabled`, update the publication's table list yourself when changing the source's +selected tables. Selecting a table in RDI does not add it to a customer-managed publication. + +### Replace an existing all-tables publication + +Changing the RDI table selection does not narrow a `FOR ALL TABLES` publication. +Changing `publication.autocreate.mode` to `filtered` also does not convert it: +PostgreSQL rejects attempts to set, add, or remove individual tables in that publication. + +Have your database administrator plan the change with the owners of every consumer of +the publication: + +1. Check existing publications in the source database: + + ```sql + SELECT pubname, puballtables, pubupdate, pubdelete + FROM pg_publication; + ``` + +1. Identify every consumer of the broad publication and coordinate a maintenance window. +1. Stop the affected RDI sources before replacing the publication. +1. Replace the broad publication with an explicit table-list publication. You can recreate + it under the same name, or create a new publication and update each consumer's publication + name. For RDI, use `publication.autocreate.mode: disabled` for an administrator-managed + publication. On Redis Cloud, contact support if the running source's collector properties + cannot be edited in the console. +1. Remove the old broad publication only after coordinating all consumers. Leaving it in + place still subjects its tables to replica-identity requirements, even if RDI uses a + different publication or is stopped. +1. Resume the affected sources and verify inserts, updates, and deletes reach Redis. + +Keep the replication slots and saved RDI offsets during this change. Do not reset the +pipeline or drop its slot as a publication repair. If other consumers still need the broad +publication, configure replica identity on its affected tables instead of removing it. + +### PostgreSQL publication error FAQ + +#### I get this PostgreSQL error. What should I do? + +You may see the following error when an application updates a published table without a +replica identity: + +```text +SQLSTATE 55000 +ERROR: cannot update table "example_table" because it does not have a replica identity and publishes updates +HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE. +``` + +Check whether the table should be captured by RDI. If it should, add a primary key or +configure a suitable replica identity. As a fallback, run: + +```sql +ALTER TABLE REPLICA IDENTITY FULL; +``` + +If the table should not be captured, remove it from an explicit table-list publication or +use `filtered` with the required permissions. For an existing `FOR ALL TABLES` publication, +follow [Replace an existing all-tables publication](#replace-an-existing-all-tables-publication). +Check all publications containing the table: another publication that publishes updates or +deletes can still require replica identity. After changing RDI configuration, redeploy the +pipeline so RDI applies it. + ## Set privileges for Debezium to create PostgreSQL publications with `pgoutput` The Debezium user needs specific permissions to work with the `pgoutput` plug-in. @@ -241,10 +434,11 @@ the replication group using the following commands: ALTER TABLE table_name OWNER TO replication_group; ``` -You must also set the value of the `publication.autocreate.mode` parameter to `filtered` -to allow Debezium to specify the publication configuration. See the +If you want Debezium to create or update a publication for the tables configured in the RDI +source, set `publication.autocreate.mode` to `filtered`. See the [Debezium documentation for `publication.autocreate.mode`](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-publication-autocreate-mode) -to learn more about this setting. +to learn more about this setting. If a database administrator manages an existing +publication, use `disabled` instead, as described in [Choose a publication mode](#choose-a-publication-mode). ## Configure PostgreSQL for replication with the Debezium connector host diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md index d559b06ab3..65b66ea348 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md @@ -151,7 +151,10 @@ CREATE PUBLICATION rdi_publication Creating the publication explicitly avoids granting table ownership or broad publication-creation permissions to the RDI role and limits the publication's -table scope. +table scope. Before creating it, [check replica identity](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#check-replica-identity-and-choose-a-publication-mode) +for every included table. Updates and deletes on published tables require a usable replica +identity. Update the publication's table list when you add or remove captured tables; +`publication.autocreate.mode: disabled` leaves that list under your control. ## 5. Configure TLS diff --git a/content/operate/rc/rdi/define.md b/content/operate/rc/rdi/define.md index 692bd553f7..e74894a4d8 100644 --- a/content/operate/rc/rdi/define.md +++ b/content/operate/rc/rdi/define.md @@ -130,10 +130,18 @@ Use **Collector properties** for additional source and sink settings. These sett ![The advanced properties dialog with separate collector source and sink properties.](/images/rc/rdi/rdi-advanced-properties.png) {width="80%"} +For PostgreSQL, [configure the publication properties](/content/operate/rc/rdi/setup.md#postgresql-publications) +before deployment. Use `disabled` for an administrator-managed table-list publication, or +`filtered` with the required permissions for RDI to manage it. + ## Select data {#dataset} Select each source in the **Sources** list and choose the data to ingest from that source. +For PostgreSQL, this selection does not narrow an existing `FOR ALL TABLES` publication. +If your administrator manages the publication, update its table list to match the selection. +See [PostgreSQL publications](/content/operate/rc/rdi/setup.md#postgresql-publications). + > [!WARNING] > Do not write data directly to keys managed by RDI. Changes from another application can cause transformation failures or data inconsistencies, and RDI can overwrite them. A pipeline reset does not flush the target database. **Flush target database** is a separate action that deletes all target data, including data written outside RDI. See [Data recovery](/content/operate/rc/rdi/faq.md#data-recovery). @@ -156,6 +164,10 @@ Select each source in the **Sources** list and choose the data to ingest from th ![The key control beside the column used to identify a record.](/images/rc/rdi/rdi-unique-key-selected.png) {width="500px"} + For PostgreSQL, choosing a key here does not create a primary key or replica identity + in the source database. [Configure replica identity](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#check-replica-identity-and-choose-a-publication-mode) + separately for tables that publish updates or deletes. + 1. Repeat for the other sources, then select **Continue**. The available schema, table, and column controls depend on the source type. Each source keeps its own selection, including when two sources contain tables with the same name. diff --git a/content/operate/rc/rdi/quick-start.md b/content/operate/rc/rdi/quick-start.md index 4c578cdc1b..15bf88fc32 100644 --- a/content/operate/rc/rdi/quick-start.md +++ b/content/operate/rc/rdi/quick-start.md @@ -129,6 +129,10 @@ If you lose any outputs, run `terraform output` to view them again. {width="80%"} 1. Under **Source configuration**, enter the `database` and `port` Terraform outputs in the matching fields. +1. Under **Collector properties**, set `publication.autocreate.mode` to `filtered` so RDI + creates a publication for the selected tables. This example uses the `postgres` user, + which has the required permissions. For an administrator-managed publication or an + existing all-tables publication, follow [PostgreSQL publications](/content/operate/rc/rdi/setup.md#postgresql-publications). 1. Select **Test source** and correct any validation errors, then select **Continue**. 1. In **Select data**, select the schemas, tables, and columns to ingest. Review the selected key for each table. diff --git a/content/operate/rc/rdi/setup.md b/content/operate/rc/rdi/setup.md index 7de00a64c0..ae4f7e8b26 100644 --- a/content/operate/rc/rdi/setup.md +++ b/content/operate/rc/rdi/setup.md @@ -33,6 +33,36 @@ See [Prepare source databases](/content/integrate/redis-data-integration/data-pi See the [RDI architecture overview](/content/integrate/redis-data-integration/architecture/_index.md#overview) for more information about CDC. +### PostgreSQL publications {#postgresql-publications} + +For PostgreSQL sources, including databases hosted on EC2, Aurora/RDS, and Supabase, +[check replica identity](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#check-replica-identity-and-choose-a-publication-mode) +before deploying. Selecting tables in the console does not narrow an existing +`FOR ALL TABLES` publication. That publication includes current and future tables, and +updates or deletes on any published table without replica identity can fail in your +application. + +For a publication managed by your database administrator, create an explicit table list +as shown in the [PostgreSQL preparation guide](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#choose-a-publication-mode). +In **Configure source**, under **Collector properties**, open **Edit advanced properties** +and enter these source properties: + +| Property | Value | +|:--|:--| +| `publication.name` | The name of the publication created by your administrator | +| `publication.autocreate.mode` | `disabled` | + +Keep the publication's table list aligned with the source's selected tables. With +`disabled`, RDI does not update the publication when you change the selection. + +To let RDI create and manage a publication for selected tables, use `filtered` instead. +This requires [publication creation and table ownership permissions](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#set-privileges-for-debezium-to-create-postgresql-publications-with-pgoutput). +Use a separate publication for each source with different table selections. + +If a source already uses `FOR ALL TABLES`, follow +[Replace an existing all-tables publication](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md#replace-an-existing-all-tables-publication). +Changing the collector property alone does not convert that publication. + ### Supabase {#supabase} You can use a hosted [Supabase](https://supabase.com/) PostgreSQL database as