We think there might be an upper bound on a single COPY operation that's not disk-bound. Effectively, we need to find a way to parallelize a COPY operation of a single table.
The approach we could take is to issue sub-selects on a primary key range. Here's an example of a 3-way split:
COPY (SELECT * FROM t WHERE id < n) t TO STDOUT;
COPY (SELECT * FROM t WHERE id >= n AND < m) TO STDOUT:
COPY (SELECT * FROM t WHERE id > m) TO STDOUT:
The issue is synchronization. These statements are executed inside 3 different transactions, so they don't have the same view of the table.
One approach we could take to sync at the end is to use upserts instead of inserts. Once logical replication boots up, even if the destination table already has the row, it will be replaced with the newest version. We just need to pick the earliest LSN of any of these copies as the starting point for the stream.
We think there might be an upper bound on a single
COPYoperation that's not disk-bound. Effectively, we need to find a way to parallelize aCOPYoperation of a single table.The approach we could take is to issue sub-selects on a primary key range. Here's an example of a 3-way split:
The issue is synchronization. These statements are executed inside 3 different transactions, so they don't have the same view of the table.
One approach we could take to sync at the end is to use upserts instead of inserts. Once logical replication boots up, even if the destination table already has the row, it will be replaced with the newest version. We just need to pick the earliest LSN of any of these copies as the starting point for the stream.