Description
When using PostgresToGCSOperator with use_server_side_cursor=True and cursor_itersize=10000, the operator issues FETCH FORWARD 1 per row instead of batching when the psycopg3 driver is active (providers-postgres >= 7.0.0).
The _PostgresServerSideCursorDecorator uses fetchone() for the psycopg3 code path, which bypasses the cursor's itersize attribute entirely. With psycopg2 it iterated over the cursor (which respects itersize).
Impact
A 2M row table went from ~5 min to 90+ min (17x slower). The operator becomes unusable for large tables with the psycopg3 driver.
Steps to Reproduce
- Use
apache-airflow-providers-postgres>=7.0.0 (psycopg3 default)
- Configure
PostgresToGCSOperator with use_server_side_cursor=True, cursor_itersize=10000
- Run against a table with >100k rows
- Observe
pg_stat_activity shows FETCH FORWARD 1 instead of FETCH FORWARD 10000
Expected Behavior
Server-side cursor should fetch in batches of cursor_itersize (e.g. FETCH FORWARD 10000).
Environment
- Airflow 3.3.1
- apache-airflow-providers-google 22.1.0
- apache-airflow-providers-postgres 7.0.0+ (bug present)
- Astro Runtime 3.3.5
Workaround
Pin apache-airflow-providers-postgres<7.0.0 to stay on psycopg2.
Description
When using
PostgresToGCSOperatorwithuse_server_side_cursor=Trueandcursor_itersize=10000, the operator issuesFETCH FORWARD 1per row instead of batching when the psycopg3 driver is active (providers-postgres >= 7.0.0).The
_PostgresServerSideCursorDecoratorusesfetchone()for the psycopg3 code path, which bypasses the cursor'sitersizeattribute entirely. With psycopg2 it iterated over the cursor (which respectsitersize).Impact
A 2M row table went from ~5 min to 90+ min (17x slower). The operator becomes unusable for large tables with the psycopg3 driver.
Steps to Reproduce
apache-airflow-providers-postgres>=7.0.0(psycopg3 default)PostgresToGCSOperatorwithuse_server_side_cursor=True, cursor_itersize=10000pg_stat_activityshowsFETCH FORWARD 1instead ofFETCH FORWARD 10000Expected Behavior
Server-side cursor should fetch in batches of
cursor_itersize(e.g.FETCH FORWARD 10000).Environment
Workaround
Pin
apache-airflow-providers-postgres<7.0.0to stay on psycopg2.