perf(sqlalchemy-bigquery): parallelize compliance tests with worker-partitioned datasets and pytest-xdist - #18534
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request re-enables the sqlalchemy-bigquery package in CI, configures Nox to use the virtualenv backend to bypass an upstream packaging issue, and introduces parallel execution for compliance tests using pytest-xdist with dynamic dataset provisioning. Feedback on these changes highlights three key improvements: first, moving the initialization of COMPLIANCE_RUN_PREFIX to pytest_configure to ensure pytest-xdist worker processes correctly inherit the environment variable; second, persisting the generated prefix in os.environ within _dataset_id_from_ident to prevent inconsistent dataset IDs across calls; and third, checking for both -n and --numprocesses in noxfile.py to avoid duplicate arguments.
…artitioned datasets and pytest-xdist
…ist processes Initialize COMPLIANCE_RUN_PREFIX in pytest_configure before workers are spawned and propagate it via workerinput and environment variables. Also cache fallback prefix in provision.py and recognize --numprocesses in noxfile.
… dburi in xdist sessionstart Implement generate_driver_url provision hook for bigquery dialect so generate_db_urls resolves properly. Set dburi on workers and ensure worker datasets exist during worker sessionstart to prevent NoSectionError.
… unit tests for provision.py
f1f1da9 to
f0c1f3b
Compare
| "protobuf": ("https://googleapis.dev/python/protobuf/latest/", None), | ||
| } | ||
|
|
||
| # Check reachability of the Python standard library inventory before attaching it. |
There was a problem hiding this comment.
FYI: This check is in direct response to an issue that came up today: docs.python.org was not available on line for a period of time and was failing tests.
|
|
||
| visit_REAL = visit_FLOAT | ||
| visit_REAL = visit_DOUBLE = visit_DOUBLE_PRECISION = visit_FLOAT | ||
|
|
There was a problem hiding this comment.
This change is due to an upstream issue in sqlalchemy and how they handle DOUBLE and DOUBLE_PRECISION, which BigQuery does not recognize.
…n provision and conftest
| import google.cloud.bigquery | ||
| from sqlalchemy.engine import make_url | ||
| from sqlalchemy.testing.provision import ( | ||
| create_db, |
There was a problem hiding this comment.
sqlalchemy provides decorators to help manage portions of db and url creation.
Most of the functions below are wrapped by the sqlalchemy decorators to align with their expectations and structured internally to handle nuances that are particular to our needs OR the BigQuery API.
… calls in conftest
2 participants
Summary
Parallelizes the
sqlalchemy-bigquerydialect compliance suite usingpytest-xdistto reduce execution time from ~140 minutes down to ~35-45 minutes.Key Changes
sqlalchemy_bigquery.provision):create_db,drop_db,follower_url_from_main) for BigQuery so each xdist worker runs in an isolated BigQuery dataset with a 1-hour expiration safety net.conftest.py):noxfile.py):compliancesession to installpytest-xdistand run tests with-n=4and--dist=loadscope(keeping test classes together on the same worker).--reruns=2 --reruns-delay=30).