Skip to content

fix(bigquery): keep the Arrow first page and wait without rows - #18650

Open
ohmayr wants to merge 1 commit into
feat/bigquery-arrow-timestamp-precisionfrom
fix/bigquery-arrow-first-page
Open

ohmayr wants to merge 1 commit into
feat/bigquery-arrow-timestamp-precisionfrom
fix/bigquery-arrow-first-page

Conversation

@ohmayr

@ohmayr ohmayr commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Keeps the jobs.query first page of Arrow results and streams only the rest, instead of dropping it and rereading every row.

  • Uses pageRowCount and totalRows to decide whether more rows remain.
  • Waits for the job with maxResults=0, so no unused JSON rows are fetched.
  • QueryJob.result() keeps an Arrow first page, and reading the results twice gives the same rows.
  • Adds unit tests and system tests comparing Arrow with JSON results, including other locations, all data types, extreme values, cached results, failed queries, and the existing query parameter cases.

@ohmayr
ohmayr requested review from a team as code owners October 13, 2026 08:25
@ohmayr
ohmayr requested review from Edvc97 and removed request for a team October 13, 2026 08:25

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enhances the handling of ARROW query results in query_and_wait by requesting zero rows during wait operations (since getQueryResults only returns STRUCT_ENCODING rows) and preserving the first page response for ARROW queries. It also introduces a helper function has_more_rows to check if rows remain by comparing pageRowCount and totalRows. The review feedback recommends making has_more_rows more robust by wrapping the integer conversions in a try-except block to safely handle potential TypeError or ValueError exceptions from malformed API responses.

Comment on lines +132 to +139
def has_more_rows(response: Dict[str, Any]) -> bool:
"""Return True if rows remain after an ``ARROW`` ``jobs.query`` response.

``ARROW`` responses say how many rows the first page holds in
``pageRowCount``, so compare that with ``totalRows`` instead of checking
for a ``pageToken``.
"""
return int(response.get("pageRowCount", 0)) < int(response.get("totalRows", 0))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current implementation of has_more_rows directly calls int() on the result of response.get(). If pageRowCount or totalRows is present in the response but has a None value (or is an empty/invalid string), this will raise a TypeError or ValueError. Using a safer parsing approach with a try-except block ensures the function is robust against unexpected or malformed API responses.

Suggested change
def has_more_rows(response: Dict[str, Any]) -> bool:
"""Return True if rows remain after an ``ARROW`` ``jobs.query`` response.
``ARROW`` responses say how many rows the first page holds in
``pageRowCount``, so compare that with ``totalRows`` instead of checking
for a ``pageToken``.
"""
return int(response.get("pageRowCount", 0)) < int(response.get("totalRows", 0))
def has_more_rows(response: Dict[str, Any]) -> bool:
"""Return True if rows remain after an ``ARROW`` ``jobs.query`` response.
``ARROW`` responses say how many rows the first page holds in
``pageRowCount``, so compare that with ``totalRows`` instead of checking
for a ``pageToken``.
"""
try:
page_row_count = int(response.get("pageRowCount") or 0)
total_rows = int(response.get("totalRows") or 0)
except (ValueError, TypeError):
return False
return page_row_count < total_rows

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pageRowCount and totalRows come from the jobs.query response, where they're always integer strings when present, and the existing code in this module already reads totalRows with int(). I'd rather not add handling for responses the API doesn't send.

@ohmayr
ohmayr added this pull request to stack #18648 October 13, 2026 08:32
@ohmayr
ohmayr force-pushed the fix/bigquery-arrow-first-page branch from 6ba439e to ab029f6 Compare October 13, 2026 10:02
@ohmayr
ohmayr requested a review from tswast October 13, 2026 20:41
@ohmayr
ohmayr force-pushed the fix/bigquery-arrow-first-page branch from ab029f6 to c8fab58 Compare October 15, 2026 02:29
When an ARROW query's results didn't fit in the jobs.query response,
QueryJob.result() dropped that response, because it only kept first
pages that had STRUCT_ENCODING rows. The stream then reread every row
from offset 0. While waiting for the job, query_and_wait() also passed
page_size to jobs.getQueryResults, which returned STRUCT_ENCODING rows
that ARROW results never use.

For ARROW queries, decide whether rows remain by comparing
pageRowCount with totalRows instead of checking for a pageToken, wait
with page_size=0 so jobs.getQueryResults is called with maxResults=0,
and keep a jobs.query response that has arrowRecordBatch as the first
page. RowIterator then returns the first page and streams the rest
from the next row, also when it's read again. Queries that don't use
ARROW are unchanged.
@ohmayr
ohmayr force-pushed the fix/bigquery-arrow-first-page branch from c8fab58 to 7208b63 Compare October 15, 2026 03:22

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant