🦺 retry transient Google Sheets API errors

[adaptcom]

Requested by: Guilherme Amorim guilherme@wander.com
Created by: Adapt

---

Summary

main_every_3h.collect_manual_fixes.collect_manual_references failed on both tries at 2026-08-20 12:00 UTC with:

APIError: APIError: [503]: The service is currently unavailable.
  gcpde/sheets.py:56 in _open_document -> gc.open_by_key(document_id)
  gspread/http_client.py:128 -> raise APIError(response)

gcpde/sheets.py had no retry anywhere, so one transient 5xx on the spreadsheet-metadata call takes down the whole task. The DAGs run with retries: 1, so both tries burn inside the same outage window and the tables stay stale until the next 3-hourly run. The same 503 killed collect_scraping.collect_price_match_sla_remove_list on both the 12:00 and 15:00 UTC runs the day before, so this is not specific to one pipeline: every ingest_sheet / ingest_all_tabs pipeline in wandercom/data-pipelines sits on this code path.

What changed

• _call_api(operation) — a tenacity-wrapped leaf-call helper, mirroring the existing patterns in gcpde/bq.py and gcpde/gcs.py. Exponential backoff (1s to 30s), 5 attempts, reraise=True, warning log between attempts.
• _is_transient_api_error retries only gspread.exceptions.APIError with status 408, 429, 500, 502, 503 or 504. Client errors (401/403/404, a revoked share or a deleted doc) still fail on the first attempt, so a real permission problem does not sit there backing off for 45 seconds.
• Wrapped the read-path leaf calls: open_by_key, Spreadsheet.worksheet, Spreadsheet.worksheets (both in _get_worksheets and list_worksheets), and Worksheet.get_all_records.

Two scope choices worth calling out in review:

1. Only leaf calls are wrapped, never a function that already calls a wrapped one. Nesting two retried calls multiplies attempts (5 x 5), which is why _get_worksheets and list_worksheets now hold the spreadsheet in a local before the wrapped worksheets() call instead of chaining.
2. Write paths (replace_from_records, replace_or_create_from_records, delete_worksheet) are deliberately untouched. Retrying a clear() + update() pair mid-flight has different failure semantics than a read, and the incident is entirely on the read path. Happy to widen it if you would rather have it uniform.

Test plan

• make checks — ruff + mypy gcpde (strict) clean.
• make test — 77 passed. 8 new tests in tests/unit/test_sheets.py: the transient/non-transient predicate matrix, retry-then-succeed, reraise after 5 attempts, no retry on 404, and _open_document recovering from a 503. Retry waits are neutralized with retry_with(wait=tenacity.wait_none()) so the suite stays fast.
• Coverage on gcpde/sheets.py is 99%; the single uncovered line (worksheet.resize in replace_or_create_from_records) predates this branch.

Notes

• No Linear ticket exists for this. DAT-2613 "Manual references not pulling in" is Canceled and unrelated.
• Consumers pick this up on the next gcpde release + bump in wandercom/data-pipelines.
• Triage thread: https://wander.slack.com/archives/C07LEEWGLRW/p1787228566431219

---

Note

Medium Risk
Changes how Sheets API failures are handled (retries, backoff, which status codes fail fast). Write paths are unchanged, so risk is mainly extra latency and possible nested-retry mistakes on the read path.

Overview
Retries transient Google Sheets API failures on the read path so a single 503/429 no longer takes down ingest jobs.

Adds _call_api (tenacity, exponential backoff 1–30s, 5 attempts) and _is_transient_api_error, which retries only APIError with 408/429/5xx. Client errors (401/403/404) still fail immediately.

Wraps leaf calls only: open_by_key, worksheet, worksheets, and get_all_records. Nested wrappers are avoided so attempts do not multiply. Write helpers (replace_*, delete_worksheet) are left without extra retries. Unit tests cover the status-code matrix, retry-then-succeed, max attempts, and no retry on 404.

^{Reviewed by Cursor Bugbot for commit 06b856c. Bugbot is set up for automated code reviews on this repo. Configure here.}