Skip to content

Bug: Entitylist preload expansion ignores continuation tokens and requests whole list at once #56

Description

@MozzamShahid

File / lines: src/gmaps_scraper/scraper.py, lines 601-632, 635-660

Problem: _extract_entitylist_payload returns only payload[0] and throws away payload[1], which is the continuation token. _expand_entitylist_preload_text then rewrites the page-size parameter to total_rows and makes a single enormous request. If Google rejects that size, the except Exception: return None silently keeps the truncated first payload.

Evidence:

candidate = payload[0]          # payload[1] (continuation token) is ignored

Suggested fix: Parse and use the continuation token. Implement a paginated fetch loop:

  1. Fetch the preload.
  2. If len(rows) < total, use the token from payload[1] to request the next page.
  3. Accumulate rows (or the full payloads) into script_texts.
  4. Cap the single-request page size and log a quality flag when pagination is used.

Difficulty: high-value
Impact: High — saved lists with many places may be truncated or silently incomplete.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions