File / lines: src/gmaps_scraper/place_scraper.py, lines 1715-1720, 2736-2778
Problem: _collect_preview_place_enrichment is invoked unconditionally for every place after the browser DOM snapshot is collected. If the DOM already contains the critical fields (name, address, phone, website, coordinates), this separate HTTP fetch is wasted work and latency.
Why it matters: In batch place scraping, every unnecessary HTTP round-trip adds up. The preview fetch duplicates work the browser already did and increases fingerprinting/rate-limit risk.
Suggested fix:
- Gate the preview fetch on missing high-value fields. Skip it when
dom_snapshot already has non-empty address, phone, website, lat, and lng.
- Make the gating configurable via a parameter (e.g.,
skip_preview_if_dom_complete) so callers can opt into the old behavior.
- Add tests showing the preview fetch is skipped when DOM is complete and still runs when fields are missing.
Difficulty: medium
Impact: High — reduces duplicate HTTP traffic and speeds batch refreshes.
File / lines:
src/gmaps_scraper/place_scraper.py, lines 1715-1720, 2736-2778Problem:
_collect_preview_place_enrichmentis invoked unconditionally for every place after the browser DOM snapshot is collected. If the DOM already contains the critical fields (name, address, phone, website, coordinates), this separate HTTP fetch is wasted work and latency.Why it matters: In batch place scraping, every unnecessary HTTP round-trip adds up. The preview fetch duplicates work the browser already did and increases fingerprinting/rate-limit risk.
Suggested fix:
dom_snapshotalready has non-emptyaddress,phone,website,lat, andlng.skip_preview_if_dom_complete) so callers can opt into the old behavior.Difficulty: medium
Impact: High — reduces duplicate HTTP traffic and speeds batch refreshes.