Skip to content

fix(utils): treat a missing robots.txt (HTTP 404) as allow-all - #4171

Merged
barjin merged 3 commits into
apify:masterfrom
breken-ai:fix/robots-txt-404-allow-all
Oct 2, 2026
Merged

barjin merged 3 commits into
apify:masterfrom
breken-ai:fix/robots-txt-404-allow-all

docs: mention robots.txt error status handling in the v4 upgrading guide

58aa351
Select commit
Loading
Failed to load commit list.
Sign in for the full log view

Annotations

2 warnings
Check PR title
succeeded Oct 2, 2026 in 12s