fix(utils): treat a missing robots.txt (HTTP 404) as allow-all

[breken-ai]

RobotsTxtFile.load() throws on any non-2xx status before it reaches the 404 branch, so that branch can never run:

if (response.status < 200 || response.status >= 300) {
    throw new Error(`Failed to load robots.txt from ${url}: HTTP ${response.status}`);
}

if (response.status === 404) {
    return new RobotsTxtFile(url, { isAllowed() { return true; }, ... });
}

So for a site with no robots.txt, RobotsTxtFile.find() rejects instead of returning an allow-all file. This order came in with #3306, which replaced the got-scraping HTTPError handling.

With respectRobotsTxtFile: true, getRobotsTxtFileForUrl() catches the rejection and allows the URL, but it caches nothing. Every URL on that host fetches /robots.txt again and logs Failed to fetch robots.txt for request .... I measured this with a BasicCrawler and 5 addRequests() URLs on a host whose robots.txt returns 404: 5 robots.txt fetches and 5 warnings on master, 1 fetch and no warnings with this change. Crawling each request runs checkRobotsTxt and repeats the same thing.

The fix moves the 404 check above the generic non-2xx rejection. Other error statuses still reject as before. The new tests in packages/utils/test/robots.test.ts cover a 404 (allow-all, no sitemaps, no crawl delay) and a 500 (still rejects). The 404 test fails on master with Failed to load robots.txt from http://no-robots.com/robots.txt: HTTP 404.

This PR was prepared with help from an AI coding assistant; I reviewed the change and ran the tests.

[barjin]

lgtm, thank you @breken-ai !