feat(HNT-2891): select the Zyte extraction mode per domain

[mmiermans]

Goal

HNT-2891

Add resolveExtractFrom, which picks the Zyte extraction source for a URL and product so the crawler pays for cheap plain-HTTP extraction where the legacy quality gates already proved it works and uses full browser rendering everywhere else. This builds on the crawl-common utils slice by using its getRegistrableDomain function to collapse subdomains onto one lookup key. docs/ARCHITECTURE.md in #14 shows where the extraction step sits in the crawl flow.

Implementation decisions

Decision	Approach	Why
Choose httpResponseBody for an allowlist of domains and browserHtml for everything else	resolveExtractFrom looks the registrable domain up in a per-product cheap-domain set and returns httpResponseBody on a hit, otherwise browserHtml	httpResponseBody is a plain HTTP fetch with no headless browser, so it is materially cheaper per Zyte call, but it fails on a JavaScript-heavy publisher. The allowlist is the set of domains the legacy crawler quality gates already proved extract correctly over plain HTTP, and browserHtml renders JavaScript so any other publisher still yields a full article. content-monorepo's curated-corpus-api uses extractFrom: 'browserHtml' as its Zyte default, which matches our fallback.
Default to browserHtml and accept the higher Zyte cost	The resolver returns browserHtml for every domain not on the cheap allowlist, so most crawled domains, mainly international publishers the legacy gates never evaluated, take the more expensive browser-rendering path	This is a deliberate fidelity-over-cost choice for HNT-2086: browser rendering yields a full article from a JavaScript-heavy publisher where plain HTTP would return a thin or empty body, so the crawler favors extraction quality over Zyte spend. The added cost is per request at the same request rate, which the per-role Zyte limiter still bounds, and a domain moves onto the cheap list as it clears quality, so the allowlist grows and the cost falls over time.
Track clearance per product, not per domain	Separate article and articleList sets, so a domain cleared for article extraction is not automatically cleared for article-list extraction	The two Zyte products render differently, so a domain that yields a clean article over plain HTTP may still need a browser for its list page. Keeping the sets separate avoids a bad list extraction on a domain cleared only for articles.
Match on the registrable domain	The lookup key is getRegistrableDomain(url) from crawl-common, so www. and other subdomains collapse to one entry	A publisher is cleared once and all of its subdomains inherit the decision, which keeps the list short and avoids missing a subdomain variant.
Fall back to browserHtml for an unknown or unparseable URL	A URL that does not parse, or a domain not on the list, returns browserHtml	The safe default is full rendering, so a new or malformed publisher URL still extracts rather than silently returning a thin plain-HTTP result.
Ship the domain list as data	zyte-cheap-domains.ts holds the two lists as plain arrays ported from the legacy content-ml-services quality gates (zyte_domains.py)	This is the bulk of the diff yet carries almost no review load, and adding or removing a cleared publisher later is a data edit rather than a code change. One recorded case is theguardian.com, which sits in both cheap lists and so takes the plain httpResponseBody path for both the article and article-list products. This keeps its crawl behavior the same as the existing content-ml-services solution, which lists it in both of its httpResponseBody domain sets, and that parity is the reason for the choice. The Guardian bot protection also returns HTTP 451 to Zyte's browserHtml browser fetch, so the cheap path is the one that reliably extracts.