feat(HNT-2086): add crawl-common utility helpers - #22
Draft
mmiermans wants to merge 1 commit into
Draft
Conversation
mmiermans
force-pushed
the
claude/hnt-2086-redis-keys
branch
from
July 6, 2026 17:08
54a1f4b to
2d2de0a
Compare
mmiermans
force-pushed
the
claude/hnt-2086-common-utils
branch
from
July 6, 2026 17:08
baaa374 to
31dc989
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
HNT-2086
Add two small
crawl-commonhelpers that later services reuse: a registrable-domain resolver for the discovery same-domain filter, and temporary per-environment config fallbacks. Both are pure functions wired through theutilsbarrel.See docs/ARCHITECTURE.md under review as #14 for how same-domain filtering fits the discovery flow.
Implementation decisions
tldtsgetRegistrableDomainreturnsgetDomainwithallowPrivateDomainsset true, backed by the Public Suffix ListgetDomainfromtldtsinservers/curated-corpus-api/src/shared/utils.ts. The Public Suffix List handles multi-part suffixes such as.co.ukthat matter for non-US publishers, andallowPrivateDomainstreats platform suffixes such asblogspot.comas the boundary so two publishers on one platform stay distinct domains.undefinedrather than throw on an unparseable URL or a bare IPnullresult toundefinedgetRegistrableDomainFromUrlthrows instead because its callers want the failure surfaced.deployed-defaultsas an explicit throwaway moduledev,stage, andprodto a Memorystore IP and a GCP project id, returns empty otherwise, and is wired only as the fallback used when the env var is unsetPROJECT_IDandREDIS_HOST. These move to thehnt-configconfigMap once the matching webservices-infra change lands, at which point this module and its two config call sites are deleted.