Skip to content

feat(HNT-2086): add crawl-common utility helpers - #22

Draft
mmiermans wants to merge 1 commit into
claude/hnt-2086-redis-keysfrom
claude/hnt-2086-common-utils
Draft

feat(HNT-2086): add crawl-common utility helpers#22
mmiermans wants to merge 1 commit into
claude/hnt-2086-redis-keysfrom
claude/hnt-2086-common-utils

Conversation

@mmiermans

@mmiermans mmiermans commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Goal

HNT-2086

Add two small crawl-common helpers that later services reuse: a registrable-domain resolver for the discovery same-domain filter, and temporary per-environment config fallbacks. Both are pure functions wired through the utils barrel.

See docs/ARCHITECTURE.md under review as #14 for how same-domain filtering fits the discovery flow.

Implementation decisions

Decision Approach Why
Resolve the registrable domain with tldts getRegistrableDomain returns getDomain with allowPrivateDomains set true, backed by the Public Suffix List Matches content-monorepo, which resolves the eTLD+1 with getDomain from tldts in servers/curated-corpus-api/src/shared/utils.ts. The Public Suffix List handles multi-part suffixes such as .co.uk that matter for non-US publishers, and allowPrivateDomains treats platform suffixes such as blogspot.com as the boundary so two publishers on one platform stay distinct domains.
Return undefined rather than throw on an unparseable URL or a bare IP The helper coalesces the null result to undefined The discovery same-domain filter is a soft edge check, so a URL it cannot parse should be filtered out, not crash the worker. content-monorepo's equivalent getRegistrableDomainFromUrl throws instead because its callers want the failure surfaced.
Ship deployed-defaults as an explicit throwaway module It maps dev, stage, and prod to a Memorystore IP and a GCP project id, returns empty otherwise, and is wired only as the fallback used when the env var is unset The mozcloud chart does not yet inject the non-secret PROJECT_ID and REDIS_HOST. These move to the hnt-config configMap once the matching webservices-infra change lands, at which point this module and its two config call sites are deleted.

@mmiermans
mmiermans force-pushed the claude/hnt-2086-redis-keys branch from 54a1f4b to 2d2de0a Compare July 6, 2026 17:08
@mmiermans
mmiermans force-pushed the claude/hnt-2086-common-utils branch from baaa374 to 31dc989 Compare July 6, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant