Skip to content

feat(HNT-2086): add the article discovery handler - #28

Draft
mmiermans wants to merge 1 commit into
claude/hnt-2086-article-handlerfrom
claude/hnt-2086-discovery-handler
Draft

feat(HNT-2086): add the article discovery handler#28
mmiermans wants to merge 1 commit into
claude/hnt-2086-article-handlerfrom
claude/hnt-2086-discovery-handler

Conversation

@mmiermans

@mmiermans mmiermans commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Goal

HNT-2111

Add handleArticleDiscovery, the pure function that extracts a publisher page's article list through Zyte, drops cross-domain and duplicate links, and fans each surviving article into one article-discoveries event per context. It returns those events plus the unique article URLs the worker will enqueue for extraction.

This builds on the article-handler slice by reusing its BigQuery-safe field mappers, together with the shared zyte and metrics clients, the per-domain extraction-mode module, the crawl-common getRegistrableDomain util, and the crawl domain types. docs/ARCHITECTURE.md under review as #14 shows the discovery flow.

Implementation decisions

Decision Approach Why
Keep only same-registrable-domain articles selectArticles resolves the page's registrable domain once with getRegistrableDomain and drops any list item on a different one The legacy crawler filters out cross-domain links so off-publisher URLs are never crawled. getRegistrableDomain uses tldts, matching content-monorepo's use of the same library in servers/curated-corpus-api for registrable-domain logic.
Use the enqueued page URL as the same-domain baseline and each event's source_url selectArticles resolves the page domain from message.url and mapToDiscoveryEvent sets source_url to message.url, not Zyte's post-redirect URL The publisher pages come from a curated first-party list that does not cross-domain redirect, and using the enqueued URL keeps each discovery event's source_url aligned with the crawl-article job that triggered it. A page that did cross-domain redirect would yield zero discoveries, a safe failure surfaced by monitoring rather than bad data.
Deduplicate by URL, keeping the first occurrence and its page position A Set of seen URLs keeps the first item per URL and records its 1-based list position The tech spec dedups per URL keeping the first occurrence. Preserving each article's original 1-based index feeds page_position, an ML feature, so gaps left by cross-domain and duplicate filtering are intentional. One crawl-article job is enqueued per unique URL and one discovery event is emitted per article and context.
Take the summary from the list-page description, not the article body mapToDiscoveryEvent sets summary from the list item's description The legacy crawler stored the Zyte articleList item's articleBody in its discovery summary, and that body is empty for about 80 percent of the 8.3M legacy source = 'PAGE' rows in rss_feed_items and reaches 500 characters or more for well under 1 percent, so in list extraction it is rarely a full body and makes a weak summary source. Zyte's articleList items also expose a per-item description, the list-page dek written to summarize each link, so summary reads from description, and matching the legacy body field would only copy a mostly empty column for comparability.
Reuse the article handler's BigQuery-safe mappers toEventAuthors and toEventTimestamp from #27 map the discovery event's authors and published_at Discovery rows write to the same tables through the same dead-letter-less BigQuery subscription with the same REQUIRED authors.name subfield, so the identical guards apply and are worth sharing rather than reimplementing.

@mmiermans
mmiermans force-pushed the claude/hnt-2086-article-handler branch from acf5713 to 83f5ad8 Compare July 6, 2026 17:08
@mmiermans
mmiermans force-pushed the claude/hnt-2086-discovery-handler branch from 4b7a119 to 21f1545 Compare July 6, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant