Skip to content

feat(HNT-2086): wire the discovery worker to its subscription - #32

Draft
mmiermans wants to merge 1 commit into
claude/hnt-2086-article-consumerfrom
claude/hnt-2086-discovery-consumer
Draft

feat(HNT-2086): wire the discovery worker to its subscription#32
mmiermans wants to merge 1 commit into
claude/hnt-2086-article-consumerfrom
claude/hnt-2086-discovery-consumer

Conversation

@mmiermans

@mmiermans mmiermans commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Goal

HNT-2086

Wire the discovery worker to the crawl-article-discovery subscription so page jobs are consumed, validated at the boundary, and handed to processDiscovery with Sentry context and per-message metrics, mirroring the article consumer. This is a thin wiring slice, and most of its added lines are an emulator integration test that drives a page job through the handler, Pub/Sub, and Redis with Zyte stubbed.

It builds on the discovery slice by consuming its processDiscovery step, and reuses the pubsub package, the message validators, and the worker helpers and config. See docs/ARCHITECTURE.md in #14 for the discovery fan-out.

Implementation decisions

Decision Approach Why
Reuse the article consumer's validate, Sentry, and metrics shape The same startSubscriber wiring, with validateCrawlArticleDiscoveryMessage at the boundary and withSentryHandler over withMessageMetrics The two consumers differ only in their subscription, validator, and handler, so keeping one shape means a reviewer reasons about the pattern once and boundary validation and error reporting behave identically across both worker roles.
Report the distinct surfaces and topics from the job's contexts to Sentry The metadata extractor dedups message.contexts into surface_ids and topics arrays with a context_count, alongside the page url and interval A discovery job carries one context per audience, unlike the single-context article job, so a failure needs the set of affected surfaces and topics to triage; deduping keeps the context readable when many contexts share a surface or topic. The HNT-2589 Sentry criteria call for surface_id and topic on the context when present on the job.

@mmiermans
mmiermans force-pushed the claude/hnt-2086-article-consumer branch from e0754cb to c66cc03 Compare July 6, 2026 17:08
@mmiermans
mmiermans force-pushed the claude/hnt-2086-discovery-consumer branch from 608d3d1 to a76ed67 Compare July 6, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant