Skip to content

feat(HNT-2113): wire the article worker to its subscription - #31

Draft
mmiermans wants to merge 1 commit into
claude/hnt-2086-process-discoveryfrom
claude/hnt-2086-article-consumer
Draft

feat(HNT-2113): wire the article worker to its subscription#31
mmiermans wants to merge 1 commit into
claude/hnt-2086-process-discoveryfrom
claude/hnt-2086-article-consumer

Conversation

@mmiermans

@mmiermans mmiermans commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Goal

HNT-2113

Wire the article worker to its Pub/Sub subscription so crawl-article jobs are consumed, validated at the boundary, and handed to processArticle with Sentry context and per-message metrics attached. This is a thin wiring slice. Most of its added lines are an emulator integration test that drives the full path against Pub/Sub and Redis with Zyte stubbed.

It builds on the Redis-guarded article step, the pubsub package, the crawl-message validators, and the worker message helpers and config, with Sentry from HNT-2589 as baseline. See docs/ARCHITECTURE.md in #14 for where this consumer fits the article flow.

Implementation decisions

Decision Approach Why
Validate the payload at the subscription boundary Pass validateCrawlArticleMessage as the subscriber's validate hook The pubsub client nacks a failed validate and routes it to the error handler under a distinct validation-error kind, so a malformed poison payload is told apart from a transient handler failure in the logs and Sentry and never reaches processArticle. This is the boundary validation HNT-2487 calls for.
Split the Sentry metadata into tags and context worker_role, has_corpus_item, and corpus_topic are tags; url, crawl_id, source_url, and enqueued_at are context The small enum-like fields make useful searchable tags for filtering failures, while the per-message identifiers ride along as context on the issue. has_corpus_item lets triage separate live-article from discovered-article failures at a glance, and corpus_topic is named so it is not mistaken for the Pub/Sub topic.

@mmiermans
mmiermans force-pushed the claude/hnt-2086-process-discovery branch from 40c02b7 to 63b7d5b Compare July 6, 2026 17:08
@mmiermans
mmiermans force-pushed the claude/hnt-2086-article-consumer branch from e0754cb to c66cc03 Compare July 6, 2026 17:08
@mmiermans mmiermans changed the title feat(HNT-2086): wire the article worker to its subscription feat(HNT-2113): wire the article worker to its subscription Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant