Skip to content

feat(HNT-2119): add the Redis-guarded discovery process step - #30

Draft
mmiermans wants to merge 1 commit into
claude/hnt-2086-process-articlefrom
claude/hnt-2086-process-discovery
Draft

feat(HNT-2119): add the Redis-guarded discovery process step#30
mmiermans wants to merge 1 commit into
claude/hnt-2086-process-articlefrom
claude/hnt-2086-process-discovery

Conversation

@mmiermans

@mmiermans mmiermans commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Goal

HNT-2119

Add processDiscovery, the Redis-guarded step that turns one crawl-article-discovery job into discovery events on the article-discoveries topic and follow-on crawl-article jobs. It skips a page still within its interval_minutes, holds a per-page lock while it lists the page's articles with Zyte, publishes one discovery event per article and context, and enqueues an extraction job for each article not fetched recently.

This builds on the discovery handler, the worker helpers that supply HandlerResult and withinMinutes, and the redis-state and pubsub packages. See docs/ARCHITECTURE.md in #14 for where the discovery fan-out sits in the system.

Implementation decisions

Decision Approach Why
Claim the page:fetch marker before the Zyte call and the publish fan-out setTimestamp writes the marker under the lock, before handleArticleDiscovery and before any event or job is published, rather than last as the tech spec step order suggests A partial failure still nacks and redelivers, but the redelivery now sees the marker and skips instead of re-paying for a Zyte article-list fetch and re-emitting duplicate events and jobs. That interval's missing rows self-heal on the next scheduled crawl, which trades a rare interval of missing discovery rows for bounded Zyte spend, consistent with the spec's at-least-once idempotent design.
Re-check page:fetch inside the lock After acquiring page:lock, withinMinutes is evaluated a second time before crawling Concurrent duplicate jobs all pass the pre-lock freshness check, then serialize on the lock. Without the re-read each would re-crawl the page. The first holder claims the marker so the rest skip, keeping the step idempotent under the spec's duplicate delivery.
Publish discovery events and enqueue jobs together A single Promise.all runs the per-context event publishes and enqueueUnfetchedArticles concurrently A discovery event is published for every article and context regardless of fetch state, since each discovery is a distinct occurrence worth recording, while only the follow-on crawl-article enqueue is gated by article:fetch. The two publish paths are independent, so running them together holds the page lock only as long as the slower of the two, reducing the window another worker can contend for the same page.
Enqueue an article job only when the URL was not fetched recently, with a fresh crawl_id and the default window enqueueUnfetchedArticles checks article:fetch per URL; each discovered job carries no corpus_item, a new randomUUID crawl_id, and refresh_interval_minutes set to the default article fetch TTL Matches the spec's per-article article:fetch skip so a URL already being crawled is not re-queued. The fresh crawl_id traces the extraction back to this discovery, and the default window makes the article worker dedup on the same cadence.
Release the lock best-effort in finally A releaseLock rejection is caught and logged, not propagated The lock self-expires on its TTL, so a release failure must not escape finally and mask the handler's own outcome, which would nack an otherwise successful message and force a needless redelivery.

@mmiermans
mmiermans force-pushed the claude/hnt-2086-process-article branch from fca1ad9 to 90aca3b Compare July 6, 2026 17:08
@mmiermans
mmiermans force-pushed the claude/hnt-2086-process-discovery branch from 40c02b7 to 63b7d5b Compare July 6, 2026 17:08
@mmiermans mmiermans changed the title feat(HNT-2086): add the Redis-guarded discovery process step feat(HNT-2119): add the Redis-guarded discovery process step Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant