Skip to content

feat(HNT-2116): add the Redis-guarded article process step - #29

Draft
mmiermans wants to merge 1 commit into
claude/hnt-2086-discovery-handlerfrom
claude/hnt-2086-process-article
Draft

feat(HNT-2116): add the Redis-guarded article process step#29
mmiermans wants to merge 1 commit into
claude/hnt-2086-discovery-handlerfrom
claude/hnt-2086-process-article

Conversation

@mmiermans

@mmiermans mmiermans commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Goal

HNT-2116

Add processArticle, the Redis-guarded step that turns a validated crawl-article job into at most one Zyte fetch and one publish to the articles topic. It skips a URL fetched within its refresh window, takes a per-article lock, re-checks freshness under the lock, and publishes only when the extracted content changed since the last fetch.

This builds on the article extraction handler and worker helper layer, composing them with the crawl Redis keys, the redis-state and pubsub clients, worker config, and the domain types, and it adds the pubsub workspace dependency to the worker. docs/ARCHITECTURE.md in #14 draws this flow.

Implementation decisions

Decision Approach Why
Claim the fetch marker before the Zyte call and the publish setTimestamp(fetchKey) runs inside the lock but ahead of extraction and publishing, where the tech spec writes it last A partial failure, whether a failed publish, an ack-deadline expiry, or a mid-handler crash, then redelivers into a skip instead of re-paying for a Zyte fetch and republishing. The rare cost is that one interval's update can be dropped, which self-heals on the next crawl. This trades a very occasional missed update for bounded Zyte spend under at-least-once delivery.
Re-check article:fetch after acquiring the lock The freshness check runs again inside the lock before extracting Concurrent duplicate jobs all pass the pre-lock check and then serialize on the lock. Without the second read each would re-extract, so the first claims the marker and the rest skip.
Exclude url and extracted_at from the content hash contentHash hashes only the meaningful content fields, not url or extracted_at url is constant per key and extracted_at changes on every fetch, so including either would make the hash never match and republish the article every time.
Publish before storing the content hash, and refresh the content key every fetch publishIfChanged publishes when the hash differs, then always rewrites the hash If the store failed after a publish, the message redelivers and republishes rather than dropping the event, and the tech spec's at-least-once design already dedups by latest-per-URL. Rewriting the key each fetch keeps its TTL in step with the fetch marker so an unchanged article does not expire and get republished.
Release the lock best-effort in a finally The release is awaited in finally and its rejection is caught and logged, not rethrown The lock self-expires on its TTL, so a release failure must not propagate out and turn a successful publish into a nack that redelivers the message.
Dedup live articles on their per-message refresh interval The window is message.refresh_interval_minutes ?? config.articleFetchTtlMinutes Live articles carry their own interval so they dedup on the agent's cadence rather than being fetched on every delivery, while discovered articles fall back to the default fetch TTL.

@mmiermans
mmiermans force-pushed the claude/hnt-2086-discovery-handler branch from 4b7a119 to 21f1545 Compare July 6, 2026 17:08
@mmiermans
mmiermans force-pushed the claude/hnt-2086-process-article branch from fca1ad9 to 90aca3b Compare July 6, 2026 17:08
@mmiermans mmiermans changed the title feat(HNT-2086): add the Redis-guarded article process step feat(HNT-2116): add the Redis-guarded article process step Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant