Skip to content

feat(HNT-2117): add the agent tick enqueue loop - #37

Draft
mmiermans wants to merge 1 commit into
claude/hnt-2086-publisher-listfrom
claude/hnt-2086-agent-tick
Draft

feat(HNT-2117): add the agent tick enqueue loop#37
mmiermans wants to merge 1 commit into
claude/hnt-2086-publisher-listfrom
claude/hnt-2086-agent-tick

Conversation

@mmiermans

@mmiermans mmiermans commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Goal

HNT-2117

Add the agent's per-tick enqueue loop. runTick publishes a discovery job for each page and a crawl-article job for each live article whose crawl interval has elapsed since it was last enqueued, and returns the counts for logging.

This builds on the redis-state slice for the enqueue markers, the pubsub package for publishing, the crawl Redis key builders, and the crawl domain message types. See docs/ARCHITECTURE.md in #14 for where the tick sits between the publisher list and the crawl queues.

Implementation decisions

Decision Approach Why
Dedup enqueues with agent-owned markers rather than the worker fetch keys runTick reads and writes its own page:enqueued and article:enqueued markers, not the worker-written page:fetch and article:fetch keys. There is a window between the agent enqueuing an item and the worker recording its fetch, so reusing the worker fetch marker for dedup would let the agent re-enqueue the same item on every tick during that window. An agent-owned enqueue marker closes that gap.
A plain check-then-set on the enqueue marker, with no distributed lock Each item reads its Redis enqueue timestamp and, when due, publishes and then writes a fresh timestamp, with no lock or atomic guard. The agent runs as a single replica, so no two ticks race the same marker. Within a tick the items are enqueued concurrently, but validatePublisherList rejects duplicate URLs, so no two list entries contend on the same key either. A distributed lock would add cost with no benefit.
Publish before marking enqueued Publish the job first, and only on success write the enqueue timestamp. A failed publish leaves no marker, so the item is retried on the next tick rather than being suppressed until its interval elapses, and the error propagates so a broken publisher fails the tick visibly.
Compare the stored timestamp for dueness, not a marker TTL enqueuedWithin compares elapsed time against interval_minutes using the value stored in Redis, and the marker's own TTL is unrelated to the interval. Encoding the window in a TTL would tie each item's cadence to the tick interval and misfire for short intervals. Comparing the stored timestamp keeps the cadence exact and independent of how often the loop runs.
Shape the live-article job with source_url equal to the article URL and a matching refresh window The job carries the unchanged corpus_item, a fresh crawl_id per enqueue, source_url set to the article URL, and refresh_interval_minutes set to the agent's live-article interval. A live article has no discovery page, so source_url is the article itself. The worker dedups against refresh_interval_minutes, so matching it to the agent's live-article interval keeps the worker's refresh window aligned with the agent's cadence rather than the worker default.

@mmiermans
mmiermans force-pushed the claude/hnt-2086-publisher-list branch from 9c08d05 to 57f2332 Compare July 6, 2026 17:09
@mmiermans
mmiermans force-pushed the claude/hnt-2086-agent-tick branch from 422b88b to b81ce44 Compare July 6, 2026 17:09
@mmiermans mmiermans changed the title feat(HNT-2086): add the agent tick enqueue loop feat(HNT-2117): add the agent tick enqueue loop Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant