feat(HNT-2117): add the agent tick enqueue loop - #37
Draft
mmiermans wants to merge 1 commit into
Draft
Conversation
mmiermans
force-pushed
the
claude/hnt-2086-publisher-list
branch
from
July 6, 2026 17:09
9c08d05 to
57f2332
Compare
mmiermans
force-pushed
the
claude/hnt-2086-agent-tick
branch
from
July 6, 2026 17:09
422b88b to
b81ce44
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
HNT-2117
Add the agent's per-tick enqueue loop.
runTickpublishes a discovery job for each page and acrawl-articlejob for each live article whose crawl interval has elapsed since it was last enqueued, and returns the counts for logging.This builds on the redis-state slice for the enqueue markers, the pubsub package for publishing, the crawl Redis key builders, and the crawl domain message types. See docs/ARCHITECTURE.md in #14 for where the tick sits between the publisher list and the crawl queues.
Implementation decisions
runTickreads and writes its ownpage:enqueuedandarticle:enqueuedmarkers, not the worker-writtenpage:fetchandarticle:fetchkeys.validatePublisherListrejects duplicate URLs, so no two list entries contend on the same key either. A distributed lock would add cost with no benefit.enqueuedWithincompares elapsed time againstinterval_minutesusing the value stored in Redis, and the marker's own TTL is unrelated to the interval.source_urlequal to the article URL and a matching refresh windowcorpus_item, a freshcrawl_idper enqueue,source_urlset to the article URL, andrefresh_interval_minutesset to the agent's live-article interval.source_urlis the article itself. The worker dedups againstrefresh_interval_minutes, so matching it to the agent's live-article interval keeps the worker's refresh window aligned with the agent's cadence rather than the worker default.