feat(HNT-2086): run the agent tick loop with Corpus refresh and metrics - #39
Draft
mmiermans wants to merge 2 commits into
Draft
feat(HNT-2086): run the agent tick loop with Corpus refresh and metrics#39mmiermans wants to merge 2 commits into
mmiermans wants to merge 2 commits into
Conversation
mmiermans
force-pushed
the
claude/hnt-2086-publisher-data
branch
from
July 6, 2026 17:09
6e5b486 to
972641e
Compare
mmiermans
force-pushed
the
claude/hnt-2086-agent-main
branch
from
July 6, 2026 17:09
038ecc5 to
5074306
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
HNT-2086
Run the crawl-agent entry point that wires the Pub/Sub, Redis, and metrics clients and drives the tick loop, turning the agent from a health-only scaffold into the live scheduler that originates every crawl job. The
mainmodule loads the publisher pages once at startup, refreshes live articles from the Corpus API on thecorpusRefreshMinutesinterval when that source is configured, and on each tick delegates torunTickand emits enqueue counts and tick duration to metrics.This builds on earlier slices by combining their publisher-list loader, tick enqueue loop, agent config, and Corpus read with the metrics, pubsub, and redis-state clients. See docs/ARCHITECTURE.md in #14 for where the agent sits in the system.
Implementation decisions
start()awaits the firstrefreshLiveArticlesbefore entering the loop, so a misconfigured or unauthorized Corpus client aborts startup, while a refresh inside the loop is wrapped so a failure keeps the last good list and retries next tickcorpusRefreshMinuteshas elapsed since the last read, defaulting to fifteen minutes, while the tick loop itself runs about once a minutesetLastTickAtafter every iteration whether the tick threw or not/healthzfails when the last tick is older than the stale threshold, so it should catch a wedged event loop, which a restart fixes, and not a transient Pub/Sub or Redis outage that a restart would not fix and that Sentry already surfaces. The tech spec defines the liveness probe as failing only when a tick has not run for more than ten minutes.shutdownstops the loop, drains Pub/Sub and closes Redis while logging and continuing on failure, closes the server, then flushes metrics and Sentry, with a forced exit after the timeout