Skip to content

feat(HNT-2086): run the agent tick loop with Corpus refresh and metrics - #39

Draft
mmiermans wants to merge 2 commits into
claude/hnt-2086-publisher-datafrom
claude/hnt-2086-agent-main
Draft

feat(HNT-2086): run the agent tick loop with Corpus refresh and metrics#39
mmiermans wants to merge 2 commits into
claude/hnt-2086-publisher-datafrom
claude/hnt-2086-agent-main

Conversation

@mmiermans

@mmiermans mmiermans commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Goal

HNT-2086

Run the crawl-agent entry point that wires the Pub/Sub, Redis, and metrics clients and drives the tick loop, turning the agent from a health-only scaffold into the live scheduler that originates every crawl job. The main module loads the publisher pages once at startup, refreshes live articles from the Corpus API on the corpusRefreshMinutes interval when that source is configured, and on each tick delegates to runTick and emits enqueue counts and tick duration to metrics.

This builds on earlier slices by combining their publisher-list loader, tick enqueue loop, agent config, and Corpus read with the metrics, pubsub, and redis-state clients. See docs/ARCHITECTURE.md in #14 for where the agent sits in the system.

Implementation decisions

Decision Approach Why
Fail fast on the first Corpus load but tolerate a later refresh failure start() awaits the first refreshLiveArticles before entering the loop, so a misconfigured or unauthorized Corpus client aborts startup, while a refresh inside the loop is wrapped so a failure keeps the last good list and retries next tick A configuration error should stop the agent immediately rather than run degraded, whereas a transient Corpus outage should reduce content freshness rather than availability, matching the tech spec's failure-mode principle that the crawler causes no New Tab outages and only degrades freshness. See the Article Crawler tech spec.
Refresh live articles on their own interval rather than on every tick The loop re-reads the Corpus list only after corpusRefreshMinutes has elapsed since the last read, defaulting to fifteen minutes, while the tick loop itself runs about once a minute Editors change section items far less often than once a minute, so a dedicated slower interval propagates editorial changes without a restart while keeping a Corpus API blip from coupling to every tick, consistent with degrading content freshness rather than availability.
Mark loop liveness even after a failed tick The loop calls setLastTickAt after every iteration whether the tick threw or not /healthz fails when the last tick is older than the stale threshold, so it should catch a wedged event loop, which a restart fixes, and not a transient Pub/Sub or Redis outage that a restart would not fix and that Sentry already surfaces. The tech spec defines the liveness probe as failing only when a tick has not run for more than ten minutes.
Drain Pub/Sub and Redis before flushing metrics and Sentry on shutdown shutdown stops the loop, drains Pub/Sub and closes Redis while logging and continuing on failure, closes the server, then flushes metrics and Sentry, with a forced exit after the timeout Draining first flushes pending publishes so no enqueue is lost, and flushing observability last means an error raised while draining still reaches Sentry, all within the Kubernetes termination grace period.

@mmiermans
mmiermans force-pushed the claude/hnt-2086-publisher-data branch from 6e5b486 to 972641e Compare July 6, 2026 17:09
@mmiermans
mmiermans force-pushed the claude/hnt-2086-agent-main branch from 038ecc5 to 5074306 Compare July 6, 2026 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant