Skip to content

feat(HNT-2897): add crawl-agent runtime configuration - #35

Draft
mmiermans wants to merge 1 commit into
claude/hnt-2086-worker-mainfrom
claude/hnt-2086-agent-config
Draft

feat(HNT-2897): add crawl-agent runtime configuration#35
mmiermans wants to merge 1 commit into
claude/hnt-2086-worker-mainfrom
claude/hnt-2086-agent-config

Conversation

@mmiermans

@mmiermans mmiermans commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Goal

HNT-2897

Read and validate crawl-agent runtime configuration into one typed object, and keep the health check green while scheduled crawling is off in the dev sandbox. This builds on the crawl-common utilities slice by reading the temporary deployedProjectId and deployedRedisHost fallbacks it added. See docs/ARCHITECTURE.md in #14 for where the agent sits in the system.

Implementation decisions

Decision Approach Why
Disable scheduled crawling on dev and short-circuit the health check to 200 when it is off crawlEnabled is environment !== 'dev', and /healthz returns 200 immediately when crawling is disabled The dev environment is a sandbox where a locally run service points at real GCP resources, so it must not crawl on a schedule. With no tick to go stale, the staleness liveness check would otherwise fail the pod, so it reports healthy while the process is up. A developer running locally sets ENVIRONMENT=local to enable crawling.
Resolve publishers.json relative to the module rather than the working directory The path comes from fileURLToPath(new URL('../publishers.json', import.meta.url)) The container runs from /app while the file ships next to the package, so a module-relative path loads regardless of the working directory. One list serves every environment, so the path is fixed rather than env-configurable.
Treat a blank numeric env var as unset and fail fast on a non-numeric one numberEnv trims the value, uses the fallback when it is empty, and throws when it is not finite A templated-but-empty Helm var would otherwise coerce to 0 and silently mis-tune an interval, while failing fast on a genuine typo surfaces the config error at load rather than at runtime.
Derive the Pub/Sub topic names from ENVIRONMENT, with explicit env vars as overrides crawlArticleDiscoveryTopic and crawlArticleTopic default to ${environment}-..., matching how the crawl-worker config names its topics and how Terraform names the resources Keeping the naming in one place needs no chart change since ENVIRONMENT is already injected, while the override lets the emulator point at local topic names.
Gate the Corpus live-article source on the JWK, and validate surfaces and refresh interval only when it is set When CORPUS_API_JWK_JSON is set the config requires at least one surface and a positive refresh interval The endpoint, issuer, and audience have app defaults, so the JWK secret is the only required input. Validating the surfaces and interval only when the source is on fails fast on a typo rather than silently crawling zero live articles, and leaving the JWK unset sources live articles from the committed file instead.
Fall back to per-environment defaults for PROJECT_ID and REDIS_HOST Both read process.env first, then deployedProjectId(environment) and deployedRedisHost(environment) from crawl-common This is a temporary shim (HNT-2086) until the chart injects both values. The env var always wins when set, and the fallback is removed once the chart is updated. It reuses the deployed-defaults module added in the crawl-common utilities slice.

@mmiermans
mmiermans force-pushed the claude/hnt-2086-worker-main branch from b5d0ef3 to f37548c Compare July 6, 2026 17:09
@mmiermans
mmiermans force-pushed the claude/hnt-2086-agent-config branch from 17d7d43 to 6bad473 Compare July 6, 2026 17:09
@mmiermans mmiermans changed the title feat(HNT-2086): add crawl-agent runtime configuration feat(HNT-2897): add crawl-agent runtime configuration Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant