feat(HNT-2897): add crawl-agent runtime configuration - #35
Draft
mmiermans wants to merge 1 commit into
Draft
Conversation
mmiermans
force-pushed
the
claude/hnt-2086-worker-main
branch
from
July 6, 2026 17:09
b5d0ef3 to
f37548c
Compare
mmiermans
force-pushed
the
claude/hnt-2086-agent-config
branch
from
July 6, 2026 17:09
17d7d43 to
6bad473
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
HNT-2897
Read and validate crawl-agent runtime configuration into one typed object, and keep the health check green while scheduled crawling is off in the dev sandbox. This builds on the crawl-common utilities slice by reading the temporary
deployedProjectIdanddeployedRedisHostfallbacks it added. See docs/ARCHITECTURE.md in #14 for where the agent sits in the system.Implementation decisions
crawlEnabledisenvironment !== 'dev', and/healthzreturns 200 immediately when crawling is disabledENVIRONMENT=localto enable crawling.publishers.jsonrelative to the module rather than the working directoryfileURLToPath(new URL('../publishers.json', import.meta.url))/appwhile the file ships next to the package, so a module-relative path loads regardless of the working directory. One list serves every environment, so the path is fixed rather than env-configurable.numberEnvtrims the value, uses the fallback when it is empty, and throws when it is not finiteENVIRONMENT, with explicit env vars as overridescrawlArticleDiscoveryTopicandcrawlArticleTopicdefault to${environment}-..., matching how the crawl-worker config names its topics and how Terraform names the resourcesENVIRONMENTis already injected, while the override lets the emulator point at local topic names.CORPUS_API_JWK_JSONis set the config requires at least one surface and a positive refresh intervalPROJECT_IDandREDIS_HOSTprocess.envfirst, thendeployedProjectId(environment)anddeployedRedisHost(environment)from crawl-common