Skip to content

feat(HNT-2899): load and validate the agent publisher list - #36

Draft
mmiermans wants to merge 1 commit into
claude/hnt-2086-agent-configfrom
claude/hnt-2086-publisher-list
Draft

feat(HNT-2899): load and validate the agent publisher list#36
mmiermans wants to merge 1 commit into
claude/hnt-2086-agent-configfrom
claude/hnt-2086-publisher-list

Conversation

@mmiermans

@mmiermans mmiermans commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Goal

HNT-2899

Add a publisher-list module so the crawl agent has a validated set of pages and live articles to enqueue. loadPublisherList reads and validates the committed JSON file and samples it down to the configured page limit, and fetchLiveArticles reads the currently scheduled section items from the Corpus API when that source is configured.

It builds on earlier slices for the crawl domain message types and validators, the Corpus section-items read, and the agent config. See docs/ARCHITECTURE.md in #14 for where the agent's page and live-article sources sit in the system.

Implementation decisions

Decision Approach Why
Fail fast on a bad committed list, but skip a bad Corpus item loadPublisherList throws when the file is missing, unparseable, or fails validatePublisherList. fetchLiveArticles validates each Corpus item on its own and skips the malformed ones with a warning, while letting client and transport errors propagate. The committed file is trusted config, so a bad entry is a deploy-time mistake worth aborting startup for. The Corpus API is external input that can carry a curated item with a blank required field, so dropping one item degrades freshness for that item rather than crash-looping the whole agent, and a failed Corpus call still fails startup so a misconfigured or unreachable client is caught immediately.
Keep a blank publisher but drop an item missing a required field Validation accepts an empty publisher string and only skips an item that is missing a truly required field such as the title. Production publishers are non-empty because the Corpus API back-fills an empty publisher with a hostname fallback on create, and its ApprovedItem model types publisher as a non-null string with no default, so a blank publisher does not arise from production data. Accepting a blank is defensive robustness against malformed or dev-seeded input rather than a legitimate production state, and it is harmless because the crawler only echoes the value back unchanged. The validator reused from an earlier slice therefore permits a blank publisher and reserves skipping for an item missing a truly required field such as the title.
Sample pages by even stride under a limit limitPages maps each output slot to floor(i * total / limit) rather than taking the first N or striding by a fixed integer. A dev deployment runs a representative subset at lower load. A fixed integer stride of ceil(total/limit) over-strides and returns fewer than limit pages when the limit does not divide the total, and taking the first N collapses to one corner of the alphabetically sorted list and loses publisher and language variety.
Fetch surfaces concurrently and dedup by URL keeping the first occurrence Promise.all over the configured surfaces, then flatten and de-duplicate by URL. One article can be scheduled on more than one surface. Promise.all preserves surface order, so the cross-surface dedup deterministically keeps the first occurrence of each URL.

@mmiermans
mmiermans force-pushed the claude/hnt-2086-agent-config branch from 17d7d43 to 6bad473 Compare July 6, 2026 17:09
@mmiermans
mmiermans force-pushed the claude/hnt-2086-publisher-list branch from 9c08d05 to 57f2332 Compare July 6, 2026 17:09
@mmiermans mmiermans changed the title feat(HNT-2086): load and validate the agent publisher list feat(HNT-2899): load and validate the agent publisher list Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant