Skip to content

feat(HNT-2900): commit the publisher list and its generator - #38

Draft
mmiermans wants to merge 1 commit into
claude/hnt-2086-agent-tickfrom
claude/hnt-2086-publisher-data
Draft

feat(HNT-2900): commit the publisher list and its generator#38
mmiermans wants to merge 1 commit into
claude/hnt-2086-agent-tickfrom
claude/hnt-2086-publisher-data

Conversation

@mmiermans

@mmiermans mmiermans commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Goal

HNT-2900

Commit the crawl-agent's generated publishers.json, a small publishers.example.json, and the Python generator that builds it from the legacy crawl PAGES export, plus a guard test that the committed data validates through the real loader. This gives the agent a self-contained startup config that a reviewer can read and CI can verify.

This builds on the message validation slice by using its validatePublisherList and the loadPublisherList loader from the earlier slice, both exercised by the guard test. See docs/ARCHITECTURE.md in #14 for how the agent loads this list at startup.

Implementation decisions

Decision Approach Why
Commit the generated data beside the loader and exclude it from the diff view publishers.json is committed as generated data and excluded from the reviewable diff, and the small publishers.example.json and the generator stand in for review. The file is generated and thousands of lines, so it is not line-by-line reviewable. Committing it makes the agent's startup config self-contained and reproducible, the example lets a reviewer read one representative entry, and the generator documents how to regenerate it.
Guard the committed file with a test that loads it through the real loader A spec loads publishers.json via loadPublisherList at its module-relative path and asserts it has pages and an empty live_articles. Because the file is otherwise unreviewed, a malformed export or a path regression fails in CI here rather than crash-looping the deployed agent.
Generate with a Python script that imports the legacy export scripts/generate-publishers.py imports the legacy pages.py module and emits the PublisherList JSON. The legacy publisher list is a Python export of the curated Google Sheet, so a Python generator imports it directly instead of reimplementing its parsing in the TypeScript service.
Map each locale to a New Tab surface id and flatten targets to contexts Each legacy target's locale becomes a NEW_TAB_<LOCALE> surface id and each of its topics becomes a lowercased discovery context, interval_minutes defaults to 20 per the tech spec's example cadence, and duplicate URLs are merged into one entry with de-duplicated contexts. NEW_TAB_EN_US and the like are the Corpus API ScheduledSurfaceGUID values, matching content-monorepo's ScheduledSurfacesEnum in packages/content-common/src/types.ts. The agent's validator requires unique page URLs, so merging keeps one entry per page carrying all its contexts.
Emit an empty live_articles array The generator always writes live_articles: []. The legacy PAGES list has no curated live articles. Those are sourced from the Corpus API in Phase 5, not from this committed file.

@mmiermans
mmiermans force-pushed the claude/hnt-2086-agent-tick branch from 422b88b to b81ce44 Compare July 6, 2026 17:09
@mmiermans
mmiermans force-pushed the claude/hnt-2086-publisher-data branch from 6e5b486 to 972641e Compare July 6, 2026 17:09
@mmiermans mmiermans changed the title feat(HNT-2086): commit the publisher list and its generator feat(HNT-2900): commit the publisher list and its generator Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant