Skip to content

feat(HNT-2086): read scheduled live articles from the Corpus API - #23

Draft
mmiermans wants to merge 1 commit into
claude/hnt-2086-common-utilsfrom
claude/hnt-2086-corpus-section-items
Draft

feat(HNT-2086): read scheduled live articles from the Corpus API#23
mmiermans wants to merge 1 commit into
claude/hnt-2086-common-utilsfrom
claude/hnt-2086-corpus-section-items

Conversation

@mmiermans

@mmiermans mmiermans commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Goal

HNT-2086

Let the crawl agent source curated live articles from the Corpus Admin API instead of a hardcoded list. getScheduledSectionItems reads a surface's currently scheduled section items and returns them as LiveArticle values, mapping the camelCase Corpus shape to the snake_case corpus_item the agent expects.

This builds on the domain-types slice by importing its CorpusItem and LiveArticle types. See the agent flow in docs/ARCHITECTURE.md under review as #14.

Implementation decisions

Decision Approach Why
Extract a shared graphqlRequest helper Both updateApprovedCorpusItem and the new getScheduledSectionItems call through one function for JWT auth, retry, transient 5xx and network backoff, and error handling Adding a second GraphQL operation would otherwise duplicate the whole auth, retry, and fail-fast path. One helper keeps that policy in a single place and keeps the two operations behavior-preserving.
Run the admin sections query and keep only LIVE sections in code getSectionsWithSectionItems is fetched with the admin JWT, then non-LIVE sections are filtered out before flattening their items The agent authenticates as an admin client, and the admin query is not date-filtered, so it returns scheduled, disabled, and expired sections too, unlike the public query. content-monorepo's servers/curated-corpus-api/src/database/queries/Section.ts documents this: the public context returns only LIVE sections while the admin context returns all active enabled and disabled ones. The LIVE-only filter therefore has to live in the client. This sections query is chosen over the date-based getScheduledCorpusItems because it matches the legacy HydrateSectionItems semantics, returns the full item including language so non-EN surfaces are not defaulted, and reuses the same curator JWT group the write path already holds so the read needs no new credential.
Deduplicate the flattened items by URL, keeping the first A seen-set drops repeat URLs across sections One article can appear in more than one LIVE section, and the agent's publisher list requires unique URLs.
Tolerate a missing authors array in the mapping A missing authors maps to an empty array rather than throwing This camelCase-to-snake_case mapping runs before validateLiveArticle can drop a malformed item, so an unguarded throw here would crash-loop the agent at startup. The validation layer, not the mapper, is where a bad item is rejected.

@mmiermans
mmiermans force-pushed the claude/hnt-2086-common-utils branch from baaa374 to 31dc989 Compare July 6, 2026 17:08
@mmiermans
mmiermans force-pushed the claude/hnt-2086-corpus-section-items branch from 2ce2154 to a212372 Compare July 6, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant