Skip to content

feat(HNT-2086): index url, domain, and topic as Sentry tags - #41

Draft
mmiermans wants to merge 1 commit into
claude/hnt-2086-dev-plumbingfrom
claude/hnt-2086-index-error-tags
Draft

feat(HNT-2086): index url, domain, and topic as Sentry tags#41
mmiermans wants to merge 1 commit into
claude/hnt-2086-dev-plumbingfrom
claude/hnt-2086-index-error-tags

Conversation

@mmiermans

Copy link
Copy Markdown
Collaborator

Goal

HNT-2086

Index the crawl url, its registrable domain, and its topic as searchable Sentry tags on both the article and discovery consumers so an engineer can filter crawl errors down to one publisher or one topic while triaging. Sentry indexes every tag and truncates each tag value at 200 characters, which is acceptable for a URL, and it applies no ceiling on the number of distinct values a tag may have.

The change touches both consumer slices, so it lands as its own commit on top of the stack rather than inside either wiring slice.

Implementation decisions

Decision Approach Why
Promote url, domain, and topic to searchable tags Set url, the domain of the url, and topic as Sentry tags on both consumers, and keep crawl_id and messageId as context Tags are indexed and searchable so an engineer can filter to one publisher or one topic when triaging a spike, Sentry indexes all of them with no ceiling on the number of distinct tag values, and the 200 character limit on a tag value truncates only the longest URLs while leaving them searchable by prefix. The identifiers that vary on every message stay as context because they are read once an issue is open rather than searched across issues.
Derive the domain from the url Reuse getRegistrableDomain from crawl-common, which resolves the registrable domain through the Public Suffix List A domain tag groups every failure from one publisher even though each article URL is distinct, and reusing the shared helper keeps the domain boundary identical to the rest of the crawler.
Report the discovery topics as one stable tag A discovery job carries one topic per context, so set the topic tag to the distinct topics sorted and joined into a single value A tag value must be a single string, and sorting before joining makes the value deterministic for a given set of topics so the same combination always matches the same search. This keeps the earlier intent of reporting the distinct topics.

@mmiermans
mmiermans force-pushed the claude/hnt-2086-dev-plumbing branch from 0edaf9e to 32e0d0e Compare July 6, 2026 17:09
@mmiermans
mmiermans force-pushed the claude/hnt-2086-index-error-tags branch from aa12c3b to 3d3cfcb Compare July 6, 2026 17:09
@mmiermans
mmiermans force-pushed the claude/hnt-2086-dev-plumbing branch from 32e0d0e to c3bc5b7 Compare August 3, 2026 16:51
@mmiermans
mmiermans force-pushed the claude/hnt-2086-index-error-tags branch from 3d3cfcb to 9ffc578 Compare August 3, 2026 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant