Weekly link-freshness engine + MCP server

[in-c0]

Builds the thing the README has had as a TODO since the first commit — "set up cron jobs/GitHub Actions to periodically re-run the scrapers and keep the dataset up-to-date" — and then exposes the result over MCP.

Why

The dataset is a table of URLs. URLs rot. Measured on main today, over the 176 rows that carry all eight columns:

outcome	share of 1,232 checks
ok — resolves at exactly the stored address	52.5%
moved — resolves only after redirecting elsewhere	41.7%
dead — 404/410	4.0%
blocked — 401/403/429	0.6%

Per column, the share still resolving at the stored address:

column	ok	note
Security Policy	76.7%
Privacy Policy	73.9%
Terms of Service	57.4%
Developer Community/Forum	56.8%
Official Documentation	38.1%	16.5% dead
Release Notes	35.2%
Rate Limiting Policy	29.5%	66.5% moved

The two policy columns rot in opposite directions: rate-limit pages almost never 404, they just move; documentation pages are the ones that actually disappear. The redirects are not cosmetic — api.slack.com → docs.slack.dev, stripe.com/docs → docs.stripe.com, cloud.google.com → docs.cloud.google.com are whole doc-platform migrations.

What is added

tools/check-links.mjs — probes every URL in every column, follows redirects, classifies the outcome. Two distinctions carry the design:

• moved vs dead. A redirect has a known-good replacement, so --fix rewrites those rows automatically. A 404 does not, and inventing one would put a fabrication in the dataset — dead URLs are reported for a human to resolve.
• blocked vs dead. 401/403/429 means we were refused, which says nothing about whether the page exists. Treating a bot-detection challenge as rot would let Cloudflare silently delete good rows.

.github/workflows/link-freshness.yml — runs it every Monday and opens a PR with the corrections. Installs only csv-parse and csv-stringify rather than the full manifest, so CI does not download Playwright browsers in order to make fetch calls.

mcp/server.mjs — serves the index over MCP with search_apis, get_api_resources and index_health. Every URL comes back with when it was last verified and what the verifier saw; with no report present it reports "unknown" rather than implying freshness. Handing a model a link without its age invites it to trust a 404, and a tool result reads as authoritative.

tools/verify-mcp.mjs — drives the server as a real MCP client end-to-end.

A defect found along the way

Only 176 of 562 rows (31%) carry all eight columns, and the missing fields are omitted rather than blanked — so every later value shifts one column left. 189 rows have a URL in Official_Documentation_URL that is plainly a privacy policy, community forum or security page. Microsoft Graph People API, Slack Interactivity API, Dropbox File Request API and Zendesk Support API are examples.

This is worse than rot: those rows are not stale, they are mislabelled, and a consumer asking for documentation gets a privacy policy with no indication anything is wrong.

Per-column statistics therefore cover complete rows only — otherwise "documentation health" would be partly a claim about privacy policies. The problem is measured separately under alignment in datasets/link-health.json, and flagged in the README's Known Issues as the highest-value contribution available. It is not fixed here — repairing it needs per-row judgement about which column each URL belongs to.

Also measured

74 of 164 documentation hosts (45%) now serve /llms.txt — including Slack, Stripe, AWS, GitHub, Twilio, PayPal and Mailchimp. Tracked per run, because if that number keeps climbing it is a direct signal about this project's remaining usefulness, and the honest thing is to watch it rather than not look.

Notes for review

• api-docs-urls.csv shows every line as changed because csv-stringify normalises quoting. The meaningful changes are the 1,472 URLs replaced with their redirect targets.
• datasets/link-health-history.csv is append-only — the series is the interesting output, not any single snapshot.
• npm test → 13 passing. node tools/verify-mcp.mjs → all MCP checks passed.
• The weekly workflow needs Settings → Actions → General → "Allow GitHub Actions to create and approve pull requests" enabled; the repo currently has it off, so the PR-opening step will fail until it is turned on. Everything before that step still runs and uploads the report as an artifact.

🤖 Generated with Claude Code