Skip to content

doc/user: publish agent-readable Markdown and llms.txt for every page - #39466

Draft
sjwiesman wants to merge 10 commits into
MaterializeInc:mainfrom
sjwiesman:docs-agent-markdown
Draft

sjwiesman wants to merge 10 commits into
MaterializeInc:mainfrom
sjwiesman:docs-agent-markdown

Conversation

@sjwiesman

@sjwiesman sjwiesman commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Makes the docs agent-friendly per the Agent-Friendly Docs spec. Against a local build served like production, npx afdocs check scores 100/100 on the default 50-page sample (production scores 59, capped because it discovers a single page).

  • Markdown for every page at <page URL>index.md, generated after the Hugo build by bin/docs-markdown, which converts each page's <article> (replaces the skill output format and its per-shortcode Markdown twins). Links are absolute.
  • Nested llms.txt: a 2K root index linking to one llms.txt per top-level section (largest 28K, previously one 86K file), covering 561 of 562 sitemap pages including per-version release notes.
  • llms.txt directive first in every HTML and Markdown page, plus <link rel="alternate" type="text/markdown">.
  • Sidebar renders only the current branch; the rest loads from a shared sidebar.html. Median converted page size drops from 46K to 12K characters. The merged sidebar is identical to the old one on all 566 pages.
  • Sitemap uses absolute <loc> URLs and is served as application/xml.
  • CloudFront function for Accept: text/markdown (ci/deploy_website/docs-content-negotiation.js), attached to the distribution by hand.

Preview:

Open before merge:

  • bin/gen-claude-skill is removed, but the update-mz-docs-skill workflow in MaterializeInc/agent-skills still runs it.
  • 15 pages still convert to over 50K characters on their own content (two over 100K: releases/, mz_internal), which keeps a full-site check from passing every time.
  • /docs/markdown-docs/ URLs will 404 after deploy; no redirects yet.
  • The CloudFront function and Vary: Accept need to be set up on the distribution.

Tests: misc/python/materialize/cli/docs_markdown_test.py, run by ci/test/lint-docs.sh.

🤖 Generated with Claude Code

sjwiesman and others added 10 commits October 1, 2026 10:48
The docs Markdown output is moving to serve agents on the web, with
absolute links and per-page index files, which no longer fits a local
skill bundle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Markdown rendition is regenerated from the built HTML in a following
commit, which removes the need for a Markdown twin of every shortcode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The <article> gives the Markdown rendition and agent tooling one element
to read the page body from. Section pages now wrap their child-page
links in a <ul>; they were bare <li> elements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bin/docs-markdown converts each page's <article> into an index.md beside
its index.html, so https://materialize.com/docs/<page>/index.md serves
Markdown for every page. Converting the rendered HTML keeps the two
renditions in parity without a Markdown twin of every shortcode. Links
and images are absolute, since an agent holding the Markdown has usually
lost the URL it fetched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every HTML page now opens with a visually hidden directive naming the
docs index and the Markdown rendition, and advertises its Markdown with
<link rel="alternate">. bin/docs-markdown copies the directive to the
top of each Markdown page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The production build sets baseURL to a bare path, so Hugo's sitemap
emitted <loc>/docs/...</loc>. The sitemap protocol requires absolute
URLs. Also serve sitemap.xml as application/xml rather than the RSS
media type Hugo deploy infers for .xml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The single llms.txt was 86K characters, past the 50K that agents can
reliably read in one fetch. The root llms.txt now links to an llms.txt
for each top-level section, each listing that section's pages grouped
by subsection; the largest is 28K. Per-version release notes are listed
in releases/llms.txt. Each page's llms.txt directive names its own
section's index.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every page rendered the full 485-entry navigation tree, about 41K
characters once converted to text, which pushed most pages past the size
agents read in one fetch. Pages now render the top-level entries and the
branch leading to themselves; the median converted page drops from 46K
to 12K characters. The sidebar script fills in the other branches from a
shared sidebar.html, so readers can still expand any section in place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A password input in server-rendered HTML reads to agent tooling as a
login wall, which flagged the webhook quickstart as auth-gated. The
widget only works with JavaScript, so its script now masks the field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clients that send Accept: text/markdown, as some coding agents do, get a
page's index.md instead of its HTML. The function is attached to the
distribution by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant