Docs markdown - #151
Open
justinmclean wants to merge 8 commits into
Open
Conversation
An is-agentic scan of magpie.apache.org found that unknown paths answer 200 with the app shell, that /llms.txt, /robots.txt and a sitemap do not exist, and that the site's developer resources cannot be found by name. It also inferred an HTTP API and marked it blocked; there is none. - astro.config.mjs, package.json: @astrojs/sitemap, so every route the build emits is enumerated at /sitemap-index.xml - public/.htaccess: ErrorDocument 404 /404.html so the ASF web server serves the site's own 404 page with the 404 status intact. Astro copies public/ into dist/, which CI pushes to the publish branch, so no workflow change - public/robots.txt: allow all, pointing at the sitemap - src/pages/llms.txt.ts: an llmstxt.org index built from the docs collection, so it tracks the synced docs rather than duplicating them. It states plainly that Magpie is a set of agent skills with no API on this domain, which is what tools currently get wrong about us - src/components/JsonLd.astro: Organization / WebSite / SoftwareSourceCode JSON-LD and a rel="llms-txt" link, with "<" escaped in the payload - src/pages/404.astro: recovery links, so a 404 is not a dead end - src/lib/site.ts: site constants, the hand-written route list, and HIDDEN_DOCS, which src/pages/docs/[...slug].astro now imports so the index and the site hide the same pages Titles in llms.txt come from frontmatter, else the page's opening H1, else the same dash-to-space fallback the docs page uses. Left for the checkout: mount <JsonLd /> in src/layouts/BaseLayout.astro. Generated-by: Claude (Fable 5.1)
Synced docs open with an SPDX comment and a doctoc block, so the H1 is never on the first line and every readme was titled "readme". Generated-by: Claude (Fable 5.1)
Closed
justinmclean
force-pushed
the
docs-markdown
branch
from
September 5, 2026 08:16
299d637 to
9ca2f55
Compare
An agent that wants a docs page currently has to parse the rendered HTML. This publishes a Markdown twin next to each page, so /docs/modes/ is also available as /docs/modes.md. The docs are already Markdown synced from apache/magpie, and Astro emits endpoints as static files, so this is a route beside the existing src/pages/docs/[...slug].astro, using the same slug and the same HIDDEN_DOCS filter, not a prebuild script. - src/pages/docs/[...slug].md.ts: one twin per public collection entry. The body carries the title, the description as a blockquote, the rendered and source URLs, then the page as synced. A body that already opens with an H1 supplies the title rather than printing it twice - src/pages/llms-full.txt.ts: every twin concatenated, for one-fetch agents - src/pages/llms.txt.ts: links go to the twins and state the convention - src/lib/site.ts: docMarkdownPath, docSourceUrl and docMarkdown helpers The sitemap filter from the previous change already excludes the twins: they are alternates, not pages. How they are served and advertised is the next change. Generated-by: Claude (Fable 5.1)
Generated-by: Claude (Fable 5.1)
justinmclean
force-pushed
the
docs-markdown
branch
from
September 5, 2026 08:26
9ca2f55 to
ef74191
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #150. Same shape as apache/iggy-website#79.
Every docs page gets a twin at the same path with
.mdin place of the slash (/docs/modes/→/docs/modes.md), so agents never parse the HTML. An Astro route beside[...slug].astro, same slug andHIDDEN_DOCSfilter.src/pages/docs/[...slug].md.ts: title, description, rendered and source URLs, then the body as synced (its H1 becomes the title, not repeated)src/pages/llms-full.txt.ts: all twins in one filellms.txtnow links to the twinsVerified: 135 twins, one per rendered page; one H1 each. Excluded from the sitemap by #150's filter.