Serve Markdown Twins - #152
Closed
justinmclean wants to merge 6 commits into
Closed
Conversation
An is-agentic scan of magpie.apache.org found that unknown paths answer 200 with the app shell, that /llms.txt, /robots.txt and a sitemap do not exist, and that the site's developer resources cannot be found by name. It also inferred an HTTP API and marked it blocked; there is none. - astro.config.mjs, package.json: @astrojs/sitemap, so every route the build emits is enumerated at /sitemap-index.xml - public/.htaccess: ErrorDocument 404 /404.html so the ASF web server serves the site's own 404 page with the 404 status intact. Astro copies public/ into dist/, which CI pushes to the publish branch, so no workflow change - public/robots.txt: allow all, pointing at the sitemap - src/pages/llms.txt.ts: an llmstxt.org index built from the docs collection, so it tracks the synced docs rather than duplicating them. It states plainly that Magpie is a set of agent skills with no API on this domain, which is what tools currently get wrong about us - src/components/JsonLd.astro: Organization / WebSite / SoftwareSourceCode JSON-LD and a rel="llms-txt" link, with "<" escaped in the payload - src/pages/404.astro: recovery links, so a 404 is not a dead end - src/lib/site.ts: site constants, the hand-written route list, and HIDDEN_DOCS, which src/pages/docs/[...slug].astro now imports so the index and the site hide the same pages Titles in llms.txt come from frontmatter, else the page's opening H1, else the same dash-to-space fallback the docs page uses. Left for the checkout: mount <JsonLd /> in src/layouts/BaseLayout.astro. Generated-by: Claude (Fable 5.1)
Synced docs open with an SPDX comment and a doctoc block, so the H1 is never on the first line and every readme was titled "readme". Generated-by: Claude (Fable 5.1)
An agent that wants a docs page currently has to parse the rendered HTML. This publishes a Markdown twin next to each page, so /docs/modes/ is also available as /docs/modes.md. The docs are already Markdown synced from apache/magpie, and Astro emits endpoints as static files, so this is a route beside the existing src/pages/docs/[...slug].astro, using the same slug and the same HIDDEN_DOCS filter, not a prebuild script. - src/pages/docs/[...slug].md.ts: one twin per public collection entry. The body carries the title, the description as a blockquote, the rendered and source URLs, then the page as synced. A body that already opens with an H1 supplies the title rather than printing it twice - src/pages/llms-full.txt.ts: every twin concatenated, for one-fetch agents - src/pages/llms.txt.ts: links go to the twins and state the convention - src/lib/site.ts: docMarkdownPath, docSourceUrl and docMarkdown helpers The sitemap filter from the previous change already excludes the twins: they are alternates, not pages. How they are served and advertised is the next change. Generated-by: Claude (Fable 5.1)
Generated-by: Claude (Fable 5.1)
The previous change writes a .md twin beside every docs page; this one makes them findable without knowing the convention. - public/.htaccess: text/markdown on .md files and the two llms files, and X-Robots-Tag: noindex on the twins, which are alternates of the HTML page, not pages - src/components/JsonLd.astro: <link rel="alternate" type="text/markdown"> on docs pages, pointing at the twin, so it is discoverable from the HTML. DocsLayout.astro needs to pass docMarkdownPath(entry) through No Accept-header negotiation, as on iggy.apache.org: the sibling file needs no server control. The is-agentic scan reported negotiation without Vary on this site; PR.md has the curl to confirm whether the server does it on its own, and the one-line Vary header to add only if it does. Generated-by: Claude (Fable 5.1)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #151
.htaccess:text/markdownon.mdand the llms files;noindexon twins<link rel="alternate" type="text/markdown">on every docs pageAfter publish:
curl -sI https://magpie.apache.org/docs/modes.md | grep -i content-type, then rescan withnpx is-agentic magpie.apache.org.