Agent discoverability - #150
Open
justinmclean wants to merge 4 commits into
Open
Conversation
An is-agentic scan of magpie.apache.org found that unknown paths answer 200 with the app shell, that /llms.txt, /robots.txt and a sitemap do not exist, and that the site's developer resources cannot be found by name. It also inferred an HTTP API and marked it blocked; there is none. - astro.config.mjs, package.json: @astrojs/sitemap, so every route the build emits is enumerated at /sitemap-index.xml - public/.htaccess: ErrorDocument 404 /404.html so the ASF web server serves the site's own 404 page with the 404 status intact. Astro copies public/ into dist/, which CI pushes to the publish branch, so no workflow change - public/robots.txt: allow all, pointing at the sitemap - src/pages/llms.txt.ts: an llmstxt.org index built from the docs collection, so it tracks the synced docs rather than duplicating them. It states plainly that Magpie is a set of agent skills with no API on this domain, which is what tools currently get wrong about us - src/components/JsonLd.astro: Organization / WebSite / SoftwareSourceCode JSON-LD and a rel="llms-txt" link, with "<" escaped in the payload - src/pages/404.astro: recovery links, so a 404 is not a dead end - src/lib/site.ts: site constants, the hand-written route list, and HIDDEN_DOCS, which src/pages/docs/[...slug].astro now imports so the index and the site hide the same pages Titles in llms.txt come from frontmatter, else the page's opening H1, else the same dash-to-space fallback the docs page uses. Left for the checkout: mount <JsonLd /> in src/layouts/BaseLayout.astro. Generated-by: Claude (Fable 5.1)
Synced docs open with an SPDX comment and a doctoc block, so the H1 is never on the first line and every readme was titled "readme". Generated-by: Claude (Fable 5.1)
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A scan of magpie.apache.org found 404s answering 200 with the app shell, no llms.txt / robots.txt / sitemap, and a phantom "blocked API". Same shape as apache/iggy-website#77; nothing visible to a person changes.
@astrojs/sitemap→ /sitemap-index.xmlpublic/.htaccess:ErrorDocument 404 /404.html;public/robots.txtsrc/pages/llms.txt.ts: index generated from the docs collection; says plainly there is no API on this domainJsonLd.astro: Organization / WebSite / SoftwareSourceCode, in BaseLayout404.astrowith recovery linkssrc/lib/site.ts: constants, route list, andHIDDEN_DOCSnow shared with[...slug].astroVerified: build clean, 135 pages in llms.txt with real titles, no
.mdin the sitemap.ErrorDocumentis checked on the live site:curl -s -o /dev/null -w "%{http_code}" https://magpie.apache.org/nope→ 404.