diff --git a/content/docs/guides/markdown-content-negotiation.mdx b/content/docs/guides/markdown-content-negotiation.mdx new file mode 100644 index 00000000..9717c988 --- /dev/null +++ b/content/docs/guides/markdown-content-negotiation.mdx @@ -0,0 +1,264 @@ +--- +title: Serve Markdown to AI Agents +seo: + title: Serve Markdown to AI Agents | TinaCMS Docs + description: Configure content negotiation for TinaCMS pages in Next.js, Astro, and Vite. +--- + +Serve TinaCMS source Markdown or MDX to AI agents when they request `Accept: text/markdown`. Browsers can continue to receive HTML at the same URL. + +To serve both formats at one URL, route requests through a server, CDN, or edge layer that handles content negotiation. A static host alone cannot do this. + +These examples use the same `negotiator` library as [TinaDocs](https://github.com/tinacms/tina-docs) to select between HTML and Markdown. It respects quality values (`q`), wildcards, and case-insensitive media types. Markdown with `q=0` is never selected. Missing headers and generic wildcards default to HTML; if neither format is acceptable, these examples fall back to HTML, matching TinaDocs. + +Install the library and its TypeScript types in your project: + +```bash +pnpm add negotiator +pnpm add -D @types/negotiator +``` + + + +## Next.js App Router + +Create a static route handler that returns the source MDX. This example assumes documents live in `content/docs` and that `/docs` maps to `content/docs/index.mdx`. + +```typescript +// src/app/api/markdown/[...slug]/route.ts +import { readFile, readdir } from "node:fs/promises"; +import path from "node:path"; + +const docsDirectory = path.join(process.cwd(), "content", "docs"); + +export const dynamic = "force-static"; +export const dynamicParams = false; + +export async function generateStaticParams() { + const files = await readdir(docsDirectory, { recursive: true }); + + return files + .filter((file) => file.endsWith(".mdx")) + .map((file) => ({ + slug: file.replace(/\.mdx$/, "").split(path.sep), + })); +} + +export async function GET( + _request: Request, + { params }: { params: Promise<{ slug: string[] }> }, +) { + const { slug } = await params; + const filePath = path.join(docsDirectory, `${slug.join(path.sep)}.mdx`); + + if (!filePath.startsWith(`${docsDirectory}${path.sep}`)) { + return new Response(null, { status: 404 }); + } + + try { + return new Response(await readFile(filePath, "utf8"), { + headers: { "Content-Type": "text/markdown; charset=utf-8" }, + }); + } catch { + return new Response(null, { status: 404 }); + } +} +``` + +Use middleware to negotiate the response and keep the public URL unchanged. This matches TinaDocs' Next.js 15 implementation. Merge it into any existing middleware and matcher configuration. + +```typescript +// src/middleware.ts +import Negotiator from "negotiator"; +import { type NextRequest, NextResponse } from "next/server"; + +export function middleware(request: NextRequest) { + const type = new Negotiator({ + headers: { accept: request.headers.get("accept") ?? "*/*" }, + }).mediaType(["text/html", "text/markdown"]); + const url = request.nextUrl.clone(); + url.pathname = `/api/markdown${url.pathname.slice("/docs".length) || "/index"}`; + + const response = + process.env.EXPORT_MODE !== "static" && type === "text/markdown" + ? NextResponse.rewrite(url) + : NextResponse.next(); + response.headers.append("Vary", "Accept"); + return response; +} + +export const config = { matcher: "/docs/:path*" }; +``` + +Static exports do not run middleware. TinaDocs uses `EXPORT_MODE=static` to select its static-export configuration; a static host still serves HTML at the docs URL regardless of `Accept`. + +The middleware appends `Vary: Accept` to both representations, but Next.js or the hosting layer can replace this header. [Vercel includes `Accept` in its cache key](https://vercel.com/docs/caching/cdn-cache) automatically. For other CDNs and reverse proxies, configure separate cache entries by `Accept` or bypass caching for docs routes; otherwise HTML and Markdown can share a cache entry. + +## Astro + +Use an Astro adapter for a Node.js server with on-demand rendering. Set `output: "server"`, or set `export const prerender = false` on your docs pages as well as the endpoint below, so middleware can inspect each request. Make `content/docs` available to the deployed server: this example reads source MDX from disk at request time. See [Astro middleware](https://docs.astro.build/en/guides/middleware/). + +Create a server endpoint for the source MDX: + +```typescript +// src/pages/_markdown/[...slug].ts +import { readFile } from "node:fs/promises"; +import path from "node:path"; +import type { APIRoute } from "astro"; + +export const prerender = false; + +const docsDirectory = path.join(process.cwd(), "content", "docs"); + +export const GET: APIRoute = async ({ params }) => { + const slug = params.slug || "index"; + const filePath = path.join(docsDirectory, `${slug}.mdx`); + + if (!filePath.startsWith(`${docsDirectory}${path.sep}`)) { + return new Response(null, { status: 404 }); + } + + try { + return new Response(await readFile(filePath, "utf8"), { + headers: { "Content-Type": "text/markdown; charset=utf-8" }, + }); + } catch { + return new Response(null, { status: 404 }); + } +}; +``` + +Add middleware to rewrite Markdown requests to the endpoint and append `Vary: Accept` to both HTML and Markdown responses: + +```typescript +// src/middleware.ts +import { defineMiddleware } from "astro:middleware"; +import Negotiator from "negotiator"; + +export const onRequest = defineMiddleware(async (context, next) => { + const pathname = context.url.pathname.replace(/\/$/, ""); + const isDocsPage = pathname === "/docs" || pathname.startsWith("/docs/"); + + if (!isDocsPage) { + return next(); + } + + const type = new Negotiator({ + headers: { accept: context.request.headers.get("accept") ?? "*/*" }, + }).mediaType(["text/html", "text/markdown"]); + const slug = pathname.slice("/docs".length).replace(/^\//, "") || "index"; + const response = + type === "text/markdown" + ? await next(`/_markdown/${slug}`) + : await next(); + const headers = new Headers(response.headers); + headers.append("Vary", "Accept"); + + return new Response(response.body, { + status: response.status, + statusText: response.statusText, + headers, + }); +}); +``` + +Astro static output cannot inspect request headers. For a static deployment, publish separate `.md` files or configure negotiation in the hosting layer. + +## Plain Vite + +Vite produces static files for production. Use this plugin to add content negotiation to its development and preview servers (`vite` and `vite preview`): + +```typescript +// vite.config.ts +import { readFile } from "node:fs/promises"; +import path from "node:path"; +import { defineConfig, type Plugin, type PreviewServer, type ViteDevServer } from "vite"; +import Negotiator from "negotiator"; + +function markdownNegotiation(): Plugin { + const docsDirectory = path.join(process.cwd(), "content", "docs"); + + const install = (server: ViteDevServer | PreviewServer) => { + server.middlewares.use(async (request, response, next) => { + const pathname = new URL(request.url || "/", "http://localhost").pathname; + + if (pathname !== "/docs" && !pathname.startsWith("/docs/")) { + return next(); + } + + response.appendHeader("Vary", "Accept"); + const type = new Negotiator({ + headers: { accept: request.headers.accept ?? "*/*" }, + }).mediaType(["text/html", "text/markdown"]); + + if (type !== "text/markdown") { + return next(); + } + + const slug = pathname.slice("/docs".length).replace(/^\//, "") || "index"; + const filePath = path.join(docsDirectory, `${slug}.mdx`); + + if (!filePath.startsWith(`${docsDirectory}${path.sep}`)) { + response.statusCode = 404; + return response.end(); + } + + try { + response.setHeader("Content-Type", "text/markdown; charset=utf-8"); + response.end(await readFile(filePath, "utf8")); + } catch { + response.statusCode = 404; + response.end(); + } + }); + }; + + return { + name: "markdown-content-negotiation", + configureServer: install, + configurePreviewServer: install, + }; +} + +export default defineConfig({ + plugins: [markdownNegotiation()], +}); +``` + +Use `vite preview` only for local verification. In production, run equivalent middleware in your application server or configure your CDN or edge platform to route requests based on `Accept`. For a static-only deployment, publish generated `.md` files at separate URLs. + +## Test the result + +Verify both representations at the same URL: + +```bash +curl -i https://example.com/docs/getting-started +curl -i -H 'Accept: text/markdown' https://example.com/docs/getting-started +``` + +The first response should use `Content-Type: text/html`. The second should use `Content-Type: text/markdown`. If a CDN or proxy caches the responses, confirm that its cache key includes `Accept` or that the response includes `Vary: Accept`. + +Also test negotiation at both `/docs` and a nested docs URL: + +| `Accept` header | Expected response | +| --- | --- | +| Missing, `*/*`, or `text/*` | HTML | +| `text/markdown` or `Text/Markdown` | Markdown | +| `text/html;q=1, text/markdown;q=0.1` | HTML | +| `text/html;q=0.1, text/markdown;q=1` | Markdown | +| `text/html, text/markdown;q=0` | HTML | +| `text/markdown;q=0.000, text/html` | HTML | +| `text/markdown; charset=utf-8; q=0, text/html` | HTML | +| `text/markdown;q=0.5` | Markdown | +| `text/html;q=0, text/markdown` | Markdown | + +Repeat requests in HTML → Markdown → HTML → Markdown order to check that the deployed cache keeps the representations separate. Check that Markdown responses contain source MDX, including frontmatter, rather than rendered HTML. On a static-only deployment, expect HTML at the docs URL for every header above. + +## Related resources + +- [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) +- [Negotiator API](https://github.com/jshttp/negotiator#api) +- [Next.js rewrites](https://nextjs.org/docs/15/app/api-reference/config/next-config-js/rewrites) +- [Astro middleware](https://docs.astro.build/en/guides/middleware/) +- [Astro endpoints](https://docs.astro.build/en/guides/endpoints/) +- [Vite plugin API](https://vite.dev/guide/api-plugin) diff --git a/content/navigation-bar/docs-navigation-bar.json b/content/navigation-bar/docs-navigation-bar.json index 997ca85f..4fc610fc 100644 --- a/content/navigation-bar/docs-navigation-bar.json +++ b/content/navigation-bar/docs-navigation-bar.json @@ -917,6 +917,11 @@ "slug": "content/docs/editing/referenced-files.mdx", "_template": "item" }, + { + "slug": "content/docs/guides/markdown-content-negotiation.mdx", + "title": "Serve Markdown to AI Agents", + "_template": "item" + }, { "slug": "content/docs/guides/react-data-fetching.mdx", "title": "Querying TinaCMS Content at Runtime",