📚 Add Markdown content negotiation guide

content/docs/guides/markdown-content-negotiation.mdx

@@ -0,0 +1,264 @@
+---
+title: Serve Markdown to AI Agents
+seo:
+  title: Serve Markdown to AI Agents | TinaCMS Docs
+  description: Configure content negotiation for TinaCMS pages in Next.js, Astro, and Vite.
+---
+
+Serve TinaCMS source Markdown or MDX to AI agents when they request `Accept: text/markdown`. Browsers can continue to receive HTML at the same URL.
+
+To serve both formats at one URL, route requests through a server, CDN, or edge layer that handles content negotiation. A static host alone cannot do this.
+
+These examples use the same `negotiator` library as [TinaDocs](https://github.com/tinacms/tina-docs) to select between HTML and Markdown. It respects quality values (`q`), wildcards, and case-insensitive media types. Markdown with `q=0` is never selected. Missing headers and generic wildcards default to HTML; if neither format is acceptable, these examples fall back to HTML, matching TinaDocs.
+
+Install the library and its TypeScript types in your project:
+
+```bash
+pnpm add negotiator
+pnpm add -D @types/negotiator
+```
+
+<Callout variant="warning" body="MDX can contain JSX and TinaCMS components. Most AI agents can still read it, but clients that require CommonMark may need a separate MDX-to-Markdown conversion step." />
+
+## Next.js App Router
+
+Create a static route handler that returns the source MDX. This example assumes documents live in `content/docs` and that `/docs` maps to `content/docs/index.mdx`.
+
+```typescript
+// src/app/api/markdown/[...slug]/route.ts
+import { readFile, readdir } from "node:fs/promises";
+import path from "node:path";
+
+const docsDirectory = path.join(process.cwd(), "content", "docs");
+
+export const dynamic = "force-static";
+export const dynamicParams = false;
+
+export async function generateStaticParams() {
+  const files = await readdir(docsDirectory, { recursive: true });
+
+  return files
+    .filter((file) => file.endsWith(".mdx"))
+    .map((file) => ({
+      slug: file.replace(/\.mdx$/, "").split(path.sep),
+    }));
+}
+
+export async function GET(
+  _request: Request,
+  { params }: { params: Promise<{ slug: string[] }> },
+) {
+  const { slug } = await params;
+  const filePath = path.join(docsDirectory, `${slug.join(path.sep)}.mdx`);
+
+  if (!filePath.startsWith(`${docsDirectory}${path.sep}`)) {
+    return new Response(null, { status: 404 });
+  }
+
+  try {
+    return new Response(await readFile(filePath, "utf8"), {
+      headers: { "Content-Type": "text/markdown; charset=utf-8" },
+    });
+  } catch {
+    return new Response(null, { status: 404 });
+  }
+}
+```
+
+Use middleware to negotiate the response and keep the public URL unchanged. This matches TinaDocs' Next.js 15 implementation. Merge it into any existing middleware and matcher configuration.
+
+```typescript
+// src/middleware.ts
+import Negotiator from "negotiator";
+import { type NextRequest, NextResponse } from "next/server";
+
+export function middleware(request: NextRequest) {
+  const type = new Negotiator({
+    headers: { accept: request.headers.get("accept") ?? "*/*" },
+  }).mediaType(["text/html", "text/markdown"]);
+  const url = request.nextUrl.clone();
+  url.pathname = `/api/markdown${url.pathname.slice("/docs".length) || "/index"}`;
+
+  const response =
+    process.env.EXPORT_MODE !== "static" && type === "text/markdown"
+      ? NextResponse.rewrite(url)
+      : NextResponse.next();
+  response.headers.append("Vary", "Accept");
+  return response;
+}
+
+export const config = { matcher: "/docs/:path*" };
+```
+
+Static exports do not run middleware. TinaDocs uses `EXPORT_MODE=static` to select its static-export configuration; a static host still serves HTML at the docs URL regardless of `Accept`.
+
+The middleware appends `Vary: Accept` to both representations, but Next.js or the hosting layer can replace this header. [Vercel includes `Accept` in its cache key](https://vercel.com/docs/caching/cdn-cache) automatically. For other CDNs and reverse proxies, configure separate cache entries by `Accept` or bypass caching for docs routes; otherwise HTML and Markdown can share a cache entry.
+
+## Astro
+
+Use an Astro adapter for a Node.js server with on-demand rendering. Set `output: "server"`, or set `export const prerender = false` on your docs pages as well as the endpoint below, so middleware can inspect each request. Make `content/docs` available to the deployed server: this example reads source MDX from disk at request time. See [Astro middleware](https://docs.astro.build/en/guides/middleware/).
+
+Create a server endpoint for the source MDX:
+
+```typescript
+// src/pages/_markdown/[...slug].ts
+import { readFile } from "node:fs/promises";
+import path from "node:path";
+import type { APIRoute } from "astro";
+
+export const prerender = false;
+
+const docsDirectory = path.join(process.cwd(), "content", "docs");
+
+export const GET: APIRoute = async ({ params }) => {
+  const slug = params.slug || "index";
+  const filePath = path.join(docsDirectory, `${slug}.mdx`);
+
+  if (!filePath.startsWith(`${docsDirectory}${path.sep}`)) {
+    return new Response(null, { status: 404 });
+  }
+
+  try {
+    return new Response(await readFile(filePath, "utf8"), {
+      headers: { "Content-Type": "text/markdown; charset=utf-8" },
+    });
+  } catch {
+    return new Response(null, { status: 404 });
+  }
+};
+```
+
+Add middleware to rewrite Markdown requests to the endpoint and append `Vary: Accept` to both HTML and Markdown responses:
+
+```typescript
+// src/middleware.ts
+import { defineMiddleware } from "astro:middleware";
+import Negotiator from "negotiator";
+
+export const onRequest = defineMiddleware(async (context, next) => {
+  const pathname = context.url.pathname.replace(/\/$/, "");
+  const isDocsPage = pathname === "/docs" || pathname.startsWith("/docs/");
+
+  if (!isDocsPage) {
+    return next();
+  }
+
+  const type = new Negotiator({
+    headers: { accept: context.request.headers.get("accept") ?? "*/*" },
+  }).mediaType(["text/html", "text/markdown"]);
+  const slug = pathname.slice("/docs".length).replace(/^\//, "") || "index";
+  const response =
+    type === "text/markdown"
+      ? await next(`/_markdown/${slug}`)
+      : await next();
+  const headers = new Headers(response.headers);
+  headers.append("Vary", "Accept");
+
+  return new Response(response.body, {
+    status: response.status,
+    statusText: response.statusText,
+    headers,
+  });
+});
+```
+
+Astro static output cannot inspect request headers. For a static deployment, publish separate `.md` files or configure negotiation in the hosting layer.
+
+## Plain Vite
+
+Vite produces static files for production. Use this plugin to add content negotiation to its development and preview servers (`vite` and `vite preview`):
+
+```typescript
+// vite.config.ts
+import { readFile } from "node:fs/promises";
+import path from "node:path";
+import { defineConfig, type Plugin, type PreviewServer, type ViteDevServer } from "vite";
+import Negotiator from "negotiator";
+
+function markdownNegotiation(): Plugin {
+  const docsDirectory = path.join(process.cwd(), "content", "docs");
+
+  const install = (server: ViteDevServer | PreviewServer) => {
+    server.middlewares.use(async (request, response, next) => {
+      const pathname = new URL(request.url || "/", "http://localhost").pathname;
+
+      if (pathname !== "/docs" && !pathname.startsWith("/docs/")) {
+        return next();
+      }
+
+      response.appendHeader("Vary", "Accept");
+      const type = new Negotiator({
+        headers: { accept: request.headers.accept ?? "*/*" },
+      }).mediaType(["text/html", "text/markdown"]);
+
+      if (type !== "text/markdown") {
+        return next();
+      }
+
+      const slug = pathname.slice("/docs".length).replace(/^\//, "") || "index";
+      const filePath = path.join(docsDirectory, `${slug}.mdx`);
+
+      if (!filePath.startsWith(`${docsDirectory}${path.sep}`)) {
+        response.statusCode = 404;
+        return response.end();
+      }
+
+      try {
+        response.setHeader("Content-Type", "text/markdown; charset=utf-8");
+        response.end(await readFile(filePath, "utf8"));
+      } catch {
+        response.statusCode = 404;
+        response.end();
+      }
+    });
+  };
+
+  return {
+    name: "markdown-content-negotiation",
+    configureServer: install,
+    configurePreviewServer: install,
+  };
+}
+
+export default defineConfig({
+  plugins: [markdownNegotiation()],
+});
+```
+
+Use `vite preview` only for local verification. In production, run equivalent middleware in your application server or configure your CDN or edge platform to route requests based on `Accept`. For a static-only deployment, publish generated `.md` files at separate URLs.
+
+## Test the result
+
+Verify both representations at the same URL:
+
+```bash
+curl -i https://example.com/docs/getting-started
+curl -i -H 'Accept: text/markdown' https://example.com/docs/getting-started
+```
+
+The first response should use `Content-Type: text/html`. The second should use `Content-Type: text/markdown`. If a CDN or proxy caches the responses, confirm that its cache key includes `Accept` or that the response includes `Vary: Accept`.
+
+Also test negotiation at both `/docs` and a nested docs URL:
+
+| `Accept` header | Expected response |
+| --- | --- |
+| Missing, `*/*`, or `text/*` | HTML |
+| `text/markdown` or `Text/Markdown` | Markdown |
+| `text/html;q=1, text/markdown;q=0.1` | HTML |
+| `text/html;q=0.1, text/markdown;q=1` | Markdown |
+| `text/html, text/markdown;q=0` | HTML |
+| `text/markdown;q=0.000, text/html` | HTML |
+| `text/markdown; charset=utf-8; q=0, text/html` | HTML |
+| `text/markdown;q=0.5` | Markdown |
+| `text/html;q=0, text/markdown` | Markdown |
+
+Repeat requests in HTML → Markdown → HTML → Markdown order to check that the deployed cache keeps the representations separate. Check that Markdown responses contain source MDX, including frontmatter, rather than rendered HTML. On a static-only deployment, expect HTML at the docs URL for every header above.
+
+## Related resources
+
+- [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/)
+- [Negotiator API](https://github.com/jshttp/negotiator#api)
+- [Next.js rewrites](https://nextjs.org/docs/15/app/api-reference/config/next-config-js/rewrites)
+- [Astro middleware](https://docs.astro.build/en/guides/middleware/)
+- [Astro endpoints](https://docs.astro.build/en/guides/endpoints/)
+- [Vite plugin API](https://vite.dev/guide/api-plugin)

content/navigation-bar/docs-navigation-bar.json

@@ -917,6 +917,11 @@
               "slug": "content/docs/editing/referenced-files.mdx",
               "_template": "item"
             },
+            {
+              "slug": "content/docs/guides/markdown-content-negotiation.mdx",
+              "title": "Serve Markdown to AI Agents",
+              "_template": "item"
+            },
             {
               "slug": "content/docs/guides/react-data-fetching.mdx",
               "title": "Querying TinaCMS Content at Runtime",