Replace llms.txt with Accept: text/markdown content negotiation

[ajwild]

Relations:

• Related PR: Render llms.txt for all pages #1160

Motivation

After reading an Evil Martians blog post, Which AI actually reads your site? Two months of LLM traffic, measured, it became clear that our approach to llms.txt wasn't particularly helpful for agents reading the site. The article found that Claude uses Accept: text/markdown to request markdown files, and ChatGPT is almost exclusively requesting HTML. As Claude is the main tool we're using at Livingdocs, and we can't do much to change the behaviour of ChatGPT, this PR optimises the website for the Accept: text/markdown approach. It does not seem worthwhile keeping the llms.txt approach when we can have a single canonical URL which an agent can share with the user.

Edit: The blog post links to a Checkly blog post, The Current State of Content Negotiation for AI Agents (Feb 2026), which contains more details regarding the requests that AI agents are making.

Changelog

• Replace llms.txt with Accept: text/markdown content negotiation
• Do not serve the .gz and .br files that are generated in the public directory
  • these were being served with the wrong mime type and should be consumed using the Content-Encoding header
  • during local development we can still preview the files using the llms.txt suffix, but they are blocked by nginx in production