feat: LLM discoverability sync from kurrentschrift

[MarkusNeusinger]

Summary

• Assistants can find the machine guide from wherever they land. The sister project kurrentschrift learned from two assistant protocols (2026-08-28, its PRs [box-basic] highcharts implementation #439/[area-basic] letsplot implementation #440) that assistants' fetch tools often allow only URLs that already appeared verbatim in fetched content — a relative link, a …-elided path or the bare /llms.txt convention leaves the API unreachable. So the guide and one complete, callable example per surface now stand at every place an agent actually reads: the SPA shell (<link rel="alternate" href="/llms.txt"> + a <noscript> block with absolute URLs — the fallback for every client the nginx map does not recognise), both robots.txt files (two terse lines at the very top of the site file, a block beside Sitemap:, the API host's comment head), every bot page's nav, and every implementation page (its own code URL spelled out + a visible <pre><code class="language-json"> retrieval record beside the JSON-LD, because HTML-to-Markdown converters drop <script> and keep <pre>). Guessed paths resolve instead of failing: /.well-known/llms.txt → 302 (it soft-404'd with the shell, verified live), api.anyplot.ai/llms.txt → 302 to the site file (it 404'd).
• Two correctness fixes. (1) bot_fetch on the prerender path was very likely mostly dropped by Plausible: on the hop Cloud Run app → Cloudflare → API, cf-connecting-ip is the app container's Google egress address — exactly the "hosting provider IP" Plausible discards — and visitor_ip preferred it over the forwarded list where the crawler stands. kurrentschrift measured one counted read in twenty with probe events; anyplot has the same topology. visitor_ip now reads the leftmost valid x-forwarded-for first, and @seo_proxy forwards the crawler explicitly. (2) The use=reference Content-Signal token was invalid (vocabulary is exactly search/ai-input/ai-train; a strict parser may discard the whole line) — dropped from both groups.
• Shared-by-decision blocks stay identical. The nginx $is_bot map gains ~*xai (a xAI-Bot UA got the empty shell — verified live) with kurrentschrift's exact comment, so the block stays byte-identical (checked with diff against kurrentschrift origin/main); AI_AGENTS gains the bare grok and xai tokens the map already served but never counted (needs mirroring in kurrentschrift — Todoist task filed). New asset_fetch event on bots.anyplot.ai records which implementation's code / which spec's detail assistants fetch through the API (cache misses — those routes stay edge-cached for the SPA; documented in plausible.md). Guards: new app/src/routes/seoCoverage.test.ts pins llms.txt ↔ route registry ↔ robots.txt ↔ shell (llms.txt gains the missing /map line it surfaced); bot-serving-check adds the .well-known redirect, robots/sitemap bypasses, the site card, a real 404, xAI-Bot and a deep-route SPA control. Doctrine: docs/reference/seo.md "Discoverability for assistants".

Plan

N/A — assessment in session; source findings: kurrentschrift docs/reference/crawler-richtlinie.md §3 "Auffindbarkeit" and its PRs #428, #435, #439, #440.

Test plan

• uv run pytest tests/unit tests/integration — 1713 + 67 passed; ruff check/format --check clean; mypy api core clean
• cd app && yarn lint && yarn fm:check && yarn type-check && yarn test && yarn build — 623 tests, build clean; dist/index.html carries the head link and the <noscript> block with absolute URLs
• Local API smoke (uvicorn against the shared DB, reads only): /robots.txt carries Content-Signal + pointers, /llms.txt → 302 https://anyplot.ai/llms.txt, bot nav ends with llms.txt · llms-full.txt, /seo-proxy/box-basic/python/plotly renders the retrieval record with real render/interactive URLs and quality_score
• app/public/robots.txt parsed with urllib.robotparser: Bytespider denied everywhere, /debug and /interactive denied for *, /llms.txt allowed — the two new top comment lines change nothing
• bot-serving-check.yml run block extracted and executed locally against the Cloud Run origin: 29 checks OK; exactly the two checks that need the new nginx config (xAI-Bot → prerender, .well-known/llms.txt → redirect) fail until the app deploy lands — expected: the daily run will be red until the frontend Cloud Build after merge, then green
• After merge: watch the app and API Cloud Builds, then curl -sI https://anyplot.ai/.well-known/llms.txt (302 → /llms.txt), curl -s https://api.anyplot.ai/robots.txt (pointer lines), curl -s -A "xAI-Bot/1.0" https://anyplot.ai/scatter-basic | grep -o '<title>[^<]*' (per-route title), and dispatch Bot Serving Check once by hand
• After merge: register the asset_fetch goal and the asset, spec, library properties on bots.anyplot.ai in Plausible (Todoist task); watch whether the 06:23 UTC bot-serving run now shows ~25 bot_fetch events on the bot site (that is the live proof of the visitor-IP fix)

[Copilot]

🟢 Approval recommended

The changes are internally consistent (tests + docs + nginx/app/api updates align) and the added guards substantially reduce drift/regression risk without introducing confirmed correctness issues.

Pull request overview

This PR improves assistant/LLM discoverability of anyplot’s machine-facing surfaces (llms.txt, OpenAPI, MCP) across the SPA shell, robots.txt, bot-rendered pages, and implementation pages, while also fixing analytics correctness for crawler traffic in the Cloud Run → Cloudflare → API topology.

Changes:

• Add redundant, absolute, “verbatim fetchable” pointers to llms.txt/llms-full.txt/OpenAPI/MCP across HTML shell, robots.txt, bot nav, and implementation pages (including a visible JSON retrieval record).
• Fix Plausible bot event attribution by preferring the leftmost valid X-Forwarded-For entry for analytics (visitor_ip), and add explicit forwarding in nginx for the prerender hop.
• Add “drift guard” tests and monitoring updates (Vitest coverage coupling, expanded bot-serving-check) plus new asset_fetch analytics for single-asset API reads.

File summaries

File	Description
tests/unit/api/test_seo_helpers.py	Updates assertions to keep the new retrieval-record <pre><code class="language-json"> while still excluding source-code blocks when code is absent.
tests/unit/api/test_routers.py	Adds coverage for API-host robots.txt pointers, /llms.txt redirect behavior, bot-nav absolute links, and asset_fetch vs navigation/machine-file exclusions.
tests/unit/api/test_analytics.py	Extends agent detection (xAI/Grok tokens), verifies visitor_ip ordering, and adds tests for classify_asset + track_asset_fetch.
docs/reference/seo.md	Documents the discoverability strategy and the corrected Content-Signal vocabulary and API-host redirect behavior.
docs/reference/plausible.md	Documents new asset_fetch event and the visitor-IP forwarding rationale/caveat (cache misses).
CHANGELOG.md	Adds [Unreleased] entries covering the discoverability sync and analytics fixes.
app/src/routes/seoCoverage.test.ts	Adds a Vitest drift-guard coupling paths.ts ↔ llms.txt ↔ robots.txt ↔ index.html shell fallback.
app/public/robots.txt	Adds top-of-file pointers to llms/OpenAPI and removes invalid use=reference token; reiterates machine-guide block near sitemap.
app/public/llms.txt	Adds missing /map route link to keep machine guide aligned with public routes.
app/nginx.conf	Adds ~*xai bot-map entry, forwards X-Forwarded-For on @seo_proxy, and adds /.well-known/llms.txt redirect.
app/index.html	Adds <link rel="alternate" href="/llms.txt"> plus a <noscript> absolute-URL fallback with complete example URLs.
api/routers/seo.py	Adds full-URL machine links to bot nav, adds visible per-implementation retrieval record, updates API robots.txt, and redirects API-host /llms.txt to site guide.
api/request_context.py	Changes visitor_ip to prefer leftmost valid X-Forwarded-For before cf-connecting-ip for Plausible analytics correctness.
api/main.py	Extends middleware to emit asset_fetch for single-asset API reads and ensures /seo-proxy is only bot_fetch.
api/analytics.py	Adds xAI/Grok bare-token detection, introduces classify_asset, and implements track_asset_fetch event emission.
.github/workflows/bot-serving-check.yml	Expands synthetic monitoring (xAI-Bot UA, .well-known/llms.txt redirect, robots/sitemap bypasses, og-image served as PNG, deep-route SPA control).

Review details

• Files reviewed: 16/16 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Copilot]

🟢 Approval recommended

The changes are well-covered by targeted unit and drift-guard tests, with only a minor naming clarity nit identified in the middleware tuple unpacking.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

api/main.py:201

• In record_bot_fetch, the unpacked tuple element is named kind, but it actually represents the asset type ("code" / "spec"). Using kind here is easy to confuse with the assistant "kind" (e.g., user_directed, training) used elsewhere in analytics; renaming improves readability and reduces the chance of wiring the wrong value into track_asset_fetch.

    asset = classify_asset(path)
    if asset is not None:
        kind, spec_id, library_id = asset
        track_asset_fetch(request, asset=kind, spec=spec_id, library=library_id, status=response.status_code)

• Files reviewed: 16/16 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟢 Approval recommended

The changes are internally consistent across nginx/app/API/docs, and the new/updated test coverage meaningfully guards the intended discoverability and analytics behavior against drift.

Review details

• Files reviewed: 16/16 changed files
• Comments generated: 0 new
• Review effort level: Lite