Count the bare Grok and xAI-Bot user agents, not only prerender them - #442
Merged
Conversation
The nginx $is_bot map has served ~*grok and ~*xai since the prerender path landed, but AI_AGENTS — the taxonomy behind bot_fetch and asset_fetch — knew only grok-deepsearch, grokbot and xai-grok. xAI's fetcher sending plain "Grok/1.0" (seen on anyplot, 2026-08-19) or "xAI-Bot" (seen here, 2026-08-28) got the page and left no event. Two bare entries close the gap, last among the xAI patterns so the specific ones keep winning; two test cases pin them. Mirrored verbatim from anyplot #10808 — the taxonomy is kept identical between the two sites by decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016joUzstV3siE3WqJtbcych
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016joUzstV3siE3WqJtbcych
There was a problem hiding this comment.
🟢 Approval recommended
The change is low risk, narrowly scoped, and pinned by focused unit tests with an accompanying changelog entry.
Pull request overview
This PR closes a bot-analytics classification gap by teaching the server-side AI_AGENTS taxonomy to recognize the bare xAI user-agent tokens (Grok/* and xAI-Bot/*) that were already being prerendered by nginx, ensuring they are also counted via Plausible bot events.
Changes:
- Add two new substring patterns (
"grok"and"xai") toapi/analytics.py::AI_AGENTSso bare xAI fetchers are classified as("grok", "user_directed"). - Add two parametrized test cases to pin the new classifications.
- Add a
CHANGELOG.mdentry under Fixed describing the behavioral impact and doctrine (mirrored taxonomy).
File summaries
| File | Description |
|---|---|
api/analytics.py |
Extends AI_AGENTS with bare grok/xai patterns so prerendered xAI fetchers are also counted in bot analytics. |
tests/test_api_analytics.py |
Adds coverage to assert Grok/1.0 and xAI-Bot/1.0 are classified as grok/user_directed. |
CHANGELOG.md |
Documents the fix under [Unreleased] -> Fixed per project convention. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Copilot review nit on #442: the parenthetical read as if it applied to both test cases; it describes only the xAI-Bot one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016joUzstV3siE3WqJtbcych
There was a problem hiding this comment.
🟢 Approval recommended
The change is small, well-scoped, and test-pinned, with only a minor comment-clarity nit remaining.
Review details
Suppressed comments (1)
tests/test_api_analytics.py:52
- This comment is a bit misleading: it reads as if both "Grok" and "xAI-Bot" have “no grok substring”, but only the xAI-Bot UA lacks a
groksubstring. Clarifying avoids confusion when future readers compare this to the actual matching behavior.
# The bare xAI tokens the nginx map serves: plain "Grok" and "xAI-Bot"
# (no grok substring) were prerendered but not counted until these
# patterns existed — mirrored from anyplot #10808.
- Files reviewed: 3/3 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🟢 Approval recommended
The new UA patterns and tests align with the existing substring-first matching contract and the change is narrowly scoped with appropriate changelog and regression coverage.
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 0 new
- Review effort level: Lite
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The nginx
$is_botmap has served~*grokand~*xaisince the prerender path landed (#433, #440), butAI_AGENTS— the taxonomy behindbot_fetchandasset_fetch— knew onlygrok-deepsearch,grokbotandxai-grok. So xAI's fetcher sending plainGrok/1.0(seen on anyplot, 2026-08-19) orxAI-Bot(seen here, 2026-08-28) got the prerendered page and left no event onbots.kurrentschrift.ink.Two bare entries —
("grok", "grok", "user_directed"),("xai", "grok", "user_directed")— close the gap, placed last among the xAI patterns so the specific ones keep winning. Two test cases pin them.Doctrine
AI_AGENTSis kept verbatim-identical with anyplot by decision (frontend-stack.md §6, crawler-richtlinie.md §3); anyplot merged the same two entries in #10808 today. The entry lists now diff empty again (comments aside). No new term, no glossary entry; CHANGELOG under Fixed.Verification
uv run --extra test pytest— 1756 passed, 17 skippeduv run --extra dev ruff check .+ruff format --check .— cleandiffof the twoAI_AGENTStuples (comments stripped) against~/projects/anyplot/api/analytics.py— identical