fix(measurement): cache robots.txt for 0s, name the property, stitch to gtag

Four measurement defects, three of them silent.

ROBOTS.TXT WAS CACHED FOR FOUR HOURS. Measured live on both zones:
`cache-control: public, max-age=14400, must-revalidate`. That is Cloudflare Pages'
default, not ours — the string 14400 appears in this repo only inside two
explanatory comments, and the built _headers had no /robots.txt rule at all, so no
zone Cache Rule was involved either. robots.txt is the one file where being wrong
for four hours can cost a crawl of everything: a bad Disallow, a missing Allow for a
new AI crawler, a corrected Sitemap line. Now max-age=0, must-revalidate — the edge
copy and the ETag are kept, the 304s already worked, every fetch just asks first.
Same for /llms.txt, /llms-full.txt and the mirrors: they are the ingestion surface,
regenerated on every deploy, and an agent holding a four-hour-old index is following
links to content that may have moved. Consolidated to one block per path so this
does not depend on Cloudflare merging rules that match the same URL.

`mp=` IN x-agent-analytics. The hardest question to answer from outside was "events
are being sent, but to WHICH property?" — and it is not academic: the two editions
shared one property by accident until 2026-08-02, with imqueue.org's traffic landing
in the one named imqueue.com. A repo cannot tell you which property owns an id, which
is exactly why the id lives in the deployment and its value now appears in the
header. G-… ids are public. The api_secret is never named, only its presence.

GTAG STITCHING. The edge used a salted IP+UA digest; gtag uses its own `_ga` client
id. Both are fine identifiers and they are different ones — so a person reading three
pages was one user to gtag and a second, unrelated user to this middleware, and no
per-user metric in the property was true for anyone appearing in both streams. The
`_ga` cookie is now read when it is already there. Nothing is created, no consent
state is touched, and a crawler never borrows one even if a cookie is presented —
asserted, because merging GPTBot into a real user's history would corrupt exactly the
numbers this exists to make true.

probe:agent-analytics WAS BROKEN AND ITS PREMISE WAS BACKWARDS.
  * `buildEvent(...).events[0]` — buildEvent became async when the visitor digest was
    added, so this read `.events` off a promise and the probe threw before sending
    anything.
  * It REFUSED to run against the site's own property, to keep crawler events out of
    the human numbers. Rule 1 reversed that on 2026-08-03: both streams share one
    property on purpose and the separation is the srv_page_view event name, since
    GA4's `Views` counts only page_view. The refusal was rejecting the only supported
    configuration. It is now the inverse — a non-site id is probably a stale copy —
    and the closing advice says to look for srv_page_view, not page_view.
  * A fourth case added: a browser arriving from a Perplexity answer, which is the
    outcome the whole programme exists to produce and the one row nobody had seen
    arrive. Verified against GA4's validation endpoint: four payloads, empty
    validationMessages, `ai=perplexity` on the fourth.

npm test green, 11 analytics checks.

Author: Mikhus

lib/agent-analytics.js

@@ -385,6 +385,36 @@ export function classifyKind(found, surface, browser = false, botish = false) {
   return AGENT_ONLY.has(surface) ? 'assistant.other' : 'unknown';
 }
 
+/**
+ * GA4's own client id for this browser, from the `_ga` cookie, or null.
+ *
+ * gtag writes `_ga=GA1.1.<client_id>.<timestamp>` (the first two labels are the
+ * cookie's own version and domain-depth, not part of the id). Handing that same
+ * value to the Measurement Protocol is what makes a server-sent event and a
+ * gtag-sent event belong to the SAME GA4 user.
+ *
+ * Without it they never do. The edge was using a salted IP+UA digest, which is a
+ * perfectly good pseudonymous id and a completely different one — so a person
+ * reading three pages appeared as one user to gtag and a second, unrelated user to
+ * this middleware, and no per-user metric in the property was true for anyone who
+ * appeared in both streams.
+ *
+ * Falls back to the digest when there is no cookie, which is the common case and
+ * the whole point: a crawler has no cookie, and a visitor who declined consent has
+ * none either. Nothing here CREATES a cookie or changes consent — it reads one that
+ * gtag already set, and only when it is there.
+ *
+ * @param {string|null} cookieHeader The raw Cookie header.
+ * @returns {string|null} GA4's client id, or null.
+ */
+export function ga4ClientId(cookieHeader) {
+  if (!cookieHeader) return null;
+
+  const match = /(?:^|;\s*)_ga=GA\d+\.\d+\.(\d+\.\d+)/.exec(cookieHeader);
+
+  return match ? match[1] : null;
+}
+
 /**
  * Does this request look like a browser fetching a DOCUMENT?
  *
@@ -483,7 +513,7 @@ export function classifySurface(pathname) {
  * (`page_referrer` needs nothing: GA4 reads it natively.)
  */
 export async function buildEvent({
-  url, userAgent, status, edition, ip, salt, referrer = null,
+  url, userAgent, status, edition, ip, salt, referrer = null, gaClientId = null,
   isDocument = false, now = Date.now(),
 }) {
   const seen = describe({ url, userAgent, isDocument });
@@ -511,7 +541,10 @@ export async function buildEvent({
   const session = `${visitor ? visitor.slice(0, 16) : slug(crawler)}-${Math.floor(now / 1800000)}`;
 
   return {
-    client_id: visitor || `${slug(operator)}.${slug(crawler)}`,
+    // gtag's own client id when the browser has one, so the two streams describe
+    // the same user; otherwise the visitor digest for a person, or a stable family
+    // label for a crawler. See ga4ClientId().
+    client_id: (browser && gaClientId) || visitor || `${slug(operator)}.${slug(crawler)}`,
     events: [{
       // NOT page_view. gtag owns that name, and GA4's `Views` metric counts only
       // page_view/screen_view — so a browser that consents is reported twice, once by
@@ -620,9 +653,17 @@ export function headerNote({ request, env, url, status, edition }) {
   // waiting for a real one and then trusting a dashboard.
   const { ai_source } = classifyReferrer(request.headers.get('referer'), url);
 
+  // mp= names WHICH GA4 property this deployment is feeding. The single hardest
+  // question to answer from outside was "the events are being sent, but to which
+  // property?" — and it is not academic here: the two editions shared one property
+  // by accident until 2026-08-02, with imqueue.org's traffic landing in the one
+  // named imqueue.com. A repo cannot tell you which property owns an id, which is
+  // exactly why the id belongs in the deployment and its VALUE belongs in this
+  // header. G-… ids are public (they ship in every page's source), so there is
+  // nothing to redact. The api_secret is never named here, only its presence.
   return `sent kind=${seen.kind} crawler=${seen.crawler} surface=${seen.surface} ` +
     `status=${status} edition=${edition} ai=${ai_source} ` +
-    `salt=${salted ? 'set' : 'MISSING'}`;
+    `mp=${env.GA4_MP_MEASUREMENT_ID} salt=${salted ? 'set' : 'MISSING'}`;
 }
 
 /**
@@ -657,6 +698,8 @@ export async function trackRequest({ request, env, url, status, edition }) {
     // buildEvent, never stored — like the IP and the UA, it only leaves here as a
     // hostname and a brand slug.
     referrer: request.headers.get('referer'),
+    // Read, never written, and only used when gtag already set it.
+    gaClientId: ga4ClientId(request.headers.get('cookie')),
     isDocument: isDocumentRequest(request),
     status,
     edition,

scripts/check-agent-analytics.js

@@ -340,6 +340,55 @@ async function main() {
   }
   ok('AI referrals are attributed by brand, and our own navigation is not one');
 
+  // --- 12. gtag stitching ---------------------------------------------------
+  // The edge used a salted IP+UA digest and gtag uses its own `_ga` client id, so a
+  // person appearing in both streams was two unrelated users and no per-user metric
+  // in the property was true for them. Reading gtag's id when it is already there
+  // fixes it; the three assertions are that it is read, that a crawler never
+  // borrows one, and that nothing is invented when there is no cookie.
+  {
+    const { ga4ClientId } = await import('../lib/agent-analytics.js');
+
+    assert.strictEqual(
+      ga4ClientId('foo=1; _ga=GA1.1.1234567890.1699999999; _ga_ABC=x'),
+      '1234567890.1699999999',
+      'the _ga cookie\'s client id must be extracted without its version/depth labels',
+    );
+    assert.strictEqual(ga4ClientId(null), null, 'no cookie header must give null');
+    assert.strictEqual(ga4ClientId('_ga=nonsense'), null, 'a malformed _ga must give null');
+
+    const browserEvent = await buildEvent({
+      url: new URL('https://imqueue.org/docs/'),
+      userAgent: BROWSER, ip: '203.0.113.9', salt: 'test-salt', isDocument: true,
+      gaClientId: '1234567890.1699999999', status: 200, edition: 'org',
+    });
+
+    assert.strictEqual(browserEvent.client_id, '1234567890.1699999999',
+      'a browser with a _ga cookie must be reported under gtag\'s client id');
+
+    // A crawler has no cookie in practice, but if one is ever presented it must not
+    // be adopted: GPTBot is not a person, and merging it into a real user's history
+    // would corrupt exactly the numbers this exists to make true.
+    const crawlerEvent = await buildEvent({
+      url: new URL('https://imqueue.org/llms.txt'),
+      userAgent: GPTBOT, ip: '203.0.113.9', salt: 'test-salt',
+      gaClientId: '1234567890.1699999999', status: 200, edition: 'org',
+    });
+
+    assert.strictEqual(crawlerEvent.client_id, 'openai.gptbot',
+      'a crawler must keep its family label even if a _ga cookie is present');
+
+    const noCookie = await buildEvent({
+      url: new URL('https://imqueue.org/docs/'),
+      userAgent: BROWSER, ip: '203.0.113.9', salt: 'test-salt', isDocument: true,
+      status: 200, edition: 'org',
+    });
+
+    assert.ok(noCookie.client_id && noCookie.client_id !== '1234567890.1699999999',
+      'with no cookie the visitor digest must still be used');
+  }
+  ok('server events share gtag\'s client id when there is one, and never borrow one');
+
   console.log(`\nAll ${checks} agent-analytics checks passed.`);
 }
 

scripts/probe-agent-analytics.js

@@ -9,8 +9,10 @@
 //
 // This is that other half, and it is deliberately opt-in:
 //
-//   export GA4_MP_MEASUREMENT_ID='G-…'   # the AGENT property, not the site's
+//   export GA4_MP_MEASUREMENT_ID='G-…'   # the site's own property for this edition
 //   export GA4_MP_API_SECRET='…'         # from that property's own data stream
+//
+// One property, not two: see the note above sitePropertyIds().
 //   npm run probe:agent-analytics
 //
 // It calls the same lib/agent-analytics.js the Cloudflare middleware calls, so a pass
@@ -26,11 +28,20 @@ const path = require('node:path');
 
 const ROOT = path.resolve(__dirname, '..');
 
-// The site's own GA4 property, read from where it actually lives rather than copied
-// here. Sending crawler events to it is the one mistake with no undo — GA4 has no
-// selective delete — so the probe refuses by default. Deriving it means this cannot
-// go stale when the site's id changes; if the read fails, the check is skipped rather
-// than guessed at.
+// The site's own GA4 property ids, read from where they actually live.
+//
+// THIS USED TO BE A REFUSAL. The probe would exit 1 if pointed at the site's own
+// property, on the grounds that crawler events must not mix into the human numbers
+// and GA4 has no selective delete. That reasoning was superseded on 2026-08-03 by
+// rule 1 in lib/agent-analytics.js: both streams land in the SAME property on
+// purpose, and the separation is done by EVENT NAME — gtag sends page_view,
+// everything server-side sends srv_page_view, and GA4's `Views` metric counts only
+// the former, so the two can never be summed by accident.
+//
+// So the site's property is now the CORRECT target and the refusal was rejecting the
+// only supported configuration. What remains is worth saying out loud, because a
+// probe writes real rows into real reports: it names which property it is about to
+// write to, and whether that is the site's.
 function sitePropertyIds() {
   try {
     const cfg = fs.readFileSync(path.join(ROOT, 'eleventy.config.js'), 'utf8');
@@ -41,10 +52,20 @@ function sitePropertyIds() {
   }
 }
 
+// The fourth case is a BROWSER arriving from a Perplexity answer, which is the
+// outcome the whole programme exists to produce and the one row nobody had ever seen
+// arrive. It needs an ip and a salt, because a browser with no salt is dropped by
+// design.
 const CASES = [
   ['GPTBot/1.2 (+https://openai.com/gptbot)',      '/llms.txt',                  200],
   ['ClaudeBot/1.0 (+claudebot@anthropic.com)',     '/tutorial/index.md',         200],
   ['PerplexityBot/1.0',                            '/probe-missing/index.md',    404],
+  [
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0 Safari/537.36',
+    '/compare/',
+    200,
+    { referrer: 'https://www.perplexity.ai/search/imqueue-vs-nats', document: true },
+  ],
 ];
 
 async function main() {
@@ -68,13 +89,19 @@ async function main() {
   }
 
   const siteIds = sitePropertyIds();
+  const isSiteProperty = siteIds.has(env.GA4_MP_MEASUREMENT_ID);
 
-  if (siteIds.has(env.GA4_MP_MEASUREMENT_ID) && !force) {
+  if (!isSiteProperty && siteIds.size && !force) {
+    // The inverse of the old refusal, and the one that is now worth having: an id
+    // that is NOT one of the site's is probably a stale copy from before the
+    // one-property decision, or someone else's property entirely. Sending there
+    // produces a clean run and no data where anyone will look for it.
     console.error(
-      `Refusing: ${env.GA4_MP_MEASUREMENT_ID} is the property the SITE reports to `
-      + '(eleventy.config.js).\nCrawler events there mix into the numbers that describe '
-      + 'humans, and GA4 has no\nselective delete. Point this at the agent property, '
-      + 'or pass --force if you mean it.',
+      `${env.GA4_MP_MEASUREMENT_ID} is not a property this site reports to.\n`
+      + `  eleventy.config.js names: ${[...siteIds].join(', ')}\n\n`
+      + 'Both streams share one property by design (rule 1 in lib/agent-analytics.js);\n'
+      + 'the separation is the srv_page_view event name, not a second property. If you\n'
+      + 'do mean to write somewhere else, pass --force.',
     );
     process.exit(1);
   }
@@ -84,28 +111,51 @@ async function main() {
 
   console.log(`property: ${env.GA4_MP_MEASUREMENT_ID}${debug ? '  (GA4_MP_DEBUG — validating, NOT recording)' : ''}\n`);
 
-  for (const [userAgent, pathname, status] of CASES) {
+  for (const [userAgent, pathname, status, extra = {}] of CASES) {
     const url = new URL(`https://imqueue.org${pathname}`);
-    const { params } = buildEvent({ url, userAgent, status, edition: 'org' }).events[0];
+    // AWAIT. buildEvent became async when the visitor digest was added (crypto.subtle
+    // returns a promise), and this line was reading `.events` off the promise —
+    // so `npm run probe:agent-analytics` threw before sending anything at all.
+    const headers = {
+      'user-agent': userAgent,
+      ...(extra.referrer ? { referer: extra.referrer } : {}),
+      ...(extra.document ? { 'sec-fetch-dest': 'document' } : {}),
+    };
+    const probeEnv = extra.document
+      ? { ...env, VISITOR_SALT: env.VISITOR_SALT || 'probe-salt' }
+      : env;
+    const built = await buildEvent({
+      url, userAgent, status, edition: 'org',
+      referrer: extra.referrer || null,
+      isDocument: Boolean(extra.document),
+      ip: extra.document ? '203.0.113.9' : null,
+      salt: probeEnv.VISITOR_SALT,
+    });
+    const { params } = built.events[0];
 
     await trackRequest({
-      request: { headers: { get: (h) => (h.toLowerCase() === 'user-agent' ? userAgent : null) } },
-      env,
+      request: { headers: { get: (h) => headers[h.toLowerCase()] ?? null } },
+      env: probeEnv,
       url,
       status,
       edition: 'org',
     });
 
-    console.log(`  sent  ${params.crawler.padEnd(15)} ${params.surface.padEnd(16)} ${params.status}  ${pathname}`);
+    console.log(
+      `  sent  ${params.crawler.padEnd(15)} ${params.surface.padEnd(16)} ${params.status}`
+      + `  ai=${(params.ai_source || '-').padEnd(11)} ${pathname}`,
+    );
   }
 
   console.log(
     debug
       ? '\nValidation output is above: an empty validationMessages array means the payload'
         + '\nis well-formed. Nothing was recorded — unset GA4_MP_DEBUG and re-run to send.'
       : '\nDone. GA4 answers 204 to valid and invalid hits alike, so no error here proves'
-        + '\nnothing — the proof is the data. Open GA4 → Reports → Realtime on that property;'
-        + '\nthree page_view events should appear within a minute or two, including the 404.'
+        + '\nnothing — the proof is the data. Open GA4 → Reports → Realtime on that property'
+        + '\nand look for srv_page_view — NOT page_view, which is gtag\'s and is what the'
+        + '\n`Views` metric counts. Three of them should appear within a minute or two,'
+        + '\nincluding the 404.'
         + '\nIf Realtime stays empty: wrong property, a secret from a different stream, or'
         + '\nGA4 filtering. Re-run with GA4_MP_DEBUG=1 to see what it says about the payload.',
   );

src/headers.liquid

@@ -12,14 +12,6 @@ eleventyExcludeFromCollections: true
 # while leaving them fully fetchable by AI crawlers and agents — robots.txt
 # allows them, noindex only stops SERP indexing, not retrieval.
 
-/*.md
-  X-Robots-Tag: noindex
-
-/llms.txt
-  X-Robots-Tag: noindex
-
-/llms-full.txt
-  X-Robots-Tag: noindex
 
 # ---- caching -------------------------------------------------------------
 # Fonts are the one asset class safe to cache forever unasked: the filenames come
@@ -44,6 +36,43 @@ eleventyExcludeFromCollections: true
 /js/*
   Cache-Control: public, max-age=31536000, immutable
 
+# robots.txt and the agent-facing index files must be CHEAP TO CORRECT.
+#
+# Measured live on both zones 2026-08-03:
+#
+#   cache-control: public, max-age=14400, must-revalidate
+#   cf-cache-status: REVALIDATED
+#
+# That is Cloudflare Pages' default, not ours — the string 14400 appears in this
+# repo only in the two comments above, and no zone Cache Rule is involved (checked:
+# the built _headers had no /robots.txt entry at all). The consequence is that a
+# wrong Disallow, a missing Allow for a new AI crawler, or a corrected Sitemap line
+# takes FOUR HOURS to reach the crawler that needs it — and robots.txt is the one
+# file on the site where being wrong for four hours can cost a crawl of everything.
+#
+# Same reasoning for /llms.txt, /llms-full.txt and the mirrors: they are the
+# ingestion surface, they are regenerated on every deploy, and an agent holding a
+# four-hour-old copy of the index is following links to content that may have moved.
+# `must-revalidate` with max-age=0 keeps the edge copy and the ETag — the 304s
+# already work — while making every fetch ask first.
+# One block per path rather than two, so this does not depend on Cloudflare merging
+# rules that match the same URL — the noindex and the Cache-Control belong to the
+# same decision anyway.
+/robots.txt
+  Cache-Control: public, max-age=0, must-revalidate
+
+/llms.txt
+  X-Robots-Tag: noindex
+  Cache-Control: public, max-age=0, must-revalidate
+
+/llms-full.txt
+  X-Robots-Tag: noindex
+  Cache-Control: public, max-age=0, must-revalidate
+
+/*.md
+  X-Robots-Tag: noindex
+  Cache-Control: public, max-age=0, must-revalidate
+
 # ---- security ------------------------------------------------------------
 # Not ranking factors, but cheap and worth having. Left out on purpose:
 #   * Strict-Transport-Security — a year-long max-age is effectively