Stop the GitHub triage relay amplifying issue spam

[jpr5]

What happened

On 2026-09-10/11 two throwaway accounts (sarahnicholas1327-lgtm, kaylaford203-beep) filed 23 SEO backlink issues on CopilotKit/CopilotKit — ~3.3 KB of marketing prose each, titled variants of "Why 1Rank.app Belongs in Your SEO Growth Stack", with a dozen-odd links to 1rank.app / zagfro.com. All are now closed.

Outpost's GitHub triage relay forwarded every one of those bodies verbatim into search-docs + search-code on mcp.copilotkit.ai, within ~4 seconds of issue creation, and then answered the issue publicly as copilotkit-outpost-bot[bot]. Confirmed against Pathfinder's query_log:

GH issue	created (UTC)	MCP query (UTC)
CopilotKit#7091	18:56:05 2026-09-11	18:56:09 / 18:56:10
CopilotKit#7039	18:33:08 2026-09-10	18:33:11 / 18:33:12

46 query_log rows, one IP (152.55.178.126, Railway), UA node, one search-docs + one search-code per issue. Those rows then ranked in Pathfinder's Top Queries panel, went into the weekly search report's Notion table, and were eligible as cluster representatives in the monthly gap-analysis LLM prompt — 3.3 KB of third-party marketing copy with live backlinks, published to Notion via a model prompt. That last one is the part worth fixing properly.

The spammer wrote once and got a machine-generated reply, 46 rows of retrieval traffic, and a foothold in three of our reports for free. That is amplification, and it is ours.

The change

Three levers. Only the first is content-aware.

1. Drop link-spam in the issues.opened webhook, before a ticket exists

apps/github-app/src/lib/spam-filter.ts. The gate sits ahead of InboundHandler deliberately: creating the ticket is what enqueues the AI job, and that job is what does the relaying and the posting. A filter further down would already have paid for it.

All four conditions must hold:

• author is not OWNER/MEMBER/COLLABORATOR/CONTRIBUTOR (a spam account is NONE by construction; a maintainer or prior contributor can never be silenced by this gate)
• body ≥ 1500 chars
• body contains no fenced code block
• ≥ 5 links point at one third-party host (GitHub, our docs and localhost don't count)

Measured against the full issue history of CopilotKit/CopilotKit — 1,264 non-spam issues plus all 23 known spam issues:

rule	spam	legit
len >= 3000 AND urls >= 6	23/23	15/1264
+ untrusted author	23/23	14/1264
this rule	23/23	0/1264

A length-and-URL rule alone is not good enough: it eats real bug reports (#2667 is 40 KB with 16 URLs, #3510 is 13 KB with 26). The two terms that buy the separation describe an advertisement rather than a report — no code in it, and the links point over and over at the same third-party host. A stack trace or a repro has code; a backlink campaign does not, because the links are the payload.

Honest about scope: this is fitted on one campaign and one repo's history. It is a discriminator, not a general spam classifier.

2. Cap the MCP search query (the durable one)

capQuery() in packages/outpost/ai/src/pathfinder.ts, default 1000 chars, PATHFINDER_MAX_QUERY_CHARS to override. Content-independent, so it bounds the next campaign whatever it advertises.

A retrieval query is an embedding input, not a transcript — the generator still sees the full body, only the search string is capped. Context from 7 days of query_log: the longest query from any client that is not a relay is 194 characters; these spam bodies ran 3,304–3,631 and scored 0.33–0.43 cosine for it, so the long tail was buying nothing. The head is kept (that is where the question lives) and the cut pulls back to a word boundary if one is in the last 15%.

3. X-Pathfinder-Source: outpost on the MCP initialize request

So this relay's traffic is attributable and, more usefully, excludable downstream. Pathfinder captures this header once, at session initialisation, and closes over it for the session's lifetime — setting it on an individual tools/call silently does nothing, so it rides on initialize and there is a test asserting exactly that asymmetry.

Value is outpost (env PATHFINDER_SOURCE) rather than github-triage: this client also serves the Discord, Slack and Teams bots, and all of it is relay traffic that should be excluded together. Tagging a Discord query "github-triage" would be a lie.

Red / green

Harness (not committed): a local HTTP server speaking MCP Streamable-HTTP, the real PathfinderClient pointed at it, fed the real bodies of all 23 spam issues and all 1,264 other issues pulled live from the GitHub API.

RED (this branch, fix reverted):

=== A. WIRE CAPTURE (real spam issue #7052, body 3631 chars) ===
X-Pathfinder-Source header at initialize: <ABSENT>
  tool=search-docs query_chars=3631 head="Mobile-First SEO for Small Businesses That Want to Be Found\n\nMany firs"
  tool=search-code query_chars=3631 head="Mobile-First SEO for Small Businesses That Want to Be Found\n\nMany firs"
  MAX query chars on the wire: 3631

=== B. SKIP DECISION (no spam-filter module — relays everything) ===
  spam issues skipped:  0/23
  legit issues skipped: 0/1264

=== C. POSITIVE CONTROL (real legit issue #2449, body 61801 chars, author_association=NONE) ===
  skipped by spam filter: false
  queries relayed: 1 (chars=61801)

VERDICT: RED

GREEN (this branch):

=== A. WIRE CAPTURE (real spam issue #7052, body 3631 chars) ===
X-Pathfinder-Source header at initialize: outpost
  tool=search-docs query_chars=999 head="Mobile-First SEO for Small Businesses That Want to Be Found\n\nMany firs"
  tool=search-code query_chars=999 head="Mobile-First SEO for Small Businesses That Want to Be Found\n\nMany firs"
  MAX query chars on the wire: 999

=== B. SKIP DECISION (spam-filter present) ===
  spam issues skipped:  23/23
  legit issues skipped: 0/1264

=== C. POSITIVE CONTROL (real legit issue #2449, body 61801 chars, author_association=NONE) ===
  skipped by spam filter: false
  queries relayed: 1 (chars=989)
  head="### ♻️ Reproduction Steps\n\nWe have a simple action : TodoWrite accepti"

VERDICT: GREEN

Note C: a 62 KB legitimate bug report from a first-time reporter is still relayed and still answered — it is only its search string that shrinks.

Mutation-tested

The new tests are not vacuous. Neutering isLikelySpamIssue to return false fails 3 of them; neutering capQuery and dropping the source header fails 6 in pathfinder.test.ts.

Local gate

pnpm lint, pnpm typecheck, pnpm build, pnpm test (2,287 passed), and prettier --check on every changed file — all green. One of the three commits is pure prettier: these files were already non-conforming on main, and CI format-checks every changed file, so the pre-existing violations had to go with them.

Not touched

apps/github-app/railway.toml and the service's Railway config path are untouched, as is anything under deploy/. No env var is required for this to work — both new knobs have defaults.

Follow-up, not in this PR

A parallel change adds the Pathfinder-side exclusion so relay traffic (now identifiable by X-Pathfinder-Source) stops ranking in Top Queries, the weekly report and the gap-analysis prompt. That is the belt to this PR's braces: it protects those surfaces regardless of what the relay forwards.