fix(seo): unblock /mcp in robots to clear "indexed though blocked"

[vreshch]

Why

Google Search Console reports "Indexed, though blocked by robots.txt" for agentage.io. Blocking a path in robots.txt is what causes that warning: Google can't crawl the URL, so it can't see the 404/noindex that would drop it, and keeps the bare URL indexed.

robots.ts disallowed /mcp to "keep it out of the index until the catalog has real content" - but that backfires two ways:

1. /mcp currently 404s on landing. Left crawlable, Google fetches it, gets 404, and drops it - exactly the intended outcome, without the warning.
2. When the mcp-catalog (a public MCP directory whose whole value is search traffic) ships at agentage.io/mcp, a standing Disallow would strand its SEO until robots.txt is changed.

Change

Remove /mcp from the disallow list. Keep /api/ and /dashboard/ blocked (crawl-budget paths we never want fetched).

- const disallow = ['/api/', '/dashboard/', '/mcp'];
+ const disallow = ['/api/', '/dashboard/'];

AI-bot allowlist + sitemap reference unchanged.

Verify

• CI e2e-gate runs the built image and asserts /robots.txt.
• After this reaches prod: curl -s https://agentage.io/robots.txt should no longer list Disallow: /mcp, then hit Validate Fix in Search Console for the "Indexed, though blocked" issue. It clears over the next crawl cycles.

Scope note

Any GSC-flagged URLs under /api or /dashboard stay blocked by choice, so those specific "indexed though blocked" entries won't clear - acceptable since we never want them indexed anyway (they cause only a cosmetic snippet-less listing).

[github-actions]

✅ CI - Verify PASSED

Commit: 989d6e7416e7bd47ae1f69a0b66aaa51955cf16a · Branch: feature/seo-unblock-mcp-robots

• ✅ Type check
• ✅ Lint
• ✅ Format check
• ✅ Unit tests
• ✅ Build
• ✅ Landing e2e
• ✅ Showcase e2e

🔗 View workflow run