improvement(seo): fix soft 404s, per-integration freshness, and docs markdown for AI crawlers

[waleedlatif1]

Summary

Technical SEO / GEO fixes for www.sim.ai and docs.sim.ai, found by crawling both sites the way search and AI crawlers do. No content was rewritten.

www.sim.ai

• Soft 404s. Unknown /integrations/*, /library/*, /blog/*, /models/*, and author slugs return 200 with contradictory index, follow + noindex metas. fix(landing): return 404 for unknown library, blog, integration, model, and author slugs #8587's dynamicParams = false only works when the route is prerendered. The production image is built with a placeholder NEXT_PUBLIC_APP_URL, so it takes the RuntimePublicEnvScript path and renders every route dynamically. There, the loading.tsx Suspense boundary commits a 200 before the page's notFound() runs. fix(landing): return 404 for unknown library, blog, integration, model, and author slugs #8587 was verified on a hosted-URL build, which hid this.
  • Removed loading.tsx from the slug segments.
  • Moved the blog/library index skeletons into an (index) route group, so the index pages keep them.
  • Documented the rule in the comments and in (landing)/CLAUDE.md.
• Underscore integration slugs (/integrations/google_drive, linked with docs-style slugs) 308 to the hyphenated page.
• Per-integration freshness.
  • Before: one catalog-wide updatedAt stamped every integration's sitemap lastmod, JSON-LD dateModified, and visible "Last updated" line. Every regen moved all 272 dates even when one entry changed.
  • Now: each integrations.json entry carries its own updatedAt, which scripts/generate-docs.ts keeps until that entry's content changes.
  • Initial dates are backfilled from the file's git history: the last commit that changed each entry.
  • The catalog-wide date still dates the /integrations hub.
• Reseller model pages (Bedrock, Vertex, Azure, …) are already generated and indexable. They are now in the sitemap, and their titles/H1 name the reseller ("Claude Haiku 4.5 on AWS Bedrock"), since their prices and model IDs differ from the first-party page.
• Internal links. Footer integration links point to the landing integration pages instead of docs. The footer Chat link and the docs Compare link no longer go through redirects.
• Structured data.
  • Home drops its hardcoded offers, which still named the retired "Community Plan" and showed prices the page doesn't display. The offers live on /pricing, which now declares the canonical #software node.
  • #software, the solutions/platform pages, comparisons, blog/library publishers, and the docs publisher now reference #organization / #software instead of minting separate entities.
  • Removed the placeholder datePublished.
• RSS autodiscovery on /blog, /library, and /changelog.
• Removed a Vary: User-Agent that Next always overwrote.

docs.sim.ai

• API reference markdown. The 303 OpenAPI operation pages served empty markdown at /llms.mdx/* and .mdx. They now serve the same markdown as the page's "Copy page" action. They stay out of llms-full.txt, at ~40 KB of resolved schema each.
• llms-full.txt goes from 23.4 MB to 12.6 MB. The processed-markdown serializer padded every table cell to its column's widest cell (one timezone enum padded the whole Plane page to 3 MB), and that padding is now collapsed. No content was removed.
• Markdown negotiation.
  • Accept: text/markdown returned HTML: the proxy pattern was /docs/*path, but docs live at the root.
  • Added /<page>.md URLs (they currently return 500 on main and 404 on staging).
  • Every page now declares <link rel="alternate" type="text/markdown">.
• Sitemap lastmod. Vercel clones --depth=10 and source.config.ts correctly refuses to date pages from a shallow clone, so the docs sitemap shipped with no dates. The build now fetches commit history first (--filter=blob:none, commits and trees only). It runs only on Vercel, and on failure it logs and ships undated, as today.
• Academy redirects. The removed academy lessons redirect to the reference page for their topic instead of all landing on /introduction.
• Integration cross-links. Each docs integration page links to its landing page. Previously links only went landing → docs.

Not included (decisions, not code): publishing the draft customer stories, pricing/enterprise FAQ copy, and edge caching for the dynamic marketing pages.

Type of Change

• Bug fix
• Other: SEO / GEO

Testing

• New E2E suite, scripts/test-landing-seo-e2e.ts. It crawls the app as a crawler does (statuses, robots, sitemap, JSON-LD, links), with 10 checks, and writes a JSON report. It runs in the e2e-mobile job against the production-mode build (next start on the placeholder-URL build, so every route is dynamic, matching production). It only issues GETs, so it can also be pointed at a deployed origin.
  • Proof it can fail: pointed at the live www.sim.ai (current code), 9 of 10 checks fail for exactly the issues above. Only "known pages render" passes.
• Generator: bun run check:docs reports the backfilled integrations.json unchanged. Mutating one block's description and regenerating re-dated only that entry (plus the catalog date); the other 271 kept theirs.
• Docs: llms-full:generate locally:
  • 12.6 MB (from 23.4 MB);
  • 0 uncompacted table delimiters;
  • operation pages render markdown such as countTableRows.
  • Negotiation, .md, redirects, and sitemap dates are to be verified on this PR's Vercel preview.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Oct 14, 2026 6:02am UTC

[cubic-dev-ai]

All reported issues were addressed across 48 files

<sub>Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic</sub>

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge, with no new actionable issues found.

Summary

This PR fixes marketing-page status codes, adds per-integration dates, and improves docs markdown for crawlers.

• Unknown marketing slugs reach a 404 before a loading screen is sent.
• Each integration page shows the date its own content changed.
• Hosted model pages name their reseller in search results and headings.
• Marketing pages share product details and link directly to useful destinations.
• Docs pages and API references provide markdown directly to crawlers.
• Docs and marketing integration links lead to the matching landing pages.
• Docs builds fetch full commit history so sitemap dates can be included.
• A crawler-style check records what marketing pages return.

_{Reviews (4) · Last reviewed commit: "test(seo): match the visible date attrib..." · Reviewed by Greptile}

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 57 files

<sub>Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic</sub>

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 58 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.