Skip to content

Coolguyzone/chore/ai rules robots #17796

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
coolguyzone merged 3 commits into from
May 19, 2026
+8 −1
Merged

Coolguyzone/chore/ai rules robots #17796

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d4094fb
Add user agent rules for ai crawlers in robots.txt
coolguyzone May 18, 2026
4bf623f
Add ai content signals to robots.txt
coolguyzone May 18, 2026
db96e70
Collapse individual bot rules to a wildcard
coolguyzone May 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion app/robots.txt/route.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,17 @@
import {isDeveloperDocs} from 'sentry-docs/isDeveloperDocs';

const sitemap = isDeveloperDocs
? 'https://develop.sentry.dev/sitemap.xml'
: 'https://docs.sentry.io/sitemap.xml';

export const GET = (_request: Request) => {
return new Response(
`
Sitemap: ${isDeveloperDocs ? 'https://develop.sentry.dev/sitemap.xml' : 'https://docs.sentry.io/sitemap.xml'}
Sitemap: ${sitemap}

User-agent: *
Copy link
Copy Markdown
Contributor

@sfanahata sfanahata May 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coolguyzone since we're wildcard allowing every bot, I'm not sure it makes sense to add in more specific bots that we're also carte blanche allowing? Beside the trainers, which we might want to update in what they're allowed to do, I wonder if we remove the explicits of each bot? Otherwise, we're still missing some, like

  • ChatGPT-User
  • PerplexityBot
  • Perplexity-User
  • Meta-ExternalAgent
  • cohere-ai
  • Diffbot

And that list will continue to get longer/need updating as more bots are created.

Copy link
Copy Markdown
Contributor Author

@coolguyzone coolguyzone May 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, in the future it might make sense if we are adding different rules for different bots but you're right, for now it makes sense just to collapse everything to a wildcard.

Allow: /
Content-Signal: ai-train=yes, search=yes, ai-input=yes
`.trim(),
{headers: {'content-type': 'text/plain'}}
);
Expand Down
Loading