From abbdcc5075af3017d8b35b15a78dd5c629fdd3f8 Mon Sep 17 00:00:00 2001 From: Sean Davis Date: Mon, 27 Jul 2026 08:04:02 -0600 Subject: [PATCH] Allow crawlers on package landing pages and biocViews robots.txt disallowed /packages/release/, /packages/devel/ and /biocViews/, which blocks the package landing pages, the browse tree and every rendered vignette from being crawled. Analytics show roughly 70% of sessions land directly on /packages/*, and Organic Search accounts for about 74% of sessions, so this blocks the site's main entry point. The numbered-release paths stay disallowed, including /packages/3.23/ and /packages/3.24/, so the versioned aliases of release and devel do not compete with the canonical URLs. Refs #398 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01YBSvPwEk6sH1rhGmjPmsFq --- assets/robots.txt | 3 --- 1 file changed, 3 deletions(-) diff --git a/assets/robots.txt b/assets/robots.txt index 3396a0ab..c7e2f187 100644 --- a/assets/robots.txt +++ b/assets/robots.txt @@ -45,8 +45,6 @@ Disallow: /packages/3.21/ Disallow: /packages/3.22/ Disallow: /packages/3.23/ Disallow: /packages/3.24/ -Disallow: /packages/devel/ -Disallow: /packages/release/ Disallow: /packages/bioc/ Disallow: /packages/data/ Disallow: /repository/ @@ -54,7 +52,6 @@ Disallow: /checkResults/ Disallow: /data/ Disallow: /datafiles/ Disallow: /installScripts/ -Disallow: /biocViews/ Disallow: /stats/ Disallow: /dataann-stats/ Disallow: /dataexp-stats/