feat(web): fetch /browse data client-side and disallow crawlers via robots.txt#1426
Merged
Conversation
…w bots Sourcebot exposes an unbounded URL space (every file x revision x commit) and crawler traffic (GPTBot, Applebot, meta-externalagent, AhrefsBot) generates heavy SSR load. Deny all crawling while keeping unfurl bots (Slack, X, LinkedIn, etc.) allowed so shared links keep their OpenGraph previews. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…C props Blob, tree, and commit-diff panels previously fetched git data (file source, folder contents, full commit diffs) in server components and passed it as props across the client boundary. React flight-encodes those props into the document at seconds of main-thread time for large payloads (a large commit page measured 38MB / ~8s of encoding), which starves the event loop and health probes under crawler load. The panels now render a shell and fetch via existing JSON API routes with react-query (per repo convention), plus a new /api/folder_contents route. Documents drop to the ~70KB shell regardless of content size, and JS-less crawlers never trigger the data fetch at all. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughBrowse preview and commit-diff panels now fetch only repository metadata on the server and load file, folder, commit, diff, and blame data in client components. New client API helpers, a folder contents API route, changelog notes, and a robots policy accompany the refactor. ChangesBrowse panel client-side refactor
Estimated code review effort: 4 (Complex) | ~60 minutes Robots.txt Crawler Policy
Sequence Diagram(s)sequenceDiagram
participant CodePreviewPanel
participant CodePreviewPanelClient
participant ClientAPI
CodePreviewPanel->>ClientAPI: getRepoInfoByName(repoName)
CodePreviewPanel->>CodePreviewPanelClient: render with repo, path, revisionName
CodePreviewPanelClient->>ClientAPI: getFileSource(...)
CodePreviewPanelClient->>ClientAPI: getFileBlame(...)
ClientAPI-->>CodePreviewPanelClient: source, blame data
CodePreviewPanelClient-->>CodePreviewPanelClient: render PureCodePreviewPanel
sequenceDiagram
participant FocusedCommitDiffPanel
participant FocusedCommitDiffPanelClient
participant useCommitDiff
participant ClientAPI
FocusedCommitDiffPanel->>ClientAPI: getRepoInfoByName(repoName)
FocusedCommitDiffPanel->>FocusedCommitDiffPanelClient: render with repo, commitSha, path
FocusedCommitDiffPanelClient->>useCommitDiff: useCommitDiff(repoName, commitSha, path)
useCommitDiff->>ClientAPI: getCommit(...)
useCommitDiff->>ClientAPI: getDiff(...)
ClientAPI-->>useCommitDiff: commit, diff
useCommitDiff-->>FocusedCommitDiffPanelClient: commit, diff
FocusedCommitDiffPanelClient-->>FocusedCommitDiffPanelClient: render file header and LightweightDiffViewer
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (2)
packages/web/src/app/api/(client)/client.ts (1)
173-240: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winOptional: Extract a shared fetch helper to reduce duplication.
The four new functions (
getCommit,getDiff,getFolderContents,getFileBlame) share the same structure: URL construction, param iteration, fetch with theX-Sourcebot-Client-Sourceheader, JSON parsing, and type casting. A small shared helper would centralize the pattern and make future additions or header changes a one-line edit.♻️ Suggested shared helper
+async function fetchJson<T>(endpoint: string, queryParams: Record<string, string | undefined>): Promise<T | ServiceError> { + const url = new URL(endpoint, window.location.origin); + for (const [key, value] of Object.entries(queryParams)) { + if (value !== undefined) { + url.searchParams.set(key, value); + } + } + const result = await fetch(url, { + method: "GET", + headers: { + "X-Sourcebot-Client-Source": "sourcebot-web-client", + }, + }).then(response => response.json()); + return result as T | ServiceError; +} + -export const getCommit = async (queryParams: { repo: string; ref: string }): Promise<CommitDetail | ServiceError> => { - const url = new URL("/api/commit", window.location.origin); - for (const [key, value] of Object.entries(queryParams)) { - url.searchParams.set(key, value); - } - - const result = await fetch(url, { - method: "GET", - headers: { - "X-Sourcebot-Client-Source": "sourcebot-web-client", - }, - }).then(response => response.json()); - - return result as CommitDetail | ServiceError; -} +export const getCommit = (queryParams: { repo: string; ref: string }) => + fetchJson<CommitDetail>("/api/commit", queryParams);The same simplification applies to
getDiff,getFolderContents, andgetFileBlame. Theundefinedfilter in the helper covers both required and optional param types uniformly.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/web/src/app/api/`(client)/client.ts around lines 173 - 240, The four API client functions repeat the same URL-building and fetch/json/header logic, so extract that common flow into a shared helper in client.ts and have getCommit, getDiff, getFolderContents, and getFileBlame delegate to it. Keep the helper responsible for appending query params, setting the X-Sourcebot-Client-Source header, performing fetch, and returning parsed JSON so future header or request changes only need one update.packages/web/src/app/(app)/browse/[...path]/components/commitDiffPanel/focusedCommitDiffPanelClient.tsx (1)
42-57: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueOptional: extract the shared pending/error UI.
These pending and error blocks are duplicated verbatim in
fullCommitDiffPanel.tsx(lines 25-40). Consider a small shared wrapper (e.g.<CommitDiffState isPending error>) to avoid divergence as the panels evolve.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/web/src/app/`(app)/browse/[...path]/components/commitDiffPanel/focusedCommitDiffPanelClient.tsx around lines 42 - 57, The pending and error UI in focusedCommitDiffPanelClient is duplicated in fullCommitDiffPanel, so extract the shared loading/error rendering into a reusable component or wrapper (for example, a CommitDiffState in the shared commit diff panel area) and use it from both panels. Keep the existing behavior of the isPending and error/data checks, but centralize the Loader2/loading text and error message markup so both focusedCommitDiffPanelClient and fullCommitDiffPanel stay in sync as they evolve.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In
`@packages/web/src/app/`(app)/browse/[...path]/components/commitDiffPanel/focusedCommitDiffPanelClient.tsx:
- Around line 42-57: The pending and error UI in focusedCommitDiffPanelClient is
duplicated in fullCommitDiffPanel, so extract the shared loading/error rendering
into a reusable component or wrapper (for example, a CommitDiffState in the
shared commit diff panel area) and use it from both panels. Keep the existing
behavior of the isPending and error/data checks, but centralize the
Loader2/loading text and error message markup so both
focusedCommitDiffPanelClient and fullCommitDiffPanel stay in sync as they
evolve.
In `@packages/web/src/app/api/`(client)/client.ts:
- Around line 173-240: The four API client functions repeat the same
URL-building and fetch/json/header logic, so extract that common flow into a
shared helper in client.ts and have getCommit, getDiff, getFolderContents, and
getFileBlame delegate to it. Keep the helper responsible for appending query
params, setting the X-Sourcebot-Client-Source header, performing fetch, and
returning parsed JSON so future header or request changes only need one update.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: c02f0a2b-b454-4983-964c-cb35fa6d5583
📒 Files selected for processing (12)
CHANGELOG.mdpackages/web/src/app/(app)/browse/[...path]/components/codePreviewPanel/codePreviewPanel.tsxpackages/web/src/app/(app)/browse/[...path]/components/codePreviewPanel/codePreviewPanelClient.tsxpackages/web/src/app/(app)/browse/[...path]/components/commitDiffPanel/focusedCommitDiffPanel.tsxpackages/web/src/app/(app)/browse/[...path]/components/commitDiffPanel/focusedCommitDiffPanelClient.tsxpackages/web/src/app/(app)/browse/[...path]/components/commitDiffPanel/fullCommitDiffPanel.tsxpackages/web/src/app/(app)/browse/[...path]/components/commitDiffPanel/useCommitDiff.tspackages/web/src/app/(app)/browse/[...path]/components/treePreviewPanel/treePreviewPanel.tsxpackages/web/src/app/(app)/browse/[...path]/components/treePreviewPanel/treePreviewPanelClient.tsxpackages/web/src/app/api/(client)/client.tspackages/web/src/app/api/(server)/folder_contents/route.tspackages/web/src/app/robots.ts
Render the path header, separator, and file toolbar (blame toggle + age legend) immediately from props, and scope the loading spinner / error to the body below. The line-count/size stat shows a skeleton while the file source is pending, then the real value once loaded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
app.sourcebot.dev was repeatedly killed by its liveness probe (6 restarts in 24h). Investigation traced the cause to
/browsepages: blob/tree/commit panels fetched git data (file sources, folder contents, full commit diffs) in server components and passed it as props across the client-component boundary. React flight-encodes those props into the document on the main thread — a single large commit page measured 38MB of RSC payload (~8s of encoding), and a binary blob produced a 62MB document. Crawler traffic (GPTBot, Applebot, meta-externalagent, AhrefsBot — ~97% of all server processing time) walked the unbounded@<sha>URL space, stacking these renders until the event loop starved and health probes timed out.Changes
robots.txt (
app/robots.ts): disallow all crawling, with an allowlist for link-preview fetchers (Slackbot, Twitterbot, LinkedInBot, Discordbot, facebookexternalhit, TelegramBot, WhatsApp) so shared links — e.g. chat pages — keep their OpenGraph unfurls. Applebot/GPTBot/meta-externalagent are deliberately not allowlisted (search/AI-training crawlers, not preview fetchers).Client-side data fetching for /browse panels:
CodePreviewPanel,TreePreviewPanel,FullCommitDiffPanel, andFocusedCommitDiffPanelnow render a shell and fetch via JSON API routes with react-query (per repo convention). Server components only fetchrepoInfofor the path header./api/folder_contentsroute (internal, not registered in the public API) wrapping the existinggetFolderContents.getCommit,getDiff,getFolderContents,getFileBlame.useCommitDiffhook preserving the initial-commit empty-tree fallback.CommitsPanelleft server-side intentionally (bounded at 35 commits/page).Documents drop to the ~70KB app shell regardless of content size, JSON serialization replaces flight encoding (~50-100x less main-thread time for large payloads), and JS-less crawlers never trigger data fetches at all.
Verification
Exercised in an isolated dev environment against an indexed repo: blob, tree, commit, blame-mode, and focused-diff pages all render correctly via client fetching; blob documents contain zero file content and commit documents zero hunk payloads (previously embedded); all API routes return 200; robots.txt serves the intended rules. Typecheck clean for all touched files.
Follow-ups (not in this PR)
/api/diffon a mega-commit still returns MBs of JSON).🤖 Generated with Claude Code
Summary by CodeRabbit
robots.txtroute that blocks crawlers by default while allowing common link-preview bots to keep OpenGraph previews working.