feat: batch GitHub API calls into single GraphQL queries - #56
feat: batch GitHub API calls into single GraphQL queries#56santoshyadavdev wants to merge 8 commits into
Conversation
Replace per-year API calls with batched endpoints that use GraphQL field aliases to fetch all years in one request. This fixes 'Resource limits exceeded' errors for users with many years of activity. New server endpoints: - POST /api/github/contributions-batch - POST /api/github/repository-contributions-batch Updated frontend service methods (getAggregatedActivity, getRepositoryInsights) to use the batch endpoints, reducing N+1 API calls down to 2 total (contributions + repo contributions). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughGitHubService and the Workers API now support batched multi-year contribution and repository contribution queries. Aggregation methods consume batched responses, merge repository data across years, and retain existing response structures. ChangesGitHub batch contribution flow
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant GitHubService
participant WorkersAPI
participant GitHubGraphQL
GitHubService->>WorkersAPI: POST username and years
WorkersAPI->>GitHubGraphQL: Run chunked aliased queries
GitHubGraphQL-->>WorkersAPI: Return contribution data by year
WorkersAPI-->>GitHubService: Return batched response
GitHubService->>GitHubService: Aggregate totals and merge repositories
Possibly related PRs
Suggested labels: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
commitstory | cffb917 | Commit Preview URL Branch Preview URL |
Jul 17 2026, 10:08 AM |
- Contributions batch: chunked into groups of 5 years per query
- Repo contributions batch: chunked into groups of 3 years per query
- Reduced maxRepositories from 100 to 20 per year to lower node count
- Added rateLimit { cost remaining } to queries for debugging
- Sequential chunk execution prevents hitting GitHub's resource limits
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@yourstory/src/server.ts`:
- Around line 378-400: Validate body.years in both batch handlers at
yourstory/src/server.ts lines 378-400 and 530-554 before constructing queries or
calling chunkArray: require an array of integers within the supported year range
and enforce a maximum list size. Return the existing 400-style validation
response for invalid, out-of-range, empty, or oversized input, while preserving
the default current-year behavior when years is omitted.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 0c59fa48-e308-475d-95c4-544934c5bbb9
📒 Files selected for processing (2)
yourstory/src/app/core/services/github.service.tsyourstory/src/server.ts
| const body = (await request.json()) as { username?: string; years?: number[] }; | ||
| const login = body.username ?? ''; | ||
| if (!login) return json({ error: 'username is required' }, 400); | ||
|
|
||
| const token = env.GITHUB_TOKEN; | ||
| if (!token) return json({ error: 'GITHUB_TOKEN is not configured' }, 503); | ||
|
|
||
| const years = body.years ?? [new Date().getFullYear()]; | ||
| if (years.length === 0) return json({ error: 'years array is empty' }, 400); | ||
|
|
||
| // Build aliased fragments for each year | ||
| const contribFields = ` | ||
| totalCommitContributions | ||
| totalIssueContributions | ||
| totalPullRequestContributions | ||
| totalPullRequestReviewContributions | ||
| restrictedContributionsCount | ||
| `; | ||
|
|
||
| const yearFragments = years.map( | ||
| (y) => | ||
| `y${y}: contributionsCollection(from: "${y}-01-01T00:00:00Z", to: "${y}-12-31T23:59:59Z") { ${contribFields} }` | ||
| ).join('\n '); |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the file and inspect relevant slices.
wc -l yourstory/src/server.ts
sed -n '340,430p' yourstory/src/server.ts
printf '\n--- SECOND RANGE ---\n'
sed -n '500,590p' yourstory/src/server.ts
printf '\n--- SEARCH FOR VALIDATION HELPERS ---\n'
rg -n "years|username|validate|schema|zod|parse" yourstory/src/server.tsRepository: santoshyadavdev/commitstory
Length of output: 13381
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the batching/query construction and any size checks.
sed -n '430,540p' yourstory/src/server.ts
printf '\n--- REPO BATCH CONTINUATION ---\n'
sed -n '591,690p' yourstory/src/server.ts
printf '\n--- SEARCH FOR MAX/YEARS LIMITS ---\n'
rg -n "MAX_.*YEAR|chunkArray\\(|years\\.length|Array\\.isArray|Number\\.isInteger|Integer|plausible|unique" yourstory/src/server.tsRepository: santoshyadavdev/commitstory
Length of output: 8534
Validate years before either batch query is assembled.
body.years is only null-coalesced today; a non-array payload can still reach chunkArray(...) and fail at runtime, and large lists can make the aliased GraphQL queries much more expensive. Add explicit array/integer/range/size checks in both batch handlers before constructing the query.
📍 Affects 1 file
yourstory/src/server.ts#L378-L400(this comment)yourstory/src/server.ts#L530-L554
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@yourstory/src/server.ts` around lines 378 - 400, Validate body.years in both
batch handlers at yourstory/src/server.ts lines 378-400 and 530-554 before
constructing queries or calling chunkArray: require an array of integers within
the supported year range and enforce a maximum list size. Return the existing
400-style validation response for invalid, out-of-range, empty, or oversized
input, while preserving the default current-year behavior when years is omitted.
- Contributions: 3 years per chunk (scalars only, lightweight) - Repo contributions: 1 year per chunk (safest, avoids node limits) - maxRepositories: reduced from 20 to 10 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@yourstory/src/server.ts`:
- Around line 504-512: Update the frontend aggregation flow in github.service.ts
to consume privateContributionsBlocked from the yearly contribution results and
propagate the unavailable status instead of treating privateContributions: 0 as
an actual total. Ensure the aggregate contract marks private contributions
incomplete, or represents the value as nullable, while preserving normal totals
when the flag is not set.
- Around line 445-450: Update the GraphQL query in the batch handler around the
rateLimit selection so its cost and remaining values are either surfaced through
the existing logs or returned payload, or remove the rateLimit field entirely if
they are not needed. Ensure both batch handlers consistently handle the
rate-limit data rather than querying and ignoring it.
- Around line 466-472: Update the hasSamlError calculation in the GraphQL error
handling block to require every error in data.errors to satisfy the existing
SAML indicators, using all() rather than any-error matching. Preserve the
current non-SAML response behavior so mixed or unrelated failures return the
first error instead of entering the SAML fallback.
- Around line 436-440: Keep SAML failure state scoped to the current chunk in
the loop within the server handler: replace the shared privateBlocked state with
a per-chunk flag initialized for each iteration, and use it only when selecting
contribution fields and applying that chunk’s fallback. Preserve normal private
contributions and restrictedContributionsCount for later chunks whose queries
succeed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: f9b8ce7a-38ef-469c-974d-90f886c2ed09
📒 Files selected for processing (1)
yourstory/src/server.ts
| let privateBlocked = false; | ||
|
|
||
| try { | ||
| for (const chunk of chunks) { | ||
| const fields = privateBlocked ? contribFieldsNoPrivate : contribFields; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show the relevant region with line numbers
sed -n '380,560p' yourstory/src/server.ts
echo '--- SEARCH privateBlocked ---'
rg -n "privateBlocked|contribFieldsNoPrivate|privateContributionsBlocked|restrictedContributionsCount" yourstory/src/server.tsRepository: santoshyadavdev/commitstory
Length of output: 8165
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
p = Path('yourstory/src/server.ts')
text = p.read_text()
for needle in ['privateBlocked', 'contribFieldsNoPrivate', 'privateContributionsBlocked', 'restrictedContributionsCount']:
print(f'--- {needle} ---')
for i, line in enumerate(text.splitlines(), start=1):
if needle in line:
print(f'{i}: {line}')
PYRepository: santoshyadavdev/commitstory
Length of output: 2056
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find how the batch response is consumed and whether privateContributions is summed.
rg -n "privateContributions|privateContributionsBlocked|contributions-batch|years\s*:\s*allResults|sum\(" yourstory/srcRepository: santoshyadavdev/commitstory
Length of output: 3000
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "contributions-batch|privateContributionsBlocked|privateContributions" yourstory/src yourstory -g '!**/*.map'Repository: santoshyadavdev/commitstory
Length of output: 5563
🌐 Web query:
GitHub GraphQL restrictedContributionsCount SAML failure contributionsCollection query error per range or per user
💡 Result:
When querying the GitHub GraphQL API for contributionsCollection, users may experience issues related to restricted contributions or SAML enforcement if their Personal Access Token (PAT) is not correctly authorized for the organization’s SAML single sign-on (SSO) session [1][2][3]. ### Understanding the Behavior 1. SAML Enforcement: If you are a member of an organization that enforces SAML SSO, the GitHub GraphQL API will restrict access to data related to that organization unless the OAuth token or PAT has been explicitly authorized for the organization's SSO session [1][2][3]. Queries that touch protected resources—even if the information seems public—may fail or return partial/zeroed results if the session is not active [1][2]. 2. ContributionsCollection and SSO: When fetching contribution data, the API will not include activity from SAML-protected organizations if there is no active, authorized SSO session associated with the token [1]. This often manifests as missing data or zero counts in the contributionsCollection response, even when the user has contributions in those organizations [1][4]. 3. Restricted Contributions: The field restrictedContributionsCount represents contributions made in private or internal repositories that the viewer cannot access [5][6]. This field only contains non-zero values if the user has specifically chosen to share their private contribution counts on their profile [5][6]. Note that restrictedContributionsCount is distinct from the SAML-related access issues; it is a privacy setting managed by the user, not a result of SAML enforcement [5]. ### Recommended Troubleshooting Steps * Authorize your PAT for SAML SSO: If you are receiving errors or missing data for a specific organization, you must authorize your PAT for that organization's SAML SSO [3]. You can do this through the GitHub web interface: navigate to your Personal Access Tokens settings, select the relevant token, and choose "Configure SSO" to authorize it for your organization [3]. * Check Token Permissions: Ensure your token has the necessary scopes (read:user, repo, etc.) required for the data you are accessing [5][7]. While classic PATs are generally recommended for avoiding expiration, ensure the permissions match the data requested [7]. * Verify User Privacy Settings: If you are querying restrictedContributionsCount and getting zero, verify that the target user has enabled "Include private contributions on my profile" in their GitHub account settings [5][6]. * Use Appropriate Endpoints: If you require information that is specifically protected by SAML, ensure the API request includes the necessary authentication headers for the SSO session. If you only need broad, non-protected data, you may need to filter your queries to avoid explicitly triggering protected resource lookups that require active SSO [2]. If you receive a specific GraphQL error with an extensions field containing saml_failure: true, this is a definitive indicator that your token lacks the required SSO authorization for that specific organization [2].
Citations:
- 1: https://stackoverflow.com/questions/73915679/githubs-graphql-api-contributionscollection-not-pulling-from-sso-organizations
- 2: gh repo sync demands upstream SAML enforcement cli/cli#11001
- 3: Issue when manually triggering the github workflow lowlighter/metrics#40
- 4: https://stackoverflow.com/questions/62619169/github-v4-graphql-cant-get-organization-user-contribution-info
- 5: https://docs.github.com/en/graphql/reference/users
- 6: https://bryanburgers.github.io/docql/github/object.ContributionsCollection.html
- 7: Commit count in stats card seems to be incorrect anuraghazra/github-readme-stats#3059
Keep the SAML fallback local to the chunk. Flipping privateBlocked here zeroes privateContributions for every later year and suppresses restrictedContributionsCount even when those queries would succeed, so the batch undercounts totals after the first SAML failure.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@yourstory/src/server.ts` around lines 436 - 440, Keep SAML failure state
scoped to the current chunk in the loop within the server handler: replace the
shared privateBlocked state with a per-chunk flag initialized for each
iteration, and use it only when selecting contribution fields and applying that
chunk’s fallback. Preserve normal private contributions and
restrictedContributionsCount for later chunks whose queries succeed.
| if (data.errors?.length) { | ||
| const hasSamlError = data.errors.some((e) => { | ||
| const msg = e.message?.toLowerCase() ?? ''; | ||
| return e.extensions?.saml_failure === true || msg.includes('saml') || msg.includes('organization'); | ||
| }); | ||
|
|
||
| if (!hasSamlError) return json({ error: data.errors[0].message }, 400); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
sed -n '430,510p' yourstory/src/server.tsRepository: santoshyadavdev/commitstory
Length of output: 3393
Avoid retrying mixed GraphQL failures. some() sends the whole response down the SAML fallback when any error looks SAML-related, so unrelated auth/query/rate-limit errors can be hidden. Retry only when all errors are SAML-related.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@yourstory/src/server.ts` around lines 466 - 472, Update the hasSamlError
calculation in the GraphQL error handling block to require every error in
data.errors to satisfy the existing SAML indicators, using all() rather than
any-error matching. Preserve the current non-SAML response behavior so mixed or
unrelated failures return the first error instead of entering the SAML fallback.
| allResults.push({ | ||
| year: y, | ||
| commits: c?.totalCommitContributions ?? 0, | ||
| issues: c?.totalIssueContributions ?? 0, | ||
| pullRequests: c?.totalPullRequestContributions ?? 0, | ||
| reviews: c?.totalPullRequestReviewContributions ?? 0, | ||
| privateContributions: 0, | ||
| privateContributionsBlocked: true, | ||
| }); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Propagate unavailable private totals through the frontend contract.
The fallback returns privateContributions: 0, but github.service.ts omits and ignores privateContributionsBlocked. The aggregate therefore presents incomplete data as a real zero. Consume this flag and mark the aggregate incomplete, or represent the value as nullable.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@yourstory/src/server.ts` around lines 504 - 512, Update the frontend
aggregation flow in github.service.ts to consume privateContributionsBlocked
from the yearly contribution results and propagate the unavailable status
instead of treating privateContributions: 0 as an actual total. Ensure the
aggregate contract marks private contributions incomplete, or represents the
value as nullable, while preserving normal totals when the flag is not set.
The restrictedContributionsCount field forces GitHub to enumerate all org memberships internally, causing 'Resource limits exceeded' for users with many org contributions. Removed from both the single-year and batch endpoints. privateContributions now returns 0. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Users with accounts older than 7 years were generating too many year-scoped GraphQL queries. Cap startYear to (currentYear - 6) so at most 7 years of data are fetched. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
yourstory/src/server.ts (1)
358-358: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winReturn
400for malformed batch request JSON.
request.json()runs before the handler’stry, so invalid JSON escapes this endpoint’s error handling instead of producing a client validation response.Proposed fix
- const body = (await request.json()) as { username?: string; years?: number[] }; + let body: { username?: string; years?: number[] }; + try { + body = (await request.json()) as { username?: string; years?: number[] }; + } catch { + return json({ error: 'invalid JSON body' }, 400); + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@yourstory/src/server.ts` at line 358, Move the request.json() call in the batch request handler inside its existing try block so malformed JSON is caught and returns HTTP 400 through the endpoint’s validation error handling. Preserve the current parsed body shape and successful request flow.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@yourstory/src/server.ts`:
- Line 358: Move the request.json() call in the batch request handler inside its
existing try block so malformed JSON is caught and returns HTTP 400 through the
endpoint’s validation error handling. Preserve the current parsed body shape and
successful request flow.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 70b031ca-2bc4-43bb-bada-3240a5bf9c0a
📒 Files selected for processing (2)
yourstory/src/app/core/services/github.service.tsyourstory/src/server.ts
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
yourstory/src/app/core/services/github.service.ts (1)
94-109: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winEnforce the year cap inside the batch helpers. These methods are public and pass
yearsthrough unchanged, so any other caller can bypassMAX_YEARSand send an oversized batch. Move validation/deduping/capping here, or make the wrappers private.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@yourstory/src/app/core/services/github.service.ts` around lines 94 - 109, Enforce MAX_YEARS within the public getContributionsBatch and getRepositoryContributionsBatch helpers by validating, deduplicating, and capping the years array before each HTTP request. Ensure both methods apply the same bounded-year behavior so callers cannot bypass the limit; alternatively make these wrappers private if validation remains exclusively elsewhere.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@yourstory/src/app/core/services/github.service.ts`:
- Around line 94-109: Enforce MAX_YEARS within the public getContributionsBatch
and getRepositoryContributionsBatch helpers by validating, deduplicating, and
capping the years array before each HTTP request. Ensure both methods apply the
same bounded-year behavior so callers cannot bypass the limit; alternatively
make these wrappers private if validation remains exclusively elsewhere.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 049359d3-a5f1-4dcc-b256-ecde9cade2ee
📒 Files selected for processing (1)
yourstory/src/app/core/services/github.service.ts
3 years still exceeded resource limits for some accounts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Years where GitHub returns 'Resource limits exceeded' are now skipped instead of crashing the entire batch request. The response includes a failedYears array so the frontend knows which data is missing. This handles accounts with complex org memberships that blow GitHub's internal node limits for certain year ranges. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 participant
Why
Users with long GitHub histories (10+ years) hit "Resource limits for this query exceeded" errors because the app fired a separate API call for each year -- potentially 20+ parallel requests for contributions and repository data.
Approach
Instead of making N per-year GraphQL requests, the new endpoints build a single query using GraphQL field aliases to fetch all years at once:
This reduces the total API calls from ~N*2 (contributions + repo contributions per year) down to 3 total (1 contributions batch + 1 repo contributions batch + 1 discussions).
Changes
Server (
server.ts):POST /api/github/contributions-batch-- accepts{ username, years[] }, returns all yearly contribution stats in one responsePOST /api/github/repository-contributions-batch-- same pattern for repository-level contributionsrestrictedContributionsCountif org requires SSO)Frontend (
github.service.ts):getAggregatedActivity()andgetRepositoryInsights()to call the new batch endpointsNotes
/api/github/contributions,/api/github/repository-contributions) are kept intact so nothing breaks if they are used elsewhere.{ years: [...] }envelope so the frontend can iterate without extra mapping.Summary by CodeRabbit
0when unavailable.