perf(api): drop the default COUNT from list queries, add hasMore and opt-in includeTotal

[royalTreasure]

closes #206

Summary

Every paged list query ran a full COUNT in a transaction next to its findMany. The queries already fetched take: n+1 to build nextCursor, so the count only fed a total most clients ignore.

Changes

• src/db.ts: queryTransfers, queryAllTransfers, queryNftTransfers and queryAccountSummaries issue no COUNT by default (no transaction either). They return hasMore from the existing n+1 fetch. New includeTotal param runs the exact count and returns total; otherwise total is absent.
• REST: ?includeTotal=true on /transfers/incoming|outgoing|address/:address, /accounts/:address/transfers and /nfts/transfers; hasMore in the responses. GraphQL: transfers(includeTotal: Boolean = false), total is now nullable, added hasMore: Boolean!. JSON:API meta carries hasMore.
• README and OpenAPI (openapi.json regenerated with npm run docs:openapi) say the total costs more.
• Tests: new src/__tests__/listPagination.test.ts (all four queries: no count by default, page-exactly-full boundary gives hasMore=false, one-past gives hasMore=true plus cursor, includeTotal returns exact total). Route test for forwarding includeTotal. Existing mocks gained hasMore; the network-scoping count test now passes includeTotal: true.

Behaviour change (please note)

total is no longer in list responses unless includeTotal=true. Clients that read total must opt in; hasMore / nextCursor replace it for paging.

Index check and timings

Postgres 17 (embedded), TokenTransfer seeded with 2,000,000 rows, one hot address with 133,333 incoming rows, table on a laptop with load average about 25, so wall-clock numbers are noisy; buffers are the steadier signal. Query shapes copied from Prisma's SQL for toAddress = ? ORDER BY ledger DESC, id DESC LIMIT 51.

query (hot address, 133k rows)	before	after
COUNT	204-418 ms, 90-248 buffers	not issued by default
page (LIMIT 51)	2-14 ms, 45-48 buffers	same query, 2-14 ms
request total	COUNT + page, about 205-430 ms	page only, about 2-14 ms

A rare address (20 rows, all in the oldest ledgers, worst case for a ledger-order scan) pages in 0.26 ms via the existing (network, toAddress) index. I also tried composite (network, toAddress, ledger DESC, id DESC) indexes: on this data the page time did not meaningfully improve (0.6 ms vs 2-14 ms, within load noise) and the planner already picks an index that avoids a big sort, so I did not add them. The existing indexes support the paged query without the count masking a gap. Data is synthetic and evenly distributed; heavily skewed real data may behave differently.

Verification

• npx jest src/__tests__/listPagination.test.ts src/__tests__/network.test.ts src/__tests__/routes/transfers.test.ts src/__tests__/jsonapi.test.ts: 4 suites, 122 passed.
• npx jest src/__tests__/listPagination.test.ts src/__tests__/staleReads.test.ts src/__tests__/graphql.test.ts src/__tests__/networkSelector.test.ts: 4 suites, 48 passed.
• New tests against the original db.ts: 20 failed (fail-before confirmed), pass after.
• npm run typecheck: clean.
• Full suite and vitest integration tests not run (no Docker; shared loaded machine). An earlier run of graphql.test.ts hit a 5000 ms timeout on one test while the machine was under heavy load; it passed on rerun.

[drips-wave]

@royalTreasure Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[Miracle656]

Verified locally on your branch merged with current main.

• npx tsc --noEmit -p tsconfig.test.json: clean
• Full jest suite: 542 passed / 45 suites, 0 failed
• npm run docs:openapi regenerates openapi.json byte-identical to what you committed, so the spec is genuinely in sync rather than hand-edited

The issue asked for before/after timings on a seeded table and you produced them with buffer counts alongside wall clock, which is the right instinct on a loaded machine — 204-418 ms of COUNT removed from every default list request, against a 2-14 ms page. Reporting that you tried composite (network, toAddress, ledger DESC, id DESC) indexes and then did not add them because the planner already avoids the sort is more useful than adding them speculatively, and it answers the part of the issue about whether the count was masking a missing index.

listPagination.test.ts is the part I'd single out. Asserting on which operations the client issued — countCalls() empty, transactionCalls === 0, take === 6 for a limit of 5 — tests the actual claim rather than the shape of the response, and describe.each over all four query functions means a fifth list query added later without the same treatment is a visible gap. Both boundary cases (page exactly full vs one row past) are the ones that get this wrong in practice. Fixtures are a valid strkey and a StrKey.encodeContract built from raw bytes, with an assertion proving it.

Also checked: queryAccountSummaries has no route callers, so dropping its default total breaks nothing silently, and optionalQueryBool rejects anything that is not true/false/1/0 rather than coercing it, so ?includeTotal=yes is a 400 instead of a quiet false.

The documented behaviour changes — total absent unless opted in, and GraphQL total: Int! becoming Int — are what #206 asked for, so they go in as intended. Worth a line in release notes for any consumer reading total today.

https://claude.ai/code/session_01USgemLt4Rnz4SGB1Srf3GB