Skip to content

feat: CursorPaginator for keyset pagination#55

Open
klaidliadon wants to merge 3 commits into
masterfrom
feat-cursor-paginator
Open

feat: CursorPaginator for keyset pagination#55
klaidliadon wants to merge 3 commits into
masterfrom
feat-cursor-paginator

Conversation

@klaidliadon
Copy link
Copy Markdown
Contributor

Keyset / cursor pagination as a sibling to the existing offset Paginator[T]. Came up while reviewing a downstream impl (0xPolygon/omsx#1233) that re-implemented the same encode/decode + LIMIT n+1 pattern locally — the primitives are generic enough to live in pgkit and the lifecycle fits the existing PrepareQuery / PrepareResult shape exactly.

Offset pagination is fine for stable, small result sets, but on append-only / time-ordered tables under concurrent writes it visibly skips or duplicates rows as new ones land between page fetches. Keyset (cursor) pagination uses the last row of the previous page as a WHERE predicate, so newly-inserted rows don't shift the next page.


What's added

cursor.go — three pieces:

  • EncodeCursor[C any](C) (string, error) / DecodeCursor[C any](string) (*C, error) — opaque, URL-safe base64+JSON. Empty input → (nil, nil) so callers compose with a nil-check. Malformed → ErrInvalidCursor so callers can map to 400, not 500.
  • CursorPaginator[T any, C any] — sibling of Paginator[T], same lifecycle:
    rows, q, err := paginator.PrepareQuery(query, page, applyCursor)
    if err := db.GetAll(ctx, q, &rows); err \!= nil { ... }
    rows, err = paginator.PrepareResult(rows, page, cursorFromRow)
  • Two type parameters (T row, C cursor) so the WHERE clause and serialized cursor are both typed.

page.goPage gains Cursor and NextCursor (both string). Unused by the offset paginator; shared on the type so callers can swap paginators without changing the request shape.

Design choices

Choice Rationale
Two type parameters (T, C) Lets applyCursor get a typed cursor argument instead of any smuggling.
Caller owns ORDER BY The cursor WHERE is order-dependent — only the caller knows which columns to compare. Mismatched ORDER BY and applyCursor silently skips or duplicates rows (documented).
Reuse PaginatorSettings / WithDefaultSize / WithMaxSize Same options as offset — WithSort / WithColumnFunc are no-ops here because the caller owns ORDER BY.
Extend the existing Page, no parallel type Field discipline by mode (offset vs cursor) mirrors how Column / Sort already coexist on Page without enforcement.
nil applyCursor / cursorFromRow panic at function entry Surfaces misconfiguration on the first request rather than the second (when a cursor first appears or more first fires).
Unsigned cursor pgkit has no tenant model. Documented that callers must re-apply scope from the request, never from the cursor. An HMACCursor variant can be added later if needed.

Tests

Pure-unit, no DB needed thanks to the PrepareQuery/PrepareResult split. Coverage:

  • Encode/decode round-trip, empty input, invalid base64, invalid JSON
  • First page (no cursor) chains LIMIT n+1, capacity-correct pre-allocated slice
  • With cursor: applyCursor invoked, WHERE clause appended
  • Invalid cursor → ErrInvalidCursor propagated through PrepareQuery
  • PrepareResult no-more case (no NextCursor, More=false)
  • PrepareResult has-more case (trims to limit, NextCursor round-trips back to last surviving row)
  • Default size from nil page, caps at max, max-below-default is lifted
  • End-to-end walk of a 5-row dataset across 3 pages, every row surfaces exactly once
  • PrepareResult propagates cursorFromRow errors
  • nil-callback panics on both PrepareQuery and PrepareResult
go test -run 'TestEncodeDecode|TestDecodeCursor|TestCursorPaginator' ./...
ok  	github.com/goware/pgkit/v2	1.157s

Existing TestPagination* tests unaffected.

Not in this PR (intentionally)

  • No Table.ListCursor convenience. The PrepareQuery/PrepareResult split is enough; a table-level wrapper can come later if usage warrants it.
  • No cursor signing. Easy to add (WithCursorSigner(...)) once a caller needs it. Skipped to keep v1 minimal.

Happy to split this into smaller commits, rename, or restructure if the shape isn't what you'd prefer.

Sibling of Paginator[T] for ordering-stable pagination under concurrent
writes, where offset is too expensive or visibly skips/duplicates rows.

- EncodeCursor / DecodeCursor + ErrInvalidCursor (opaque base64+JSON,
  unsigned — callers re-apply tenant scope per request)
- CursorPaginator[T, C] mirrors the offset Paginator[T] lifecycle
  (PrepareQuery -> GetAll -> PrepareResult) so call sites stay uniform
- Page gains Cursor / NextCursor on the shared type so callers can swap
  paginators without changing the Page type
- Caller owns ORDER BY; applyCursor must match it (documented + tested)
- nil applyCursor / cursorFromRow panic on entry to fail on the first
  request rather than the second
Replace the two callback positional args on PrepareQuery / PrepareResult
with a Cursor[Self, Row] interface constraint, mirroring the self-pointer
convention pgkit already uses on Table[T, P, I] -> Record[T, I].

- Cursor[Self, Row] interface: *Self + Apply + From, hangs methods on the
  pointer so From can populate the value and Apply can read it
- CursorPaginator gains a third type parameter PC Cursor[C, T]; callers
  spell it like NewCursorPaginator[*Row, Cursor, *Cursor](...)
- Drops the nil-check + panic on applyCursor / cursorFromRow — missing
  methods are now a compile error
- Cursor behavior co-located on the cursor type (Apply, From) instead of
  split between the type + a free function at the call site
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant