Skip to content

fix(sync): restart a drained walk from the top; carry the depth window to Gmail - #12

Merged
YellowSnnowmann merged 1 commit into
tinyhumansai:mainfrom
YellowSnnowmann:fix/gmail-cursor-reset-and-depth-window
Sep 4, 2026
Merged

YellowSnnowmann merged 1 commit into
tinyhumansai:mainfrom
YellowSnnowmann:fix/gmail-cursor-reset-and-depth-window

Conversation

@YellowSnnowmann

@YellowSnnowmann YellowSnnowmann commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

Two defects found while tracing why a Gmail source that reported "100 items synced" stopped picking up new mail (openhuman-side write-up in the follow-up PR that re-pins this release).

1. The cursor parked on the last page. run_sync completed when the provider had no next page but saved state.cursor pointing at the page it had just read. Every later run resumed there, re-read that one page, skipped it, and completed — and never asked for page one again, where a newest-first mailbox puts everything that arrived since. Hidden by small mailboxes (one page keeps the cursor at None); it bit every account deeper than one page the moment its first walk reached the end.

Fix: a completing run forgets its position (SyncState::restart_from_top) — only the position; the seen-set and versions stay, so the restart re-reads page one as skips, not re-ingests.

2. "Sync depth (days)" was a no-op. SyncLimits::depth_days existed and nothing read it, and the wire request had no field for it, so the host's setting could not even arrive.

Fix, additive on the wire (contract 1.7 → 1.8):

  • ConnectorSyncRequest.depth_days: Option<u32> — absent decodes as None, None is not serialised, so an older module never sees an unknown key and an older host is unaffected.
  • The module applies the host's value or nothing — never the library default — so a module update alone does not start truncating anyone's mailbox. The default stays for direct callers of run_sync.
  • PageSpec.depth_window: Option<DepthWindow> — the provider-side syntax for the bound. Gmail carries it as its own after:YYYY/MM/DD search term; the other four toolkits have no such argument and read unbounded rather than approximating one client-side (which would spend exactly the requests the bound exists to save).

Tests

Pinned at every layer: the state method (restarting_from_the_top_forgets_only_the_cursor), the loop (a_drained_provider_restarts_the_next_run_from_the_top: second run requests None first, ingests only the new item, skips the three it already had), the Gmail page read (after: term when a window is set; no query at all when it is not; the window ignored by toolkits that cannot express it), the wire round-trip, and the module boundary through the bus (a_depth_window_reaches_gmail_as_a_search_query, a_request_without_a_window_reads_unbounded).

cargo fmt --check, cargo clippy --all-targets --all-features -D warnings, cargo build --all-targets --all-features, cargo test --all-features — all green locally.

Release

Needs a minor release (new optional wire field, contract 1.8). The openhuman follow-up re-pins the module and threads the per-source sync_depth_days setting into the request.

Closes #13

…w to Gmail

Two defects the host could not work around, found by tracing why a Gmail
source that reported "100 items synced" stopped seeing new mail.

Cursor parked on the last page. `run_sync` completed when the provider had no
next page but left `state.cursor` pointing at the page it had just read, and
saved it. Every later run resumed there, re-read that one page, skipped it,
and completed — page one, where a newest-first mailbox puts everything that
arrived since, was never requested again. Hidden by small mailboxes, whose
single page keeps the cursor at `None`; it bit every account deeper than one
page the moment its first walk reached the end. A completing run now forgets
its position (`SyncState::restart_from_top`), and only the position: the
seen-set and versions stay, so the restart re-reads page one as skips.

"Sync depth (days)" was a no-op. `SyncLimits::depth_days` existed and nothing
read it; the wire request had no field for it, so the host's setting could not
even arrive. `ConnectorSyncRequest` gains an optional `depth_days` (contract
1.8, additive: absent reads as `None`, `None` is not sent), the module applies
it to the run's limits — the host's value or nothing, never the library
default, so a module update alone does not start truncating anyone's mailbox
— and `PageSpec` gains `depth_window`, the provider-side syntax for the bound.
Gmail carries it as its own `after:YYYY/MM/DD` search term, so the request
costs one page of recent mail rather than a walk that a client-side filter
would have paid for and thrown away; the other four toolkits have no such
argument and read unbounded rather than approximating one.

Tests pin both at every layer: the state method, the loop (a drained walk
restarts from `None` and ingests only what is new), the Gmail page read (the
`after:` term when a window is set, no `query` at all when it is not, and the
window ignored by toolkits that cannot express it), the wire round-trip, and
the module boundary through the bus.
@coderabbitai

coderabbitai Bot commented Sep 3, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 35495254-6e92-4f11-a37b-a3c19bf4e56f

Warning

Your free Security trial is over. An organization admin can upgrade to Advanced for continuous pull request security review or dismiss this notice.


Comment @coderabbitai help to get the list of available commands.

@YellowSnnowmann
YellowSnnowmann marked this pull request as ready for review September 4, 2026 04:53
@tinysweeper

tinysweeper Bot commented Sep 4, 2026

Copy link
Copy Markdown

How this change flows

4 changed behaviours across 10 relationships. 5 surrounding behaviours are shown (60 graph nodes walked). 31 further behaviours left out to keep the diagram readable.

flowchart LR
  n0["PageSpec<br/>changed"]:::changed
  n1["fetch_page<br/>changed"]:::changed
  n2["json<br/>changed"]:::changed
  n3["...ith_nothing_to_read_completes_immediately<br/>changed"]:::changed
  n4["service_over"]:::impacted
  n5["new"]:::impacted
  n6["page_from"]:::impacted
  n7["context"]:::impacted
  n8["default_registry"]:::impacted
  n1 -->|uses| n0
  n1 -->|calls| n2
  n1 -->|calls| n6
  n3 -->|calls| n5
  n3 -->|tests| n5
  n3 -->|calls| n7
  n3 -->|tests| n7
  n4 -->|calls| n8
  n6 -->|uses| n0
  n7 -->|calls| n5
  classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
  classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
  classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
  classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Loading

Green: changed behaviour. Grey: surrounding behaviour. Arrows name the call, use, implementation, or test relationship. Orange: has findings. Red: has a finding that blocks the merge.

tinysweeper 0.1.0

@tinysweeper tinysweeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tinysweeper found nothing blocking. Approving.

          $0.0072 · 78,360 in / 1,236 out · 0 cached (0%) · deepseek/deepseek-v4-flash, openrouter/openai/text-embedding-3-small · 778 embedded
critique: $0.0066 · 73,741 in / 563 out   · 0 cached (0%) · deepseek/deepseek-v4-flash

@YellowSnnowmann
YellowSnnowmann merged commit 13d8d40 into tinyhumansai:main Sep 4, 2026
11 checks passed
YellowSnnowmann added a commit to YellowSnnowmann/openhuman that referenced this pull request Sep 4, 2026
tinyhumansai/tinyconnectors#12 shipped in v0.8.0 (contract 1.8, adds
`ConnectorSyncRequest.depth_days`). Move `vendor/tinyconnectors` from the
PR commit to the release tag and re-pin the registry record: version,
release URL, and the eleven per-platform archive digests taken from the
release assets. The workspace version bump moves `tinyconnectors-bus` in
Cargo.lock.

Module Pin Gate and submodule-monotonic gate pass locally.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sync: the cursor parks on the last page after a full walk, and depth_days never reaches the provider

1 participant