Skip to content

build(deps): bump pyarrow to 25.0.1 for timestamp materialisation - #457

Merged
gloryfromca merged 4 commits into
mainfrom
fix/pyarrow-25-timestamp-import
Sep 24, 2026
Merged

gloryfromca merged 4 commits into
mainfrom
fix/pyarrow-25-timestamp-import

Conversation

@gloryfromca

@gloryfromca gloryfromca commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Summary

pyarrow 24.0.0 → 25.0.1 in uv.lock, plus a pyarrow>=25.0.1 floor in pyproject.toml so installs from PyPI into an environment that already has 24 get it too (lancedb pulls pyarrow in regardless — a floor, not a new package; nothing else in the lock moves).

Why: every LanceDB read path ends in to_list() = pyarrow to_pylist(). Two things changed between 24 and 25.0.1:

  1. Zone resolution (25.0.0, GH-48xxx prefer_zoneinfo) — on 24, converting each tz-aware timestamp value tries import pytz first (datetime.cc:377); pytz is not installed and Python does not cache failed imports, so every value re-walks sys.path: 48 failed imports per 16-row result with three timestamp columns, 3 000 per 1 000 rows. On Windows each stat goes through Defender — py-spy on the 10-hour soak (PR feat(windows): unblock the import path and cover it in CI #454) showed the event-loop thread spending ~half its busy time in importlib under to_list(); search p50 climbed from 0.5 s to 3.2 s under load while the same query took 12 ms idle. 25 goes to zoneinfo directly (0 pytz attempts, measured with an import hook).
  2. Row materialisation (25.0.1 only, GH-50326) — to_pylist() no longer builds a Scalar per element; this is where most of the macOS speed-up comes from, and why the floor is 25.0.1 and not 25 (25.0.0 also has a mimalloc crash at interpreter exit, GH-50471).

Measurements

macOS, 1 000 rows, median ms per to_pylist() call (review's independent benchmark):

columns 24.0.0 25.0.0 25.0.1
3 tz timestamps + vector[1024] 186.8 125.3 18.7
3 tz timestamps only 57.5 4.0 3.9
vector[1024] only 128.0 121.7 14.8

Windows soak box (Defender on), 100 real rows of the soak episode table: 311.9 ms → 13.4 ms per call (300 failed import pytz → 0). On Windows the pytz path dominates; on macOS the Scalar removal does.

Casting the timestamp columns to int64 before to_pylist() was measured too (18 % on 24, eight call sites in four modules) and rejected.

Compatibility

  • src/everos touches pyarrow directly only for schema construction (pa.schema/field/timestamp, is_timestamp, DataType.equals) and Table.sort_by/slice/to_pylist; none of it changed in 25.
  • 23 Arrow type families × null/NaN/±inf/lists/structs/maps/dictionaries/three time zones: to_pylist() output identical between 24 and 25.0.1 (review's parity script); an EverOS-shaped LanceModel round trip (create/add/merge_insert/vector search/FTS/model_validate) passes under -W error on both.
  • Caveat worth knowing: lancedb 0.34.0 (2026-07-02) predates pyarrow 25 (07-10), and lancedb's own tests extra pins pyarrow<25 from 0.36.0 on with no stated reason. Our Linux CI (unit ×2, integration ×2), the macOS parity/smoke scripts and the Windows box's full unit suite (2581 passed; the two failures were a GBK-locale test bug fixed in feat(windows): unblock the import path and cover it in CI #454, identical on 24) are the evidence.
  • Wheels: cp310–cp314 and cp314t for win_amd64, macOS arm64/x86_64, manylinux/musllinux x86_64/aarch64 (cp313t wheels were dropped upstream in 25.0.0; not a platform this project supports).

Verification

  • uv lock --upgrade-package pyarrow → only pyarrow changed; uv lock after the pyproject floor → only the project's requires-dist entry added.
  • pytest tests/unit: 2550 passed / 4 skipped; pytest tests/integration: 183 passed / 5 skipped / 7 deselected. make lint green.
  • Windows: full unit suite on the soak box with 25.0.1 in the venv (see Compatibility); before/after benchmark in the comment below.

Review round

Adversarial review confirmed the lock diff and the per-value import mechanism, reproduced the numbers, and corrected two things now reflected above: the 10× on macOS is mostly 25.0.1's GH-50326 rather than the zoneinfo change, and there is no Windows CI job on main yet (that job arrives with #454).

🤖 Generated with Claude Code

Every LanceDB read path ends in `to_list()`, which is pyarrow's
`to_pylist()`. On pyarrow 24 each tz-aware timestamp value runs
`import pytz` first and falls back to `zoneinfo`; pytz is not installed,
and Python does not cache a failed import, so every value walks the whole
`sys.path`. A 16-row result with three timestamp columns does 48 failed
imports per call. On Windows each of those stats goes through Defender,
and py-spy on the 10-hour soak showed the event-loop thread spending about
half of its busy time inside importlib under `to_list()` — search p50 rose
from 0.5 s to 3.2 s under load while the same query took 12 ms idle.

pyarrow 25 resolves the zone through `zoneinfo` directly (zero pytz
attempts, measured) and its `to_pylist()` is about ten times faster on the
same rows (4.8 ms -> 0.5 ms per call locally). lancedb only requires
`pyarrow>=16`; nothing else in the lock moves. Unit 2550 passed /
4 skipped, integration 183 passed / 5 skipped on the bumped lock.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@gloryfromca

Copy link
Copy Markdown
Member Author

Windows before/after on the soak box (Windows 11 Enterprise, Defender on), same 100 rows of the soak episode table (3 tz-aware timestamp columns + 1024-dim vector), to_pylist() per call:

pyarrow failed import pytz per call ms/call
24.0.0 (locked) 300 311.9
25.0.1 (this PR) 0 13.4

A failed import costs about 1 ms on this machine (each sys.path entry is a stat through Defender), which is why the soak's search latency was dominated by it. 23× on Windows, 10× on macOS for the same call.

A lock-only bump reaches `uv sync`, CI and the soak machine, but a user
who `pip install everos` into an environment that already has pyarrow 24
keeps the slow path. lancedb pulls pyarrow in regardless, so this adds a
floor, not a package. 25.0.1 specifically: 25.0.0 resolves zones through
zoneinfo (no per-value `import pytz`) but still builds a Scalar per
element in `to_pylist()` and carries a mimalloc crash at interpreter exit
(GH-50471); the ~10x materialisation speed-up is 25.0.1's GH-50326.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@gloryfromca
gloryfromca force-pushed the fix/pyarrow-25-timestamp-import branch from c53d8d0 to 9238ebc Compare September 24, 2026 05:56
@arelchan
arelchan self-requested a review September 24, 2026 08:45
@gloryfromca
gloryfromca merged commit b736df6 into main Sep 24, 2026
10 checks passed
@gloryfromca
gloryfromca deleted the fix/pyarrow-25-timestamp-import branch September 24, 2026 08:54
This was referenced Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants