build(deps): bump pyarrow to 25.0.1 for timestamp materialisation - #457
Merged
Merged
Conversation
Every LanceDB read path ends in `to_list()`, which is pyarrow's `to_pylist()`. On pyarrow 24 each tz-aware timestamp value runs `import pytz` first and falls back to `zoneinfo`; pytz is not installed, and Python does not cache a failed import, so every value walks the whole `sys.path`. A 16-row result with three timestamp columns does 48 failed imports per call. On Windows each of those stats goes through Defender, and py-spy on the 10-hour soak showed the event-loop thread spending about half of its busy time inside importlib under `to_list()` — search p50 rose from 0.5 s to 3.2 s under load while the same query took 12 ms idle. pyarrow 25 resolves the zone through `zoneinfo` directly (zero pytz attempts, measured) and its `to_pylist()` is about ten times faster on the same rows (4.8 ms -> 0.5 ms per call locally). lancedb only requires `pyarrow>=16`; nothing else in the lock moves. Unit 2550 passed / 4 skipped, integration 183 passed / 5 skipped on the bumped lock. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Member
Author
|
Windows before/after on the soak box (Windows 11 Enterprise, Defender on), same 100 rows of the soak
A failed import costs about 1 ms on this machine (each |
A lock-only bump reaches `uv sync`, CI and the soak machine, but a user who `pip install everos` into an environment that already has pyarrow 24 keeps the slow path. lancedb pulls pyarrow in regardless, so this adds a floor, not a package. 25.0.1 specifically: 25.0.0 resolves zones through zoneinfo (no per-value `import pytz`) but still builds a Scalar per element in `to_pylist()` and carries a mimalloc crash at interpreter exit (GH-50471); the ~10x materialisation speed-up is 25.0.1's GH-50326. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
gloryfromca
force-pushed
the
fix/pyarrow-25-timestamp-import
branch
from
September 24, 2026 05:56
c53d8d0 to
9238ebc
Compare
0xKT
approved these changes
Sep 24, 2026
arelchan
self-requested a review
September 24, 2026 08:45
arelchan
approved these changes
Sep 24, 2026
This was referenced Sep 24, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pyarrow 24.0.0 → 25.0.1inuv.lock, plus apyarrow>=25.0.1floor inpyproject.tomlso installs from PyPI into an environment that already has 24 get it too (lancedb pulls pyarrow in regardless — a floor, not a new package; nothing else in the lock moves).Why: every LanceDB read path ends in
to_list()= pyarrowto_pylist(). Two things changed between 24 and 25.0.1:prefer_zoneinfo) — on 24, converting each tz-aware timestamp value triesimport pytzfirst (datetime.cc:377); pytz is not installed and Python does not cache failed imports, so every value re-walkssys.path: 48 failed imports per 16-row result with three timestamp columns, 3 000 per 1 000 rows. On Windows eachstatgoes through Defender — py-spy on the 10-hour soak (PR feat(windows): unblock the import path and cover it in CI #454) showed the event-loop thread spending ~half its busy time in importlib underto_list(); search p50 climbed from 0.5 s to 3.2 s under load while the same query took 12 ms idle. 25 goes tozoneinfodirectly (0 pytz attempts, measured with an import hook).to_pylist()no longer builds a Scalar per element; this is where most of the macOS speed-up comes from, and why the floor is 25.0.1 and not 25 (25.0.0 also has a mimalloc crash at interpreter exit, GH-50471).Measurements
macOS, 1 000 rows, median ms per
to_pylist()call (review's independent benchmark):Windows soak box (Defender on), 100 real rows of the soak
episodetable: 311.9 ms → 13.4 ms per call (300 failedimport pytz→ 0). On Windows the pytz path dominates; on macOS the Scalar removal does.Casting the timestamp columns to int64 before
to_pylist()was measured too (18 % on 24, eight call sites in four modules) and rejected.Compatibility
src/everostouches pyarrow directly only for schema construction (pa.schema/field/timestamp,is_timestamp,DataType.equals) andTable.sort_by/slice/to_pylist; none of it changed in 25.to_pylist()output identical between 24 and 25.0.1 (review's parity script); an EverOS-shaped LanceModel round trip (create/add/merge_insert/vector search/FTS/model_validate) passes under-W erroron both.testsextra pinspyarrow<25from 0.36.0 on with no stated reason. Our Linux CI (unit ×2, integration ×2), the macOS parity/smoke scripts and the Windows box's full unit suite (2581 passed; the two failures were a GBK-locale test bug fixed in feat(windows): unblock the import path and cover it in CI #454, identical on 24) are the evidence.Verification
uv lock --upgrade-package pyarrow→ only pyarrow changed;uv lockafter the pyproject floor → only the project's requires-dist entry added.pytest tests/unit: 2550 passed / 4 skipped;pytest tests/integration: 183 passed / 5 skipped / 7 deselected.make lintgreen.Review round
Adversarial review confirmed the lock diff and the per-value import mechanism, reproduced the numbers, and corrected two things now reflected above: the 10× on macOS is mostly 25.0.1's GH-50326 rather than the zoneinfo change, and there is no Windows CI job on
mainyet (that job arrives with #454).🤖 Generated with Claude Code