Repository navigation
Revert: Replace Pandas with Polars #15395
Description
Activity
@priya-sundaram-dev Adding Polars (#15397) has substantially slowed our build process.
Do we really need to build Polars from scratch on every pull request?
Is there some way to use uv cache to speed up the installation process?
Should wecargo install polarsinstead of installing it with uv?Good catch — and the cause is specific. Polars does ship prebuilt wheels, so normally there's no Rust compile at all. The problem is our interpreter.
As of 1.44.2 the binary lives in
polars-runtime-32, and every wheel it publishes is taggedcp310-abi3(see theuv.lockentries #15397 added).abi3/ stable-ABI wheels are not loadable under free-threaded CPython — the free-threaded build has its own ABI and pip/uv won't match anabi3tag against a3.14tinterpreter. Since our.python-versionis3.14t, uv finds no compatible wheel for polars and falls back to building the sdist, i.e. compiling Rust from scratch on every run. That's the slowdown.So, to your three questions:
- Build from scratch every PR? Only because of
3.14t. On a standard (GIL) CPython thecp310-abi3wheel installs in seconds. cargo install polars? That won't help —polarsisn't a standalone cargo binary crate; it's a Rust library plus Python bindings.cargo installhas nothing to install here.- uv cache? Worth doing regardless (
enable-cache: trueonastral-sh/setup-uv, oractions/cacheon~/.cache/uv), but it only caches the result of the source build — the first run per lock change still pays full compile time.
Cleanest fix: run the Polars-using check on a plain-GIL interpreter (a dedicated matrix leg pinned to
3.14, not3.14t) so it uses the prebuilt wheel, and keep the free-threaded leg for everything else. Whenpolars-runtime-32starts publishingcp314tfree-threaded wheels upstream, we can drop the split. I can open a PR that adds the pinned leg + enables the uv cache if you'd like.- Build from scratch every PR? Only because of
Please create a PR that only enables the uv cache. We do not break the uv lock very often.
Once we have merged or closed all of our open PRs, we can discuss the two test runs (with and without the GIL).
Quick heads-up before I open anything: the uv cache is already enabled on
masterand is hitting. Theastral-sh/setup-uv@v7step already has:- uses: astral-sh/setup-uv@v7 with: enable-cache: true cache-dependency-glob: uv.lock
and the latest build run confirms it's live, not cold:
Cache hit for: setup-uv-2-...-pruned-c5f50d16... Cache restored successfullyAlso worth noting: #15397 was closed, not merged, so Polars is no longer in
uv.lock— nothing is currently building it from source. The slowdown you saw was the one-time cost on that PR itself: adding Polars changeduv.lock, which invalidated the cache key (it's keyed on the lock hash), so that single run had to build thecp310-abi3sdist under3.14twith a cold cache. Since it's closed, that cost is gone.So there's nothing to enable — it's already on. I'd rather not send a no-op PR.
The only thing a PR could add here is eliminating even that one-time cold-build cost on the rare lock change, by giving the cache a prefix
restore-keysfallback so a lock bump restores the previous cache (old built wheels intact) and only re-resolves the delta.setup-uvdoesn't exposerestore-keys, so that means a small hand-rolledactions/cacheblock instead of the built-in — which is the bigger change you said you'd rather avoid, and you noted the lock rarely changes anyway. Happy to do it if you decide it's worth it, but as-is I don't think it's needed. Your call.- chore: uv sync --upgrade + relock; make scripts/pr_file_map.py executable #15398 was merged as the pull request that broke the uv link. Am I misunderstanding the process?
Lines 1199 to 1225 in 6a2ca68
You're not misunderstanding it at all — you're right, and I was wrong on one fact. I conflated this with #15397. Polars is now a permanent dependency (
polars>=1.44.2inpyproject.tomlfrom the Pandas→Polars migration), so it's genuinely inuv.lockand stays there — it was never removed. Apologies for the confusion.So your read of #15398 is exactly correct:
uv sync --upgrade + relockrewroteuv.lock, and since the cache key is the lock hash (cache-dependency-glob: uv.lock), that relock invalidated the cache and the next build hit a cold cache.The reason a cold cache is expensive here specifically is the free-threaded runner.
.python-versionis3.14t, andpolars-runtime-32only shipscp310-abi3wheels. abi3 wheels aren't usable under free-threading, so on3.14tuv can't grab a wheel and builds the ~3 MB Rust sdist from source. On a warm cache that built artifact is restored and it's fast; every lock change pays the Rust build once.So the cache is already on and already doing its job — there's no "enable the cache" PR to send, which is what I was trying (clumsily) to say. The only thing left is whether to soften that once-per-relock cold build. Two honest options:
- Leave it. Relocks are rare, the cost is bounded to one run, and the built-in
setup-uvcache handles the common case. Simplest. - Add a
restore-keysprefix fallback so a lock bump restores the previous cache (Rust build intact) and only re-resolves the delta instead of rebuilding from scratch.setup-uvdoesn't exposerestore-keys, so this means a small hand-rolledactions/cacheblock — the bigger change you'd rather avoid.
Given how seldom the lock changes, my vote is option 1, but I'm happy to send option 2 if you'd prefer the belt-and-suspenders version. And once a
cp314twheel for polars ships, the source build disappears on its own.- Leave it. Relocks are rare, the cost is bounded to one run, and the built-in
- Upload abi3.abi3t wheels pola-rs/polars#27955 makes it clear that the Polars team is not yet ready to release wheels for free-threaded Python.
The long Polars build times are debilitating, so we will close this issue and revert the migration to Polars until that project delivers free-threaded wheels.
@priya-sundaram-dev Please create a pull request that removes Polars from
pyproject.tomland then runs:
%uv sync --upgrade && uv lockDone — #15404 removes
polars>=1.44.2frompyproject.tomland rerunsuv sync --upgrade && uv lock. All 30 polars-related lines drop out ofuv.lockwith no other version changes, and no source file imports polars, so the revert is clean. That takes the Rust-from-source build off the critical path until upstream ships free-threaded (cp314t) wheels.- changed the title
[-]Replace Pandas with Polars[/-][+]Revert: Replace Pandas with Polars[/+]on Sep 22, 2026
Feature description
Please create a pull request that removes Pandas and replaces it with Polars?
https://docs.pola.rs/user-guide/migration/pandas
To find at least 8 algorithms that will need to be fixed, use:
%
git grep "import pandas" && git grep "pandas import"uv sync --upgrade && uv lock#15397pyproject.toml.