Push hugging face storage resolution into vortex-cloud rather than vortex-python - #9199
Open
robert3005 wants to merge 3 commits into
Open
Push hugging face storage resolution into vortex-cloud rather than vortex-python#9199robert3005 wants to merge 3 commits into
robert3005 wants to merge 3 commits into
Conversation
…e registry
Vortex could not read from the Hugging Face Hub. `hf://` was understood only by
`vortex.datasets`, which parsed the URI in Python and rewrote it into an
authenticated `HTTPStore`, so `vx.open("hf://...")` failed and Polars, DuckDB and
DataFusion could not open Hub URLs at all.
A Hub repository is a set of files behind an HTTP endpoint that honours range
requests, so serving one needs no cloud SDK: an `object_store::http::HttpStore`
rooted at the repository's `resolve` prefix, carrying a bearer token when one is
available, is the whole implementation. Reads therefore keep every
`ClientOptions` setting a caller passes -- timeouts, retries, proxy
configuration, `allow_http` -- unlike the OpenDAL-backed schemes, whose bridge
owns its own HTTP client.
Unlike every other scheme in the registry, the store is rooted at a
(repository, revision) pair rather than at the URL authority, since both occupy
URL path segments. It therefore reports the in-repository path itself and lets
the registry mount it that deep; getting this wrong would send the repository
name to the Hub as part of the file path, so it is covered from both the
build-and-cache and the cached-store branch.
Listing is not supported. The Hub does not implement WebDAV `PROPFIND`, which is
how `object_store`'s HTTP store lists a prefix. Opening a known path -- `head`
plus ranged `get`, both plain HTTP -- is what a Vortex scan needs, and callers
that must expand a glob can list through the Hub's own API first.
The `hf` feature adds no cloud SDK, only `http` and `percent-encoding`, so
`vortex-python` enables it unconditionally rather than making it opt-in the way
`opendal` is.
Towards #5379.
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lc5zw7Le2T3pakDEUdKTYd
…reads `hf://` `vortex.datasets` built Hub URLs and authorization headers by hand because the reader could not resolve `hf://` itself. It can now, so the URL building (`hf_hub_url`), the saved-login lookup (`get_token`), the endpoint-relative path rewriting, and the separate prefix-rooted store for revisions containing `/` all collapse into emitting `hf://` URIs and letting the reader resolve them. Two cases an `hf://` URI cannot express keep a store of their own, both rooted at the encoded `resolve` prefix: * a `token` string, which the reader has no way to see; * `token=False`, which must suppress the credentials the reader would otherwise read from `HF_TOKEN` or the saved login. Routing this through `hf://` would have quietly authenticated a read the caller asked to be anonymous, so it is covered by its own test. `token=True` asks for exactly the saved login the reader already finds, so it joins the default on the `hf://` path. Listing stays here. The Hub serves no listing over the object-store protocol, so expanding a glob or a directory still goes through the Hub API, which is why `_parse_hf_uri` and the `HfApi` call remain. File paths are percent-encoded on their way into the URI, since they become URI segments; Vortex decodes them back into the object key. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5zw7Le2T3pakDEUdKTYd
Merging this PR will degrade performance by 12.49%
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
Footnotes
|
…x.open` `vortex.datasets` still hand-built a Hugging Face `resolve` URL and an authorization header for the two cases an `hf://` URI cannot express: a token held in a variable, and a read that must stay anonymous despite the environment offering credentials. Expose the store class those cases actually want. `HfStore(repo_id, *, repo_type, revision, token, endpoint)` wraps the `HfConfig` / `make_hf_store` pair the way `CosStore` wraps `CosConfig`. `token` follows `huggingface_hub`'s own convention, so a caller can pass its value straight through: `None`/`True` take whatever the environment offers, `False` forces an anonymous read, and a string is used directly. `revision` is passed literally -- unlike in a URL, the store percent-encodes a revision containing `/` itself. `vx.open` previously accepted only the `pyo3-object_store` classes, which is why `datasets.py` carried a comment that `CosStore` could not be read from. It now takes the same `AnyVortexStore` extraction `read_url`/`write` use, so `HfStore` works there -- and `CosStore` does too, removing that limitation. Two things this surfaced: * `HfConfig` built `ClientOptions::default()`, ignoring `ALLOW_HTTP`. Every other scheme picks it up because `parse_url_opts` reads its configuration from the environment, so a plain-HTTP `HF_ENDPOINT` -- a self-hosted Hub, or a test double -- behaved inconsistently. It is honoured now, which is what lets the new end-to-end read test exercise the real path against a local server. * `HfRepoType` gained `FromStr` so the binding can take `repo_type="dataset"`, accepting the plural the Hub spells in its URLs as well. With the store class in place, `datasets.py` no longer knows the Hub endpoint at all: the `huggingface_hub.constants` import and the last `HTTPStore` use are gone. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5zw7Le2T3pakDEUdKTYd
robert3005
force-pushed
the
claude/vortex-hugging-face-format-1asxf3
branch
from
August 5, 2026 19:30
4014ea0 to
80f90fc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We should treat hf:// urls as any other object store, right now we are keeping all of it in vortex-python which is not quite as versatile as having a full fledged object store.
fixes #5379