feat(io): let FileIO wrap a caller-provided opendal operator#618
Open
jordepic wants to merge 1 commit into
Open
feat(io): let FileIO wrap a caller-provided opendal operator#618jordepic wants to merge 1 commit into
jordepic wants to merge 1 commit into
Conversation
Embedders that manage their own storage backend today have no way to hand it to FileIO: Storage::build is a closed match over registered schemes. FileIOBuilder::with_operator accepts a prebuilt operator (for example a customized local-filesystem service) and short-circuits the scheme dispatch through a new Storage::Custom variant. Paths reach the operator in scheme-relative form, exactly as the built-in local-filesystem backend receives them, so the operator's root decides what they resolve against.
Author
|
cc @JingsongLi — this is the minimal Rust analog of Java Paimon's configurable FileIO ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #616
Storage::buildresolves the backing opendal operator through a closed scheme match, so an embedding engine cannot supply its own operator (custom service, metrics/caching layer, preconfigured client). Java Paimon exposes this axis through theFileIOinterface, theFileIOLoaderSPI, andCatalogContext'spreferIO/fallbackIO; this is the minimal Rust analog.Brief change log
FileIOBuilder::with_operator(op)stores a prebuiltopendal::Operator; when set, it takes precedence over scheme resolution inbuild.Storage::Custom { op }variant carries it through;create()reuses the fs-relative path handling.Tests
cargo test -p paimon --lib io::(85 tests) passes; the builder path is exercised by the existing FileIO suites plus downstream use in StreamFusion, which drives all its state-table IO through an operator injected here.API and Format
Additive API on
FileIOBuilder; no storage-format change.Documentation
Doc comments on the new builder method.