Motivation
Storage::build resolves the backing opendal operator through a closed match on URI schemes, so an embedding engine cannot supply its own operator — for example one wrapped with a custom service, a metrics/caching layer, or a preconfigured object-store client.
Java Paimon treats configurable file IO as a first-class concern: FileIO is an interface, FileIOLoader is a ServiceLoader SPI matched by scheme, and CatalogContext accepts preferIO/fallbackIO loaders programmatically. paimon-rust has no equivalent injection point.
Proposal
The minimal hook: FileIOBuilder::with_operator(op) accepting a prebuilt opendal::Operator that takes precedence over scheme resolution. A scheme-keyed loader registry mirroring Java's FileIOLoader could follow as a larger change, but the builder hook alone unblocks embedders.
Use case
StreamFusion (github.com/datafusion-contrib/StreamFusion) stores Flink operator state in local Paimon PK tables; its write path needs a thin custom fs service (the stock service's per-write create_dir_all dominated a checkpoint-path CPU profile). It currently carries this hook as a fork patch.
Motivation
Storage::buildresolves the backing opendal operator through a closed match on URI schemes, so an embedding engine cannot supply its own operator — for example one wrapped with a custom service, a metrics/caching layer, or a preconfigured object-store client.Java Paimon treats configurable file IO as a first-class concern:
FileIOis an interface,FileIOLoaderis a ServiceLoader SPI matched by scheme, andCatalogContextacceptspreferIO/fallbackIOloaders programmatically. paimon-rust has no equivalent injection point.Proposal
The minimal hook:
FileIOBuilder::with_operator(op)accepting a prebuiltopendal::Operatorthat takes precedence over scheme resolution. A scheme-keyed loader registry mirroring Java'sFileIOLoadercould follow as a larger change, but the builder hook alone unblocks embedders.Use case
StreamFusion (github.com/datafusion-contrib/StreamFusion) stores Flink operator state in local Paimon PK tables; its write path needs a thin custom fs service (the stock service's per-write
create_dir_alldominated a checkpoint-path CPU profile). It currently carries this hook as a fork patch.