diff --git a/bindings/README.md b/bindings/README.md index dc448ce19c20..7f8d60266da9 100644 --- a/bindings/README.md +++ b/bindings/README.md @@ -35,6 +35,12 @@ For example, while the `opendal` crate might be at version `0.55.0`, a binding might be at version `0.47.0` or `0.49.2`. For updates and compatibility, use the specific binding version instead of the `opendal` crate version. +## Design Proposals + +* [Dynamic service and layer extensions](docs/dynamic-extensions/README.md) + explores independently installable packages for the Python, Ruby, and + Node.js bindings. The proposal is not implemented yet. + ## Getting Started Every binding should provide a `README.md` file to help users get started. diff --git a/bindings/docs/dynamic-extensions/README.md b/bindings/docs/dynamic-extensions/README.md new file mode 100644 index 000000000000..fd3a29feabe6 --- /dev/null +++ b/bindings/docs/dynamic-extensions/README.md @@ -0,0 +1,363 @@ + + +# Dynamic Service and Layer Extensions + +Status: pre-RFC design exploration. OpenDAL has not accepted or implemented the +extension interface described by these documents. + +This design allows a language binding to install services and layers as +independent packages. For example, an application can install only S3, +Timeout, and Foyer without rebuilding or replacing the base binding. + +The leading prototype candidate uses a shared native runtime package with a +small common protocol. Python, Ruby, and Node.js use different language +interfaces over the same runtime and native extension model. The proposed +release process publishes the runtime with the language bindings as one +coordinated release family. +Selecting that candidate remains conditional on a successful cross-platform +packaging prototype and the OpenDAL RFC process. + +## Documents + +- [Design alternatives](alternatives.md) compares four possible extension + seams and explains why Design C is the leading prototype candidate. +- [Compatibility and ABI](compatibility.md) defines version and target checks, + loader checks, lifetime rules, and supported stability guarantees. +- [Native symbol isolation](symbol-isolation.md) defines export allowlists, + dependency ownership, artifact monitoring, and co-loading tests. +- [Python design](python.md) defines Python packages, discovery, typing, and + migration. +- [Ruby design](ruby.md) defines Ruby gems, blocking construction, and + middleware migration. +- [Node.js design](nodejs.md) defines npm packages, Node-API integration, + worker behavior, and ESM/CommonJS loading. + +These documents are preliminary design input. They are not an accepted RFC or +a commitment to package names, release dates, or compatibility guarantees. A +subsequent proposal must follow the [OpenDAL RFC +process](../../../core/core/src/docs/rfcs/README.md) before implementation. + +## Requirements + +The design must satisfy all of the following requirements: + +- A service or layer package can be installed without rebuilding the base + language binding. +- A service package owns its configuration schema, URI interpretation, + credentials, redaction behavior, and registry metadata. +- A layer package preserves the complete native `Layer` behavior, including + both service and operation-context composition. +- Stateful layers can be reused intentionally. Reusing one Throttle handle + shares one limiter; reusing one Foyer handle shares one cache. +- Layer construction can be asynchronous. Foyer must not force asynchronous + resource creation into a synchronous registration callback. +- The registry supports at least 1,000 installed manifests without loading + every native library during base import. +- Official and third-party packages use the same extension interface and + compatibility checks. +- The loader rejects incompatible native code before exchanging Rust values or + storing callbacks. +- Active operators, layers, operation bodies, and tasks keep their extension + code loaded. + +## Current Implementation Evidence + +The proposal derives its constraints from these current implementations: + +- The [core split RFC](../../../core/core/src/docs/rfcs/6828_core.md) prepares + service and layer crates for a future extension ecosystem but leaves dynamic + loading unresolved. +- The current [operator registry](../../../core/core/src/types/operator/registry.rs) + stores plain factories, while + [`OperatorUri`](../../../core/core/src/types/operator/uri.rs) defines the + initial URI and explicit-option merge. +- The native [`Layer` trait](../../../core/core/src/raw/layer.rs) composes both + service and operation-context planes. +- [Timeout](../../../core/layers/timeout/src/lib.rs), + [Foyer](../../../core/layers/foyer/src/lib.rs), and + [Throttle](../../../core/layers/throttle/src/lib.rs) exercise context + replacement, asynchronous state, and shared state respectively. +- The [S3 configurator](../../../core/services/s3/src/config.rs) and + [WebDAV configurator](../../../core/services/webdav/src/config.rs) demonstrate + why each service must own URI interpretation. +- The [HDFS service](../../../core/services/hdfs/README.md) documents its + libhdfs/JVM environment, which must remain outside the base binding. +- The [Python](../../python/Cargo.toml), [Ruby](../../ruby/Cargo.toml), and + [Node.js](../../nodejs/Cargo.toml) manifests show the current monolithic + feature and native-library layouts. +- The [security threat model](../../../SECURITY-THREAT-MODEL.md) treats binding + ownership, lifetime, and safe FFI behavior as part of OpenDAL's contract. + +## Candidate Architecture + +```text +Python package Ruby gem npm package + | | | + v v v +Python adapter Ruby adapter Node-API adapter + \ | / + +-----------------+--------------------+ + | + v + OpenDAL extension runtime + registry, Tokio, core types, loader, + errors, handles, library leases + / \ + v v + service extension layer extension + S3 / WebDAV / HDFS Timeout / Foyer / Throttle +``` + +The diagram shows a logical architecture. The selected distribution model puts +the implementation in a shared runtime package instead of embedding a private +copy in each main binding package. Ecosystem-specific packages may wrap +target-specific artifacts, but they must resolve the same runtime release and +runtime identity when loaded into one process. + +### Why one common runtime graph is required + +Services and layers from independent packages must compose into the same +language-level `Operator`. A native layer must be able to wrap an operator from +another package while preserving both `Layer::apply_service` and +`Layer::apply_context`. That requires shared OpenDAL types, Tokio resources, +HTTP and executor context, registry ownership, and handle identity. + +If each package owned a separate OpenDAL runtime graph, Rust `Operator`, `Layer`, +and `OperationContext` values could not cross package boundaries through a +supported stable ABI. Cross-package composition would then require either a +second stable operation interface or a complete operation adapter in each +language. The project does not plan to maintain those larger interfaces, and a +language adapter cannot preserve arbitrary native layer semantics. + +The common runtime graph therefore owns composition machinery and extensions +register release-specific factories into it. Service packages still own their +configuration, URI interpretation, credentials, redaction, and package-local +dependencies. "Common" means that all bindings in one process resolve one +runtime identity rather than loading binding-private runtime graphs. + +The protocol can remain small even though the runtime owns substantial native +state. Independently built packages need only a stable way to acquire a runtime +API, declare their required protocol level, inspect the runtime's supported +protocol range, and invoke runtime-owned factories. +The API implementation can expose a small bootstrap surface, such as an API +lookup and a construction or registration entry point, backed by an extensible +function table. The function signatures, table layouts, value grammar, handle +ownership, and error rules together form the protocol. + +### Binding adapter + +Each binding adapter translates language values, async behavior, exceptions, +and garbage collection into runtime-owned handles. It does not implement +service configuration or native layer composition. + +The adapter preserves each language's normal interface: + +- Python keeps `Operator`, `AsyncOperator`, `await`, type stubs, and Python + package discovery. +- Ruby keeps blocking `Operator` operations, `require`, keyword arguments, and + Ruby exceptions. +- Node.js keeps promises, synchronous variants where supported, JavaScript + streams, ESM/CommonJS exports, and Node-API. + +The project should share the native extension contract, not force one public +language API across all three bindings. The JSON manifest, bootstrap metadata, +configuration value grammar, factory semantics, error categories, and +conformance suite are common. Constructor names, typing, duration syntax, +blocking behavior, and package discovery remain binding-specific. + +### Extension runtime + +The runtime module owns process-scoped native resources. Python and Ruby +initially use one `NativeRuntime` per loaded runtime module and process. The +Node.js runtime calls that owner `ProcessRuntime` and places one +`EnvironmentAdapter` per `napi_env` over it. + +The runtime module owns: + +- The OpenDAL core and Tokio runtime used by extension objects. +- Service and layer registries. +- Scheme dispatch while preserving the original construction request for + package-owned URI parsing and configurators. +- Operator, layer, and operation-body handles. +- Panic containment and language-neutral error details. +- Native library activation and process-lifetime pinning. + +The current `OperatorRegistry` is not sufficient for this role. It stores +plain function pointers and replaces an existing scheme during registration. +The extension registry must record package ownership and perform atomic, +conflict-detecting registration. + +### Language package + +Each installable package contains: + +- Language code or type declarations. +- A JSON manifest for exactly one service or layer, with the package ID, + canonical component ID, service aliases when applicable, native artifact + path, unique entry symbol, and required OpenDAL version. +- A target-specific native library, or source needed to build one. +- Package-specific documentation and conformance tests. + +Registering the JSON manifest must not load the native library. The runtime +activates the library when a caller first constructs a declared service or +layer. This isolates HDFS and similar native dependencies. + +### Native extension + +A native extension contains exactly one service or layer factory. The package +exports one generated, package-unique bootstrap symbol. The final artifact must +hide every other package-local symbol and pass the [native symbol isolation +contract](symbol-isolation.md). Language-runtime initializers remain explicit +allowlist exceptions. + +The bootstrap exposes the package identity, target, and exact OpenDAL version +through a small C calling convention. The [compatibility +contract](compatibility.md#bootstrap-encoding-alternatives) compares a bounded +JSON document with an exact-release C-layout descriptor instead of removing +either encoding before prototyping. Only after all checks pass may the runtime +enter the release-specific internal interface. That internal interface is not a +stable Rust ABI. + +Node.js packages are an exception to the unique-initializer rule when the +native artifact is itself a Node-API addon: Node defines the addon initializer. +The Node adapter must use a language-appropriate initializer and validate the +same package identity in the returned bootstrap metadata. + +## Construction Requests + +A service factory accepts one of two request forms: + +```text +UriRequest { + raw_uri, + explicit_string_options, +} + +ConfigRequest { + structured_values, +} +``` + +`structured_values` uses the language-neutral `ConfigValue` grammar +defined by [the compatibility contract](compatibility.md#configuration-value-contract). +Each binding validates and converts its public values before invoking a native +factory; a native package never receives Python, Ruby, or JavaScript objects. + +The URI request preserves the original URI and explicit options. The package +constructs its local `OperatorUri` and calls its own `Configurator::from_uri`. +The runtime must not replace service-specific behavior with a universal +precedence rule. + +A layer factory accepts structured configuration and returns an asynchronous +result: + +```text +create_layer(layer_id, structured_values) -> future +``` + +`LayerHandle` retains state and can be applied to more than one operator. +Applying a layer returns a new operator. The runtime never sorts layers: later +applications are outer layers, matching OpenDAL core behavior. + +## Registration Rules + +- Scheme and layer IDs use lowercase canonical strings. +- One manifest claims exactly one service or layer. A service manifest can also + declare aliases for its canonical scheme. +- Re-registering the identical package and manifest is idempotent. +- A different owner claiming the component ID or one of its aliases rejects the + manifest. +- Registry locks are not held while loading code or running a factory. +- Capabilities are read from a constructed operator, not declared statically in + registration metadata. +- A failed native activation affects only the requested package. +- Extensions remain loaded for the rest of the process in the first version. + +## Package Families + +The selected release family has three package roles: + +```text +shared runtime package native runtime, protocol, registries, and handles +main binding package Python, Ruby, or Node.js public API and adapter +service/layer package manifest, language API, and native implementation +``` + +The main binding package declares the `required_runtime_protocol` that its +adapter needs. The runtime package exposes `minimum_runtime_protocol` and +`runtime_protocol` so the adapter can verify that requirement before using the +runtime API. A service or layer package also depends on the runtime package, +while its native artifact follows the exact-release extension compatibility +rules. A minimal installation uses the runtime package and one main binding +package; applications then add selected extension packages. + +The proposed release process publishes these packages together. Coordinated +publication gives every binding and official extension a consistent runtime +implementation, build contract, and compatibility matrix. Keeping the +implementation in its own package also makes upgrades and dependency +diagnostics easier to manage than embedding equivalent native code independently +in every binding. + +Independently installable does not imply independently ABI-versioned: every +native extension package must be rebuilt for every OpenDAL release, even when +the internal interface appears unchanged. + +The 1,000-manifest requirement measures registry behavior, not a commitment to +publish 1,000 official package families. Before a split release, each ecosystem +needs reserved package names, trusted publishing, coordinated release tooling, +and rollback rules. OpenDAL can publish only supported high-value extensions +while leaving the same SDK available to third parties. + +## Security Model + +Native extensions are trusted in-process code. Compatibility validation +prevents accidental mismatches; it does not sandbox, authenticate, or constrain +a malicious package. + +OpenDAL still owns these in-scope properties: + +- No panic or language exception crosses any FFI boundary. +- Safe binding operations do not cause use-after-free or data races. +- Official packages redact credentials from errors and diagnostics. +- Registry metadata and build diagnostics do not contain secrets. +- An extension callback cannot run after its library is unloaded. + +The loader must use an explicit package-provided artifact path. It must not scan +arbitrary library search paths and execute every matching file. + +## Delivery Gates + +The design is not ready for an RFC decision until prototypes demonstrate: + +1. A minimal runtime plus separately packaged FS and Retry tracers. Memory + remains in the runtime because OpenDAL core always provides it. +2. S3 and WebDAV URI behavior without host-owned configuration schemas. +3. Lazy HDFS failure without affecting S3, WebDAV, or `hdfs-native`. +4. Complete Timeout context behavior and shared Throttle state. +5. Asynchronous, stateful Foyer construction and lifetime handling. +6. Atomic registration and lazy lookup with 1,000 synthetic packages. +7. Target packaging on supported Linux, macOS, and Windows variants. +8. Python, Ruby, and Node.js adapters passing the same native conformance suite. +9. Final-artifact export, import, and native-dependency reports matching + explicit platform allowlists. +10. Co-loading independently compiled extensions without symbol interposition, + including real operations that exercise runtime-owned facilities. + +If those gates pass, the next deliverable is a `0000_*.md` RFC. These pre-RFC +documents remain supporting analysis rather than evidence that the candidate +has already been accepted. diff --git a/bindings/docs/dynamic-extensions/alternatives.md b/bindings/docs/dynamic-extensions/alternatives.md new file mode 100644 index 000000000000..0eb549d00181 --- /dev/null +++ b/bindings/docs/dynamic-extensions/alternatives.md @@ -0,0 +1,348 @@ + + +# Dynamic Extension Design Alternatives + +Status: pre-RFC design comparison. No alternative has been accepted or +implemented. + +This document compares four ways to deliver OpenDAL services and layers to +language bindings. Designs B and D remain useful comparisons even though their +maintenance or semantic costs make them unsuitable for the current proposal. + +## Decision Guide + +```text +Must packages compose after installation? + no -> Design A: feature-selected monolithic binding + yes -> Must arbitrary native layers preserve core semantics? + no -> Design B: language-owned operator adapters + yes -> Must old binaries survive OpenDAL upgrades? + no -> Design C: exact-release shared runtime + yes -> Design D: stable C operation interface +``` + +OpenDAL should prototype Design C. Design A remains useful for custom builds. +Design B does not preserve arbitrary native layers and repeats an operation +adapter in every language. Design D preserves cross-version binaries only by +requiring OpenDAL to maintain a second operation model. + +## Runtime Distribution Alternatives + +Design C still needs a distribution choice. The protocol and the package are +separate concepts: independently built components always need a protocol, even +if that protocol consists of only a small API lookup and construction surface. +The runtime package determines who ships and owns its implementation. + + + +| Model | Distribution | Strengths | Limitations | +| ----- | ------------ | --------- | ----------- | +| Binding-embedded runtime | Every main binding package carries a private runtime implementation | Simplest installation; no separate runtime dependency; binding maintainers control loading | Duplicates native artifacts; can create multiple runtime identities in one process; lets bindings drift in behavior and build settings | +| Host-provided runtime | The main binding loads a runtime and exposes its API to extension packages | Avoids a separately visible runtime package; extensions reuse the host instance | Couples discovery to each host language; makes extensions depend on host loading order and adapter-specific mechanisms; does not naturally coordinate multiple languages | +| Shared runtime package | Main bindings and extension packages depend on one separately versioned runtime package | Gives all languages one runtime identity and implementation; centralizes compatibility checks, fixes, and native resources; fits coordinated OpenDAL releases | Requires cross-ecosystem native artifact resolution; needs package-manager rules against incompatible duplicate runtimes; adds an explicit dependency | + + + +OpenDAL selects the shared runtime package for the prototype. The project +will publish the runtime and language bindings as a coordinated release family, +which strengthens consistency and simplifies release management. +Although each binding could carry the same implementation, a separate package +provides one place to version, test, diagnose, and update that implementation. +Target-specific wheels, gems, and npm packages may act as ecosystem delivery +wrappers, but they must represent the same runtime release and protocol. + +## Summary + + + +| Property | A. Static build | B. Language adapters | C. Exact-release runtime | D. Stable C interface | +| ------------------------------ | ----------------- | -------------------- | ------------------------ | -------------------------- | +| Independently install services | No | Yes | Yes | Yes | +| Independently install layers | No | Language decorators | Yes | Yes | +| Full native layer semantics | Yes | No | Yes | Only represented semantics | +| One core/runtime graph | Yes | Usually no | Yes | Usually no | +| Third-party source extensions | Rebuild host | Yes | Yes, exact release | Yes | +| Cross-version native binaries | Not applicable | Language-dependent | No | Yes, within ABI rules | +| Cross-language native package | No | No | Possible, not promised | Yes, by design | +| Native dependency isolation | No | Yes | Yes | Yes | +| Implementation cost | Lowest | Moderate | High | Highest | +| Primary risk | Artifact variants | Semantic loss | Packaging and linking | ABI breadth and safety | + + + +## Design A: Feature-Selected Monolithic Binding + +### Static Build Seam + +Cargo features select services and layers when the language extension is +compiled. The resulting wheel, gem, or npm native package contains one OpenDAL +core and all selected implementations. + +```text +language caller -> one native binding -> compiled services and layers +``` + +### Static Build Strengths + +- Preserves all native behavior without a new runtime interface. +- Uses ordinary Rust ownership and one Tokio/core graph. +- Has the smallest implementation and verification cost. +- Works well for downstream users who build one controlled deployment image. + +### Static Build Limitations + +- A service cannot be added after the binding is built. +- Two feature variants normally provide the same import or module name and + cannot be installed together. +- One uncommon native dependency can constrain the whole artifact. A build + containing libhdfs-backed HDFS may fail to load where Java or Hadoop libraries + are absent. +- A large published feature set increases build time, artifact size, supply + chain surface, and platform exclusions. + +### Static Build Use + +Keep this path for custom source builds and hermetic distributions. It does not +meet the independently installable package requirement. + +## Design B: Language-Owned Operator Adapters + +### Language Adapter Seam + +Each service package owns an operator and exposes a Python, Ruby, or JavaScript +operation protocol. The base binding delegates every operation through that +language protocol. Layers decorate language objects. + +```text +language Operator facade -> language operation protocol -> package-owned backend + ^ + | + language decorator +``` + +### Language Adapter Strengths + +- Uses normal language package-loading mechanisms. +- Allows pure-language third-party services without a Rust ABI. +- Lets each native package own its core version and runtime because Rust values + do not cross the package boundary. +- Isolates HDFS dependencies in the HDFS package. + +### Language Adapter Limitations + +- Repeats or generates the complete operation adapter for every language. +- Adds language calls to streaming and operation-body paths unless the adapter + adds another native batching interface. +- Prevents a native OpenDAL layer from wrapping an operator owned by another + extension. +- Eventually requires readers, writers, listers, deleters, copiers, + cancellation, HTTP context, and executor context to reproduce native layers. + +Timeout is decisive. `TimeoutLayer` wraps service calls and replaces the +executor in `OperationContext`. A method decorator cannot reproduce the +executor behavior used by concurrent block operations. Foyer is stateful and +intercepts operation bodies; a language decorator can implement a different +cache but cannot claim native `FoyerLayer` equivalence without a larger +protocol. + +### Language Adapter Use + +Keep this design as the service-only comparison. It does not meet the proposal's +complete native-layer requirement, and OpenDAL does not plan to maintain a full +operation adapter separately for every language. + +## Design C: Exact-Release Shared Native Runtime + +### Shared Runtime Seam + +The binding adapter and all native packages use one shared native runtime +module. That module owns OpenDAL and Tokio types and registries. Each extension +registers one erased service or layer factory compiled for exactly the same +OpenDAL release. Node.js can place an environment adapter over those +process-scoped resources; it does not create a second OpenDAL graph for each +Worker. + +```text +language adapter -> extension runtime <- service/layer native libraries + | + v + runtime-owned Operator +``` + +The loader reads compatibility metadata through a small C calling convention. +The [compatibility contract](compatibility.md#bootstrap-encoding-alternatives) +compares a JSON document with a C-layout descriptor for that metadata. The +loader validates the exact OpenDAL version and target before entering a +release-specific internal interface. + +### Shared Runtime Strengths + +- Preserves the native `Operator`, `Layer`, and `OperationContext` model. +- Keeps one OpenDAL core, Tokio runtime, and shared HTTP/context graph. +- Makes service and layer packages small and keeps configuration local. +- Supports stateful layer handles and asynchronous layer construction. +- Isolates native dependencies through lazy activation. +- Shares the extension SDK and conformance suite across language bindings. + +### Shared Runtime Limitations + +- Does not promise Rust ABI stability. Every native package rebuilds for each + OpenDAL release. +- Requires cross-platform shared-library discovery, repair, + rpath/install-name behavior, Windows DLL lookup, and symbol visibility work. +- Requires final-artifact export allowlists and co-loading tests so private + dependency symbols cannot resolve across the runtime and extensions. +- Requires package managers to prevent or clearly reject mixed release trains. +- Needs artifact-level proof before one physical extension artifact can be + shared across language ecosystems. +- Treats native packages as trusted code and does not unload them in the initial + design. + +### Shared Runtime Use + +This is the recommended balance when install-time composition and complete +native layers matter, but old extension binaries do not need to survive OpenDAL +upgrades. + +## Design D: Stable C Operation Interface + +### Stable C Interface Seam + +The base binding and each native extension exchange only C-layout function +tables, opaque reference-counted handles, fixed-width scalars, buffers, errors, +and versioned wire values. Rust values never cross the package boundary. + +```text +language adapter -> base operation graph + | + v + stable C ABI + | + v + extension operation graph +``` + +To support arbitrary services and layers, the interface must represent: + +- Service construction, capabilities, and every operation. +- Readers, writers, listers, deleters, copiers, and streams. +- Futures, polling or completion, waking, and cancellation. +- Buffer ownership and creator-side destruction. +- Errors and extensible operation options. +- HTTP and executor resources in `OperationContext`. +- Separate service and context hooks for layers. + +### Stable C Interface Strengths + +- Allows extensions to use different OpenDAL core and Rust compiler versions. +- Lets a correctly versioned extension binary survive base-binding upgrades. +- Can share native extensions across Python, Ruby, and Node.js on one target. +- Allows other C-ABI languages to implement extensions. + +### Stable C Interface Limitations + +- Creates a second stabilized representation of OpenDAL's raw operation model. +- Creates a large unsafe verification surface around async cancellation, stream + ownership, context composition, and wire evolution. +- Can load an old extension that lacks new operation semantics, requiring + explicit negotiation and unsupported results for every evolution. +- Allows each extension to embed its own OpenDAL and Tokio graph, increasing + size and making cross-extension layers more expensive. + +Long-term ABI governance would require size-tagged function tables, append-only +minor evolution, explicit breaking revisions, creator-provided destructors, +and documented task polling and cancellation semantics. Binary compatibility +would mean safe loading and explicit capability negotiation, not automatic +support for every new operation. + +### Stable C Interface Use + +Keep this design as the comparison for cross-version binary compatibility. +OpenDAL does not plan to implement it because its maintenance and verification +costs exceed the current requirement. A service-only C interface would not meet +the native-layer requirement. + +## Constraint Comparison + +### S3 and WebDAV + +All designs keep URI interpretation inside the service implementation. The base +binding passes a raw URI and explicit options instead of maintaining a central +configuration schema. + +This matters because generic precedence is insufficient: + +- `OperatorUri` applies query options first and explicit options second. +- S3 then derives bucket and root from the URI name and path. +- WebDAV derives `https://authority` as its endpoint, overwriting an endpoint + option supplied earlier. + +Designs B, C, and D preserve these rules when the package owns construction. A +base-owned universal schema is rejected in every design. + +### HDFS + +Design A cannot guarantee that the base artifact loads without HDFS native +dependencies when HDFS is compiled into it. Designs B, C, and D can register a +lightweight manifest and activate HDFS only at construction. `hdfs` and +`hdfs-native` remain independent packages and schemes. + +### Foyer + +Foyer requires asynchronous cache creation and a reusable stateful handle. +Design C holds the native layer in the shared runtime. Design D represents it +through a reference-counted layer handle. Design B can provide a language-level +cache but cannot claim arbitrary native layer composition. + +Foyer's current key uses path and optional version, not service identity. No +design should advertise one cache handle as isolated across unrelated storages +until the package adds namespacing or restricts the handle to one logical +storage. + +### Timeout and Throttle + +Designs A, C, and D preserve both Timeout service wrapping and executor +wrapping. Design B cannot do so through method delegation alone. + +One Throttle layer owns shared limiter state. Designs C and D preserve the +identity of the layer handle so applying it to two operators shares a quota. +The extension adapter must validate positive bandwidth and burst values before +calling a constructor that currently asserts them. + +Every viable design must preserve the canonical Timeout/Retry ordering defined +by the [layer compatibility rules](compatibility.md#layer-compatibility-rules). + +## Why Node.js Does Not Change the Decision + +[Node-API](https://nodejs.org/api/n-api.html) is ABI-stable across supported +Node.js versions, so it is a strong boundary between JavaScript and the base +addon. It does not stabilize OpenDAL's internal Rust types or external libraries +used by the addon. Therefore Node.js benefits from Design C for the same reason +as Python and Ruby. + +Node.js already publishes target-specific native npm packages, which provides a +useful packaging pattern for extension artifacts. It still needs an exact +OpenDAL version check, lazy activation, per-Worker adapter state, and a clear +rule against transferring JavaScript wrappers between Node environments. + +## Recommendation + +Prototype Design C with one language-neutral extension SDK and three binding +adapters. Preserve Design A as a supported custom-build path. Keep Designs B +and D documented as rejected alternatives so future proposals can evaluate +whether their requirements justify the maintenance or semantic costs. diff --git a/bindings/docs/dynamic-extensions/compatibility.md b/bindings/docs/dynamic-extensions/compatibility.md new file mode 100644 index 000000000000..83959d135667 --- /dev/null +++ b/bindings/docs/dynamic-extensions/compatibility.md @@ -0,0 +1,612 @@ + + +# Extension Compatibility and ABI + +Status: pre-RFC contract for the shared runtime candidate. OpenDAL does not +provide or guarantee this interface today. + +The design uses two compatibility axes. A language binding declares the runtime +protocol level that it requires. A native extension must target exactly the +OpenDAL version used by the runtime. The target identity remains a separate +platform check, not another versioning scheme. + +The key constraint is deliberate: + +> The candidate design provides stable public language interfaces, but it does +> not provide a stable Rust extension ABI across OpenDAL releases. + +Every native extension rebuilds for every OpenDAL release. Equal layouts in two +releases do not create a supported compatibility range. + +## Public Language Interfaces + +Python, Ruby, and Node.js continue to version their public APIs according to +their existing binding policies. CPython `abi3`, Node-API, and Ruby or Magnus +compatibility describe the interface between a binding adapter and its language +runtime. They are informational for the native extension loader and do not +replace the exact OpenDAL version check. + +## Runtime Protocol Compatibility + +The binding-to-runtime boundary uses a common, language-neutral protocol. The +protocol is distinct from the runtime package version and from the +exact-release native extension interface. + +Each main binding package declares one `required_runtime_protocol`. The runtime +package reports an inclusive supported range: + +- `minimum_runtime_protocol` identifies the oldest protocol contract that the + runtime still supports. +- `runtime_protocol` identifies the newest protocol capability that the runtime + provides. + +A binding is compatible when: + +```text +minimum_runtime_protocol + <= required_runtime_protocol + <= runtime_protocol +``` + +For example, a binding that requires protocol 20 can use a runtime that supports +protocols 18 through 23. Package metadata provides an early diagnostic, but the +loaded runtime reports the authoritative values before returning an API table. + +The runtime reports the range and acquires the requested API through one +bootstrap function equivalent to: + +```c +typedef struct { + uint32_t struct_size; + uint32_t minimum_runtime_protocol; + uint32_t runtime_protocol; +} opendal_runtime_protocol_info_v1; + +typedef struct opendal_runtime_api opendal_runtime_api; + +int32_t opendal_runtime_get_api_v1( + uint32_t required_runtime_protocol, + opendal_runtime_protocol_info_v1 *protocol_info, + const opendal_runtime_api **api +); +``` + +The runtime fills `protocol_info` even when the requested level is incompatible +and returns a null `api` in that case. On success, it returns an API table that +conforms to `required_runtime_protocol`. The binding performs this call before +converting configuration or acquiring a runtime-owned handle. The binding uses +no capability introduced after its required level, and the runtime does not +send that interaction a value, callback, or handle kind introduced after that +level. A newer runtime therefore preserves the complete behavior of every +protocol at or above `minimum_runtime_protocol`. + +The binding's requested level is also the interaction ceiling for an extension. +The loader rejects an extension whose `required_runtime_protocol` is greater +than the binding's requested level, even when the runtime itself provides that +newer level. An extension can require an older level because the exact-release +runtime and extension enter the interaction at the binding's requested level. + +An incompatible change to this bootstrap function uses a new exported symbol, +such as `opendal_runtime_get_api_v2`; it does not add a separately negotiated +ABI-major field to the normal protocol check. + +The protocol should expose the smallest practical surface. The exported +bootstrap function acquires a size-tagged API table; a registration or +construction function in that table performs the main operation. Even this +two-function interaction has a protocol: its function signatures, table layout, +`ConfigValue` representation, status codes, handle ownership, and lifetime +rules are the compatibility contract. + +Adding an ordinary service configuration field does not raise the runtime +protocol level because each service package owns its schema. Adding a new +shared `ConfigValue` variant, handle kind, factory capability, or lifetime rule +does raise the protocol level. The binding that first uses that capability then +raises `required_runtime_protocol`. The protocol does not contain language- or +service-specific identifiers such as `opendal.python.s3`. + +OpenDAL distributes the protocol implementation in the shared runtime package +and publishes that package with the language bindings. Ecosystem-specific +delivery wrappers must resolve the same runtime identity when multiple bindings +load in one process. + +## Configuration Value Contract + +Factories receive language-neutral configuration values. The runtime protocol +defines this closed grammar at the binding's requested protocol level: + +```text +ConfigValue = + Null + | Bool(bool) + | I64(i64) + | U64(u64) + | F64(finite IEEE-754 binary64) + | Utf8(string) + | Bytes(byte sequence) + | List(sequence) + | Map(map) + | SignedDuration(seconds: i64, nanoseconds: i32) +``` + +`SignedDuration` is the only duration representation at the shared factory +boundary. It matches `jiff::SignedDuration`: the nanosecond field has an +absolute value below one second and has the same sign as the seconds field when +both are non-zero. This preserves the full signed range without squeezing total +nanoseconds into one `i64`. + +Binding-specific interfaces can accept seconds, milliseconds, or native +duration objects, but their adapters must define nanosecond rounding and reject +non-finite, non-canonical, or overflowing values. Each service or layer decides +whether its configuration accepts negative or zero durations. Numeric +conversion never silently truncates, saturates, or wraps. + +The decoder enforces a maximum nesting depth of 32, a maximum UTF-8 map key of +4 KiB, a maximum individual string or byte value of 16 MiB, a maximum of 65,536 +entries in one list or map, and a maximum encoded request size of 64 MiB. It +rejects invalid UTF-8, duplicate map keys, non-finite floats, unknown value tags, +and values outside the declared numeric ranges before calling package code. + +A package owns its configuration format, fields, defaults, validation, +credentials, redaction behavior, and any schema version. `ConfigValue` defines +only the shared transport vocabulary; it does not impose one schema model on +all bindings or packages. + +Each package therefore decides: + +- Whether a missing field selects a default or produces an error. +- Whether `Null` differs from a missing field. +- Whether to reject, ignore, or preserve unknown fields. +- How to represent and evolve package-specific configuration versions. +- Which credential and token fields require redaction. + +The runtime enforces the transport limits above and never logs raw configuration +values. Package validation errors must not render secret values. + +`UriRequest` remains separate. It contains one UTF-8 URI and a map of UTF-8 +option names to UTF-8 option values, matching current iterator construction. +The runtime applies the same size limits and never includes raw option values in +loader diagnostics. Service code remains responsible for URI semantics and +service-specific validation. + +## JSON Manifest + +Each language package registers one JSON manifest without activating native +code. A manifest declares exactly one service or layer. A service can also +declare aliases for its canonical scheme. + +An illustrative service manifest is: + +```json +{ + "opendal_version": "0.55.0", + "required_runtime_protocol": 20, + "package_id": "opendal-service-s3", + "package_version": "0.55.0", + "component": { + "kind": "service", + "id": "s3", + "aliases": [] + }, + "native_artifact_path": "lib/opendal_service_s3.so", + "native_entry_symbol": "opendal_service_s3_bootstrap_v1", + "target_identity": "x86_64-unknown-linux-gnu" +} +``` + +The runtime protocol defines the common document fields and their meaning. The +loader first verifies that `required_runtime_protocol` does not exceed the +binding's already validated requested level, then rejects a native extension +manifest whose `opendal_version` differs from the runtime. It does not negotiate +native extension compatibility from the protocol level. `package_version` +remains package metadata and does not establish native compatibility. + +The registry validates document size, UTF-8, JSON structure, IDs, aliases, and +artifact paths before storing the manifest. It must not store credentials, URI +options, or service configuration in registration metadata. + +## Bootstrap Encoding Alternatives + +The installed discovery manifest remains JSON so the registry can inspect it +without loading native code. The native bootstrap repeats its compatibility +metadata after the library loads. The bootstrap encoding is still a design +choice: OpenDAL should compare a bounded JSON document with an exact-release +C-layout descriptor instead of discarding either option before prototyping. + +For directly loaded native libraries, both encodings use a package-unique +exported function with a fixed C calling convention. They exist only to reject +an incompatible native artifact before the runtime enters the release-specific +interface. + +Both encodings sit behind the same stable bootstrap envelope and function +signature: + +```c +enum { + OPENDAL_BOOTSTRAP_JSON = 1, + OPENDAL_BOOTSTRAP_C_LAYOUT_V1 = 2, +}; + +typedef struct { + uint32_t struct_size; + uint32_t payload_encoding; + const unsigned char *payload; + size_t payload_len; +} opendal_bootstrap_result_v1; + +typedef int32_t (*opendal_bootstrap_fn_v1)( + opendal_bootstrap_result_v1 *result +); +``` + +The host initializes `struct_size` to `sizeof(opendal_bootstrap_result_v1)` and +zeroes the other fields before calling the package-unique symbol. Every version +1 bootstrap symbol ends in `_bootstrap_v1` and uses this signature. A future +incompatible envelope uses a new symbol suffix, so the loader never guesses a +function signature from unvalidated package metadata. + +The function returns one of these status codes: + +- `0`: Success. The result contains one recognized, bounded payload. +- `1`: Invalid argument, including a null result pointer. +- `2`: Unsupported bootstrap envelope, including an undersized `struct_size`. +- `3`: The package could not provide bootstrap metadata. + +The loader treats every non-zero or unknown status as `BootstrapInvalid` and +does not read the payload fields. On success, the payload is immutable +package-owned memory that remains valid while the library is loaded. The loader +rejects a null payload, a zero or excessive length, or an unknown encoding +before decoding it. + +A Node-API addon initializer cannot use a package-unique C initializer. It +returns the same status, payload-encoding discriminant, and bounded payload +through Node-API values. The environment adapter applies the same validation +before passing metadata to the process runtime. + +Prototype runtimes can accept both payload encodings through this envelope for +comparison. A published OpenDAL release selects one encoding for its supported +SDK and official packages. The common function signature remains the same, so a +stale manifest cannot make the loader call the bootstrap with the wrong ABI. + + + +| Property | JSON payload | C-layout payload | +| ------------------ | ------------------------------------------------- | ------------------------------------------------ | +| Bootstrap call | Common version 1 envelope | Common version 1 envelope | +| Installed manifest | JSON | JSON | +| Native metadata | UTF-8 names and values | Size-tagged structure with bounded byte slices | +| Payload ownership | Immutable package memory | Immutable package memory | +| Validation surface | Length, UTF-8, JSON, fields | Pointer, length, size, alignment, and fields | +| Human inspection | Direct | Requires a decoding tool | +| Node-API transport | String or byte buffer | Wrapper around the native structure | +| Evolution | Schema follows the exact OpenDAL release | New layout needs a new payload discriminant | +| Main risk | Parser complexity and non-canonical serialization | Unsafe pointer, length, and alignment validation | + + + +### JSON Payload + +The JSON payload is a bounded UTF-8 document. An illustrative document is: + +```json +{ + "opendal_version": "0.55.0", + "required_runtime_protocol": 20, + "package_id": "opendal-service-s3", + "package_version": "0.55.0", + "component_kind": "service", + "component_id": "s3", + "target_identity": "x86_64-unknown-linux-gnu", + "entry_symbol": "opendal_service_s3_entry" +} +``` + +The loader validates the envelope length before parsing. The RFC must define +canonical encoding where bytewise comparison matters; field comparison must not +depend on JSON object order. + +### C-Layout Payload + +The C-layout payload points to this illustrative version 1 metadata structure: + +```c +typedef struct { + const unsigned char *data; + size_t len; +} opendal_bytes; + +typedef struct { + uint32_t struct_size; + uint32_t required_runtime_protocol; + opendal_bytes opendal_version; + opendal_bytes package_id; + opendal_bytes package_version; + opendal_bytes component_kind; + opendal_bytes component_id; + opendal_bytes target_identity; + opendal_bytes entry_symbol; +} opendal_c_metadata_v1; +``` + +The package contract requires every returned pointer to reference immutable +package-owned memory for the declared lifetime. The loader can reject null or +misaligned pointers and invalid structural bounds, but it cannot prove that an +arbitrary in-process pointer is mapped safely. After the checkable pointer and +alignment checks, the loader requires `payload_len` to cover the `struct_size` +field. It then requires `struct_size` to contain every version 1 field and not +exceed `payload_len` before reading any byte slice. Version 1 defines the +complete layout needed to read the OpenDAL version; changing that layout +requires a new payload encoding discriminant. The RFC must define maximum slice +lengths, encoding, and structure lifetime. + +### Shared Bootstrap Rules + +Whichever encoding the prototype selects, the bootstrap follows these rules: + +- Every exported function uses an explicit C calling convention. +- No panic or foreign exception crosses the call. +- Every package-unique bootstrap symbol uses the common envelope signature. +- The loader checks the required runtime protocol, OpenDAL version, package + identity, component identity, target identity, and entry symbol before + invoking the release-specific entry point or factory. +- The JSON manifest and native bootstrap must identify the same package and + component. +- The loader reports incompatible metadata; it does not attempt ABI adaptation. + +The bootstrap does not expose `Operator`, `Layer`, trait objects, Rust strings, +Rust enums, futures, Tokio handles, or allocator ownership. + +JSON is the leading candidate because the discovery manifest and native +metadata can share parsing and scalar-value conventions, and the Node-API +adapter can transport it without native structure access. The C-layout payload +remains a candidate if the prototype demonstrates simpler or safer activation +on the supported native targets. The selection must follow cross-platform +loader tests, not document preference alone. + +## OpenDAL Release Compatibility + +The runtime, extension SDK, and official extension packages form one coordinated +OpenDAL release. The SDK build tool pins the exact dependencies, compiler, +target, Cargo profile, Rust flags, panic strategy, and linkage policy used by +that release. It generates both the package manifest and embedded bootstrap +document instead of asking extension authors to copy compatibility metadata into +source code. + +Package-local dependencies such as S3 signing, XML parsing, `hdrs`, Foyer, or a +rate limiter can use different versions when their types and globals remain +inside the package and their dynamic symbols satisfy the [native symbol +isolation contract](symbol-isolation.md). Any value exchanged through the +internal interface follows the exact SDK contract for that OpenDAL release. + +The project never replaces a published runtime artifact with different bits +under the same OpenDAL version. A changed artifact requires a new release. + +## Change Impact + + + +| Change | OpenDAL version | Extension action | +| ------------------------------------------- | --------------- | ------------------------------ | +| Install another compatible component | Unchanged | Register the new manifest | +| Change package-private code or dependencies | Unchanged | Republish only that package | +| Change a public binding method only | Per binding | No native rebuild | +| Change the manifest or bootstrap contract | New release | Rebuild every native extension | +| Change an SDK handle or factory layout | New release | Rebuild every native extension | +| Change ABI-visible OpenDAL code or features | New release | Rebuild every native extension | +| Change compiler, panic, or linkage inputs | New release | Rebuild every native extension | + + + +## Exact-Release Internal Interface + +After bootstrap validation, the runtime enters a release-specific factory +interface generated by the extension SDK. That interface can exchange +runtime-owned operator and layer handles or exact-release Rust adapters. + +The following constraints apply: + +- The interface is compatible with exactly one OpenDAL release. +- The [Rust ABI has no stability guarantees](https://doc.rust-lang.org/reference/items/external-blocks.html). +- The loader rejects a mismatch before registering factory pointers. +- Package-local code catches panics before returning across the boundary. +- Ownership remains on the creating side unless an SDK handle explicitly + transfers it. +- Opaque runtime handles are preferred over exposing Rust types directly. +- All cross-artifact calls enter through validated bootstrap or SDK functions; + exact-release matching does not permit ambient Rust symbol resolution. + +## Target Identity + +An equal OpenDAL version does not make an artifact portable across targets. The +loader must also match: + +- Operating system and architecture. +- Pointer width, endianness, and calling convention. +- Linux libc family and minimum version where applicable. +- macOS deployment target and architecture. +- Windows toolchain and runtime family. +- Required CPU target features. +- Language adapter variant when the runtime is not physically shared. + +Cross-language reuse of one physical native artifact is supported only after +artifact-level tests prove that the adapters load the same runtime identity on +that target. + +## Loader State Machine + +```text +unregistered + | + | register validated JSON manifest + v +registered + | + | first construction + v +loading -> bootstrap validated -> factory installed -> active + | | | + +---- error -----+--------------------+--> failed +``` + +The loader provides these properties: + +1. One manifest registers one component and its service aliases atomically. +2. An identical registration is idempotent. +3. A different owner for the component or an alias rejects registration. +4. Registry locks are released before filesystem access, `dlopen`, addon + loading, or package code. +5. Only the requested package activates. +6. One failed package does not poison unrelated registrations. +7. Concurrent first construction activates a package once and shares the + result or failure deterministically. +8. The runtime retains a library lease before storing any callback. +9. The initial implementation never unloads an activated native extension. + +The current core registry silently replaces an existing scheme. The extension +registry must reject conflicts instead. + +## Package Manager Constraints + +Package metadata provides an early compatibility diagnostic. The native loader +remains authoritative. + + + +| Ecosystem | Main binding constraint | Native extension constraint | Additional requirement | +| --------- | ---------------------------------------- | --------------------------------------- | --------------------------------------------------------- | +| Python | Required runtime protocol | Exact OpenDAL runtime release | Bootstrap version check; wheel target must match | +| Ruby | Required runtime protocol | Exact OpenDAL runtime release | Bootstrap version check; source/native policy is explicit | +| Node.js | Required runtime protocol | Exact runtime and target dependencies | Reject a nested incompatible `ProcessRuntime` | + + + +Dependency installation alone does not register an extension. Each binding uses +its defined discovery mechanism and preserves construction-time native +activation. + +## Service Compatibility Rules + +- A service factory receives the raw URI and explicit string options. +- The service package owns `Configurator::from_uri`, aliases, validation, + credentials, redaction, and registry metadata. +- Query options are merged before explicit options by `OperatorUri`, but the + service configurator can subsequently derive or replace fields from URI + authority and path. +- Capabilities belong to the constructed operator. Registration metadata does + not promise static capabilities. +- A package retains every native resource used by active operations. +- Construction errors identify the package and operation without copying + secrets into loader diagnostics. + +S3 must continue deriving bucket and root from its URI. WebDAV must continue +deriving an HTTPS endpoint from authority, including its current behavior of +overwriting an endpoint option. HDFS activation must be lazy and isolated from +`hdfs-native`. + +## Layer Compatibility Rules + +- A layer factory may complete asynchronously. +- A layer handle retains native state and its extension library lease. +- Applying a layer returns a new operator and does not mutate the source + operator. +- Later applications are outer layers. The runtime does not sort or deduplicate + layers. +- Both `apply_service` and `apply_context` behavior must be preserved. +- Reusing one handle preserves that layer's sharing identity. + +Timeout and Retry have one cancellation-safe order. Applying Timeout first and +Retry second places Retry outside Timeout, so each retry attempt has its own +timeout. Applying Retry first and Timeout second places Timeout outside Retry; +the timeout can drop Retry's future before Retry restores operation-body state. +Bindings must preserve caller order and reject this known unsafe outer Timeout +composition when their metadata makes it visible. + +## Error Contract + +The runtime reports stable error categories: + +```text +NotInstalled +ManifestInvalid +Conflict +Incompatible +UnsupportedTarget +NativeLoadFailed +BootstrapInvalid +FactoryFailed +LayerInitializationFailed +``` + +Language adapters map these categories to native exception classes while +retaining package ID, component ID, and construction operation. Diagnostics do +not include credentials or an unredacted configuration map. + +## Security Constraints + +Native extensions are trusted in-process code. Neither JSON validation nor an +exact OpenDAL version provides a sandbox, signature verification, provenance, +process isolation, or protection from a malicious extension. + +The loader and official SDK must: + +- Validate all untrusted lengths and identifiers before use. +- Catch panics at every exported native entry and runtime-invoked callback. +- Enforce the export allowlist defined by the [native symbol isolation + contract](symbol-isolation.md). +- Load only a path supplied by an installed, explicitly selected package. +- Pin libraries while any callback, vtable, task, or object may reference them. +- Keep credentials out of manifests and conflict errors. + +Package signing and registry provenance remain package-release concerns. They +do not change the in-process trust model. + +Official artifacts use unwind-capable panic handling at extension entries. The +SDK and release pipeline reject `panic=abort` for this interface. + +## Unsupported Guarantees + +The exact-release design does not guarantee: + +- Loading an extension built for a different OpenDAL version. +- Native library unloading. +- Safe use of arbitrary extension code or secret redaction by third parties. +- Serialization of live caches, limiters, runtimes, JVMs, or connection pools. +- One physical extension binary across Python, Ruby, and Node.js. +- Static service capabilities before configuration. +- Runtime installation of a missing package. +- Backend correctness, authorization, durability, or resource limits beyond + the existing OpenDAL threat model. + +## Required Conformance + +Before publishing a native extension interface, CI must verify: + +- 1,000 manifest registrations without eager native activation. +- Single-component conflicts, alias conflicts, and idempotent registration. +- Version and target mismatch rejection before storing callbacks. +- Concurrent activation and process-lifetime library pinning. +- S3 and WebDAV URI and configuration semantics and secret-free errors. +- HDFS load isolation from the base, S3, WebDAV, and `hdfs-native`. +- Foyer asynchronous construction, cancellation, invalidation, and handle + lifetime. +- Timeout service and executor behavior through a separately packaged layer. +- Timeout-inside-Retry acceptance and unsafe outer-Timeout rejection. +- Throttle shared limiter identity and argument validation. +- Language garbage collection, task cancellation, and callback races under + sanitizers where available. +- Every supported OS, architecture, libc/deployment floor, and language adapter + variant at the artifact level. diff --git a/bindings/docs/dynamic-extensions/nodejs.md b/bindings/docs/dynamic-extensions/nodejs.md new file mode 100644 index 000000000000..3cde1b9ffef8 --- /dev/null +++ b/bindings/docs/dynamic-extensions/nodejs.md @@ -0,0 +1,369 @@ + + +# Node.js Dynamic Extension Design + +Status: pre-RFC binding-specific design proposal. The current Node.js binding +remains one native addon with compile-time service and layer selection. + +The exact-release extension design is useful for Node.js. Node-API makes the base +addon portable across supported Node.js versions, but it does not stabilize the +Rust interface between separately built OpenDAL native packages. + +This document applies the [shared extension architecture](README.md) with +Node-specific package loading, addon initialization, Worker, and event-loop +constraints. +The [shared compatibility contract](compatibility.md) is canonical for native +ABI, configuration, lifetime, and loader rules; this document defines Node.js +deltas. + +## Current Constraints + +- `bindings/nodejs` builds one napi-rs `cdylib` using Node-API version 6. +- Services are selected by Cargo features. The default includes S3 and WebDAV; + the published feature set has explicit exclusions and target differences. +- `Operator` owns both asynchronous and blocking OpenDAL operators. +- Current layers are binding-local `NodeLayer` trait objects wrapped in + napi-rs `External` values. +- An `External` from one independently built addon is not a supported handle + for another addon. Its Rust type and layout belong to the addon that created + it. +- The generated loader already selects target packages by operating system, + architecture, and Linux libc, but it loads only one monolithic `.node` file. +- The package currently exposes only its root and `package.json` subpaths. +- ESM and CommonJS wrappers maintain central layer export lists and must stay in + sync. + +The dynamic design should reuse the target-package pattern without exposing +napi-rs implementation types as the extension contract. + +## npm Package Layout + +The proposed family is: + +```text +opendal Node-API adapter and JavaScript interface +@opendal/runtime runtime loader and target resolver +@opendal/runtime-linux-x64-gnu target-specific runtime addon +@opendal/service-s3 S3 JavaScript stub and types +@opendal/service-s3-linux-x64-gnu target-specific S3 native addon/library +@opendal/layer-timeout Timeout stub and types +@opendal/layer-foyer Foyer stub and types +``` + +Each root extension package declares: + +- An exact peer dependency on the compatible `@opendal/runtime` release. +- A compatible peer dependency on the `opendal` adapter when it exposes a + JavaScript API through that package. +- Target packages as optional dependencies with `os`, `cpu`, and `libc` + metadata where available. +- ESM and CommonJS entry points that use one registration implementation. +- A mandatory embedded OpenDAL version check. +- A clear error for optional dependencies omitted during installation. + +The `opendal` adapter declares its `required_runtime_protocol`. +`@opendal/runtime` exposes its minimum and current protocol levels for the +adapter to check. Applications depend on `opendal` and their selected +extensions; the package manager resolves `@opendal/runtime` and its target +package. Package names remain provisional pending npm namespace and release +prototypes. + +One root package plus several target packages per extension creates a large +publication matrix. A registry with 1,000 manifest records can be efficient; +that does not prove that publishing or loading 1,000 native npm package families +is operationally practical. + +## Runtime Ownership + +The native architecture gives process resources and Node environment state +different owners: + +```text +loaded opendal native module + ProcessRuntime + OpenDAL core, Tokio, registries, activation state, handle identity + process-lifetime native library leases + +main napi_env Worker napi_env + EnvironmentAdapter A EnvironmentAdapter B + JS wrappers and callbacks JS wrappers and callbacks + Promise completion bridge Promise completion bridge +``` + +One loaded runtime native module owns one `ProcessRuntime`. The exact peer +dependency should normally produce one such module in a process; nested +incompatible runtime installations can load distinct modules with distinct +process-runtime identities. The `ProcessRuntime` owns manifest conflicts, +activate-once state, native factories, Tokio/core resources, native handles, and +library leases. + +Every `napi_env` owns one `EnvironmentAdapter`. It owns JavaScript wrappers, +references, resolver/activation callbacks, Promise completion bridges, and +environment cleanup hooks. Registering through an environment adapter commits +the manifest atomically into its `ProcessRuntime`; an identical process-level +registration is idempotent and a different owner is a conflict. Native package +activation runs once per `ProcessRuntime`, not once per Worker. + +Node-API environments can be initialized and destroyed multiple times and can +run concurrently in Workers. A `napi_env`, `napi_value`, reference, JavaScript +callback, or public wrapper must never move between environment adapters. Node +documents these environment-lifecycle rules in its [Node-API +documentation](https://nodejs.org/api/n-api.html). + +Environment cleanup stops new calls, cancels or finishes environment-owned +asynchronous work, releases all JavaScript references, and removes its local +callbacks. It does not unload process-pinned extension libraries or invalidate +native handles owned by another environment. A Worker can therefore terminate +independently of operations in another Worker. + +## Native Registration Adapter + +The first Node prototype should compare two packaging implementations: + +1. The `ProcessRuntime` loads a language-neutral native library directly from + the package manifest. +2. An environment-bound JavaScript activator loads a target-specific Node-API + bootstrap addon in the calling `napi_env`. + +Both implementations use the same bootstrap metadata and selected encoding, +OpenDAL version, factory contract, and conformance suite. A literal separately +installed shared Rust `dylib` is not a design assumption; npm/pnpm/Yarn layouts, +rpaths, and Windows DLL discovery must prove it first. + +Direct host loading is the preferred starting point because it naturally +preserves JSON registration, process-level activate-once behavior, and +construction-time native activation. + +The Node-API bootstrap variant requires this explicit environment-bound +activation sequence: + +1. The JavaScript registration stub gives its `EnvironmentAdapter` a JSON + manifest and a local activation callback. It loads no native target package. +2. First construction asks the adapter, on its JavaScript thread, to invoke that + callback. +3. The callback resolves the package's target artifact and synchronously loads + its `.node` addon in the same `napi_env`. ESM and CommonJS wrappers call one + shared loader implementation. +4. The addon uses the normal Node-API initializer and returns the bootstrap + status, encoding discriminant, and payload. The payload is JSON bytes or a + `napi_external` pointing to C-layout metadata. It does not return a napi-rs + class or an `External` shared with the base. +5. The base validates the status, length, and selected encoding before giving + its release-specific entry to the `ProcessRuntime`. The process runtime + installs the factory and records an explicit process-lifetime library lease. +6. The environment adapter releases its activation callback after success. The + factory does not retain `napi_env`, `napi_value`, JavaScript references, or + thread-safe functions. + +The variant is viable only if the platform prototype can retain a native +library lease independently from the initiating environment. A variant that +loads the HDFS addon during registration, or whose callbacks become invalid +when the initiating Worker exits, fails the design requirements. + +The base schedules extension futures on runtime-owned native resources and +adapts completion into the calling environment. + +## Registration Interface + +Explicit registration avoids bundler-dependent side effects: + +```javascript +import { Operator } from "opendal"; +import { registerS3 } from "@opendal/service-s3"; + +registerS3(); + +const op = Operator.fromUri("s3://photos/archive", { + region: "us-east-1", +}); +``` + +`registerS3()` is idempotent for the same package and `ProcessRuntime`. It +registers a JSON manifest without activating native code. Under the bootstrap +addon prototype it also installs one environment-local activation callback; +under direct host loading the manifest's artifact path is sufficient. + +A package may offer a documented side-effect registration subpath for +convenience, but it must mark that subpath appropriately for bundlers. The +explicit function remains the unambiguous interface. + +## Proposed Layer Interface + +`Operator.layer()` should accept a base-owned opaque `Layer`, not +`ExternalObject` from napi-rs internals. + +Synchronous factories work for layers without asynchronous resources: + +```javascript +import { TimeoutLayer } from "@opendal/layer-timeout"; +import { ThrottleLayer } from "@opendal/layer-throttle"; + +const timeout = new TimeoutLayer(); +timeout.timeout = 60_000; +timeout.ioTimeout = 10_000; + +const limit = new ThrottleLayer(10 * 1024, 10 * 1024 * 1024); + +const layered = op.layer(limit.build()).layer(timeout.build()); +``` + +The compatibility `opendal` package preserves these current constructors, +setters, and `.build()` calls. Internally, `.build()` returns a base-owned opaque +`Layer` instead of a package-local napi-rs `External`. If the generated +`ExternalObject` TypeScript name cannot remain as a deprecated alias, the +type-name change must wait for the binding's next breaking public release; it +does not justify weakening the native handle boundary. + +Foyer uses a Promise-returning factory because JavaScript constructors cannot +be asynchronous: + +```javascript +import { FoyerLayer } from "@opendal/layer-foyer"; + +const cache = await FoyerLayer.create({ + memoryCapacity: 64 << 20, + storagePath: "/var/cache/opendal", +}); + +const cached = op.layer(cache); +``` + +The binding should not provide a synchronous Foyer constructor that blocks the +event loop. A separately documented worker/off-thread helper can be evaluated +later. + +Layer packages validate JavaScript numbers before calling Rust. Timeout values +use non-negative integer milliseconds and convert to the shared +[`SignedDuration`](compatibility.md#configuration-value-contract) +representation. +The adapter rejects values outside `SignedDuration`'s `i64`-seconds range and +never truncates, saturates, or wraps them. Throttle bandwidth and burst must be +positive integers in the supported `u32` range so invalid input cannot reach the +core constructor's assertions. Other options convert through the same shared +`ConfigValue` grammar; native factories never receive JavaScript objects. + +Applying one Throttle or Foyer handle to several operators preserves shared +native state. Later `.layer()` calls remain outer layers. The adapter preserves +the [canonical Timeout/Retry order](compatibility.md#layer-compatibility-rules) +and rejects the known unsafe composition when both layer IDs are visible. + +## Async Operations and Cancellation + +The base adapter owns the conversion between runtime futures and JavaScript +Promises. Extension factories and operations do not call Node-API from Tokio +worker threads. + +Promise cancellation policy must be explicit because JavaScript Promises do not +provide universal cancellation. Where an operation accepts an `AbortSignal`, +the adapter forwards it into the runtime and drops or aborts the native future +according to OpenDAL semantics. + +Worker termination must not leave a callback targeting a destroyed `napi_env`. +Process-level native work may outlive one environment only when it has no +environment-owned completion callback and its resources have an explicit owner. + +## ESM, CommonJS, and Bundlers + +- ESM and CommonJS exports converge on one `EnvironmentAdapter` per Node + environment and the same `ProcessRuntime` for that loaded runtime module. +- Calling registration through both module systems is idempotent. +- Export maps include a supported registration/runtime subpath instead of + relying on generated private files. +- Side-effect-only registration modules declare their side effects so bundlers + do not remove them. +- Package stubs resolve native artifacts relative to their own installed + package, not the current working directory. +- Errors distinguish an unsupported target from installation with + `--omit=optional` and from an incompatible runtime. + +Dynamic native extensions are scoped to native Node-API targets. WASI and other +environments without compatible dynamic loading require a static bundled +design and should not silently fall back to this interface. + +## Version and Error Behavior + +Node-API version compatibility and OpenDAL runtime compatibility remain +separate. The generated package-version checks provide an early diagnostic; the +embedded exact OpenDAL version check remains authoritative. + +Extension lifecycle errors should be JavaScript `Error` subclasses or errors +with stable codes: + +```text +OPENDAL_EXTENSION_NOT_INSTALLED +OPENDAL_EXTENSION_LOAD_FAILED +OPENDAL_EXTENSION_INCOMPATIBLE +OPENDAL_EXTENSION_CONFLICT +OPENDAL_LAYER_INITIALIZATION_FAILED +``` + +Errors retain package ID, scheme/layer ID, target, and construction operation. +They do not expose credentials or an unredacted option object. Normal OpenDAL +errors retain their structured kind instead of becoming only a formatted +reason string. + +## Multiple Runtime Versions + +npm can install nested copies of a package. An extension stub may therefore see +a different `opendal` instance from the one that created an operator. + +The design applies three defenses: + +1. Exact runtime peer dependencies make the intended singleton visible to the + package manager. +2. Registration records the specific `ProcessRuntime` identity. +3. A layer/operator wrapper verifies process-runtime identity before native + handle use and throws `OPENDAL_EXTENSION_INCOMPATIBLE` on mismatch. + +The adapter must never reinterpret a handle from another `ProcessRuntime`, even +when package versions appear equal. JavaScript wrappers also remain confined to +their creating `EnvironmentAdapter`. + +## Migration + +1. Introduce `ProcessRuntime`, `EnvironmentAdapter`, and the extension registry + inside the current addon. +2. Replace the public `ExternalObject` detail with a base-owned opaque + layer wrapper while retaining current constructors. +3. Make compiled services/layers use the internal extension factory model. +4. Publish runtime and target packages using the existing platform-loader + experience. +5. Extract S3 and Timeout as tracer package families and remove them from the + base package after their packages are available. +6. Add Foyer async creation and HDFS lazy activation as design gates. +7. Test both direct native loading and Node-API bootstrap addons before + selecting the physical linking model. +8. Publish the third-party SDK only after Worker, ESM/CommonJS, target, and + lifetime conformance passes. + +## Node.js Conformance Gates + +- ESM/CommonJS double registration, one environment adapter per `napi_env`, and + process-level activate-once behavior. +- Main thread plus multiple Workers importing, using, and terminating adapters. +- Two incompatible nested runtime versions rejecting cross-instance handles. +- Missing optional target package and unsupported target diagnostics. +- glibc/musl, macOS, and Windows artifact selection. +- Bundler retention for any documented side-effect registration entry. +- S3/WebDAV construction without central JavaScript config schemas. +- HDFS registration without Java/Hadoop and isolated activation failure. +- Foyer Promise construction, rejection, cleanup, and reusable handle state. +- Timeout executor behavior and Throttle shared identity through extracted + packages. +- No callback into a destroyed Node environment or unloaded native library. +- 1,000 synthetic registrations without loading 1,000 native addons. diff --git a/bindings/docs/dynamic-extensions/prototype/.cargo/config.toml b/bindings/docs/dynamic-extensions/prototype/.cargo/config.toml new file mode 100644 index 000000000000..4feecfab2d43 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/.cargo/config.toml @@ -0,0 +1,7 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0. + +[target.'cfg(target_os = "linux")'] +rustflags = ["-C", "link-arg=-Wl,--exclude-libs,ALL"] diff --git a/bindings/docs/dynamic-extensions/prototype/.gitignore b/bindings/docs/dynamic-extensions/prototype/.gitignore new file mode 100644 index 000000000000..6274ef8dae7f --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/.gitignore @@ -0,0 +1,6 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0. + +/target/ diff --git a/bindings/docs/dynamic-extensions/prototype/Cargo.lock b/bindings/docs/dynamic-extensions/prototype/Cargo.lock new file mode 100644 index 000000000000..bdab43529aa4 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/Cargo.lock @@ -0,0 +1,1253 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + +[[package]] +name = "base64" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b25655df2c3cdd83c5e5b293b88acd880332b2ddadd7c30ac43144fdc0033da9" + +[[package]] +name = "bitflags" +version = "1.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" + +[[package]] +name = "block-buffer" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cmov" +version = "0.5.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c9ea0ac24bc397ab3c98583a3c9ba74fa56b09a4449bbe172b9b1ddb016027a" + +[[package]] +name = "const-oid" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" + +[[package]] +name = "const-random" +version = "0.1.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "87e00182fe74b066627d63b85fd550ac2998d4b0bd86bfed477a0ae4c7c71359" +dependencies = [ + "const-random-macro", +] + +[[package]] +name = "const-random-macro" +version = "0.1.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9d839f2a20b0aee515dc581a6172f2321f96cab76c1a38a4c584a194955390e" +dependencies = [ + "getrandom 0.2.17", + "once_cell", + "tiny-keccak", +] + +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + +[[package]] +name = "crc-fast" +version = "1.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e75b2483e97a5a7da73ac68a05b629f9c53cff58d8ed1c77866079e18b00dba5" +dependencies = [ + "digest 0.10.7", + "spin", +] + +[[package]] +name = "crunchy" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "crypto-common" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "ctutils" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e" +dependencies = [ + "cmov", +] + +[[package]] +name = "defmt" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" +dependencies = [ + "bitflags", + "defmt-macros", +] + +[[package]] +name = "defmt-macros" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" +dependencies = [ + "defmt-parser", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "defmt-parser" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" +dependencies = [ + "thiserror", +] + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "crypto-common 0.1.7", +] + +[[package]] +name = "digest" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" +dependencies = [ + "block-buffer", + "const-oid", + "crypto-common 0.2.2", + "ctutils", +] + +[[package]] +name = "displaydoc" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "dlv-list" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "442039f5147480ba31067cb00ada1adae6892028e40e45fc5de7b7df6dcc1b5f" +dependencies = [ + "const-random", +] + +[[package]] +name = "form_urlencoded" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" +dependencies = [ + "percent-encoding", +] + +[[package]] +name = "futures" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a88cf1f829d945f548cf8fec32c61b1f202b6d93b45848602fc02af4b12ad218" +dependencies = [ + "futures-channel", + "futures-core", + "futures-executor", + "futures-io", + "futures-sink", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-channel" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "262590f4fe6afeb0bc83be1daa64e52657fe185690a958af7f3ad0e92085c5ae" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2cd50c473c80f6d7c3670a752354b8e569b1a7cbfdc0419ec88e5edad85e0dc7" + +[[package]] +name = "futures-executor" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6754879cc9f2c66f88c6e5c35344bb0bdb0708b0352b1201815667c7eabc7458" +dependencies = [ + "futures-core", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-io" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4577ecaa3c4f96589d473f679a71b596316f6641bc350038b962a5daf0085d7a" + +[[package]] +name = "futures-macro" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d6d3cde68c518367be28956066ddfef33813991b77a55005a69dae04bf3b10b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "futures-sink" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e34418ac499d6305c2fb5ad0ed2f6ac998c5f8ca209b4510f7f94242c647e307" + +[[package]] +name = "futures-task" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b231ed28831efb4a61a08580c4bc233ec56bc009f4cd8f52da2c3cb97df0c109" + +[[package]] +name = "futures-util" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a77a90a256fce34da66415271e30f94ee91c57b04b8a2c042d9cf3220179deaa" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "slab", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "libc", + "wasi", +] + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "libc", + "r-efi", +] + +[[package]] +name = "hashbrown" +version = "0.14.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" + +[[package]] +name = "hex" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" + +[[package]] +name = "hmac" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6303bc9732ae41b04cb554b844a762b4115a61bfaa81e3e83050991eeb56863f" +dependencies = [ + "digest 0.11.3", +] + +[[package]] +name = "host" +version = "0.0.0" +dependencies = [ + "libloading", + "opendal-core", +] + +[[package]] +name = "http" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "918d3568bebf352712bc2ef3d46a8bcf1a75b373be6539de198e9105cbbf9ce0" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "hybrid-array" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "707114b52a152fa7bdb290cd7cd5912d9467273b6d74e21b8d81aca1f8533f6b" +dependencies = [ + "typenum", +] + +[[package]] +name = "icu_collections" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c" +dependencies = [ + "displaydoc", + "potential_utf", + "utf8_iter", + "yoke", + "zerofrom", + "zerovec", +] + +[[package]] +name = "icu_locale_core" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29" +dependencies = [ + "displaydoc", + "litemap", + "tinystr", + "writeable", + "zerovec", +] + +[[package]] +name = "icu_normalizer" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4" +dependencies = [ + "icu_collections", + "icu_normalizer_data", + "icu_properties", + "icu_provider", + "smallvec", + "zerovec", +] + +[[package]] +name = "icu_normalizer_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38" + +[[package]] +name = "icu_properties" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de" +dependencies = [ + "icu_collections", + "icu_locale_core", + "icu_properties_data", + "icu_provider", + "zerotrie", + "zerovec", +] + +[[package]] +name = "icu_properties_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14" + +[[package]] +name = "icu_provider" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421" +dependencies = [ + "displaydoc", + "icu_locale_core", + "writeable", + "yoke", + "zerofrom", + "zerotrie", + "zerovec", +] + +[[package]] +name = "idna" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de" +dependencies = [ + "idna_adapter", + "smallvec", + "utf8_iter", +] + +[[package]] +name = "idna_adapter" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" +dependencies = [ + "icu_normalizer", + "icu_properties", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "jiff" +version = "0.2.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc" +dependencies = [ + "defmt", + "jiff-core", + "jiff-static", + "jiff-tzdb-platform", + "js-sys", + "log", + "portable-atomic", + "portable-atomic-util", + "serde_core", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "jiff-core" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7feca88439efe53da3754500c1851dedf3cb36c524dd5cf8225cc0794de95d09" +dependencies = [ + "defmt", +] + +[[package]] +name = "jiff-static" +version = "0.2.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204" +dependencies = [ + "jiff-core", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "jiff-tzdb" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "142bd39932ad231f10513df9ab62661fead8719872150b7ad02a2df79f4e141e" + +[[package]] +name = "jiff-tzdb-platform" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "875a5a69ac2bab1a891711cf5eccbec1ce0341ea805560dcd90b7a2e925132e8" +dependencies = [ + "jiff-tzdb", +] + +[[package]] +name = "js-sys" +version = "0.3.103" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "libloading" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" +dependencies = [ + "cfg-if", + "windows-link", +] + +[[package]] +name = "litemap" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0" + +[[package]] +name = "log" +version = "0.4.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" + +[[package]] +name = "md-5" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98" +dependencies = [ + "cfg-if", + "digest 0.11.3", +] + +[[package]] +name = "mea" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2640d335e7273dacdcf51044026139b2e269c3bb0dfc3f8cb3496b85e3f6a42c" +dependencies = [ + "slab", +] + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "opendal-core" +version = "0.58.1" +dependencies = [ + "anyhow", + "base64 0.23.0", + "bytes", + "futures", + "http", + "jiff", + "log", + "md-5", + "mea", + "percent-encoding", + "quick-xml", + "reqsign-core", + "serde", + "serde_json", + "tokio", + "url", + "uuid", + "web-time", +] + +[[package]] +name = "opendal-layer-timeout" +version = "0.58.1" +dependencies = [ + "opendal-core", + "tokio", +] + +[[package]] +name = "opendal-service-s3" +version = "0.58.1" +dependencies = [ + "base64 0.23.0", + "bytes", + "crc-fast", + "http", + "log", + "md-5", + "opendal-core", + "quick-xml", + "reqsign-aws-v4", + "reqsign-core", + "reqsign-file-read-tokio", + "serde", + "url", +] + +[[package]] +name = "ordered-multimap" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "49203cdcae0030493bad186b28da2fa25645fa276a51b6fec8010d281e02ef79" +dependencies = [ + "dlv-list", + "hashbrown", +] + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "portable-atomic" +version = "1.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d20d5497ef88037a52ff98267d066e7f11fcc5e99bbfbd58a42336193aacec3" + +[[package]] +name = "portable-atomic-util" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" +dependencies = [ + "portable-atomic", +] + +[[package]] +name = "potential_utf" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564" +dependencies = [ + "zerovec", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quick-xml" +version = "0.41.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e660451e55124f798a69a5af3f49ccfbefbd41910eefd25caf2393e1f3473ec1" +dependencies = [ + "memchr", + "serde", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "reqsign-aws-v4" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc883bc56889f3e4a419265c87facea222a921debc5c6f15c7fd8b68ec4b36b2" +dependencies = [ + "anyhow", + "bytes", + "form_urlencoded", + "hex", + "http", + "log", + "percent-encoding", + "quick-xml", + "reqsign-core", + "rust-ini", + "serde", + "serde_json", + "serde_urlencoded", + "sha1", +] + +[[package]] +name = "reqsign-core" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e38b44697c60a823705ccef85cb04d8e0527c9d16ed7c58bf1c6395bdd24ceb" +dependencies = [ + "anyhow", + "base64 0.22.1", + "bytes", + "futures", + "hex", + "hmac", + "http", + "jiff", + "log", + "percent-encoding", + "sha1", + "sha2", + "windows-sys", +] + +[[package]] +name = "reqsign-file-read-tokio" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "688ff0ae421b8d4b92b53fdafaf53df2de28f428a9962edcf21702990b26f74b" +dependencies = [ + "anyhow", + "reqsign-core", + "tokio", +] + +[[package]] +name = "rust-ini" +version = "0.21.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "796e8d2b6696392a43bea58116b667fb4c29727dc5abd27d6acf338bb4f688c7" +dependencies = [ + "cfg-if", + "ordered-multimap", +] + +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + +[[package]] +name = "ryu" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" + +[[package]] +name = "s3-extension" +version = "0.0.0" +dependencies = [ + "opendal-core", + "opendal-service-s3", +] + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "serde_json" +version = "1.0.151" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "serde_urlencoded" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" +dependencies = [ + "form_urlencoded", + "itoa", + "ryu", + "serde", +] + +[[package]] +name = "sha1" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest 0.11.3", +] + +[[package]] +name = "sha2" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest 0.11.3", +] + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "smallvec" +version = "1.15.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" + +[[package]] +name = "spin" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "023a211cb3138dbc438680b32560ad89f699977624c9f8dbb95a47d5b4c07dd3" + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "thiserror" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09a43598840e33d5b0331f38c5e30d13bb11c11210a4b58f0d9b18a5a5eefcd9" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "43cbfe0cf76104d42a574802844187e84a305e531ed54455f11fbde0f10541cd" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "timeout-extension" +version = "0.0.0" +dependencies = [ + "opendal-core", + "opendal-layer-timeout", +] + +[[package]] +name = "tiny-keccak" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2c9d3793400a45f954c52e73d068316d76b6f4e36977e3fcebb13a2721e80237" +dependencies = [ + "crunchy", +] + +[[package]] +name = "tinystr" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d" +dependencies = [ + "displaydoc", + "zerovec", +] + +[[package]] +name = "tokio" +version = "1.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "202caea871b69668250d242070849eb495be178ed697a3e98aebce5bc81a0bed" +dependencies = [ + "bytes", + "pin-project-lite", + "tokio-macros", +] + +[[package]] +name = "tokio-macros" +version = "2.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "url" +version = "2.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" +dependencies = [ + "form_urlencoded", + "idna", + "percent-encoding", + "serde", +] + +[[package]] +name = "utf8_iter" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" + +[[package]] +name = "uuid" +version = "1.24.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf3923a6f5c4c6382e0b653c4117f48d631ea17f38ed86e2a828e6f7412f5239" +dependencies = [ + "getrandom 0.4.3", + "js-sys", + "serde_core", + "wasm-bindgen", +] + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "wasm-bindgen" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn 2.0.119", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "web-time" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "writeable" +version = "0.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" + +[[package]] +name = "yoke" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "709fe23a0424b6a435d82152b1bd3fdfb0833487d5fa90d05d42762a9891fef5" +dependencies = [ + "stable_deref_trait", + "yoke-derive", + "zerofrom", +] + +[[package]] +name = "yoke-derive" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerofrom" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272" +dependencies = [ + "zerofrom-derive", +] + +[[package]] +name = "zerofrom-derive" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerotrie" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf" +dependencies = [ + "displaydoc", + "yoke", + "zerofrom", +] + +[[package]] +name = "zerovec" +version = "0.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239" +dependencies = [ + "yoke", + "zerofrom", + "zerovec-derive", +] + +[[package]] +name = "zerovec-derive" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/bindings/docs/dynamic-extensions/prototype/Cargo.toml b/bindings/docs/dynamic-extensions/prototype/Cargo.toml new file mode 100644 index 000000000000..7fc5619917e6 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/Cargo.toml @@ -0,0 +1,31 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[workspace] +members = ["host", "s3-extension", "timeout-extension"] +resolver = "2" + +[workspace.package] +edition = "2024" +license = "Apache-2.0" +publish = false +rust-version = "1.91" +version = "0.0.0" + +[workspace.dependencies] +libloading = "0.8.9" +opendal-core = { path = "../../../../core/core", default-features = false } +opendal-layer-timeout = { path = "../../../../core/layers/timeout" } +opendal-service-s3 = { path = "../../../../core/services/s3" } diff --git a/bindings/docs/dynamic-extensions/prototype/README.md b/bindings/docs/dynamic-extensions/prototype/README.md new file mode 100644 index 000000000000..96b6b76c886b --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/README.md @@ -0,0 +1,78 @@ + + +# Exact-Release Dynamic Loading Prototype + +This prototype records one experiment behind the dynamic extension design. It +builds three final artifacts: + +- A host executable with its own statically linked `opendal-core`. +- An S3 `cdylib` with its own `opendal-core` and service dependencies. +- A Timeout `cdylib` with its own `opendal-core` and Tokio dependency. + +The runner builds each artifact in a separate Cargo invocation and target +directory, so Cargo does not unify their `opendal-core` feature graphs. The host +then loads both extensions with local visibility. The S3 extension creates an +`Operator`, the Timeout extension applies the real `TimeoutLayer`, and the host +reads the resulting operator information. All three artifacts compile against +the same source checkout, lockfile, compiler, target, profile, and Rust flags. + +On Linux, run: + +```console +./run-linux.sh +``` + +The script also audits the final ELF dynamic symbol tables. Each extension must +export only its package-unique bootstrap symbol. The local Cargo configuration +passes `--exclude-libs,ALL` to prevent symbols from statically linked dependency +archives from becoming dynamic exports. Explicit version scripts document the +intended allowlists, while the audit remains authoritative because Rust's +`cdylib` link can add exports after a user-supplied version script. + +## What It Proves + +The prototype shows that an exact-release internal adapter can move one current +`Operator` through separately linked S3 and Timeout artifacts on the tested +Linux toolchain. It also provides concrete artifacts for developing the [symbol +isolation contract](../symbol-isolation.md). It does not implement or conform +to the proposed SDK interface. + +## What It Does Not Prove + +This is design evidence, not a supported ABI or production loader: + +- The prototype exchanges a Rust value through an opaque pointer. The Rust ABI + remains unstable, so any change in compiler, target, profile, flags, features, + or dependency graph invalidates the experiment. +- The host ultimately destroys an allocation created and transformed by other + artifacts. This intentionally violates the proposed ownership contract. The + production SDK must use opaque handles and creator-side destructors rather + than treating this experiment as an ownership model. +- The prototype applies `TimeoutLayer` but does not execute an operation. + Therefore it does not prove that independently linked Tokio copies observe + compatible runtime state. +- The prototype does not provide the proposed shared runtime package, runtime + protocol negotiation, manifest validation, error contract, or library lease + implementation. +- The prototype covers ELF/Linux only. The design still requires equivalent + Mach-O and PE export enforcement and co-loading tests. +- The bootstrap functions return null on failure and omit structured errors. + The production contract must use the validated bootstrap envelope. + +The prototype should evolve into a conformance fixture only after the RFC +selects the exact-release interface representation. diff --git a/bindings/docs/dynamic-extensions/prototype/audit-elf-exports.sh b/bindings/docs/dynamic-extensions/prototype/audit-elf-exports.sh new file mode 100755 index 000000000000..59a84bbbd1ab --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/audit-elf-exports.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env sh +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eu + +if [ "$#" -ne 2 ]; then + echo "usage: $0 ARTIFACT EXPECTED_SYMBOL" >&2 + exit 2 +fi + +artifact=$1 +expected=$2 +actual=$( + nm --dynamic --defined-only --extern-only --format=posix "$artifact" \ + | awk '{ print $1 }' \ + | sed 's/@.*//' \ + | LC_ALL=C sort -u +) + +if [ "$actual" != "$expected" ]; then + echo "unexpected exports in $artifact" >&2 + echo "expected: $expected" >&2 + echo "actual:" >&2 + echo "$actual" >&2 + exit 1 +fi + +echo "$artifact exports only $expected" diff --git a/bindings/docs/dynamic-extensions/prototype/host/Cargo.toml b/bindings/docs/dynamic-extensions/prototype/host/Cargo.toml new file mode 100644 index 000000000000..a49573dba213 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/host/Cargo.toml @@ -0,0 +1,26 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +edition.workspace = true +license.workspace = true +name = "host" +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[dependencies] +libloading.workspace = true +opendal-core.workspace = true diff --git a/bindings/docs/dynamic-extensions/prototype/host/src/main.rs b/bindings/docs/dynamic-extensions/prototype/host/src/main.rs new file mode 100644 index 000000000000..303d1aa45d97 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/host/src/main.rs @@ -0,0 +1,80 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::env; +use std::ffi::c_void; +use std::process::ExitCode; + +use libloading::{Library, Symbol}; +use opendal_core::Operator; + +type CreateOperator = unsafe extern "C" fn() -> *mut c_void; +type ApplyLayer = unsafe extern "C" fn(*mut c_void) -> *mut c_void; + +fn main() -> ExitCode { + match unsafe { run() } { + Ok(()) => ExitCode::SUCCESS, + Err(message) => { + eprintln!("{message}"); + ExitCode::FAILURE + } + } +} + +unsafe fn run() -> Result<(), String> { + let mut args = env::args().skip(1); + let s3_path = args.next().ok_or("missing S3 extension path")?; + let timeout_path = args.next().ok_or("missing Timeout extension path")?; + if args.next().is_some() { + return Err("expected exactly two extension paths".to_string()); + } + + let s3 = unsafe { Library::new(&s3_path) }.map_err(|err| err.to_string())?; + let create: Symbol<'_, CreateOperator> = unsafe { + s3.get(b"opendal_service_s3_bootstrap_v1\0") + .map_err(|err| err.to_string())? + }; + let operator = unsafe { create() }; + if operator.is_null() { + return Err("S3 extension failed to create an operator".to_string()); + } + + let timeout = unsafe { Library::new(&timeout_path) }.map_err(|err| err.to_string())?; + let apply: Symbol<'_, ApplyLayer> = unsafe { + timeout + .get(b"opendal_layer_timeout_bootstrap_v1\0") + .map_err(|err| err.to_string())? + }; + let operator = unsafe { apply(operator) }; + if operator.is_null() { + return Err("Timeout extension failed to apply its layer".to_string()); + } + + let operator = unsafe { Box::from_raw(operator.cast::()) }; + let info = operator.info(); + println!( + "scheme={} name={} root={}", + info.scheme(), + info.name(), + info.root() + ); + + // Drop the operator before either library so its service and layer vtables + // still point to loaded code. + drop(operator); + drop(timeout); + drop(s3); + Ok(()) +} diff --git a/bindings/docs/dynamic-extensions/prototype/run-linux.sh b/bindings/docs/dynamic-extensions/prototype/run-linux.sh new file mode 100755 index 000000000000..7283b8d85420 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/run-linux.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env sh +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eu + +prototype_dir=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd) +s3_target="$prototype_dir/target/s3" +timeout_target="$prototype_dir/target/timeout" +host_target="$prototype_dir/target/host" +s3="$s3_target/release/libs3_extension.so" +timeout="$timeout_target/release/libtimeout_extension.so" +host="$host_target/release/host" + +CARGO_TARGET_DIR="$s3_target" cargo build --release --locked \ + --manifest-path "$prototype_dir/Cargo.toml" --package s3-extension +CARGO_TARGET_DIR="$timeout_target" cargo build --release --locked \ + --manifest-path "$prototype_dir/Cargo.toml" --package timeout-extension +CARGO_TARGET_DIR="$host_target" cargo build --release --locked \ + --manifest-path "$prototype_dir/Cargo.toml" --package host +"$prototype_dir/audit-elf-exports.sh" \ + "$s3" opendal_service_s3_bootstrap_v1 +"$prototype_dir/audit-elf-exports.sh" \ + "$timeout" opendal_layer_timeout_bootstrap_v1 +"$host" "$s3" "$timeout" diff --git a/bindings/docs/dynamic-extensions/prototype/s3-extension/Cargo.toml b/bindings/docs/dynamic-extensions/prototype/s3-extension/Cargo.toml new file mode 100644 index 000000000000..6a30fb418d6b --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/s3-extension/Cargo.toml @@ -0,0 +1,29 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +edition.workspace = true +license.workspace = true +name = "s3-extension" +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] + +[dependencies] +opendal-core.workspace = true +opendal-service-s3.workspace = true diff --git a/bindings/docs/dynamic-extensions/prototype/s3-extension/build.rs b/bindings/docs/dynamic-extensions/prototype/s3-extension/build.rs new file mode 100644 index 000000000000..501b9f44298a --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/s3-extension/build.rs @@ -0,0 +1,29 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::env; +use std::path::PathBuf; + +fn main() { + if env::var("CARGO_CFG_TARGET_OS").as_deref() == Ok("linux") { + let manifest_dir = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); + let export_map = manifest_dir.join("exports.map"); + println!("cargo:rerun-if-changed={}", export_map.display()); + println!( + "cargo:rustc-cdylib-link-arg=-Wl,--version-script={}", + export_map.display() + ); + } +} diff --git a/bindings/docs/dynamic-extensions/prototype/s3-extension/exports.map b/bindings/docs/dynamic-extensions/prototype/s3-extension/exports.map new file mode 100644 index 000000000000..09035fa37b48 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/s3-extension/exports.map @@ -0,0 +1,21 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +OPENDAL_S3_EXTENSION_1 { + global: + opendal_service_s3_bootstrap_v1; + local: + *; +}; diff --git a/bindings/docs/dynamic-extensions/prototype/s3-extension/src/lib.rs b/bindings/docs/dynamic-extensions/prototype/s3-extension/src/lib.rs new file mode 100644 index 000000000000..70744aa151bc --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/s3-extension/src/lib.rs @@ -0,0 +1,34 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::ffi::c_void; +use std::panic::{AssertUnwindSafe, catch_unwind}; + +use opendal_core::Operator; +use opendal_service_s3::S3; + +#[unsafe(no_mangle)] +pub extern "C" fn opendal_service_s3_bootstrap_v1() -> *mut c_void { + catch_unwind(AssertUnwindSafe(|| { + match Operator::new(S3::default().bucket("prototype-bucket").region("us-east-1")) { + Ok(operator) => Box::into_raw(Box::new(operator)).cast::(), + Err(err) => { + eprintln!("failed to construct prototype S3 operator: {err}"); + std::ptr::null_mut() + } + } + })) + .unwrap_or_default() +} diff --git a/bindings/docs/dynamic-extensions/prototype/timeout-extension/Cargo.toml b/bindings/docs/dynamic-extensions/prototype/timeout-extension/Cargo.toml new file mode 100644 index 000000000000..a8673d4640f9 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/timeout-extension/Cargo.toml @@ -0,0 +1,29 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +edition.workspace = true +license.workspace = true +name = "timeout-extension" +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] + +[dependencies] +opendal-core.workspace = true +opendal-layer-timeout.workspace = true diff --git a/bindings/docs/dynamic-extensions/prototype/timeout-extension/build.rs b/bindings/docs/dynamic-extensions/prototype/timeout-extension/build.rs new file mode 100644 index 000000000000..501b9f44298a --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/timeout-extension/build.rs @@ -0,0 +1,29 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::env; +use std::path::PathBuf; + +fn main() { + if env::var("CARGO_CFG_TARGET_OS").as_deref() == Ok("linux") { + let manifest_dir = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); + let export_map = manifest_dir.join("exports.map"); + println!("cargo:rerun-if-changed={}", export_map.display()); + println!( + "cargo:rustc-cdylib-link-arg=-Wl,--version-script={}", + export_map.display() + ); + } +} diff --git a/bindings/docs/dynamic-extensions/prototype/timeout-extension/exports.map b/bindings/docs/dynamic-extensions/prototype/timeout-extension/exports.map new file mode 100644 index 000000000000..d6cca9c4d8c2 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/timeout-extension/exports.map @@ -0,0 +1,21 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +OPENDAL_TIMEOUT_EXTENSION_1 { + global: + opendal_layer_timeout_bootstrap_v1; + local: + *; +}; diff --git a/bindings/docs/dynamic-extensions/prototype/timeout-extension/src/lib.rs b/bindings/docs/dynamic-extensions/prototype/timeout-extension/src/lib.rs new file mode 100644 index 000000000000..0828cbfabb93 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/timeout-extension/src/lib.rs @@ -0,0 +1,39 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::ffi::c_void; +use std::panic::{AssertUnwindSafe, catch_unwind}; + +use opendal_core::Operator; +use opendal_layer_timeout::TimeoutLayer; + +#[unsafe(no_mangle)] +/// Applies `TimeoutLayer` to an operator from the exact prototype build. +/// +/// # Safety +/// +/// `operator` must be a non-null pointer returned by the S3 extension from the +/// same compiler, target, profile, flags, source checkout, and lockfile. +pub unsafe extern "C" fn opendal_layer_timeout_bootstrap_v1(operator: *mut c_void) -> *mut c_void { + if operator.is_null() { + return std::ptr::null_mut(); + } + + catch_unwind(AssertUnwindSafe(|| { + let operator = unsafe { Box::from_raw(operator.cast::()) }; + Box::into_raw(Box::new(operator.layer(TimeoutLayer::default()))).cast::() + })) + .unwrap_or_default() +} diff --git a/bindings/docs/dynamic-extensions/python.md b/bindings/docs/dynamic-extensions/python.md new file mode 100644 index 000000000000..b1f5260bdbda --- /dev/null +++ b/bindings/docs/dynamic-extensions/python.md @@ -0,0 +1,354 @@ + + +# Python Dynamic Extension Design + +Status: pre-RFC binding-specific design proposal. The current Python binding +remains a single native distribution. + +This document applies the [shared extension architecture](README.md) to Python. +It preserves `Operator` and `AsyncOperator` while moving service and layer +implementation dependencies into independently installable distributions. +The [shared compatibility contract](compatibility.md) is canonical for native +ABI, configuration, lifetime, and loader rules; this document defines Python +deltas. + +## Current Constraints + +The current Python binding has several compile-time assumptions that cannot +serve as a dynamic extension interface: + +- `bindings/python` builds one PyO3 `cdylib`, `opendal._opendal`. +- Published wheels enable the binding's `services-all` feature, subject to its + explicit exclusions and platform conditions. +- `Scheme` is a feature-gated Rust enum. It cannot represent a service installed + after the base extension was compiled. +- `opendal.services` and `opendal.layers` are PyO3 submodules inserted into + `sys.modules`, not filesystem packages that other distributions can extend. +- `Layer` stores `Box` in the base native library. A PyO3 + subclass marker does not make that Rust trait object transferable from an + independently linked extension. +- `opendal.config.ServiceConfig` is one generated closed union of compiled + services. +- Operator pickle state records construction URI/options but does not record + applied layers. + +The migration must change these internals without requiring every caller to +adopt a new operator abstraction. + +## Distribution Layout + +The proposed release family is: + +```text +opendal-runtime provides the shared native runtime +opendal owns the `opendal` import package and Python adapter +opendal-service-s3 contributes the S3 manifest, native code, and typing +opendal-service-hdfs contributes libhdfs-backed HDFS lazily +opendal-layer-timeout contributes Timeout +opendal-layer-foyer contributes Foyer +``` + +The `opendal` distribution declares its `required_runtime_protocol`. The +`opendal-runtime` distribution exposes its minimum and current protocol levels +for the binding to check. Every native service/layer distribution requires an +exact `opendal-runtime` release and embeds that OpenDAL version in its bootstrap +metadata. Installing the base wheel resolves the runtime dependency: + +```console +python -m pip install opendal +``` + +An application installs the main binding and selected packages; the package +manager resolves `opendal-runtime`: + +```console +python -m pip install \ + opendal \ + opendal-service-s3 \ + opendal-layer-timeout \ + opendal-layer-foyer +``` + +The extension package names are provisional. + +## Import Layout + +The intended typed import layout is: + +```text +opendal regular package owned by the base distribution +opendal.services namespace subpackage +opendal.services.s3 supplied by opendal-service-s3 +opendal.services.hdfs supplied by opendal-service-hdfs +opendal.layers namespace subpackage +opendal.layers.timeout supplied by opendal-layer-timeout +opendal.layers.foyer supplied by opendal-layer-foyer +``` + +Python packaging supports splitting namespace subpackages across +distributions, but every participant must follow one consistent layout. See +the [PyPA namespace package guide](https://packaging.python.org/en/latest/guides/packaging-namespace-packages/). + +Before using this layout, the base binding must move the current native +`opendal.services` and `opendal.layers` definitions under a private native +module and expose real Python package directories. Existing flat names can be +re-exported during migration. + +If wheel-install and namespace ownership prototypes are not reliable across the +supported installers, the first tracer packages may use unambiguous top-level +imports such as `opendal_service_s3`. The runtime extension contract does not +depend on the cosmetic import layout. + +## Discovery and Activation + +Python entry points advertise installed manifests: + +```toml +[project.entry-points."opendal.services"] +s3 = "opendal.services.s3:_register" + +[project.entry-points."opendal.layers"] +foyer = "opendal.layers.foyer:_register" +``` + +[Entry points](https://packaging.python.org/en/latest/specifications/entry-points/) +allow the runtime to find an installed package without importing every package. +The resolver follows these rules: + +1. `import opendal` loads only the base adapter and runtime. +2. An explicit service/layer import registers its JSON manifest. +3. Construction of an unregistered scheme must look up the one entry point with + the matching canonical name and load only that registration stub. It reports + a conflict if more than one distribution claims the name. +4. Native code activates only when the caller constructs that service/layer. +5. Resolver results and deterministic failures are cached. +6. Discovery never installs a missing distribution at runtime. + +Explicit imports remain useful for deterministic startup and access to typed +configuration classes. Entry-point discovery preserves the current concise URI +path for callers that only need strings. + +## Proposed Operator Interface + +Existing construction remains valid: + +```python +import opendal + +op = opendal.Operator("s3", bucket="photos", region="us-east-1") + +async_op = opendal.AsyncOperator.from_uri( + "s3://photos/archive?region=us-east-1", + endpoint="https://s3.example.com", +) +``` + +Strings are the canonical dynamic scheme identifiers. The existing `Scheme` +enum may remain as a frozen compatibility aid for previously bundled official +services, but it is not an inventory of installed extensions. + +Typed configuration moves into its service distribution: + +```python +from opendal import AsyncOperator +from opendal.services.s3 import S3Config + +config = S3Config(scheme="s3", bucket="photos", region="us-east-1") +op = AsyncOperator.from_config(config) +``` + +The base `from_config` runtime path accepts a generic mapping or service recipe. +Package-local generated `TypedDict` or dataclass definitions provide field +checking without extending one base `ServiceConfig` union. The service package +owns structured serialization and validation for the matching OpenDAL release. + +The adapter converts mappings to the shared +[`ConfigValue`](compatibility.md#configuration-value-contract) grammar. It +rejects unsupported Python objects, cyclic containers, oversized values, +unknown fields, and numeric overflow before package construction. Package-local +types can expose Python-native values, but no `PyObject` crosses the factory +seam. + +URI construction sends the original URI plus explicit string options to the +service factory. It does not convert them through a central Python config +schema. This preserves S3 and WebDAV configurator behavior. + +## Proposed Layer Interface + +Simple layer factories remain synchronous: + +```python +from opendal.layers.throttle import ThrottleLayer +from opendal.layers.timeout import TimeoutLayer + +limit = ThrottleLayer(bandwidth=10 * 1024, burst=10 * 1024 * 1024) +timeout = TimeoutLayer(timeout=60.0, io_timeout=10.0) + +layered = op.layer(limit).layer(timeout) +``` + +Resource-backed construction is asynchronous: + +```python +from opendal.layers.foyer import FoyerLayer + +cache = await FoyerLayer.create( + memory_capacity=64 << 20, + storage_path="/var/cache/opendal", +) + +cached = async_op.layer(cache) +``` + +A blocking helper may be provided for synchronous applications only if it uses +the same runtime factory, releases the GIL while waiting, and has defined +cancellation/cleanup behavior. It must not create a second Tokio runtime inside +the Foyer package. + +Every concrete Python layer wraps a base-owned opaque `LayerHandle`. It does +not expose a package-local Rust trait object. Applying it returns a new operator +and preserves the native layer's service and context hooks. + +One layer object may carry shared state: + +- Applying one Throttle object to two operators shares its quota. +- Applying two independently constructed Throttle objects creates two quotas. +- Applying one Foyer object reuses one cache, subject to the cache namespace + restriction documented by that package. + +Throttle accepts only positive integer `bandwidth` and `burst` values in the +supported `u32` range. Python validation must reject invalid values before the +native constructor can assert. + +Timeout values remain finite, positive seconds at the Python interface. The +adapter uses the current `Duration::try_from_secs_f64` rule, which rounds to the +nearest nanosecond with ties to even, and then emits `SignedDuration`. It +rejects values outside `SignedDuration`'s `i64`-seconds range instead of +saturating them. + +Later `.layer()` calls are outer layers. The binding preserves the +[canonical Timeout/Retry order](compatibility.md#layer-compatibility-rules) and +rejects the known unsafe composition when it can observe both layer IDs. + +## Async Behavior + +The shared runtime owns operation futures. `AsyncOperator` converts them into +Python awaitables through the base adapter. A service or layer package does not +capture Python event loops, `PyObject` references, or PyO3 runtime state in its +native factory. + +Cancellation of a Python awaitable must reach the runtime operation. The +adapter must not detach a future merely because its Python wrapper was dropped. + +Blocking `Operator` and `AsyncOperator` should retain the same constructed +native operator graph when converted or cloned. Rebuilding from a scheme and +options would lose stateful Foyer and Throttle identity. + +## Errors + +The Python adapter should distinguish extension failures before mapping normal +OpenDAL operation errors: + +```text +ExtensionNotInstalled +ExtensionLoadError +ExtensionIncompatible +ExtensionConflict +LayerInitializationError +``` + +`ExtensionNotInstalled` may include the canonical distribution name as an +installation hint. It must not run `pip`, modify the environment, or infer a +third-party package name from untrusted input. + +Configuration and operation errors continue to use OpenDAL's Python exception +hierarchy. Extension diagnostics include package and scheme/layer IDs but omit +credentials and unredacted option maps. + +## Serialization + +A dynamic operator cannot rely on native pointer serialization. New pickle +support must choose one of these explicit policies: + +- Serialize a versioned service recipe plus ordered layer recipes, then + reconstruct fresh native resources. +- Reject pickling when an applied service/layer has no declarative + reconstruction policy. + +It must never silently discard layers. Reconstructing a Foyer layer creates or +reopens a cache according to package policy; it does not preserve live in-memory +entries. Reconstructing Throttle starts new token history. + +Existing layered pickles did not record layer recipes, so migration code cannot +recover that lost information retroactively. + +## Packaging Constraints + +The current release matrix includes CPython 3.10-specific wheels, CPython 3.11 +`abi3` wheels, and free-threaded CPython wheels. The dynamic design must prove +which artifacts can actually be shared: + +- The Python adapter follows its existing CPython/`abi3` compatibility rules. +- A language-neutral service/layer library should not link CPython. +- The extension still needs one artifact per supported native target and libc + or deployment floor. +- One extension wheel can cover several Python versions only if every base wheel + requires a protocol in the runtime's supported range and resolves the same + exact shared runtime release. +- Wheel repair must retain the intended shared runtime relationship instead of + copying private runtime libraries into every extension under conflicting + names. +- Free-threaded Python requires explicit lifetime and concurrency tests; `abi3` + does not imply free-threaded compatibility. + +The libhdfs-backed HDFS wheel may have a smaller platform allowlist or ship as a +source distribution. Installing `opendal`, S3, WebDAV, or +`hdfs-native` must not load HDFS code or require Java/Hadoop. + +## Migration + +1. Add the runtime, bootstrap interface, and registry internally while services/layers + remain compiled into the base wheel. +2. Make built-in adapters use the same internal factory interface intended for + external packages. +3. Turn `opendal.services` and `opendal.layers` into filesystem/namespace + packages and re-export existing names. +4. Extract S3 and Timeout as tracer distributions. Keep Memory in the runtime + because OpenDAL core always provides it. +5. Remove extracted components from the base wheel after their packages are + available. +6. Generate package-local configuration types from the Rust service metadata. +7. Validate WebDAV, HDFS, Foyer, and Throttle before declaring the interface + complete. +8. Introduce versioned serialization or explicit non-picklability for dynamic + operators. + +## Python Conformance Gates + +- Base import with no optional extensions installed. +- Explicit imports and lazy entry-point lookup with 1,000 synthetic manifests. +- Namespace-package coexistence under pip and other supported installers. +- Sync and async S3/WebDAV construction through URI and typed config paths. +- HDFS registration/import without native activation and scoped load failure. +- Foyer async creation, cancellation, garbage collection, and reuse. +- Timeout executor behavior through a separately packaged layer. +- Throttle argument validation and shared-handle identity. +- Pickle reconstruction or explicit rejection with missing/incompatible + packages. +- CPython 3.10, `abi3`, and free-threaded artifact composition on every + supported target. diff --git a/bindings/docs/dynamic-extensions/ruby.md b/bindings/docs/dynamic-extensions/ruby.md new file mode 100644 index 000000000000..8a43569c211b --- /dev/null +++ b/bindings/docs/dynamic-extensions/ruby.md @@ -0,0 +1,322 @@ + + +# Ruby Dynamic Extension Design + +Status: pre-RFC binding-specific design proposal. The current Ruby binding +remains one gem and one Magnus native extension. + +This document applies the [shared extension architecture](README.md) to Ruby. +It incorporates the version-locked shared runtime alternative from the earlier +Ruby design and aligns that native model with Python and Node.js. +The [shared compatibility contract](compatibility.md) is canonical for native +ABI, configuration, lifetime, and loader rules; this document defines Ruby +deltas. + +## Current Constraints + +The current Ruby binding provides a useful migration base but not a native +extension seam: + +- `bindings/ruby` builds one `opendal_ruby` `cdylib` and one `opendal` gem. +- `OpenDal::Operator.new(scheme, options)` is blocking-only and constructs + through the compiled core registry. +- Ruby does not currently expose `Operator.from_uri`, `Operator.via_iter`, or an + async operator. +- Retry, concurrent-limit, Throttle, and Timeout middleware implementations are + compiled into the same native extension. +- `Operator#middleware` uses Ruby duck typing, but an independently built native + middleware still cannot access the wrapped Rust `Operator` in another DSO. +- Operation failures currently map broadly to Ruby `RuntimeError`. +- The release process builds a source gem and a small best-effort native-gem + matrix. A source-build path remains important. + +The dynamic design must not describe proposed methods or guarantees as current +behavior. + +## Gem Layout + +The proposed release family is: + +```text +opendal-runtime provides the shared native runtime +opendal owns `require "opendal"` and the Ruby adapter +opendal-service-s3 provides S3 registration and native artifacts +opendal-service-hdfs provides libhdfs-backed HDFS lazily +opendal-layer-timeout provides Timeout +opendal-layer-foyer provides Foyer +``` + +The `opendal` gem declares its `required_runtime_protocol`. The +`opendal-runtime` gem exposes its minimum and current protocol levels for the +binding to check. Each native service/layer gem requires an exact +`opendal-runtime` release and embeds that OpenDAL version in its bootstrap +metadata. Installing the base gem resolves the runtime dependency: + +```console +gem install opendal +``` + +Applications select the main binding and extensions in their `Gemfile`; Bundler +resolves `opendal-runtime`: + +```ruby +gem "opendal", "= " +gem "opendal-service-s3" +gem "opendal-layer-timeout" +gem "opendal-layer-foyer" +``` + +## Registration and Activation + +Each extension gem contains a Ruby registration stub, a JSON manifest, and gem +metadata mapping its canonical service or layer ID to that stub: + +```ruby +require "opendal/runtime" + +OpenDal::Runtime.register_manifest( + File.expand_path("../../../opendal-extension.json", __dir__) +) +``` + +The expected require paths are: + +```ruby +require "opendal" +require "opendal/services/s3" +require "opendal/layers/timeout" +require "opendal/layers/foyer" +``` + +Requiring an extension reads and registers metadata but does not activate its +native library. The first service/layer construction performs native loading +and the exact OpenDAL version check. + +Construction of an unregistered scheme must resolve +the one matching registration stub from installed gem metadata. The resolver +reports duplicate claims, caches results and deterministic failures, handles +aliases deterministically, and never requires every native extension at +startup. Installing dependencies alone is not treated as registration. + +Explicit `require` remains the preferred deterministic registration path. An +application that wants to detect native dependency failures during controlled +startup must also construct or explicitly probe the service/layer, because +registration alone intentionally performs no native load. + +## Proposed Operator Interface + +`Operator.new` remains the compatibility constructor: + +```ruby +require "opendal" +require "opendal/services/s3" + +op = OpenDal::Operator.new("s3", { + "bucket" => "photos", + "region" => "us-east-1", +}) +``` + +The binding can add URI and explicit registry construction as additive methods: + +```ruby +op = OpenDal::Operator.via_iter("s3", { + "bucket" => "photos", + "region" => "us-east-1", +}) + +op = OpenDal::Operator.from_uri( + "s3://photos/archive?region=us-east-1", + {"endpoint" => "https://s3.example.com"} +) +``` + +After those methods exist, `Operator.new` delegates to `via_iter`. Scheme +strings remain canonical so third-party services do not require edits to a +base enum. + +The service gem receives the original URI and explicit string options. S3, +WebDAV, HDFS, and third-party gems retain their own configurator behavior, +validation, credentials, and redaction. + +Typed Ruby configuration objects and hashes convert to the shared +[`ConfigValue`](compatibility.md#configuration-value-contract) grammar. The +base adapter rejects symbols or objects without a declared conversion, cyclic +containers, oversized values, unknown fields, and numeric overflow before +calling package code. Native factories never retain Ruby objects. + +## Proposed Layer Interface + +New code uses `OpenDal::Layers` and `Operator#layer`: + +```ruby +require "opendal/layers/throttle" +require "opendal/layers/timeout" + +limit = OpenDal::Layers::Throttle.new(10 * 1024, 10 * 1024 * 1024) +timeout = OpenDal::Layers::Timeout.new(60, 10) + +layered = op.layer(limit).layer(timeout) +``` + +The Ruby object wraps a runtime-owned native `LayerHandle`, not a package-local +Rust object exposed through Magnus. `Operator#layer` returns a new operator and +preserves native service and context hooks. + +The current names remain compatibility adapters: + +- `Operator#middleware(value)` delegates to `Operator#layer(value)`. +- `OpenDal::Middleware::*` aliases the corresponding `OpenDal::Layers::*` + classes during a deprecation period. The layer classes preserve the current + positional constructors so the aliases do not change existing calls. +- A pure Ruby object implementing only `apply_to` remains a Ruby decorator and + must not be described as equivalent to an arbitrary native layer. + +Timeout values remain finite, non-negative seconds at the Ruby interface. The +adapter uses the current `Duration::try_from_secs_f64` rule, which rounds to the +nearest nanosecond with ties to even, and then emits `SignedDuration`. It rejects +values outside `SignedDuration`'s `i64`-seconds range instead of saturating them. + +Throttle accepts only positive integer `bandwidth` and `burst` values in the +supported `u32` range. Ruby validation must reject invalid values before the +native constructor can assert. + +Later layer calls are outer layers. The binding preserves the +[canonical Timeout/Retry order](compatibility.md#layer-compatibility-rules) and +rejects the known unsafe composition when it can observe both layer IDs. + +## Asynchronous Layer Construction + +Ruby operations remain blocking in the initial design. A layer such as Foyer +still needs asynchronous native initialization: + +```ruby +require "opendal/layers/foyer" + +cache = OpenDal::Layers::Foyer.build( + memory_capacity: 64 << 20, + storage_path: "/var/cache/opendal" +) + +cached = op.layer(cache) +``` + +`Foyer.build` submits the async factory to the shared runtime and waits while +releasing the GVL. It must use the runtime's Tokio instance, clean up partial +resources on failure, and return only after it owns a valid `LayerHandle`. + +The package must not start a private Tokio runtime or hold Ruby values inside +its native future. A future Ruby async interface can adapt the same runtime +future without changing the extension interface. + +## Stateful Layers + +One Ruby layer object preserves one native sharing identity: + +- Applying one Throttle object to several operators shares one quota. +- Constructing two Throttle objects creates independent quotas. +- Applying one Foyer object shares one cache subject to its documented + namespace restriction. +- Derived operators keep the layer alive after the original Ruby wrapper is + collected. + +The first version does not define `Marshal` support for operators or live layer +handles. A future declarative recipe format must reconstruct new native state +rather than claiming to serialize cache contents, limiter history, a JVM, or a +Tokio runtime. + +## Errors + +The runtime should expose Ruby exception classes for extension lifecycle +failures: + +```text +OpenDal::ExtensionNotInstalled +OpenDal::ExtensionLoadError +OpenDal::ExtensionIncompatible +OpenDal::ExtensionConflict +OpenDal::LayerInitializationError +``` + +Normal OpenDAL error kinds should also map to stable Ruby exception classes +rather than losing all structure in `RuntimeError`. Compatibility aliases or a +common superclass can preserve existing rescue behavior. + +Errors include package ID, scheme/layer ID, and construction operation. They do +not include credentials or unredacted option hashes. + +## Native Gem and Loader Constraints + +- The base Magnus extension follows the binding's supported Ruby versions and + platforms. +- A language-neutral service/layer library should not link Ruby or Magnus. +- Native extension gems still need artifacts for every supported OS, + architecture, libc/deployment floor, and OpenDAL version. +- Source gems build against the exact SDK/runtime metadata and verify the + resulting embedded OpenDAL version. +- The runtime loads an explicit artifact path from the gem manifest and keeps + the library pinned. +- Linux symbol visibility, macOS install names, and Windows DLL discovery must + be tested with gems installed in normal Bundler layouts. +- A native-gem failure may fall back to a documented source build, but it must + not silently load a different OpenDAL release. + +The current native-gem matrix is best effort. Dynamic extensions should not +claim broader binary coverage until runtime plus adapter artifacts pass an +installation test on that platform. + +## Ractor, Threads, and Fork + +The first design does not promise Ractor shareability. Runtime registries and +native handles may be process-global Rust state, but they must not retain +Ractor-local Ruby objects. + +Blocking operations and layer initialization release the GVL only through +well-defined base-adapter helpers. Package code must not call Ruby from shared +runtime worker threads. + +Runtime, JVM, connection-pool, and Foyer state is unsupported after `fork` +unless a package later defines explicit reinitialization behavior. + +## Migration + +1. Introduce `NativeRuntime` and an extension registry inside the current gem. +2. Route compiled services and middleware through internal factories using the + proposed SDK shapes. +3. Add `via_iter`, `from_uri`, `layer`, `OpenDal::Layers`, and structured errors + without removing current methods. +4. Extract S3 and Timeout as tracer gems and remove them from the base gem after + their packages are available. +5. Add gem-metadata resolution for compatibility constructors. +6. Validate WebDAV configurator behavior and HDFS lazy activation. +7. Validate Foyer initialization and Throttle sharing before publishing the + third-party SDK. +8. Expand native gem targets only after artifact-level installation tests pass. + +## Ruby Conformance Gates + +- Base runtime installation and `require "opendal"` without optional extensions. +- Explicit requires and selected lazy resolution without eager native loading. +- Existing `Operator.new` and `middleware` compatibility behavior. +- Proposed URI construction after `from_uri` is implemented. +- S3/WebDAV secret-free construction errors. +- HDFS failure scoped to the HDFS gem while other services remain usable. +- Foyer initialization with the GVL released, including cancellation/cleanup. +- Layer wrapper garbage collection while derived operators remain active. +- Source gem plus every claimed native gem target. +- Thread, Ractor-rejection, and fork-rejection behavior documented and tested. diff --git a/bindings/docs/dynamic-extensions/symbol-isolation.md b/bindings/docs/dynamic-extensions/symbol-isolation.md new file mode 100644 index 000000000000..5220e88b4472 --- /dev/null +++ b/bindings/docs/dynamic-extensions/symbol-isolation.md @@ -0,0 +1,176 @@ + + +# Native Extension Symbol Isolation + +Status: pre-RFC contract for the shared runtime candidate. OpenDAL does not +provide or guarantee this isolation today. + +Each final native artifact statically contains much of its Rust dependency +graph. Loading two artifacts into one process can therefore create two copies +of `opendal-core`, Tokio, and other dependencies. Matching source versions do +not turn those copies into one runtime, and Rust visibility does not determine +which symbols a platform dynamic loader exports. + +The shared runtime design requires two separate properties: + +1. **Symbol isolation** prevents one native artifact from accidentally + resolving private code or data from another artifact. +2. **Runtime state ownership** gives process-coupled facilities one explicit + owner and exposes them to extensions through runtime handles or SDK + functions. + +Symbol isolation makes private dependency copies possible. It does not make +duplicated runtime state safe. + +## Isolation Invariants + +The runtime and every native extension must satisfy these invariants: + +- The runtime exports only its versioned runtime bootstrap and explicitly + documented platform integration symbols. +- A service or layer extension exports only its generated, package-unique + bootstrap symbol and any initializer required by its language runtime. +- Every other package symbol has local or hidden visibility in the final + artifact. +- An artifact does not rely on unresolved Rust symbols being supplied by the + main binding, runtime, or another extension. +- A private dependency type, vtable, allocator-owned value, thread-local, or + mutable global does not cross the extension boundary. +- The side that creates an allocation destroys it unless an SDK handle + explicitly transfers ownership. +- The runtime pins an extension while any callback, vtable, task, or object can + execute code from that extension. + +An exact OpenDAL release match does not relax these rules. Release matching +selects the release-specific SDK contract; it does not authorize ambient +symbol resolution between artifacts. + +## Dependency Ownership + + + +| Dependency kind | Examples | Required treatment | +| --------------- | -------- | ------------------ | +| Runtime-owned state | Operator registry, executor, timers, shared HTTP context, library leases | The shared runtime owns the state. Extensions use runtime handles or SDK functions. | +| Extension-private implementation | Signing, XML parsing, checksums, service-specific caches | The extension may contain a private copy when its symbols stay local and its values do not cross the boundary. | +| Process-global native facility | JVM, TLS libraries with global configuration, native client libraries | Packaging selects one compatible process-wide instance or isolates the facility out of process. Symbol hiding alone is insufficient. | +| Exact-release interface | Factory adapters, opaque handle operations, creator-side destructors | The extension SDK generates the interface for one coordinated OpenDAL release. | +| Language initializer | CPython, Ruby, or Node-API module entry | The artifact exports only the initializer required by that language in addition to any explicitly selected OpenDAL bootstrap. | + + + +This classification applies to every transitive dependency, not only Tokio. +The SDK must decide whether a dependency is runtime-owned, extension-private, +or process-global before its code enters an extension artifact. + +## Export Surfaces + +The intended native export surfaces are small and reviewable. For example: + +```text +shared runtime: + opendal_runtime_get_api_v1 + +S3 extension: + opendal_service_s3_bootstrap_v1 + +Timeout extension: + opendal_layer_timeout_bootstrap_v1 +``` + +A language-native addon may instead expose the initializer required by its +language runtime. For example, Node-API controls the addon initializer name. +The initializer must return or register the same validated package metadata +and API rather than creating an unversioned second interface. + +Public Rust items such as `Operator`, `Layer`, or Tokio functions are source +interfaces. They are not part of the dynamic export allowlist and do not form a +stable ABI. + +## Platform Enforcement + +The SDK build pipeline owns symbol visibility at the final link step: + +- On ELF targets, it uses a version script or equivalent export list, localizes + archive symbols, and inspects both dynamic symbols and `DT_NEEDED` entries. + `RTLD_LOCAL` remains defense in depth; it is not the isolation mechanism. +- On macOS, it uses an exported-symbols list and inspects exports, imports, and + install names in the final Mach-O artifact. +- On Windows, it generates an explicit `.def` file or equivalent export list + and inspects the PE export/import tables and dependent DLLs. + +The build must fail when the toolchain cannot enforce the selected export +surface. "Where supported" is not sufficient for an official extension target. + +## Artifact Monitoring + +Every release pipeline produces a normalized report for the runtime, each main +binding artifact, and each official extension artifact. The report records: + +- Exported dynamic symbols. +- Imported dynamic symbols and their provider libraries. +- Direct native library dependencies. +- Target identity and the SDK inputs that selected the allowlists. + +The pipeline compares the report with a checked-in or generated allowlist and +fails on any unreviewed addition. This catches transitive crates that introduce +`no_mangle` or C exports even when their Rust symbols remain hidden. Release +artifacts retain the report so later toolchain or dependency updates can be +compared with the accepted baseline. + +Artifact inspection must cover the final repaired wheel, gem, npm package, or +shared runtime artifact. Auditing an intermediate Cargo output does not detect +changes introduced by packaging repair, symbol stripping, or native dependency +relocation. + +## Co-loading Tests + +Export inspection cannot prove runtime behavior. The conformance suite also +loads multiple independently compiled extensions into one process and verifies: + +1. Each loader lookup resolves only the package-unique bootstrap requested by + the manifest. +2. No deliberately duplicated private sentinel resolves across artifacts. +3. An operator created by a service extension composes with a layer extension + through the exact-release SDK interface. +4. Real operations exercise runtime-coupled facilities such as timers, + executors, credential file loading, HTTP, and cancellation. +5. All extension-owned objects are destroyed before a test unloads a library. + +The [prototype](prototype/README.md) supplies an initial Linux export audit and +explores layout-sensitive `Operator` pointer exchange between independently +linked S3 and Timeout artifacts. It does not implement the proposed extension +SDK, its ownership contract, or a shared runtime. Therefore it does not satisfy +the third, fourth, or fifth conformance properties. + +## Non-goals + +Symbol isolation does not provide: + +- A security sandbox for native extensions. +- A stable Rust ABI across OpenDAL releases. +- Compatibility between arbitrary process-global native libraries. +- Shared state merely because two artifacts contain the same dependency + version. +- Safe library unloading without the lifetime rules in the compatibility + contract. + +Native extensions remain trusted in-process code under the project +[security threat model](../../../SECURITY-THREAT-MODEL.md). The isolation policy +prevents accidental linking and interposition; it does not constrain malicious +native code. diff --git a/bindings/python/dynamic-extensions/.cargo/config.toml b/bindings/python/dynamic-extensions/.cargo/config.toml new file mode 100644 index 000000000000..75f24ae96dae --- /dev/null +++ b/bindings/python/dynamic-extensions/.cargo/config.toml @@ -0,0 +1,17 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[target.'cfg(target_os = "linux")'] +rustflags = ["-C", "link-arg=-Wl,--exclude-libs,ALL"] diff --git a/bindings/python/dynamic-extensions/.gitignore b/bindings/python/dynamic-extensions/.gitignore new file mode 100644 index 000000000000..d7d2adf88a85 --- /dev/null +++ b/bindings/python/dynamic-extensions/.gitignore @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +/target/ diff --git a/bindings/python/dynamic-extensions/Cargo.lock b/bindings/python/dynamic-extensions/Cargo.lock new file mode 100644 index 000000000000..295521b82f6d --- /dev/null +++ b/bindings/python/dynamic-extensions/Cargo.lock @@ -0,0 +1,1313 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + +[[package]] +name = "base64" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b25655df2c3cdd83c5e5b293b88acd880332b2ddadd7c30ac43144fdc0033da9" + +[[package]] +name = "bitflags" +version = "1.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "block-buffer" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cmov" +version = "0.5.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c9ea0ac24bc397ab3c98583a3c9ba74fa56b09a4449bbe172b9b1ddb016027a" + +[[package]] +name = "const-oid" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" + +[[package]] +name = "const-random" +version = "0.1.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "87e00182fe74b066627d63b85fd550ac2998d4b0bd86bfed477a0ae4c7c71359" +dependencies = [ + "const-random-macro", +] + +[[package]] +name = "const-random-macro" +version = "0.1.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9d839f2a20b0aee515dc581a6172f2321f96cab76c1a38a4c584a194955390e" +dependencies = [ + "getrandom 0.2.17", + "once_cell", + "tiny-keccak", +] + +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + +[[package]] +name = "crc-fast" +version = "1.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e75b2483e97a5a7da73ac68a05b629f9c53cff58d8ed1c77866079e18b00dba5" +dependencies = [ + "digest 0.10.7", + "spin", +] + +[[package]] +name = "crunchy" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "crypto-common" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "ctutils" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e" +dependencies = [ + "cmov", +] + +[[package]] +name = "defmt" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" +dependencies = [ + "bitflags 1.3.2", + "defmt-macros", +] + +[[package]] +name = "defmt-macros" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" +dependencies = [ + "defmt-parser", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "defmt-parser" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" +dependencies = [ + "thiserror", +] + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "crypto-common 0.1.7", +] + +[[package]] +name = "digest" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" +dependencies = [ + "block-buffer", + "const-oid", + "crypto-common 0.2.2", + "ctutils", +] + +[[package]] +name = "displaydoc" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "dlv-list" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "442039f5147480ba31067cb00ada1adae6892028e40e45fc5de7b7df6dcc1b5f" +dependencies = [ + "const-random", +] + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "form_urlencoded" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" +dependencies = [ + "percent-encoding", +] + +[[package]] +name = "fs-extension" +version = "0.0.0" +dependencies = [ + "opendal-core", + "opendal-dynamic-extension-sdk", + "opendal-service-fs", + "serde_json", + "tokio", +] + +[[package]] +name = "futures" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a88cf1f829d945f548cf8fec32c61b1f202b6d93b45848602fc02af4b12ad218" +dependencies = [ + "futures-channel", + "futures-core", + "futures-executor", + "futures-io", + "futures-sink", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-channel" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "262590f4fe6afeb0bc83be1daa64e52657fe185690a958af7f3ad0e92085c5ae" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2cd50c473c80f6d7c3670a752354b8e569b1a7cbfdc0419ec88e5edad85e0dc7" + +[[package]] +name = "futures-executor" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6754879cc9f2c66f88c6e5c35344bb0bdb0708b0352b1201815667c7eabc7458" +dependencies = [ + "futures-core", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-io" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4577ecaa3c4f96589d473f679a71b596316f6641bc350038b962a5daf0085d7a" + +[[package]] +name = "futures-macro" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d6d3cde68c518367be28956066ddfef33813991b77a55005a69dae04bf3b10b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "futures-sink" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e34418ac499d6305c2fb5ad0ed2f6ac998c5f8ca209b4510f7f94242c647e307" + +[[package]] +name = "futures-task" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b231ed28831efb4a61a08580c4bc233ec56bc009f4cd8f52da2c3cb97df0c109" + +[[package]] +name = "futures-util" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a77a90a256fce34da66415271e30f94ee91c57b04b8a2c042d9cf3220179deaa" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "slab", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "libc", + "wasi", +] + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "libc", + "r-efi", +] + +[[package]] +name = "hashbrown" +version = "0.14.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" + +[[package]] +name = "hex" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" + +[[package]] +name = "hmac" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6303bc9732ae41b04cb554b844a762b4115a61bfaa81e3e83050991eeb56863f" +dependencies = [ + "digest 0.11.3", +] + +[[package]] +name = "http" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "918d3568bebf352712bc2ef3d46a8bcf1a75b373be6539de198e9105cbbf9ce0" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "hybrid-array" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "707114b52a152fa7bdb290cd7cd5912d9467273b6d74e21b8d81aca1f8533f6b" +dependencies = [ + "typenum", +] + +[[package]] +name = "icu_collections" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c" +dependencies = [ + "displaydoc", + "potential_utf", + "utf8_iter", + "yoke", + "zerofrom", + "zerovec", +] + +[[package]] +name = "icu_locale_core" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29" +dependencies = [ + "displaydoc", + "litemap", + "tinystr", + "writeable", + "zerovec", +] + +[[package]] +name = "icu_normalizer" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4" +dependencies = [ + "icu_collections", + "icu_normalizer_data", + "icu_properties", + "icu_provider", + "smallvec", + "zerovec", +] + +[[package]] +name = "icu_normalizer_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38" + +[[package]] +name = "icu_properties" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de" +dependencies = [ + "icu_collections", + "icu_locale_core", + "icu_properties_data", + "icu_provider", + "zerotrie", + "zerovec", +] + +[[package]] +name = "icu_properties_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14" + +[[package]] +name = "icu_provider" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421" +dependencies = [ + "displaydoc", + "icu_locale_core", + "writeable", + "yoke", + "zerofrom", + "zerotrie", + "zerovec", +] + +[[package]] +name = "idna" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de" +dependencies = [ + "idna_adapter", + "smallvec", + "utf8_iter", +] + +[[package]] +name = "idna_adapter" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" +dependencies = [ + "icu_normalizer", + "icu_properties", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "jiff" +version = "0.2.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc" +dependencies = [ + "defmt", + "jiff-core", + "jiff-static", + "jiff-tzdb-platform", + "js-sys", + "log", + "portable-atomic", + "portable-atomic-util", + "serde_core", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "jiff-core" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7feca88439efe53da3754500c1851dedf3cb36c524dd5cf8225cc0794de95d09" +dependencies = [ + "defmt", +] + +[[package]] +name = "jiff-static" +version = "0.2.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204" +dependencies = [ + "jiff-core", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "jiff-tzdb" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "142bd39932ad231f10513df9ab62661fead8719872150b7ad02a2df79f4e141e" + +[[package]] +name = "jiff-tzdb-platform" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "875a5a69ac2bab1a891711cf5eccbec1ce0341ea805560dcd90b7a2e925132e8" +dependencies = [ + "jiff-tzdb", +] + +[[package]] +name = "js-sys" +version = "0.3.103" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "libloading" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" +dependencies = [ + "cfg-if", + "windows-link", +] + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "litemap" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0" + +[[package]] +name = "log" +version = "0.4.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" + +[[package]] +name = "md-5" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98" +dependencies = [ + "cfg-if", + "digest 0.11.3", +] + +[[package]] +name = "mea" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2640d335e7273dacdcf51044026139b2e269c3bb0dfc3f8cb3496b85e3f6a42c" +dependencies = [ + "slab", +] + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "opendal-core" +version = "0.58.1" +dependencies = [ + "anyhow", + "base64 0.23.0", + "bytes", + "futures", + "http", + "jiff", + "log", + "md-5", + "mea", + "percent-encoding", + "quick-xml", + "reqsign-core", + "serde", + "serde_json", + "tokio", + "url", + "uuid", + "web-time", +] + +[[package]] +name = "opendal-dynamic-extension-sdk" +version = "0.0.0" + +[[package]] +name = "opendal-runtime-poc" +version = "0.0.0" +dependencies = [ + "libloading", + "opendal-core", + "opendal-dynamic-extension-sdk", +] + +[[package]] +name = "opendal-service-fs" +version = "0.58.1" +dependencies = [ + "bytes", + "log", + "opendal-core", + "serde", + "tokio", + "xattr", +] + +[[package]] +name = "opendal-service-s3" +version = "0.58.1" +dependencies = [ + "base64 0.23.0", + "bytes", + "crc-fast", + "http", + "log", + "md-5", + "opendal-core", + "quick-xml", + "reqsign-aws-v4", + "reqsign-core", + "reqsign-file-read-tokio", + "serde", + "url", +] + +[[package]] +name = "ordered-multimap" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "49203cdcae0030493bad186b28da2fa25645fa276a51b6fec8010d281e02ef79" +dependencies = [ + "dlv-list", + "hashbrown", +] + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "portable-atomic" +version = "1.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d20d5497ef88037a52ff98267d066e7f11fcc5e99bbfbd58a42336193aacec3" + +[[package]] +name = "portable-atomic-util" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" +dependencies = [ + "portable-atomic", +] + +[[package]] +name = "potential_utf" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564" +dependencies = [ + "zerovec", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quick-xml" +version = "0.41.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e660451e55124f798a69a5af3f49ccfbefbd41910eefd25caf2393e1f3473ec1" +dependencies = [ + "memchr", + "serde", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "reqsign-aws-v4" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc883bc56889f3e4a419265c87facea222a921debc5c6f15c7fd8b68ec4b36b2" +dependencies = [ + "anyhow", + "bytes", + "form_urlencoded", + "hex", + "http", + "log", + "percent-encoding", + "quick-xml", + "reqsign-core", + "rust-ini", + "serde", + "serde_json", + "serde_urlencoded", + "sha1", +] + +[[package]] +name = "reqsign-core" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e38b44697c60a823705ccef85cb04d8e0527c9d16ed7c58bf1c6395bdd24ceb" +dependencies = [ + "anyhow", + "base64 0.22.1", + "bytes", + "futures", + "hex", + "hmac", + "http", + "jiff", + "log", + "percent-encoding", + "sha1", + "sha2", + "windows-sys", +] + +[[package]] +name = "reqsign-file-read-tokio" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "688ff0ae421b8d4b92b53fdafaf53df2de28f428a9962edcf21702990b26f74b" +dependencies = [ + "anyhow", + "reqsign-core", + "tokio", +] + +[[package]] +name = "rust-ini" +version = "0.21.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "796e8d2b6696392a43bea58116b667fb4c29727dc5abd27d6acf338bb4f688c7" +dependencies = [ + "cfg-if", + "ordered-multimap", +] + +[[package]] +name = "rustix" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +dependencies = [ + "bitflags 2.13.1", + "errno", + "libc", + "linux-raw-sys", + "windows-sys", +] + +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + +[[package]] +name = "ryu" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" + +[[package]] +name = "s3-extension" +version = "0.0.0" +dependencies = [ + "opendal-core", + "opendal-dynamic-extension-sdk", + "opendal-service-s3", + "serde_json", + "tokio", +] + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "serde_json" +version = "1.0.151" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "serde_urlencoded" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" +dependencies = [ + "form_urlencoded", + "itoa", + "ryu", + "serde", +] + +[[package]] +name = "sha1" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest 0.11.3", +] + +[[package]] +name = "sha2" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest 0.11.3", +] + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "smallvec" +version = "1.15.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" + +[[package]] +name = "spin" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "023a211cb3138dbc438680b32560ad89f699977624c9f8dbb95a47d5b4c07dd3" + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "thiserror" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09a43598840e33d5b0331f38c5e30d13bb11c11210a4b58f0d9b18a5a5eefcd9" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "43cbfe0cf76104d42a574802844187e84a305e531ed54455f11fbde0f10541cd" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "tiny-keccak" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2c9d3793400a45f954c52e73d068316d76b6f4e36977e3fcebb13a2721e80237" +dependencies = [ + "crunchy", +] + +[[package]] +name = "tinystr" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d" +dependencies = [ + "displaydoc", + "zerovec", +] + +[[package]] +name = "tokio" +version = "1.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "202caea871b69668250d242070849eb495be178ed697a3e98aebce5bc81a0bed" +dependencies = [ + "bytes", + "pin-project-lite", + "tokio-macros", +] + +[[package]] +name = "tokio-macros" +version = "2.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "url" +version = "2.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" +dependencies = [ + "form_urlencoded", + "idna", + "percent-encoding", + "serde", +] + +[[package]] +name = "utf8_iter" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" + +[[package]] +name = "uuid" +version = "1.24.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf3923a6f5c4c6382e0b653c4117f48d631ea17f38ed86e2a828e6f7412f5239" +dependencies = [ + "getrandom 0.4.3", + "js-sys", + "serde_core", + "wasm-bindgen", +] + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "wasm-bindgen" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn 2.0.119", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "web-time" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "writeable" +version = "0.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" + +[[package]] +name = "xattr" +version = "1.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32e45ad4206f6d2479085147f02bc2ef834ac85886624a23575ae137c8aa8156" +dependencies = [ + "libc", + "rustix", +] + +[[package]] +name = "yoke" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "709fe23a0424b6a435d82152b1bd3fdfb0833487d5fa90d05d42762a9891fef5" +dependencies = [ + "stable_deref_trait", + "yoke-derive", + "zerofrom", +] + +[[package]] +name = "yoke-derive" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerofrom" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272" +dependencies = [ + "zerofrom-derive", +] + +[[package]] +name = "zerofrom-derive" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerotrie" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf" +dependencies = [ + "displaydoc", + "yoke", + "zerofrom", +] + +[[package]] +name = "zerovec" +version = "0.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239" +dependencies = [ + "yoke", + "zerofrom", + "zerovec-derive", +] + +[[package]] +name = "zerovec-derive" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/bindings/python/dynamic-extensions/Cargo.toml b/bindings/python/dynamic-extensions/Cargo.toml new file mode 100644 index 000000000000..5d2c054e2f96 --- /dev/null +++ b/bindings/python/dynamic-extensions/Cargo.toml @@ -0,0 +1,33 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[workspace] +members = ["runtime", "sdk", "service-fs", "service-s3"] +resolver = "2" + +[workspace.package] +edition = "2024" +license = "Apache-2.0" +publish = false +rust-version = "1.91" +version = "0.0.0" + +[workspace.dependencies] +libloading = "0.8.9" +opendal-core = { path = "../../../core/core", default-features = false } +opendal-dynamic-extension-sdk = { path = "sdk" } +opendal-service-fs = { path = "../../../core/services/fs" } +opendal-service-s3 = { path = "../../../core/services/s3" } +tokio = { version = "1", features = ["rt-multi-thread", "time"] } diff --git a/bindings/python/dynamic-extensions/README.md b/bindings/python/dynamic-extensions/README.md new file mode 100644 index 000000000000..7f9329564d4f --- /dev/null +++ b/bindings/python/dynamic-extensions/README.md @@ -0,0 +1,97 @@ + + +# Python Dynamic Extension POC + +This in-tree POC tests the package and loading model proposed in +[`bindings/docs/dynamic-extensions/python.md`](../../docs/dynamic-extensions/python.md). +It does not change the released Python binding. + +The current-PyO3 [comparison prototype](pyo3-comparison/README.md) separately +tests an FS service and MIME layer with both capsule and direct PyO3 adapters. +This directory's original POC instead tests the wider versioned operation-table +boundary proposed by the design document. + +The workspace builds three independently linked native artifacts: + +- `opendal-runtime-poc` owns protocol negotiation, service registration, + opaque Python operator handles, library leases, and error transport. +- `s3-extension` owns the S3 builder, operator, dependencies, and operations. +- `fs-extension` owns the FS builder, operator, dependencies, and operations. + +The Python package roots model three separately installed distributions. The +main package exposes `opendal.Operator`. Importing `opendal.services.s3` or +`opendal.services.fs` resolves that package's native artifact and registers its +logical manifest without loading native service code. The manifest does not +contain an artifact path or target identity, and configuration crosses the +runtime interface as string pairs. + +On Linux, run: + +```console +./run-python-linux.sh +``` + +The separately staged packages support this usage: + +```python +import opendal.services.fs +import opendal.services.s3 +from opendal import Operator + +with Operator("s3", bucket="my-bucket", region="us-east-1") as s3: + print(s3.info) + +with Operator("fs", root="/tmp/opendal") as fs: + fs.write("hello.txt", b"Hello, OpenDAL!") + print(fs.read("hello.txt")) +``` + +The runner executes the complete [`python/example.py`](python/example.py) in +addition to the assertions in `python/test_poc.py`. + +The script builds each native artifact in a separate Cargo target directory, +checks the final ELF export allowlists, stages the three Python package roots, +and verifies these behaviors: + +1. S3 construction fails before importing its service package. +2. Importing the S3 package registers metadata without loading native code. +3. Constructing an S3 operator loads its package-unique bootstrap and validates + its package, component, entry symbol, protocol, and OpenDAL identities. +4. Importing the FS package registers FS and completes a real write and read. +5. Closing a Python operator invokes the extension-provided destructor before + releasing its library lease. + +## Deliberate Gaps + +This POC keeps the interface small enough to answer the Python packaging +question. It exposes only construction, information, read, write, and +destruction. Each extension owns the Tokio runtime used by its operations. +Consequently, it does not yet prove the selected shared-runtime design's most +important property: one OpenDAL and Tokio graph that can compose arbitrary +native layers. + +An earlier iteration transferred independently linked `Operator` values into a +runtime-owned Tokio graph. A real FS operation aborted because the FS library's +Tokio thread-local state could not observe the runtime library's Tokio context. +Another iteration used a Rust `dylib`, but separate S3 and FS builds generated +different runtime binaries because downstream monomorphizations changed the +dylib. The production design must solve this linkage problem or accept a wider +operation interface before extracting layers. + +The POC also omits wheel building, automatic entry-point discovery, async +Python operations, aliases, typed configuration, and non-Linux targets. diff --git a/bindings/python/dynamic-extensions/audit-elf-exports.sh b/bindings/python/dynamic-extensions/audit-elf-exports.sh new file mode 100755 index 000000000000..97845f6c514c --- /dev/null +++ b/bindings/python/dynamic-extensions/audit-elf-exports.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env sh +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eu + +if [ "$#" -lt 2 ]; then + echo "usage: $0 ARTIFACT EXPECTED_SYMBOL..." >&2 + exit 2 +fi + +artifact=$1 +shift +expected=$(printf '%s\n' "$@" | LC_ALL=C sort -u) +actual=$( + nm --dynamic --defined-only --extern-only --format=posix "$artifact" \ + | awk '{ print $1 }' \ + | sed 's/@.*//' \ + | LC_ALL=C sort -u +) + +if [ "$actual" != "$expected" ]; then + echo "unexpected exports in $artifact" >&2 + echo "expected: $expected" >&2 + echo "actual:" >&2 + echo "$actual" >&2 + exit 1 +fi + +echo "$artifact exports only the expected symbols" diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/.gitignore b/bindings/python/dynamic-extensions/pyo3-comparison/.gitignore new file mode 100644 index 000000000000..d7d2adf88a85 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/.gitignore @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +/target/ diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/Cargo.lock b/bindings/python/dynamic-extensions/pyo3-comparison/Cargo.lock new file mode 100644 index 000000000000..3044d0306635 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/Cargo.lock @@ -0,0 +1,1133 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "base64" +version = "0.23.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5" + +[[package]] +name = "bitflags" +version = "1.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "block-buffer" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "const-oid" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" + +[[package]] +name = "crypto-common" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "defmt" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" +dependencies = [ + "bitflags 1.3.2", + "defmt-macros", +] + +[[package]] +name = "defmt-macros" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" +dependencies = [ + "defmt-parser", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "defmt-parser" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" +dependencies = [ + "thiserror", +] + +[[package]] +name = "digest" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" +dependencies = [ + "block-buffer", + "const-oid", + "crypto-common", +] + +[[package]] +name = "displaydoc" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "form_urlencoded" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" +dependencies = [ + "percent-encoding", +] + +[[package]] +name = "futures" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a88cf1f829d945f548cf8fec32c61b1f202b6d93b45848602fc02af4b12ad218" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-sink", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-channel" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "262590f4fe6afeb0bc83be1daa64e52657fe185690a958af7f3ad0e92085c5ae" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2cd50c473c80f6d7c3670a752354b8e569b1a7cbfdc0419ec88e5edad85e0dc7" + +[[package]] +name = "futures-io" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4577ecaa3c4f96589d473f679a71b596316f6641bc350038b962a5daf0085d7a" + +[[package]] +name = "futures-macro" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d6d3cde68c518367be28956066ddfef33813991b77a55005a69dae04bf3b10b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "futures-sink" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e34418ac499d6305c2fb5ad0ed2f6ac998c5f8ca209b4510f7f94242c647e307" + +[[package]] +name = "futures-task" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b231ed28831efb4a61a08580c4bc233ec56bc009f4cd8f52da2c3cb97df0c109" + +[[package]] +name = "futures-util" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a77a90a256fce34da66415271e30f94ee91c57b04b8a2c042d9cf3220179deaa" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "slab", +] + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "libc", + "r-efi", +] + +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "http" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "918d3568bebf352712bc2ef3d46a8bcf1a75b373be6539de198e9105cbbf9ce0" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "hybrid-array" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "707114b52a152fa7bdb290cd7cd5912d9467273b6d74e21b8d81aca1f8533f6b" +dependencies = [ + "typenum", +] + +[[package]] +name = "icu_collections" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c" +dependencies = [ + "displaydoc", + "potential_utf", + "utf8_iter", + "yoke", + "zerofrom", + "zerovec", +] + +[[package]] +name = "icu_locale_core" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29" +dependencies = [ + "displaydoc", + "litemap", + "tinystr", + "writeable", + "zerovec", +] + +[[package]] +name = "icu_normalizer" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4" +dependencies = [ + "icu_collections", + "icu_normalizer_data", + "icu_properties", + "icu_provider", + "smallvec", + "zerovec", +] + +[[package]] +name = "icu_normalizer_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38" + +[[package]] +name = "icu_properties" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de" +dependencies = [ + "icu_collections", + "icu_locale_core", + "icu_properties_data", + "icu_provider", + "zerotrie", + "zerovec", +] + +[[package]] +name = "icu_properties_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14" + +[[package]] +name = "icu_provider" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421" +dependencies = [ + "displaydoc", + "icu_locale_core", + "writeable", + "yoke", + "zerofrom", + "zerotrie", + "zerovec", +] + +[[package]] +name = "idna" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de" +dependencies = [ + "idna_adapter", + "smallvec", + "utf8_iter", +] + +[[package]] +name = "idna_adapter" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" +dependencies = [ + "icu_normalizer", + "icu_properties", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "jiff" +version = "0.2.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc" +dependencies = [ + "defmt", + "jiff-core", + "jiff-static", + "jiff-tzdb-platform", + "js-sys", + "log", + "portable-atomic", + "portable-atomic-util", + "serde_core", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "jiff-core" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7feca88439efe53da3754500c1851dedf3cb36c524dd5cf8225cc0794de95d09" +dependencies = [ + "defmt", +] + +[[package]] +name = "jiff-static" +version = "0.2.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204" +dependencies = [ + "jiff-core", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "jiff-tzdb" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "142bd39932ad231f10513df9ab62661fead8719872150b7ad02a2df79f4e141e" + +[[package]] +name = "jiff-tzdb-platform" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "875a5a69ac2bab1a891711cf5eccbec1ce0341ea805560dcd90b7a2e925132e8" +dependencies = [ + "jiff-tzdb", +] + +[[package]] +name = "js-sys" +version = "0.3.103" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "litemap" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0" + +[[package]] +name = "log" +version = "0.4.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" + +[[package]] +name = "md-5" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98" +dependencies = [ + "cfg-if", + "digest", +] + +[[package]] +name = "mea" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "31fc7d159de0085ab6dd7ff145a9819442cfd3d098f783263120503c3f3e58b0" +dependencies = [ + "slab", +] + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "mime" +version = "0.3.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a" + +[[package]] +name = "mime_guess" +version = "2.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f7c44f8e672c00fe5308fa235f821cb4198414e1c77935c1ab6948d3fd78550e" +dependencies = [ + "mime", + "unicase", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "opendal-core" +version = "0.58.1" +dependencies = [ + "anyhow", + "base64", + "bytes", + "futures", + "http", + "jiff", + "log", + "md-5", + "mea", + "percent-encoding", + "quick-xml", + "serde", + "serde_json", + "tokio", + "url", + "uuid", + "web-time", +] + +[[package]] +name = "opendal-layer-mime-guess" +version = "0.58.1" +dependencies = [ + "mime_guess", + "opendal-core", +] + +[[package]] +name = "opendal-service-fs" +version = "0.58.1" +dependencies = [ + "bytes", + "log", + "opendal-core", + "serde", + "tokio", + "xattr", +] + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "pin-project" +version = "1.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2466b2336ed02bcdca6b294417127b90ec92038d1d5c4fbeac971a922e0e0924" +dependencies = [ + "pin-project-internal", +] + +[[package]] +name = "pin-project-internal" +version = "1.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c96395f0a926bc13b1c17622aaddda1ecb55d49c8f1bf9777e4d877800a43f8b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "portable-atomic" +version = "1.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d20d5497ef88037a52ff98267d066e7f11fcc5e99bbfbd58a42336193aacec3" + +[[package]] +name = "portable-atomic-util" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" +dependencies = [ + "portable-atomic", +] + +[[package]] +name = "potential_utf" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564" +dependencies = [ + "zerovec", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "pyo3" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd274650b21d4bfc26a0a47587962c1edb425f69287324355cd040c3ea66071c" +dependencies = [ + "libc", + "once_cell", + "portable-atomic", + "pyo3-build-config", + "pyo3-ffi", + "pyo3-macros", +] + +[[package]] +name = "pyo3-build-config" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c5e2a7d2f0d013342f295c048ad19237add5154a55b1c5a254c0ec93d4109078" +dependencies = [ + "target-lexicon", +] + +[[package]] +name = "pyo3-comparison-base" +version = "0.0.0" +dependencies = [ + "pyo3", + "pyo3-comparison-bridge", +] + +[[package]] +name = "pyo3-comparison-bridge" +version = "0.0.0" +dependencies = [ + "opendal-core", + "pin-project", + "pyo3", + "tokio", +] + +[[package]] +name = "pyo3-comparison-fs-capsule" +version = "0.0.0" +dependencies = [ + "opendal-core", + "opendal-service-fs", + "pyo3", + "pyo3-comparison-bridge", +] + +[[package]] +name = "pyo3-comparison-fs-direct" +version = "0.0.0" +dependencies = [ + "opendal-core", + "opendal-service-fs", + "pyo3", + "pyo3-comparison-bridge", +] + +[[package]] +name = "pyo3-comparison-mime-capsule" +version = "0.0.0" +dependencies = [ + "opendal-layer-mime-guess", + "pyo3", + "pyo3-comparison-bridge", +] + +[[package]] +name = "pyo3-comparison-mime-direct" +version = "0.0.0" +dependencies = [ + "opendal-layer-mime-guess", + "pyo3", + "pyo3-comparison-bridge", +] + +[[package]] +name = "pyo3-ffi" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ca85c467da1bbc8d866eea5deff9cf29ea5f7785054a17da36e65bda9c05845b" +dependencies = [ + "libc", + "pyo3-build-config", +] + +[[package]] +name = "pyo3-macros" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ac53762fd065daa3194dd09337a38bd793a188100fd1a9304c4ab312d901771" +dependencies = [ + "proc-macro2", + "pyo3-macros-backend", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "pyo3-macros-backend" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ca3a1557399783172dc5bf39cfca835157732532cba56b71d2292161e53b362" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "quick-xml" +version = "0.41.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e660451e55124f798a69a5af3f49ccfbefbd41910eefd25caf2393e1f3473ec1" +dependencies = [ + "memchr", + "serde", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "rustix" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +dependencies = [ + "bitflags 2.13.1", + "errno", + "libc", + "linux-raw-sys", + "windows-sys", +] + +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "serde_json" +version = "1.0.151" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "smallvec" +version = "1.15.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "target-lexicon" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca" + +[[package]] +name = "thiserror" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09a43598840e33d5b0331f38c5e30d13bb11c11210a4b58f0d9b18a5a5eefcd9" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "43cbfe0cf76104d42a574802844187e84a305e531ed54455f11fbde0f10541cd" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "tinystr" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d" +dependencies = [ + "displaydoc", + "zerovec", +] + +[[package]] +name = "tokio" +version = "1.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "202caea871b69668250d242070849eb495be178ed697a3e98aebce5bc81a0bed" +dependencies = [ + "bytes", + "pin-project-lite", + "tokio-macros", +] + +[[package]] +name = "tokio-macros" +version = "2.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "unicase" +version = "2.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dbc4bc3a9f746d862c45cb89d705aa10f187bb96c76001afab07a0d35ce60142" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "url" +version = "2.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" +dependencies = [ + "form_urlencoded", + "idna", + "percent-encoding", + "serde", +] + +[[package]] +name = "utf8_iter" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" + +[[package]] +name = "uuid" +version = "1.24.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf3923a6f5c4c6382e0b653c4117f48d631ea17f38ed86e2a828e6f7412f5239" +dependencies = [ + "getrandom", + "js-sys", + "serde_core", + "wasm-bindgen", +] + +[[package]] +name = "wasm-bindgen" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn 2.0.119", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "web-time" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "writeable" +version = "0.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" + +[[package]] +name = "xattr" +version = "1.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32e45ad4206f6d2479085147f02bc2ef834ac85886624a23575ae137c8aa8156" +dependencies = [ + "libc", + "rustix", +] + +[[package]] +name = "yoke" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "709fe23a0424b6a435d82152b1bd3fdfb0833487d5fa90d05d42762a9891fef5" +dependencies = [ + "stable_deref_trait", + "yoke-derive", + "zerofrom", +] + +[[package]] +name = "yoke-derive" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerofrom" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272" +dependencies = [ + "zerofrom-derive", +] + +[[package]] +name = "zerofrom-derive" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerotrie" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf" +dependencies = [ + "displaydoc", + "yoke", + "zerofrom", +] + +[[package]] +name = "zerovec" +version = "0.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239" +dependencies = [ + "yoke", + "zerofrom", + "zerovec-derive", +] + +[[package]] +name = "zerovec-derive" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/Cargo.toml b/bindings/python/dynamic-extensions/pyo3-comparison/Cargo.toml new file mode 100644 index 000000000000..be0abb7a0bb8 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/Cargo.toml @@ -0,0 +1,41 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[workspace] +members = [ + "base", + "bridge", + "fs-capsule", + "fs-direct", + "mime-capsule", + "mime-direct", +] +resolver = "2" + +[workspace.package] +edition = "2024" +license = "Apache-2.0" +publish = false +rust-version = "1.91" +version = "0.0.0" + +[workspace.dependencies] +opendal-core = { path = "../../../../core/core", features = ["blocking"] } +opendal-layer-mime-guess = { path = "../../../../core/layers/mime-guess" } +opendal-service-fs = { path = "../../../../core/services/fs" } +pin-project = "1" +pyo3 = { version = "0.29.0", features = ["extension-module"] } +pyo3-comparison-bridge = { path = "bridge" } +tokio = { version = "1", features = ["fs", "rt-multi-thread"] } diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/NOTES.md b/bindings/python/dynamic-extensions/pyo3-comparison/NOTES.md new file mode 100644 index 000000000000..8bbee9c2525e --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/NOTES.md @@ -0,0 +1,46 @@ + + +# Prototype Notes + +Question: Can separately built FS and MIME layer packages return and transform +the base package's PyO3 `Operator` with and without capsules? + +Observed result: + +- The capsule FS factory returns the exact base-package Python `Operator` type. +- FS write, read, and `stat` succeed after the operator crosses into the base + package, proving that the service-local runtime layer and executor enter the + FS package's Tokio context for these operations. +- The capsule MIME package accepts that operator and changes `hello.txt` from no + content type to `text/plain`. +- The direct FS package returns an object displayed as + `opendal_poc.Operator`, but its Python type object differs from the base + package's type object. +- The direct MIME package rejects both the FS package's operator and the base + package's operator with `Operator object is not an instance of Operator`. + +Conclusion: ordinary Rust dependency reuse does not share a PyO3 class between +independently linked extensions. A capsule can preserve the base Python class +interface, but this prototype's capsule payload is still a Rust `Operator` and +therefore requires an exact build. A production capsule should carry a +versioned function table or another explicitly validated ownership contract +instead of treating the Rust layout as stable. + +Known limit: the runtime-switch layer covers service methods, readers, writers, +and the executor used by this experiment. A production implementation must +also wrap returned listers, deleters, and copiers. diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/README.md b/bindings/python/dynamic-extensions/pyo3-comparison/README.md new file mode 100644 index 000000000000..02899e9c4361 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/README.md @@ -0,0 +1,58 @@ + + +# PyO3 Extension Comparison Prototype + +PROTOTYPE: delete or absorb this directory after the design question is +answered. + +This experiment asks whether independently built FS and MIME layer packages can +preserve the base package's PyO3 `Operator` interface. It compares two adapters: + +- The capsule adapter passes an exact-build `opendal_core::Operator` through + named `PyCapsule` values. +- The direct adapter reuses the Rust `PyOperator` definition through ordinary + Cargo dependencies and lets PyO3 perform extraction. + +Both adapters compile from the same sources in separate target directories. +The FS package injects a service-local Tokio runtime layer. The MIME package +applies `MimeGuessLayer`, whose effect is visible through +`Operator.content_type("hello.txt")`. + +Run both paths: + +```console +./run-linux.sh +``` + +Run the interactive state viewer: + +```console +./run-linux.sh --interactive +``` + +The capsule is an unsafe exact-build experiment, not a stable ABI. Its payload +contains a Rust `Operator`, so compiler, dependency graph, flags, and OpenDAL +source must match even though the capsule name is versioned. + +This comparison adapts the capsule delegation and runtime-switch patterns from +the earlier [split Python binding prototype][split-prototype]. It directly +tests the independently linked `#[pyclass]` constraint discussed in +[PyO3 issue #1444][pyo3-1444] against the current OpenDAL core APIs. + +[pyo3-1444]: https://github.com/PyO3/pyo3/issues/1444 +[split-prototype]: https://github.com/chitralverma/opendal-python-bindings/blob/main/pyo3-opendal/src/layers.rs diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/base/Cargo.toml b/bindings/python/dynamic-extensions/pyo3-comparison/base/Cargo.toml new file mode 100644 index 000000000000..15221e2f5eb5 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/base/Cargo.toml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +name = "pyo3-comparison-base" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] +name = "base" + +[dependencies] +pyo3.workspace = true +pyo3-comparison-bridge.workspace = true diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/base/src/lib.rs b/bindings/python/dynamic-extensions/pyo3-comparison/base/src/lib.rs new file mode 100644 index 000000000000..9e94095b14a2 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/base/src/lib.rs @@ -0,0 +1,22 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use pyo3::prelude::*; +use pyo3_comparison_bridge::PyOperator; + +#[pymodule(gil_used = false)] +fn opendal_poc(module: &Bound<'_, PyModule>) -> PyResult<()> { + module.add_class::() +} diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/bridge/Cargo.toml b/bindings/python/dynamic-extensions/pyo3-comparison/bridge/Cargo.toml new file mode 100644 index 000000000000..9c9c144649ff --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/bridge/Cargo.toml @@ -0,0 +1,28 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +name = "pyo3-comparison-bridge" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[dependencies] +opendal-core.workspace = true +pin-project.workspace = true +pyo3.workspace = true +tokio.workspace = true diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/bridge/src/lib.rs b/bindings/python/dynamic-extensions/pyo3-comparison/bridge/src/lib.rs new file mode 100644 index 000000000000..27a1fc340105 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/bridge/src/lib.rs @@ -0,0 +1,127 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +mod runtime_layer; + +use std::ffi::CStr; +use std::sync::LazyLock; + +use opendal_core::Operator; +use pyo3::intern; +use pyo3::prelude::*; +use pyo3::types::{PyBytes, PyCapsule, PyCapsuleMethods}; + +pub use runtime_layer::RuntimeLayer; + +const OPERATOR_CAPSULE_NAME: &CStr = c"opendal.poc.operator.v1"; + +pub fn runtime() -> &'static tokio::runtime::Runtime { + static RUNTIME: LazyLock = LazyLock::new(|| { + tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .expect("prototype Tokio runtime should build") + }); + &RUNTIME +} + +pub fn format_error(error: opendal_core::Error) -> PyErr { + pyo3::exceptions::PyRuntimeError::new_err(error.to_string()) +} + +pub fn to_operator_capsule(py: Python<'_>, op: Operator) -> PyResult> { + PyCapsule::new_with_value(py, op, OPERATOR_CAPSULE_NAME) +} + +pub fn from_operator_capsule(capsule: &Bound<'_, PyCapsule>) -> PyResult { + let pointer = capsule + .pointer_checked(Some(OPERATOR_CAPSULE_NAME))? + .cast::(); + Ok(unsafe { pointer.as_ref().clone() }) +} + +#[pyclass(module = "opendal_poc", name = "Operator")] +pub struct PyOperator { + op: Operator, +} + +impl PyOperator { + pub fn from_async(op: Operator) -> Self { + Self { op } + } + + pub fn async_operator(&self) -> &Operator { + &self.op + } +} + +#[pymethods] +impl PyOperator { + #[new] + fn new() -> PyResult { + let _guard = runtime().enter(); + let op = Operator::new(opendal_core::services::Memory::default()).map_err(format_error)?; + Ok(Self { op }) + } + + #[staticmethod] + fn _from_capsule(capsule: &Bound<'_, PyCapsule>) -> PyResult { + Ok(Self { + op: from_operator_capsule(capsule)?, + }) + } + + fn layer(&self, py: Python<'_>, layer: &Bound<'_, PyAny>) -> PyResult { + let capsule = to_operator_capsule(py, self.op.clone())?; + let result = layer.call_method1(intern!(py, "_layer_apply"), (capsule,))?; + let result = result.cast::()?; + Self::_from_capsule(result) + } + + fn scheme(&self) -> String { + self.op.info().scheme().to_string() + } + + fn content_type(&self, py: Python<'_>, path: String) -> PyResult> { + let op = self.op.clone(); + py.detach(move || { + runtime() + .block_on(op.stat(&path)) + .map(|metadata| metadata.content_type().map(str::to_string)) + .map_err(format_error) + }) + } + + fn write(&self, py: Python<'_>, path: String, content: Vec) -> PyResult<()> { + let op = self.op.clone(); + py.detach(move || { + runtime() + .block_on(op.write(&path, content)) + .map(|_| ()) + .map_err(format_error) + }) + } + + fn read<'py>(&self, py: Python<'py>, path: String) -> PyResult> { + let op = self.op.clone(); + let content = py.detach(move || { + runtime() + .block_on(op.read(&path)) + .map(|buffer| buffer.to_vec()) + .map_err(format_error) + })?; + Ok(PyBytes::new(py, &content)) + } +} diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/bridge/src/runtime_layer.rs b/bindings/python/dynamic-extensions/pyo3-comparison/bridge/src/runtime_layer.rs new file mode 100644 index 000000000000..323167b01466 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/bridge/src/runtime_layer.rs @@ -0,0 +1,229 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::future::Future; +use std::sync::Arc; +use std::task::{Context, Poll}; + +use opendal_core::raw::*; +use opendal_core::*; +use pin_project::pin_project; + +#[derive(Clone, Debug)] +pub struct RuntimeLayer { + handle: tokio::runtime::Handle, +} + +impl RuntimeLayer { + pub fn new(handle: tokio::runtime::Handle) -> Self { + Self { handle } + } +} + +impl Layer for RuntimeLayer { + fn apply_service(&self, inner: Servicer) -> Servicer { + Arc::new(RuntimeService { + inner, + handle: self.handle.clone(), + }) + } + + fn apply_context(&self, _service: Servicer, inner: OperationContext) -> OperationContext { + inner.with_executor(Executor::with(RuntimeExecutor { + handle: self.handle.clone(), + })) + } +} + +#[derive(Clone)] +struct RuntimeExecutor { + handle: tokio::runtime::Handle, +} + +impl Execute for RuntimeExecutor { + fn execute(&self, future: BoxedStaticFuture<()>) { + drop(self.handle.spawn(future)); + } +} + +#[derive(Debug)] +struct RuntimeService { + inner: Servicer, + handle: tokio::runtime::Handle, +} + +impl Service for RuntimeService { + type Reader = RuntimeReader; + type Writer = RuntimeWriter; + type Lister = oio::Lister; + type Deleter = oio::Deleter; + type Copier = oio::Copier; + + fn info(&self) -> ServiceInfo { + self.inner.info() + } + + fn capability(&self) -> Capability { + self.inner.capability() + } + + async fn create_dir( + &self, + ctx: &OperationContext, + path: &str, + args: OpCreateDir, + ) -> Result { + RuntimeFuture::new(self.inner.create_dir(ctx, path, args), self.handle.clone()).await + } + + async fn stat(&self, ctx: &OperationContext, path: &str, args: OpStat) -> Result { + RuntimeFuture::new(self.inner.stat(ctx, path, args), self.handle.clone()).await + } + + fn read(&self, ctx: &OperationContext, path: &str, args: OpRead) -> Result { + let _guard = self.handle.enter(); + self.inner.read(ctx, path, args).map(|inner| RuntimeReader { + inner, + handle: self.handle.clone(), + }) + } + + fn write(&self, ctx: &OperationContext, path: &str, args: OpWrite) -> Result { + let _guard = self.handle.enter(); + self.inner + .write(ctx, path, args) + .map(|inner| RuntimeWriter { + inner, + handle: self.handle.clone(), + }) + } + + fn delete(&self, ctx: &OperationContext) -> Result { + let _guard = self.handle.enter(); + self.inner.delete(ctx) + } + + fn list(&self, ctx: &OperationContext, path: &str, args: OpList) -> Result { + let _guard = self.handle.enter(); + self.inner.list(ctx, path, args) + } + + fn copy( + &self, + ctx: &OperationContext, + from: &str, + to: &str, + args: OpCopy, + opts: OpCopier, + ) -> Result { + let _guard = self.handle.enter(); + self.inner.copy(ctx, from, to, args, opts) + } + + async fn rename( + &self, + ctx: &OperationContext, + from: &str, + to: &str, + args: OpRename, + ) -> Result { + RuntimeFuture::new(self.inner.rename(ctx, from, to, args), self.handle.clone()).await + } + + async fn presign( + &self, + ctx: &OperationContext, + path: &str, + args: OpPresign, + ) -> Result { + RuntimeFuture::new(self.inner.presign(ctx, path, args), self.handle.clone()).await + } +} + +struct RuntimeReader { + inner: oio::Reader, + handle: tokio::runtime::Handle, +} + +impl oio::Read for RuntimeReader { + async fn open(&self, range: BytesRange) -> Result<(RpRead, Box)> { + let (response, stream) = + RuntimeFuture::new(self.inner.open(range), self.handle.clone()).await?; + Ok(( + response, + Box::new(RuntimeReadStream { + inner: stream, + handle: self.handle.clone(), + }), + )) + } + + async fn read(&self, range: BytesRange) -> Result<(RpRead, Buffer)> { + RuntimeFuture::new(self.inner.read(range), self.handle.clone()).await + } +} + +struct RuntimeReadStream { + inner: Box, + handle: tokio::runtime::Handle, +} + +impl oio::ReadStream for RuntimeReadStream { + async fn read(&mut self) -> Result { + RuntimeFuture::new(self.inner.read(), self.handle.clone()).await + } +} + +struct RuntimeWriter { + inner: oio::Writer, + handle: tokio::runtime::Handle, +} + +impl oio::Write for RuntimeWriter { + async fn write(&mut self, buffer: Buffer) -> Result<()> { + RuntimeFuture::new(self.inner.write(buffer), self.handle.clone()).await + } + + async fn close(&mut self) -> Result { + RuntimeFuture::new(self.inner.close(), self.handle.clone()).await + } + + async fn abort(&mut self) -> Result<()> { + RuntimeFuture::new(self.inner.abort(), self.handle.clone()).await + } +} + +#[pin_project] +struct RuntimeFuture { + #[pin] + inner: F, + handle: tokio::runtime::Handle, +} + +impl RuntimeFuture { + fn new(inner: F, handle: tokio::runtime::Handle) -> Self { + Self { inner, handle } + } +} + +impl Future for RuntimeFuture { + type Output = F::Output; + + fn poll(self: std::pin::Pin<&mut Self>, context: &mut Context<'_>) -> Poll { + let this = self.project(); + let _guard = this.handle.enter(); + this.inner.poll(context) + } +} diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/compare.py b/bindings/python/dynamic-extensions/pyo3-comparison/compare.py new file mode 100644 index 000000000000..747ceee02feb --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/compare.py @@ -0,0 +1,175 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import argparse +import json +import tempfile +from dataclasses import asdict, dataclass +from pathlib import Path +from typing import Callable + +import opendal_fs_capsule +import opendal_fs_direct +import opendal_mime_capsule +import opendal_mime_direct +import opendal_poc + + +@dataclass +class VariantState: + attempted: bool = False + operator_type: str | None = None + operator_type_identity: int | None = None + is_base_operator: bool | None = None + fs_round_trip: bool | None = None + fs_stat_succeeded: bool | None = None + fs_stat_before_layer: str | None = None + fs_stat_after_layer: str | None = None + layer_applied: bool | None = None + layer_error: str | None = None + base_operator_layer_error: str | None = None + error: str | None = None + + +@dataclass +class ComparisonState: + base_operator_type_identity: int + base_memory_scheme: str + capsule: VariantState + direct: VariantState + + +def initial_state() -> ComparisonState: + return ComparisonState( + base_operator_type_identity=id(opendal_poc.Operator), + base_memory_scheme=opendal_poc.Operator().scheme(), + capsule=VariantState(), + direct=VariantState(), + ) + + +def run_capsule(root: Path) -> VariantState: + state = VariantState(attempted=True) + try: + variant_root = root / "capsule" + variant_root.mkdir(exist_ok=True) + operator = opendal_fs_capsule.create(str(variant_root)) + state.operator_type = repr(type(operator)) + state.operator_type_identity = id(type(operator)) + state.is_base_operator = isinstance(operator, opendal_poc.Operator) + operator.write("hello.txt", b"hello from capsule") + state.fs_round_trip = operator.read("hello.txt") == b"hello from capsule" + state.fs_stat_before_layer = operator.content_type("hello.txt") + state.fs_stat_succeeded = True + layered = operator.layer(opendal_mime_capsule.MimeGuessLayer()) + state.fs_stat_after_layer = layered.content_type("hello.txt") + state.layer_applied = state.fs_stat_after_layer == "text/plain" + except Exception as error: + state.error = f"{type(error).__name__}: {error}" + return state + + +def run_direct(root: Path) -> VariantState: + state = VariantState(attempted=True) + try: + variant_root = root / "direct" + variant_root.mkdir(exist_ok=True) + operator = opendal_fs_direct.create(str(variant_root)) + state.operator_type = repr(type(operator)) + state.operator_type_identity = id(type(operator)) + state.is_base_operator = isinstance(operator, opendal_poc.Operator) + operator.write("hello.txt", b"hello from direct") + state.fs_round_trip = operator.read("hello.txt") == b"hello from direct" + state.fs_stat_before_layer = operator.content_type("hello.txt") + state.fs_stat_succeeded = True + try: + layered = opendal_mime_direct.apply(operator) + except Exception as error: + state.layer_error = f"{type(error).__name__}: {error}" + state.layer_applied = False + else: + state.fs_stat_after_layer = layered.content_type("hello.txt") + state.layer_applied = state.fs_stat_after_layer == "text/plain" + + try: + opendal_mime_direct.apply(opendal_poc.Operator()) + except Exception as error: + state.base_operator_layer_error = f"{type(error).__name__}: {error}" + except Exception as error: + state.error = f"{type(error).__name__}: {error}" + return state + + +def render(state: ComparisonState, *, clear: bool) -> None: + if clear: + print("\033[2J\033[H", end="") + print("\033[1mPyO3 extension comparison\033[0m") + print(json.dumps(asdict(state), indent=2, sort_keys=True)) + print() + print( + "\033[1m[c]\033[0m capsule " + "\033[1m[d]\033[0m direct " + "\033[1m[a]\033[0m both " + "\033[1m[r]\033[0m reset " + "\033[1m[q]\033[0m quit" + ) + + +def apply_action( + state: ComparisonState, + action: str, + root: Path, +) -> ComparisonState: + actions: dict[str, Callable[[Path], VariantState]] = { + "c": run_capsule, + "d": run_direct, + } + if action == "r": + return initial_state() + if action == "a": + state.capsule = run_capsule(root) + state.direct = run_direct(root) + elif action in actions: + setattr(state, "capsule" if action == "c" else "direct", actions[action](root)) + return state + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--all", action="store_true") + parser.add_argument("--interactive", action="store_true") + args = parser.parse_args() + + with tempfile.TemporaryDirectory(prefix="opendal-pyo3-comparison-") as root: + root_path = Path(root) + state = initial_state() + + if args.all or not args.interactive: + state = apply_action(state, "a", root_path) + render(state, clear=False) + return + + while True: + render(state, clear=True) + action = input("> ").strip().lower()[:1] + if action == "q": + return + state = apply_action(state, action, root_path) + + +if __name__ == "__main__": + main() diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/fs-capsule/Cargo.toml b/bindings/python/dynamic-extensions/pyo3-comparison/fs-capsule/Cargo.toml new file mode 100644 index 000000000000..6bc6447fa3b4 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/fs-capsule/Cargo.toml @@ -0,0 +1,32 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +name = "pyo3-comparison-fs-capsule" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] +name = "fs_capsule" + +[dependencies] +opendal-core.workspace = true +opendal-service-fs.workspace = true +pyo3.workspace = true +pyo3-comparison-bridge.workspace = true diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/fs-capsule/src/lib.rs b/bindings/python/dynamic-extensions/pyo3-comparison/fs-capsule/src/lib.rs new file mode 100644 index 000000000000..b43cbd242165 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/fs-capsule/src/lib.rs @@ -0,0 +1,40 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use opendal_core::Operator; +use opendal_service_fs::Fs; +use pyo3::prelude::*; +use pyo3_comparison_bridge::{RuntimeLayer, format_error, runtime, to_operator_capsule}; + +fn build_fs(root: &str) -> PyResult { + let local_runtime = runtime(); + let _guard = local_runtime.enter(); + Operator::new(Fs::default().root(root)) + .map(|operator| operator.layer(RuntimeLayer::new(local_runtime.handle().clone()))) + .map_err(format_error) +} + +#[pyfunction] +fn create<'py>(py: Python<'py>, root: &str) -> PyResult> { + let capsule = to_operator_capsule(py, build_fs(root)?)?; + py.import("opendal_poc")? + .getattr("Operator")? + .call_method1("_from_capsule", (capsule,)) +} + +#[pymodule(gil_used = false)] +fn opendal_fs_capsule(module: &Bound<'_, PyModule>) -> PyResult<()> { + module.add_function(wrap_pyfunction!(create, module)?) +} diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/fs-direct/Cargo.toml b/bindings/python/dynamic-extensions/pyo3-comparison/fs-direct/Cargo.toml new file mode 100644 index 000000000000..3b3129cc39a2 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/fs-direct/Cargo.toml @@ -0,0 +1,32 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +name = "pyo3-comparison-fs-direct" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] +name = "fs_direct" + +[dependencies] +opendal-core.workspace = true +opendal-service-fs.workspace = true +pyo3.workspace = true +pyo3-comparison-bridge.workspace = true diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/fs-direct/src/lib.rs b/bindings/python/dynamic-extensions/pyo3-comparison/fs-direct/src/lib.rs new file mode 100644 index 000000000000..4bd319f28cce --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/fs-direct/src/lib.rs @@ -0,0 +1,38 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use opendal_core::Operator; +use opendal_service_fs::Fs; +use pyo3::prelude::*; +use pyo3_comparison_bridge::{PyOperator, RuntimeLayer, format_error, runtime}; + +fn build_fs(root: &str) -> PyResult { + let local_runtime = runtime(); + let _guard = local_runtime.enter(); + Operator::new(Fs::default().root(root)) + .map(|operator| operator.layer(RuntimeLayer::new(local_runtime.handle().clone()))) + .map_err(format_error) +} + +#[pyfunction] +fn create(root: &str) -> PyResult { + build_fs(root).map(PyOperator::from_async) +} + +#[pymodule(gil_used = false)] +fn opendal_fs_direct(module: &Bound<'_, PyModule>) -> PyResult<()> { + module.add_class::()?; + module.add_function(wrap_pyfunction!(create, module)?) +} diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/mime-capsule/Cargo.toml b/bindings/python/dynamic-extensions/pyo3-comparison/mime-capsule/Cargo.toml new file mode 100644 index 000000000000..0c60d6c03fa5 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/mime-capsule/Cargo.toml @@ -0,0 +1,31 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +name = "pyo3-comparison-mime-capsule" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] +name = "mime_capsule" + +[dependencies] +opendal-layer-mime-guess.workspace = true +pyo3.workspace = true +pyo3-comparison-bridge.workspace = true diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/mime-capsule/src/lib.rs b/bindings/python/dynamic-extensions/pyo3-comparison/mime-capsule/src/lib.rs new file mode 100644 index 000000000000..fde68a665a31 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/mime-capsule/src/lib.rs @@ -0,0 +1,44 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use opendal_layer_mime_guess::MimeGuessLayer as CoreMimeGuessLayer; +use pyo3::prelude::*; +use pyo3::types::PyCapsule; +use pyo3_comparison_bridge::{from_operator_capsule, to_operator_capsule}; + +#[pyclass(name = "MimeGuessLayer")] +struct MimeGuessLayer; + +#[pymethods] +impl MimeGuessLayer { + #[new] + fn new() -> Self { + Self + } + + fn _layer_apply<'py>( + &self, + py: Python<'py>, + capsule: &Bound<'py, PyCapsule>, + ) -> PyResult> { + let operator = from_operator_capsule(capsule)?; + to_operator_capsule(py, operator.layer(CoreMimeGuessLayer::default())) + } +} + +#[pymodule(gil_used = false)] +fn opendal_mime_capsule(module: &Bound<'_, PyModule>) -> PyResult<()> { + module.add_class::() +} diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/mime-direct/Cargo.toml b/bindings/python/dynamic-extensions/pyo3-comparison/mime-direct/Cargo.toml new file mode 100644 index 000000000000..9d562fd5f542 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/mime-direct/Cargo.toml @@ -0,0 +1,31 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +name = "pyo3-comparison-mime-direct" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] +name = "mime_direct" + +[dependencies] +opendal-layer-mime-guess.workspace = true +pyo3.workspace = true +pyo3-comparison-bridge.workspace = true diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/mime-direct/src/lib.rs b/bindings/python/dynamic-extensions/pyo3-comparison/mime-direct/src/lib.rs new file mode 100644 index 000000000000..b1f37eb19c79 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/mime-direct/src/lib.rs @@ -0,0 +1,34 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use opendal_layer_mime_guess::MimeGuessLayer; +use pyo3::prelude::*; +use pyo3_comparison_bridge::PyOperator; + +#[pyfunction] +fn apply(operator: PyRef<'_, PyOperator>) -> PyOperator { + PyOperator::from_async( + operator + .async_operator() + .clone() + .layer(MimeGuessLayer::default()), + ) +} + +#[pymodule(gil_used = false)] +fn opendal_mime_direct(module: &Bound<'_, PyModule>) -> PyResult<()> { + module.add_class::()?; + module.add_function(wrap_pyfunction!(apply, module)?) +} diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/run-linux.sh b/bindings/python/dynamic-extensions/pyo3-comparison/run-linux.sh new file mode 100755 index 000000000000..0d867c2613b1 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/run-linux.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env sh +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eu + +prototype_dir=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd) +target_root="$prototype_dir/target" +stage="$target_root/python-stage" +audit="$prototype_dir/../audit-elf-exports.sh" + +build_package() { + package=$1 + target=$2 + CARGO_TARGET_DIR="$target_root/$target" cargo build --release --locked --offline \ + --manifest-path "$prototype_dir/Cargo.toml" --package "$package" +} + +build_package pyo3-comparison-base base +build_package pyo3-comparison-fs-capsule fs-capsule +build_package pyo3-comparison-fs-direct fs-direct +build_package pyo3-comparison-mime-capsule mime-capsule +build_package pyo3-comparison-mime-direct mime-direct + +mkdir -p "$stage" +cp "$target_root/base/release/libbase.so" "$stage/opendal_poc.so" +cp "$target_root/fs-capsule/release/libfs_capsule.so" "$stage/opendal_fs_capsule.so" +cp "$target_root/fs-direct/release/libfs_direct.so" "$stage/opendal_fs_direct.so" +cp "$target_root/mime-capsule/release/libmime_capsule.so" "$stage/opendal_mime_capsule.so" +cp "$target_root/mime-direct/release/libmime_direct.so" "$stage/opendal_mime_direct.so" + +"$audit" "$stage/opendal_poc.so" PyInit_opendal_poc +"$audit" "$stage/opendal_fs_capsule.so" PyInit_opendal_fs_capsule +"$audit" "$stage/opendal_fs_direct.so" PyInit_opendal_fs_direct +"$audit" "$stage/opendal_mime_capsule.so" PyInit_opendal_mime_capsule +"$audit" "$stage/opendal_mime_direct.so" PyInit_opendal_mime_direct + +if [ "$#" -eq 0 ]; then + set -- --all +fi + +PYTHONPATH="$stage" python3 "$prototype_dir/compare.py" "$@" diff --git a/bindings/python/dynamic-extensions/python/example.py b/bindings/python/dynamic-extensions/python/example.py new file mode 100644 index 000000000000..e946f993ca73 --- /dev/null +++ b/bindings/python/dynamic-extensions/python/example.py @@ -0,0 +1,29 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import tempfile + +import opendal.services.fs +import opendal.services.s3 +from opendal import Operator + + +with Operator("s3", bucket="my-bucket", region="us-east-1") as s3: + print(s3.info) + +with tempfile.TemporaryDirectory() as root: + with Operator("fs", root=root) as fs: + fs.write("hello.txt", b"Hello, OpenDAL!") + print(fs.read("hello.txt")) diff --git a/bindings/python/dynamic-extensions/python/main/opendal/__init__.py b/bindings/python/dynamic-extensions/python/main/opendal/__init__.py new file mode 100644 index 000000000000..2aa5d0941cc8 --- /dev/null +++ b/bindings/python/dynamic-extensions/python/main/opendal/__init__.py @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from pkgutil import extend_path + +__path__ = extend_path(__path__, __name__) + +from ._runtime import Operator, RuntimeProtocolError + +__all__ = ["Operator", "RuntimeProtocolError"] diff --git a/bindings/python/dynamic-extensions/python/main/opendal/_runtime.py b/bindings/python/dynamic-extensions/python/main/opendal/_runtime.py new file mode 100644 index 000000000000..21b46d50e1ea --- /dev/null +++ b/bindings/python/dynamic-extensions/python/main/opendal/_runtime.py @@ -0,0 +1,304 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import ctypes +import json +import threading +from collections.abc import Mapping +from pathlib import Path +from typing import Any + +RUNTIME_PROTOCOL = 1 +STATUS_OK = 0 +STATUS_BUFFER_TOO_SMALL = 6 + + +class RuntimeProtocolError(RuntimeError): + pass + + +class ByteSlice(ctypes.Structure): + _fields_ = [("data", ctypes.c_void_p), ("len", ctypes.c_size_t)] + + +class KeyValue(ctypes.Structure): + _fields_ = [("key", ByteSlice), ("value", ByteSlice)] + + +class OutputBuffer(ctypes.Structure): + _fields_ = [ + ("data", ctypes.c_void_p), + ("capacity", ctypes.c_size_t), + ("len", ctypes.c_size_t), + ] + + +class ServiceRegistrationV1(ctypes.Structure): + _fields_ = [ + ("struct_size", ctypes.c_size_t), + ("required_runtime_protocol", ctypes.c_uint32), + ("package_id", ByteSlice), + ("component_id", ByteSlice), + ("entry_symbol", ByteSlice), + ("library_path", ByteSlice), + ] + + +class RuntimeProtocolInfoV1(ctypes.Structure): + _fields_ = [ + ("struct_size", ctypes.c_size_t), + ("minimum_runtime_protocol", ctypes.c_uint32), + ("runtime_protocol", ctypes.c_uint32), + ] + + +RegisterServiceFn = ctypes.CFUNCTYPE( + ctypes.c_int32, + ctypes.POINTER(ServiceRegistrationV1), + ctypes.POINTER(OutputBuffer), +) +CreateOperatorFn = ctypes.CFUNCTYPE( + ctypes.c_int32, + ByteSlice, + ctypes.POINTER(KeyValue), + ctypes.c_size_t, + ctypes.POINTER(ctypes.c_void_p), + ctypes.POINTER(OutputBuffer), +) +OperatorInfoFn = ctypes.CFUNCTYPE( + ctypes.c_int32, + ctypes.c_void_p, + ctypes.POINTER(OutputBuffer), + ctypes.POINTER(OutputBuffer), +) +OperatorWriteFn = ctypes.CFUNCTYPE( + ctypes.c_int32, + ctypes.c_void_p, + ByteSlice, + ByteSlice, + ctypes.POINTER(OutputBuffer), +) +OperatorReadFn = ctypes.CFUNCTYPE( + ctypes.c_int32, + ctypes.c_void_p, + ByteSlice, + ctypes.POINTER(OutputBuffer), + ctypes.POINTER(OutputBuffer), +) +OperatorDestroyFn = ctypes.CFUNCTYPE(None, ctypes.c_void_p) + + +class RuntimeApiV1(ctypes.Structure): + _fields_ = [ + ("struct_size", ctypes.c_size_t), + ("register_service", RegisterServiceFn), + ("create_operator", CreateOperatorFn), + ("operator_info", OperatorInfoFn), + ("operator_write", OperatorWriteFn), + ("operator_read", OperatorReadFn), + ("operator_destroy", OperatorDestroyFn), + ] + + +def _bytes(value: str | bytes) -> tuple[ByteSlice, ctypes.Array[ctypes.c_char]]: + raw = value.encode() if isinstance(value, str) else value + storage = ctypes.create_string_buffer(raw) + return ByteSlice(ctypes.cast(storage, ctypes.c_void_p), len(raw)), storage + + +def _output(capacity: int = 4096) -> tuple[OutputBuffer, ctypes.Array[ctypes.c_char]]: + storage = ctypes.create_string_buffer(capacity) + return OutputBuffer(ctypes.cast(storage, ctypes.c_void_p), capacity, 0), storage + + +def _message(output: OutputBuffer, storage: ctypes.Array[ctypes.c_char]) -> str: + length = min(output.len, len(storage)) + return bytes(storage[:length]).decode(errors="replace") + + +_native_path = Path(__file__).parent / "_native" / "libopendal_runtime_poc.so" +_native = ctypes.CDLL(str(_native_path)) +_native.opendal_runtime_get_api_v1.argtypes = [ + ctypes.c_uint32, + ctypes.POINTER(RuntimeProtocolInfoV1), + ctypes.POINTER(ctypes.POINTER(RuntimeApiV1)), +] +_native.opendal_runtime_get_api_v1.restype = ctypes.c_int32 + +_protocol = RuntimeProtocolInfoV1( + struct_size=ctypes.sizeof(RuntimeProtocolInfoV1), + minimum_runtime_protocol=0, + runtime_protocol=0, +) +_api_pointer = ctypes.POINTER(RuntimeApiV1)() +_status = _native.opendal_runtime_get_api_v1( + RUNTIME_PROTOCOL, ctypes.byref(_protocol), ctypes.byref(_api_pointer) +) +if _status != STATUS_OK or not _api_pointer: + raise RuntimeProtocolError( + "runtime protocol negotiation failed: " + f"required={RUNTIME_PROTOCOL}, " + f"supported={_protocol.minimum_runtime_protocol}..{_protocol.runtime_protocol}" + ) +_api = _api_pointer.contents + + +def _register_service(manifest: Mapping[str, Any], library_path: Path) -> None: + component = manifest.get("component") + if not isinstance(component, Mapping) or component.get("kind") != "service": + raise RuntimeProtocolError("manifest must describe one service") + package_id = manifest.get("package_id") + component_id = component.get("id") + entry_symbol = manifest.get("native_entry_symbol") + required_protocol = manifest.get("required_runtime_protocol") + if not all(isinstance(value, str) for value in (package_id, component_id, entry_symbol)): + raise RuntimeProtocolError( + "manifest package ID, component ID, and entry symbol must be strings" + ) + if not isinstance(required_protocol, int): + raise RuntimeProtocolError("manifest runtime protocol must be an integer") + if not library_path.is_file(): + raise RuntimeProtocolError(f"native service artifact does not exist: {library_path}") + + package, package_storage = _bytes(package_id) + service, service_storage = _bytes(component_id) + entry, entry_storage = _bytes(entry_symbol) + path, path_storage = _bytes(str(library_path.resolve())) + registration = ServiceRegistrationV1( + struct_size=ctypes.sizeof(ServiceRegistrationV1), + required_runtime_protocol=required_protocol, + package_id=package, + component_id=service, + entry_symbol=entry, + library_path=path, + ) + error, error_storage = _output() + status = _api.register_service(ctypes.byref(registration), ctypes.byref(error)) + _ = (package_storage, service_storage, entry_storage, path_storage) + if status != STATUS_OK: + raise RuntimeProtocolError(_message(error, error_storage)) + + +def _stringify(value: object) -> str: + if isinstance(value, bool): + return "true" if value else "false" + if isinstance(value, (str, int, float, Path)): + return str(value) + raise TypeError(f"unsupported configuration value: {type(value).__name__}") + + +def _options(values: Mapping[str, object]) -> tuple[Any, list[Any]]: + pairs: list[KeyValue] = [] + keepalive: list[Any] = [] + for key, value in values.items(): + encoded_key, key_storage = _bytes(key) + encoded_value, value_storage = _bytes(_stringify(value)) + pairs.append(KeyValue(encoded_key, encoded_value)) + keepalive.extend((key_storage, value_storage)) + array_type = KeyValue * len(pairs) + return array_type(*pairs), keepalive + + +class Operator: + def __init__(self, scheme: str, **config: object) -> None: + self._lock = threading.RLock() + self._handle = ctypes.c_void_p() + scheme = scheme.strip().lower().replace("_", "-") + encoded_scheme, scheme_storage = _bytes(scheme) + options, option_storage = _options(config) + handle = ctypes.c_void_p() + error, error_storage = _output() + status = _api.create_operator( + encoded_scheme, + options, + len(options), + ctypes.byref(handle), + ctypes.byref(error), + ) + _ = (scheme_storage, option_storage) + if status != STATUS_OK: + raise RuntimeProtocolError(_message(error, error_storage)) + self._handle = handle + + def close(self) -> None: + with self._lock: + handle = self._handle + if handle: + _api.operator_destroy(handle) + self._handle = ctypes.c_void_p() + + def __enter__(self) -> Operator: + return self + + def __exit__(self, *_: object) -> None: + self.close() + + def __del__(self) -> None: + self.close() + + def _require_handle(self) -> ctypes.c_void_p: + if not self._handle: + raise RuntimeProtocolError("operator is closed") + return self._handle + + @property + def info(self) -> dict[str, str]: + with self._lock: + output, output_storage = _output() + error, error_storage = _output() + status = _api.operator_info( + self._require_handle(), ctypes.byref(output), ctypes.byref(error) + ) + if status != STATUS_OK: + raise RuntimeProtocolError(_message(error, error_storage)) + return json.loads(_message(output, output_storage)) + + def write(self, path: str, data: bytes) -> None: + with self._lock: + encoded_path, path_storage = _bytes(path) + encoded_data, data_storage = _bytes(data) + error, error_storage = _output() + status = _api.operator_write( + self._require_handle(), encoded_path, encoded_data, ctypes.byref(error) + ) + _ = (path_storage, data_storage) + if status != STATUS_OK: + raise RuntimeProtocolError(_message(error, error_storage)) + + def read(self, path: str) -> bytes: + with self._lock: + encoded_path, path_storage = _bytes(path) + output, output_storage = _output() + error, error_storage = _output() + status = _api.operator_read( + self._require_handle(), + encoded_path, + ctypes.byref(output), + ctypes.byref(error), + ) + if status == STATUS_BUFFER_TOO_SMALL: + output, output_storage = _output(output.len) + status = _api.operator_read( + self._require_handle(), + encoded_path, + ctypes.byref(output), + ctypes.byref(error), + ) + _ = path_storage + if status != STATUS_OK: + raise RuntimeProtocolError(_message(error, error_storage)) + return bytes(output_storage[: output.len]) diff --git a/bindings/python/dynamic-extensions/python/main/opendal/services/__init__.py b/bindings/python/dynamic-extensions/python/main/opendal/services/__init__.py new file mode 100644 index 000000000000..42d619211cb0 --- /dev/null +++ b/bindings/python/dynamic-extensions/python/main/opendal/services/__init__.py @@ -0,0 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from pkgutil import extend_path + +__path__ = extend_path(__path__, __name__) diff --git a/bindings/python/dynamic-extensions/python/service-fs/opendal/services/fs/__init__.py b/bindings/python/dynamic-extensions/python/service-fs/opendal/services/fs/__init__.py new file mode 100644 index 000000000000..a46e1fb6254b --- /dev/null +++ b/bindings/python/dynamic-extensions/python/service-fs/opendal/services/fs/__init__.py @@ -0,0 +1,27 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from pathlib import Path + +from opendal._runtime import _register_service + +MANIFEST = { + "required_runtime_protocol": 1, + "package_id": "opendal-service-fs-poc", + "component": {"kind": "service", "id": "fs", "aliases": ["file"]}, + "native_entry_symbol": "opendal_service_fs_bootstrap_v1", +} + +_register_service(MANIFEST, Path(__file__).parent / "_native" / "libfs_extension.so") diff --git a/bindings/python/dynamic-extensions/python/service-s3/opendal/services/s3/__init__.py b/bindings/python/dynamic-extensions/python/service-s3/opendal/services/s3/__init__.py new file mode 100644 index 000000000000..087a84b77c01 --- /dev/null +++ b/bindings/python/dynamic-extensions/python/service-s3/opendal/services/s3/__init__.py @@ -0,0 +1,27 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from pathlib import Path + +from opendal._runtime import _register_service + +MANIFEST = { + "required_runtime_protocol": 1, + "package_id": "opendal-service-s3-poc", + "component": {"kind": "service", "id": "s3", "aliases": []}, + "native_entry_symbol": "opendal_service_s3_bootstrap_v1", +} + +_register_service(MANIFEST, Path(__file__).parent / "_native" / "libs3_extension.so") diff --git a/bindings/python/dynamic-extensions/python/test_poc.py b/bindings/python/dynamic-extensions/python/test_poc.py new file mode 100644 index 000000000000..0162f4688e0c --- /dev/null +++ b/bindings/python/dynamic-extensions/python/test_poc.py @@ -0,0 +1,71 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import tempfile +from pathlib import Path + +import opendal + + +def main() -> None: + try: + opendal.Operator("s3", bucket="prototype-bucket") + except opendal.RuntimeProtocolError as err: + assert "import its package first" in str(err) + else: + raise AssertionError("S3 must not be registered before its package import") + + import opendal.services.s3 as s3 + + assert s3.MANIFEST["component"]["id"] == "s3" + s3_library = Path(s3.__file__).parent / "_native" / "libs3_extension.so" + held_library = s3_library.with_suffix(".so.held") + s3_library.rename(held_library) + try: + try: + opendal.Operator("s3", bucket="prototype-bucket") + except opendal.RuntimeProtocolError: + pass + else: + raise AssertionError("import must not load the S3 native artifact") + finally: + held_library.rename(s3_library) + + with opendal.Operator( + "s3", bucket="prototype-bucket", region="us-east-1" + ) as operator: + assert operator.info == { + "scheme": "s3", + "name": "prototype-bucket", + "root": "/", + } + + import opendal.services.fs as fs + + assert fs.MANIFEST["component"]["id"] == "fs" + with tempfile.TemporaryDirectory() as root: + with opendal.Operator("fs", root=root) as operator: + operator.write("hello.txt", b"hello from the shared runtime") + assert operator.read("hello.txt") == b"hello from the shared runtime" + assert Path(root, "hello.txt").read_bytes() == b"hello from the shared runtime" + assert operator.info["scheme"] == "fs" + + print("imported s3 and fs service packages") + print("deferred native service loading until operator construction") + print("constructed S3 operator and completed FS write/read through runtime handles") + + +if __name__ == "__main__": + main() diff --git a/bindings/python/dynamic-extensions/run-python-linux.sh b/bindings/python/dynamic-extensions/run-python-linux.sh new file mode 100755 index 000000000000..487df48a9b9b --- /dev/null +++ b/bindings/python/dynamic-extensions/run-python-linux.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env sh +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eu + +extension_dir=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd) +runtime_target="$extension_dir/target/python-runtime" +s3_target="$extension_dir/target/python-s3" +fs_target="$extension_dir/target/python-fs" +stage="$extension_dir/target/python-stage" + +CARGO_TARGET_DIR="$runtime_target" cargo build --release --locked \ + --manifest-path "$extension_dir/Cargo.toml" --package opendal-runtime-poc +CARGO_TARGET_DIR="$s3_target" cargo build --release --locked \ + --manifest-path "$extension_dir/Cargo.toml" --package s3-extension +CARGO_TARGET_DIR="$fs_target" cargo build --release --locked \ + --manifest-path "$extension_dir/Cargo.toml" --package fs-extension + +runtime="$runtime_target/release/libopendal_runtime_poc.so" +s3="$s3_target/release/libs3_extension.so" +fs="$fs_target/release/libfs_extension.so" + +"$extension_dir/audit-elf-exports.sh" "$runtime" opendal_runtime_get_api_v1 +"$extension_dir/audit-elf-exports.sh" "$s3" opendal_service_s3_bootstrap_v1 +"$extension_dir/audit-elf-exports.sh" "$fs" opendal_service_fs_bootstrap_v1 + +mkdir -p \ + "$stage/main/opendal/_native" \ + "$stage/s3/opendal/services/s3/_native" \ + "$stage/fs/opendal/services/fs/_native" +cp -R "$extension_dir/python/main/." "$stage/main/" +cp -R "$extension_dir/python/service-s3/." "$stage/s3/" +cp -R "$extension_dir/python/service-fs/." "$stage/fs/" +cp "$runtime" "$stage/main/opendal/_native/" +cp "$s3" "$stage/s3/opendal/services/s3/_native/" +cp "$fs" "$stage/fs/opendal/services/fs/_native/" + +PYTHONPATH="$stage/main:$stage/s3:$stage/fs" \ + python3 "$extension_dir/python/test_poc.py" +PYTHONPATH="$stage/main:$stage/s3:$stage/fs" \ + python3 "$extension_dir/python/example.py" diff --git a/bindings/python/dynamic-extensions/runtime/Cargo.toml b/bindings/python/dynamic-extensions/runtime/Cargo.toml new file mode 100644 index 000000000000..d6390c78c55f --- /dev/null +++ b/bindings/python/dynamic-extensions/runtime/Cargo.toml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +edition.workspace = true +license.workspace = true +name = "opendal-runtime-poc" +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] + +[dependencies] +libloading.workspace = true +opendal-core.workspace = true +opendal-dynamic-extension-sdk.workspace = true diff --git a/bindings/python/dynamic-extensions/runtime/build.rs b/bindings/python/dynamic-extensions/runtime/build.rs new file mode 100644 index 000000000000..501b9f44298a --- /dev/null +++ b/bindings/python/dynamic-extensions/runtime/build.rs @@ -0,0 +1,29 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::env; +use std::path::PathBuf; + +fn main() { + if env::var("CARGO_CFG_TARGET_OS").as_deref() == Ok("linux") { + let manifest_dir = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); + let export_map = manifest_dir.join("exports.map"); + println!("cargo:rerun-if-changed={}", export_map.display()); + println!( + "cargo:rustc-cdylib-link-arg=-Wl,--version-script={}", + export_map.display() + ); + } +} diff --git a/bindings/python/dynamic-extensions/runtime/exports.map b/bindings/python/dynamic-extensions/runtime/exports.map new file mode 100644 index 000000000000..e448b1837b28 --- /dev/null +++ b/bindings/python/dynamic-extensions/runtime/exports.map @@ -0,0 +1,21 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +OPENDAL_RUNTIME_POC_1 { + global: + opendal_runtime_get_api_v1; + local: + *; +}; diff --git a/bindings/python/dynamic-extensions/runtime/src/lib.rs b/bindings/python/dynamic-extensions/runtime/src/lib.rs new file mode 100644 index 000000000000..87179d98d2e1 --- /dev/null +++ b/bindings/python/dynamic-extensions/runtime/src/lib.rs @@ -0,0 +1,397 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::collections::HashMap; +use std::ffi::c_void; +use std::mem::size_of; +use std::panic::{AssertUnwindSafe, catch_unwind}; +use std::sync::{Arc, LazyLock, Mutex}; + +use libloading::{Library, Symbol}; +use opendal_dynamic_extension_sdk::{ + ByteSlice, CreateOperatorFn, DestroyOperatorFn, ExtensionApiV1, ExtensionBootstrapV1, KeyValue, + OperatorInfoFn, OperatorReadFn, OperatorWriteFn, OutputBuffer, RUNTIME_PROTOCOL, RuntimeApiV1, + RuntimeProtocolInfoV1, STATUS_CONFLICT, STATUS_INCOMPATIBLE, STATUS_INVALID_ARGUMENT, + STATUS_LOAD_FAILED, STATUS_OK, STATUS_OPERATION_FAILED, ServiceRegistrationV1, write_output, +}; + +struct LoadedService { + _library: Library, + create_operator: CreateOperatorFn, + destroy_operator: DestroyOperatorFn, + operator_info: OperatorInfoFn, + operator_write: OperatorWriteFn, + operator_read: OperatorReadFn, +} + +struct RegisteredService { + package_id: String, + component_id: String, + entry_symbol: String, + library_path: String, + loaded: Mutex>>, +} + +struct RuntimeOperator { + operator: *mut c_void, + service: Arc, +} + +impl Drop for RuntimeOperator { + fn drop(&mut self) { + unsafe { (self.service.destroy_operator)(self.operator) }; + } +} + +static SERVICES: LazyLock>>> = + LazyLock::new(|| Mutex::new(HashMap::new())); + +static RUNTIME_API: RuntimeApiV1 = RuntimeApiV1 { + struct_size: size_of::(), + register_service, + create_operator, + operator_info, + operator_write, + operator_read, + operator_destroy, +}; + +fn fail(error: *mut OutputBuffer, status: i32, message: impl AsRef) -> i32 { + let _ = unsafe { write_output(error, message.as_ref().as_bytes()) }; + status +} + +fn required_struct(actual: usize, name: &str) -> Result<(), String> { + if actual < size_of::() { + return Err(format!("{name} is smaller than the version 1 layout")); + } + Ok(()) +} + +unsafe fn load_service( + registration: &RegisteredService, +) -> Result, (i32, String)> { + let mut loaded = registration.loaded.lock().map_err(|_| { + ( + STATUS_OPERATION_FAILED, + "service load mutex poisoned".to_string(), + ) + })?; + if let Some(service) = loaded.as_ref() { + return Ok(Arc::clone(service)); + } + + let library = unsafe { Library::new(®istration.library_path) } + .map_err(|err| (STATUS_LOAD_FAILED, err.to_string()))?; + let bootstrap: Symbol<'_, ExtensionBootstrapV1> = + unsafe { library.get(registration.entry_symbol.as_bytes()) } + .map_err(|err| (STATUS_LOAD_FAILED, err.to_string()))?; + let mut extension = std::ptr::null(); + let status = unsafe { bootstrap(&mut extension) }; + if status != STATUS_OK || extension.is_null() { + return Err((STATUS_LOAD_FAILED, "extension bootstrap failed".to_string())); + } + let extension: &ExtensionApiV1 = unsafe { &*extension }; + required_struct::(extension.struct_size, "extension API") + .map_err(|message| (STATUS_INCOMPATIBLE, message))?; + if extension.required_runtime_protocol > RUNTIME_PROTOCOL { + return Err(( + STATUS_INCOMPATIBLE, + "extension requires a newer runtime protocol".to_string(), + )); + } + + let extension_version = unsafe { extension.opendal_version.as_str() } + .map_err(|message| (STATUS_INCOMPATIBLE, message.to_string()))?; + if extension_version != opendal_core::raw::VERSION { + return Err(( + STATUS_INCOMPATIBLE, + format!( + "extension OpenDAL version {extension_version} does not match runtime {}", + opendal_core::raw::VERSION + ), + )); + } + let extension_package = unsafe { extension.package_id.as_str() } + .map_err(|message| (STATUS_INCOMPATIBLE, message.to_string()))?; + if extension_package != registration.package_id { + return Err(( + STATUS_INCOMPATIBLE, + "extension package does not match registration".to_string(), + )); + } + let extension_component = unsafe { extension.component_id.as_str() } + .map_err(|message| (STATUS_INCOMPATIBLE, message.to_string()))?; + if extension_component != registration.component_id { + return Err(( + STATUS_INCOMPATIBLE, + "extension component does not match registration".to_string(), + )); + } + let extension_entry = unsafe { extension.entry_symbol.as_str() } + .map_err(|message| (STATUS_INCOMPATIBLE, message.to_string()))?; + if extension_entry != registration.entry_symbol { + return Err(( + STATUS_INCOMPATIBLE, + "extension entry symbol does not match registration".to_string(), + )); + } + + let service = Arc::new(LoadedService { + _library: library, + create_operator: extension.create_operator, + destroy_operator: extension.destroy_operator, + operator_info: extension.operator_info, + operator_write: extension.operator_write, + operator_read: extension.operator_read, + }); + *loaded = Some(Arc::clone(&service)); + Ok(service) +} + +unsafe extern "C" fn register_service( + registration: *const ServiceRegistrationV1, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + if registration.is_null() { + return fail(error, STATUS_INVALID_ARGUMENT, "registration is null"); + } + let registration = &*registration; + if let Err(message) = required_struct::( + registration.struct_size, + "service registration", + ) { + return fail(error, STATUS_INVALID_ARGUMENT, message); + } + if registration.required_runtime_protocol > RUNTIME_PROTOCOL { + return fail( + error, + STATUS_INCOMPATIBLE, + "service requires a newer runtime protocol", + ); + } + + let package_id = match registration.package_id.as_str() { + Ok(value) if !value.is_empty() => value, + Ok(_) => return fail(error, STATUS_INVALID_ARGUMENT, "package ID is empty"), + Err(message) => return fail(error, STATUS_INVALID_ARGUMENT, message), + }; + let component_id = match registration.component_id.as_str() { + Ok(value) if !value.is_empty() => value, + Ok(_) => return fail(error, STATUS_INVALID_ARGUMENT, "component ID is empty"), + Err(message) => return fail(error, STATUS_INVALID_ARGUMENT, message), + }; + let entry_symbol = match registration.entry_symbol.as_str() { + Ok(value) if !value.is_empty() => value, + Ok(_) => return fail(error, STATUS_INVALID_ARGUMENT, "entry symbol is empty"), + Err(message) => return fail(error, STATUS_INVALID_ARGUMENT, message), + }; + let library_path = match registration.library_path.as_str() { + Ok(value) if !value.is_empty() => value, + Ok(_) => return fail(error, STATUS_INVALID_ARGUMENT, "library path is empty"), + Err(message) => return fail(error, STATUS_INVALID_ARGUMENT, message), + }; + + let mut services = SERVICES.lock().expect("service registry mutex poisoned"); + if let Some(existing) = services.get(component_id) { + if existing.package_id == package_id { + return STATUS_OK; + } + return fail( + error, + STATUS_CONFLICT, + format!( + "service {component_id} is already owned by {}", + existing.package_id + ), + ); + } + services.insert( + component_id.to_string(), + Arc::new(RegisteredService { + package_id: package_id.to_string(), + component_id: component_id.to_string(), + entry_symbol: entry_symbol.to_string(), + library_path: library_path.to_string(), + loaded: Mutex::new(None), + }), + ); + STATUS_OK + })) + .unwrap_or_else(|_| { + fail( + error, + STATUS_OPERATION_FAILED, + "service registration panicked", + ) + }) +} + +unsafe extern "C" fn create_operator( + component_id: ByteSlice, + options: *const KeyValue, + options_len: usize, + operator: *mut *mut c_void, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + if operator.is_null() { + return fail(error, STATUS_INVALID_ARGUMENT, "operator output is null"); + } + *operator = std::ptr::null_mut(); + if options_len > 0 && options.is_null() { + return fail(error, STATUS_INVALID_ARGUMENT, "options pointer is null"); + } + let component_id = match component_id.as_str() { + Ok(value) => value, + Err(message) => return fail(error, STATUS_INVALID_ARGUMENT, message), + }; + let registration = { + let services = SERVICES.lock().expect("service registry mutex poisoned"); + match services.get(component_id) { + Some(registration) => Arc::clone(registration), + None => { + return fail( + error, + STATUS_INVALID_ARGUMENT, + format!( + "service {component_id} is not registered; import its package first" + ), + ); + } + } + }; + let service = match load_service(®istration) { + Ok(service) => service, + Err((status, message)) => return fail(error, status, message), + }; + + let mut inner = std::ptr::null_mut(); + let status = (service.create_operator)(options, options_len, &mut inner, error); + if status != STATUS_OK { + return status; + } + if inner.is_null() { + return fail( + error, + STATUS_OPERATION_FAILED, + "service returned a null operator", + ); + } + *operator = Box::into_raw(Box::new(RuntimeOperator { + operator: inner, + service, + })) + .cast::(); + STATUS_OK + })) + .unwrap_or_else(|_| fail(error, STATUS_OPERATION_FAILED, "operator creation panicked")) +} + +unsafe fn runtime_operator<'a>(operator: *mut c_void) -> Result<&'a RuntimeOperator, &'static str> { + if operator.is_null() { + return Err("operator handle is null"); + } + Ok(unsafe { &*operator.cast::() }) +} + +unsafe extern "C" fn operator_info( + operator: *mut c_void, + output: *mut OutputBuffer, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match runtime_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, STATUS_INVALID_ARGUMENT, message), + }; + (operator.service.operator_info)(operator.operator, output, error) + })) + .unwrap_or_else(|_| fail(error, STATUS_OPERATION_FAILED, "operator info panicked")) +} + +unsafe extern "C" fn operator_write( + operator: *mut c_void, + path: ByteSlice, + data: ByteSlice, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match runtime_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, STATUS_INVALID_ARGUMENT, message), + }; + (operator.service.operator_write)(operator.operator, path, data, error) + })) + .unwrap_or_else(|_| fail(error, STATUS_OPERATION_FAILED, "operator write panicked")) +} + +unsafe extern "C" fn operator_read( + operator: *mut c_void, + path: ByteSlice, + output: *mut OutputBuffer, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match runtime_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, STATUS_INVALID_ARGUMENT, message), + }; + (operator.service.operator_read)(operator.operator, path, output, error) + })) + .unwrap_or_else(|_| fail(error, STATUS_OPERATION_FAILED, "operator read panicked")) +} + +unsafe extern "C" fn operator_destroy(operator: *mut c_void) { + if operator.is_null() { + return; + } + let _ = catch_unwind(AssertUnwindSafe(|| unsafe { + drop(Box::from_raw(operator.cast::())); + })); +} + +#[unsafe(no_mangle)] +/// Negotiates the runtime protocol and returns the runtime function table. +/// +/// # Safety +/// +/// `protocol_info` and `api` must point to writable storage. The caller must +/// initialize `protocol_info.struct_size` before calling this function. +pub unsafe extern "C" fn opendal_runtime_get_api_v1( + required_runtime_protocol: u32, + protocol_info: *mut RuntimeProtocolInfoV1, + api: *mut *const RuntimeApiV1, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + if protocol_info.is_null() || api.is_null() { + return STATUS_INVALID_ARGUMENT; + } + let protocol_info = &mut *protocol_info; + if protocol_info.struct_size < size_of::() { + return STATUS_INVALID_ARGUMENT; + } + protocol_info.minimum_runtime_protocol = RUNTIME_PROTOCOL; + protocol_info.runtime_protocol = RUNTIME_PROTOCOL; + *api = std::ptr::null(); + if required_runtime_protocol != RUNTIME_PROTOCOL { + return STATUS_INCOMPATIBLE; + } + *api = &RUNTIME_API; + STATUS_OK + })) + .unwrap_or(STATUS_OPERATION_FAILED) +} diff --git a/bindings/python/dynamic-extensions/sdk/Cargo.toml b/bindings/python/dynamic-extensions/sdk/Cargo.toml new file mode 100644 index 000000000000..af43252dcb61 --- /dev/null +++ b/bindings/python/dynamic-extensions/sdk/Cargo.toml @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +edition.workspace = true +license.workspace = true +name = "opendal-dynamic-extension-sdk" +publish.workspace = true +rust-version.workspace = true +version.workspace = true diff --git a/bindings/python/dynamic-extensions/sdk/src/lib.rs b/bindings/python/dynamic-extensions/sdk/src/lib.rs new file mode 100644 index 000000000000..0bd04710f2b8 --- /dev/null +++ b/bindings/python/dynamic-extensions/sdk/src/lib.rs @@ -0,0 +1,224 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::ffi::c_void; +use std::slice; +use std::str; + +pub const RUNTIME_PROTOCOL: u32 = 1; +pub const STATUS_OK: i32 = 0; +pub const STATUS_INVALID_ARGUMENT: i32 = 1; +pub const STATUS_INCOMPATIBLE: i32 = 2; +pub const STATUS_CONFLICT: i32 = 3; +pub const STATUS_LOAD_FAILED: i32 = 4; +pub const STATUS_OPERATION_FAILED: i32 = 5; +pub const STATUS_BUFFER_TOO_SMALL: i32 = 6; + +#[repr(C)] +#[derive(Clone, Copy)] +pub struct ByteSlice { + pub data: *const u8, + pub len: usize, +} + +// Protocol byte slices point to immutable memory whose owner must retain it +// for the documented call or table lifetime. +unsafe impl Send for ByteSlice {} +unsafe impl Sync for ByteSlice {} + +impl ByteSlice { + pub const fn from_static(value: &'static str) -> Self { + Self { + data: value.as_ptr(), + len: value.len(), + } + } + + /// # Safety + /// + /// The caller must keep `data` valid for `len` bytes for the returned + /// slice's lifetime. + pub unsafe fn as_bytes<'a>(self) -> Result<&'a [u8], &'static str> { + if self.len == 0 { + return Ok(&[]); + } + if self.data.is_null() { + return Err("byte slice has a null pointer"); + } + Ok(unsafe { slice::from_raw_parts(self.data, self.len) }) + } + + /// # Safety + /// + /// The caller must satisfy [`ByteSlice::as_bytes`]. + pub unsafe fn as_str<'a>(self) -> Result<&'a str, &'static str> { + str::from_utf8(unsafe { self.as_bytes()? }).map_err(|_| "byte slice is not valid UTF-8") + } +} + +#[repr(C)] +#[derive(Clone, Copy)] +pub struct KeyValue { + pub key: ByteSlice, + pub value: ByteSlice, +} + +#[repr(C)] +pub struct OutputBuffer { + pub data: *mut u8, + pub capacity: usize, + pub len: usize, +} + +/// Copy bytes into a caller-owned output buffer. +/// +/// The function always records the required length. It returns +/// [`STATUS_BUFFER_TOO_SMALL`] without copying when the supplied capacity is +/// insufficient. +/// +/// # Safety +/// +/// `output` must be null or point to a valid [`OutputBuffer`]. Its `data` must +/// be valid for `capacity` writable bytes when the capacity is non-zero. +pub unsafe fn write_output(output: *mut OutputBuffer, value: &[u8]) -> i32 { + if output.is_null() { + return STATUS_INVALID_ARGUMENT; + } + + let output = unsafe { &mut *output }; + output.len = value.len(); + if output.capacity < value.len() { + return STATUS_BUFFER_TOO_SMALL; + } + if value.is_empty() { + return STATUS_OK; + } + if output.data.is_null() { + return STATUS_INVALID_ARGUMENT; + } + + unsafe { std::ptr::copy(value.as_ptr(), output.data, value.len()) }; + STATUS_OK +} + +/// Creates an extension-owned operator handle. +/// +/// Input slices remain caller-owned and are valid only for the call. On +/// success, `operator` receives a non-null handle that the caller must pass +/// only to callbacks from the same extension table and destroy exactly once. +pub type CreateOperatorFn = unsafe extern "C" fn( + options: *const KeyValue, + options_len: usize, + operator: *mut *mut c_void, + error: *mut OutputBuffer, +) -> i32; +/// Destroys a handle returned by [`CreateOperatorFn`]. +pub type DestroyOperatorFn = unsafe extern "C" fn(operator: *mut c_void); + +/// Describes one extension for the lifetime of its loaded native library. +/// +/// The table and all byte slices in it point to immutable package-owned +/// storage. Callers must serialize operations and destruction for each handle; +/// this POC does not define concurrent callback behavior. Operation callbacks +/// borrow their input buffers for the call and report text errors through +/// caller-owned [`OutputBuffer`] values. +#[repr(C)] +pub struct ExtensionApiV1 { + pub struct_size: usize, + pub required_runtime_protocol: u32, + pub opendal_version: ByteSlice, + pub package_id: ByteSlice, + pub component_id: ByteSlice, + pub entry_symbol: ByteSlice, + pub create_operator: CreateOperatorFn, + pub destroy_operator: DestroyOperatorFn, + pub operator_info: OperatorInfoFn, + pub operator_write: OperatorWriteFn, + pub operator_read: OperatorReadFn, +} + +/// Returns an immutable extension table owned by the loaded native library. +pub type ExtensionBootstrapV1 = unsafe extern "C" fn(extension: *mut *const ExtensionApiV1) -> i32; + +/// Provides the logical manifest fields and package-resolved artifact path. +/// +/// Every byte slice is caller-owned and remains valid only for the registration +/// call. The runtime copies fields that must outlive the call and retains the +/// loaded library while any operator created from it remains alive. +#[repr(C)] +pub struct ServiceRegistrationV1 { + pub struct_size: usize, + pub required_runtime_protocol: u32, + pub package_id: ByteSlice, + pub component_id: ByteSlice, + pub entry_symbol: ByteSlice, + pub library_path: ByteSlice, +} + +/// Registers one service and writes a diagnostic into `error` on failure. +pub type RegisterServiceFn = unsafe extern "C" fn( + registration: *const ServiceRegistrationV1, + error: *mut OutputBuffer, +) -> i32; +/// Creates a runtime-owned wrapper around an extension operator handle. +pub type CreateRuntimeOperatorFn = unsafe extern "C" fn( + component_id: ByteSlice, + options: *const KeyValue, + options_len: usize, + operator: *mut *mut c_void, + error: *mut OutputBuffer, +) -> i32; +/// Writes JSON operator information into caller-owned output storage. +pub type OperatorInfoFn = unsafe extern "C" fn( + operator: *mut c_void, + output: *mut OutputBuffer, + error: *mut OutputBuffer, +) -> i32; +/// Writes bytes through an operator without taking ownership of input slices. +pub type OperatorWriteFn = unsafe extern "C" fn( + operator: *mut c_void, + path: ByteSlice, + data: ByteSlice, + error: *mut OutputBuffer, +) -> i32; +/// Reads bytes into caller-owned output storage. +pub type OperatorReadFn = unsafe extern "C" fn( + operator: *mut c_void, + path: ByteSlice, + output: *mut OutputBuffer, + error: *mut OutputBuffer, +) -> i32; +/// Destroys a runtime operator wrapper exactly once. +pub type OperatorDestroyFn = unsafe extern "C" fn(operator: *mut c_void); + +/// Runtime callbacks available for the lifetime of the runtime library. +#[repr(C)] +pub struct RuntimeApiV1 { + pub struct_size: usize, + pub register_service: RegisterServiceFn, + pub create_operator: CreateRuntimeOperatorFn, + pub operator_info: OperatorInfoFn, + pub operator_write: OperatorWriteFn, + pub operator_read: OperatorReadFn, + pub operator_destroy: OperatorDestroyFn, +} + +/// Reports the protocol interval supported by the loaded runtime. +#[repr(C)] +pub struct RuntimeProtocolInfoV1 { + pub struct_size: usize, + pub minimum_runtime_protocol: u32, + pub runtime_protocol: u32, +} diff --git a/bindings/python/dynamic-extensions/service-fs/Cargo.toml b/bindings/python/dynamic-extensions/service-fs/Cargo.toml new file mode 100644 index 000000000000..12ed498bc352 --- /dev/null +++ b/bindings/python/dynamic-extensions/service-fs/Cargo.toml @@ -0,0 +1,32 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +edition.workspace = true +license.workspace = true +name = "fs-extension" +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] + +[dependencies] +opendal-core = { workspace = true, features = ["blocking"] } +opendal-dynamic-extension-sdk.workspace = true +opendal-service-fs.workspace = true +serde_json = "1" +tokio.workspace = true diff --git a/bindings/python/dynamic-extensions/service-fs/build.rs b/bindings/python/dynamic-extensions/service-fs/build.rs new file mode 100644 index 000000000000..501b9f44298a --- /dev/null +++ b/bindings/python/dynamic-extensions/service-fs/build.rs @@ -0,0 +1,29 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::env; +use std::path::PathBuf; + +fn main() { + if env::var("CARGO_CFG_TARGET_OS").as_deref() == Ok("linux") { + let manifest_dir = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); + let export_map = manifest_dir.join("exports.map"); + println!("cargo:rerun-if-changed={}", export_map.display()); + println!( + "cargo:rustc-cdylib-link-arg=-Wl,--version-script={}", + export_map.display() + ); + } +} diff --git a/bindings/python/dynamic-extensions/service-fs/exports.map b/bindings/python/dynamic-extensions/service-fs/exports.map new file mode 100644 index 000000000000..eaadbb8b536b --- /dev/null +++ b/bindings/python/dynamic-extensions/service-fs/exports.map @@ -0,0 +1,21 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +OPENDAL_FS_EXTENSION_1 { + global: + opendal_service_fs_bootstrap_v1; + local: + *; +}; diff --git a/bindings/python/dynamic-extensions/service-fs/src/lib.rs b/bindings/python/dynamic-extensions/service-fs/src/lib.rs new file mode 100644 index 000000000000..4d0da9d9dbae --- /dev/null +++ b/bindings/python/dynamic-extensions/service-fs/src/lib.rs @@ -0,0 +1,218 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::ffi::c_void; +use std::mem::size_of; +use std::panic::{AssertUnwindSafe, catch_unwind}; +use std::slice; + +use opendal_core::Operator; +use opendal_dynamic_extension_sdk::{ + ByteSlice, ExtensionApiV1, KeyValue, OutputBuffer, RUNTIME_PROTOCOL, STATUS_INVALID_ARGUMENT, + STATUS_OK, STATUS_OPERATION_FAILED, write_output, +}; +use opendal_service_fs::Fs; + +static API: ExtensionApiV1 = ExtensionApiV1 { + struct_size: size_of::(), + required_runtime_protocol: RUNTIME_PROTOCOL, + opendal_version: ByteSlice::from_static(opendal_core::raw::VERSION), + package_id: ByteSlice::from_static("opendal-service-fs-poc"), + component_id: ByteSlice::from_static("fs"), + entry_symbol: ByteSlice::from_static("opendal_service_fs_bootstrap_v1"), + create_operator, + destroy_operator, + operator_info, + operator_write, + operator_read, +}; + +static TOKIO_RUNTIME: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .expect("prototype FS runtime should build") + }); + +fn fail(error: *mut OutputBuffer, message: impl AsRef) -> i32 { + let _ = unsafe { write_output(error, message.as_ref().as_bytes()) }; + STATUS_OPERATION_FAILED +} + +unsafe fn option_pairs<'a>( + options: *const KeyValue, + options_len: usize, +) -> Result, &'static str> { + if options_len == 0 { + return Ok(Vec::new()); + } + if options.is_null() { + return Err("options pointer is null"); + } + unsafe { slice::from_raw_parts(options, options_len) } + .iter() + .map(|pair| unsafe { Ok((pair.key.as_str()?, pair.value.as_str()?)) }) + .collect() +} + +unsafe extern "C" fn create_operator( + options: *const KeyValue, + options_len: usize, + operator: *mut *mut c_void, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + if operator.is_null() { + return STATUS_INVALID_ARGUMENT; + } + *operator = std::ptr::null_mut(); + let options = match option_pairs(options, options_len) { + Ok(options) => options, + Err(message) => return fail(error, message), + }; + + let mut builder = Fs::default(); + for (key, value) in options { + builder = match key { + "root" => builder.root(value), + "atomic_write_dir" => builder.atomic_write_dir(value), + _ => return fail(error, format!("unsupported FS option {key}")), + }; + } + + match Operator::new(builder) { + Ok(value) => { + *operator = Box::into_raw(Box::new(value)).cast::(); + STATUS_OK + } + Err(err) => fail(error, err.to_string()), + } + })) + .unwrap_or_else(|_| fail(error, "FS operator construction panicked")) +} + +unsafe extern "C" fn destroy_operator(operator: *mut c_void) { + if operator.is_null() { + return; + } + let _ = catch_unwind(AssertUnwindSafe(|| unsafe { + drop(Box::from_raw(operator.cast::())); + })); +} + +unsafe fn as_operator<'a>(operator: *mut c_void) -> Result<&'a Operator, &'static str> { + if operator.is_null() { + return Err("FS operator is null"); + } + Ok(unsafe { &*operator.cast::() }) +} + +unsafe extern "C" fn operator_info( + operator: *mut c_void, + output: *mut OutputBuffer, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match as_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, message), + }; + let info = operator.info(); + let value = serde_json::json!({ + "scheme": info.scheme().to_string(), + "name": info.name(), + "root": info.root(), + }); + write_output(output, value.to_string().as_bytes()) + })) + .unwrap_or_else(|_| fail(error, "FS operator info panicked")) +} + +unsafe extern "C" fn operator_write( + operator: *mut c_void, + path: ByteSlice, + data: ByteSlice, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match as_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, message), + }; + let path = match path.as_str() { + Ok(path) => path, + Err(message) => return fail(error, message), + }; + let data = match data.as_bytes() { + Ok(data) => data, + Err(message) => return fail(error, message), + }; + let _guard = TOKIO_RUNTIME.enter(); + let blocking = match opendal_core::blocking::Operator::new(operator.clone()) { + Ok(blocking) => blocking, + Err(err) => return fail(error, err.to_string()), + }; + match blocking.write(path, data.to_vec()) { + Ok(_) => STATUS_OK, + Err(err) => fail(error, err.to_string()), + } + })) + .unwrap_or_else(|_| fail(error, "FS write panicked")) +} + +unsafe extern "C" fn operator_read( + operator: *mut c_void, + path: ByteSlice, + output: *mut OutputBuffer, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match as_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, message), + }; + let path = match path.as_str() { + Ok(path) => path, + Err(message) => return fail(error, message), + }; + let _guard = TOKIO_RUNTIME.enter(); + let blocking = match opendal_core::blocking::Operator::new(operator.clone()) { + Ok(blocking) => blocking, + Err(err) => return fail(error, err.to_string()), + }; + match blocking.read(path) { + Ok(value) => write_output(output, &value.to_vec()), + Err(err) => fail(error, err.to_string()), + } + })) + .unwrap_or_else(|_| fail(error, "FS read panicked")) +} + +#[unsafe(no_mangle)] +/// Returns the FS extension function table. +/// +/// # Safety +/// +/// `extension` must point to writable storage for an extension table pointer. +pub unsafe extern "C" fn opendal_service_fs_bootstrap_v1( + extension: *mut *const ExtensionApiV1, +) -> i32 { + if extension.is_null() { + return STATUS_INVALID_ARGUMENT; + } + unsafe { *extension = &API }; + STATUS_OK +} diff --git a/bindings/python/dynamic-extensions/service-s3/Cargo.toml b/bindings/python/dynamic-extensions/service-s3/Cargo.toml new file mode 100644 index 000000000000..0c6a97e4e10b --- /dev/null +++ b/bindings/python/dynamic-extensions/service-s3/Cargo.toml @@ -0,0 +1,32 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +edition.workspace = true +license.workspace = true +name = "s3-extension" +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] + +[dependencies] +opendal-core = { workspace = true, features = ["blocking"] } +opendal-dynamic-extension-sdk.workspace = true +opendal-service-s3.workspace = true +serde_json = "1" +tokio.workspace = true diff --git a/bindings/python/dynamic-extensions/service-s3/build.rs b/bindings/python/dynamic-extensions/service-s3/build.rs new file mode 100644 index 000000000000..501b9f44298a --- /dev/null +++ b/bindings/python/dynamic-extensions/service-s3/build.rs @@ -0,0 +1,29 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::env; +use std::path::PathBuf; + +fn main() { + if env::var("CARGO_CFG_TARGET_OS").as_deref() == Ok("linux") { + let manifest_dir = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); + let export_map = manifest_dir.join("exports.map"); + println!("cargo:rerun-if-changed={}", export_map.display()); + println!( + "cargo:rustc-cdylib-link-arg=-Wl,--version-script={}", + export_map.display() + ); + } +} diff --git a/bindings/python/dynamic-extensions/service-s3/exports.map b/bindings/python/dynamic-extensions/service-s3/exports.map new file mode 100644 index 000000000000..09035fa37b48 --- /dev/null +++ b/bindings/python/dynamic-extensions/service-s3/exports.map @@ -0,0 +1,21 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +OPENDAL_S3_EXTENSION_1 { + global: + opendal_service_s3_bootstrap_v1; + local: + *; +}; diff --git a/bindings/python/dynamic-extensions/service-s3/src/lib.rs b/bindings/python/dynamic-extensions/service-s3/src/lib.rs new file mode 100644 index 000000000000..7b904c14febe --- /dev/null +++ b/bindings/python/dynamic-extensions/service-s3/src/lib.rs @@ -0,0 +1,222 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::ffi::c_void; +use std::mem::size_of; +use std::panic::{AssertUnwindSafe, catch_unwind}; +use std::slice; + +use opendal_core::Operator; +use opendal_dynamic_extension_sdk::{ + ByteSlice, ExtensionApiV1, KeyValue, OutputBuffer, RUNTIME_PROTOCOL, STATUS_INVALID_ARGUMENT, + STATUS_OK, STATUS_OPERATION_FAILED, write_output, +}; +use opendal_service_s3::S3; + +static API: ExtensionApiV1 = ExtensionApiV1 { + struct_size: size_of::(), + required_runtime_protocol: RUNTIME_PROTOCOL, + opendal_version: ByteSlice::from_static(opendal_core::raw::VERSION), + package_id: ByteSlice::from_static("opendal-service-s3-poc"), + component_id: ByteSlice::from_static("s3"), + entry_symbol: ByteSlice::from_static("opendal_service_s3_bootstrap_v1"), + create_operator, + destroy_operator, + operator_info, + operator_write, + operator_read, +}; + +static TOKIO_RUNTIME: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .expect("prototype S3 runtime should build") + }); + +fn fail(error: *mut OutputBuffer, message: impl AsRef) -> i32 { + let _ = unsafe { write_output(error, message.as_ref().as_bytes()) }; + STATUS_OPERATION_FAILED +} + +unsafe fn option_pairs<'a>( + options: *const KeyValue, + options_len: usize, +) -> Result, &'static str> { + if options_len == 0 { + return Ok(Vec::new()); + } + if options.is_null() { + return Err("options pointer is null"); + } + unsafe { slice::from_raw_parts(options, options_len) } + .iter() + .map(|pair| unsafe { Ok((pair.key.as_str()?, pair.value.as_str()?)) }) + .collect() +} + +unsafe extern "C" fn create_operator( + options: *const KeyValue, + options_len: usize, + operator: *mut *mut c_void, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + if operator.is_null() { + return STATUS_INVALID_ARGUMENT; + } + *operator = std::ptr::null_mut(); + let options = match option_pairs(options, options_len) { + Ok(options) => options, + Err(message) => return fail(error, message), + }; + + let mut builder = S3::default(); + for (key, value) in options { + builder = match key { + "bucket" => builder.bucket(value), + "region" => builder.region(value), + "root" => builder.root(value), + "endpoint" => builder.endpoint(value), + "access_key_id" => builder.access_key_id(value), + "secret_access_key" => builder.secret_access_key(value), + _ => return fail(error, format!("unsupported S3 option {key}")), + }; + } + + match Operator::new(builder) { + Ok(value) => { + *operator = Box::into_raw(Box::new(value)).cast::(); + STATUS_OK + } + Err(err) => fail(error, err.to_string()), + } + })) + .unwrap_or_else(|_| fail(error, "S3 operator construction panicked")) +} + +unsafe extern "C" fn destroy_operator(operator: *mut c_void) { + if operator.is_null() { + return; + } + let _ = catch_unwind(AssertUnwindSafe(|| unsafe { + drop(Box::from_raw(operator.cast::())); + })); +} + +unsafe fn as_operator<'a>(operator: *mut c_void) -> Result<&'a Operator, &'static str> { + if operator.is_null() { + return Err("S3 operator is null"); + } + Ok(unsafe { &*operator.cast::() }) +} + +unsafe extern "C" fn operator_info( + operator: *mut c_void, + output: *mut OutputBuffer, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match as_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, message), + }; + let info = operator.info(); + let value = serde_json::json!({ + "scheme": info.scheme().to_string(), + "name": info.name(), + "root": info.root(), + }); + write_output(output, value.to_string().as_bytes()) + })) + .unwrap_or_else(|_| fail(error, "S3 operator info panicked")) +} + +unsafe extern "C" fn operator_write( + operator: *mut c_void, + path: ByteSlice, + data: ByteSlice, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match as_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, message), + }; + let path = match path.as_str() { + Ok(path) => path, + Err(message) => return fail(error, message), + }; + let data = match data.as_bytes() { + Ok(data) => data, + Err(message) => return fail(error, message), + }; + let _guard = TOKIO_RUNTIME.enter(); + let blocking = match opendal_core::blocking::Operator::new(operator.clone()) { + Ok(blocking) => blocking, + Err(err) => return fail(error, err.to_string()), + }; + match blocking.write(path, data.to_vec()) { + Ok(_) => STATUS_OK, + Err(err) => fail(error, err.to_string()), + } + })) + .unwrap_or_else(|_| fail(error, "S3 write panicked")) +} + +unsafe extern "C" fn operator_read( + operator: *mut c_void, + path: ByteSlice, + output: *mut OutputBuffer, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match as_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, message), + }; + let path = match path.as_str() { + Ok(path) => path, + Err(message) => return fail(error, message), + }; + let _guard = TOKIO_RUNTIME.enter(); + let blocking = match opendal_core::blocking::Operator::new(operator.clone()) { + Ok(blocking) => blocking, + Err(err) => return fail(error, err.to_string()), + }; + match blocking.read(path) { + Ok(value) => write_output(output, &value.to_vec()), + Err(err) => fail(error, err.to_string()), + } + })) + .unwrap_or_else(|_| fail(error, "S3 read panicked")) +} + +#[unsafe(no_mangle)] +/// Returns the S3 extension function table. +/// +/// # Safety +/// +/// `extension` must point to writable storage for an extension table pointer. +pub unsafe extern "C" fn opendal_service_s3_bootstrap_v1( + extension: *mut *const ExtensionApiV1, +) -> i32 { + if extension.is_null() { + return STATUS_INVALID_ARGUMENT; + } + unsafe { *extension = &API }; + STATUS_OK +} diff --git a/bindings/ruby/dynamic-extensions/.gitignore b/bindings/ruby/dynamic-extensions/.gitignore new file mode 100644 index 000000000000..d7d2adf88a85 --- /dev/null +++ b/bindings/ruby/dynamic-extensions/.gitignore @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +/target/ diff --git a/bindings/ruby/dynamic-extensions/Cargo.lock b/bindings/ruby/dynamic-extensions/Cargo.lock new file mode 100644 index 000000000000..f6b68fb620cd --- /dev/null +++ b/bindings/ruby/dynamic-extensions/Cargo.lock @@ -0,0 +1,297 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "aho-corasick" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" +dependencies = [ + "memchr", +] + +[[package]] +name = "bindgen" +version = "0.72.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "993776b509cfb49c750f11b8f07a46fa23e0a1386ffc01fb1e7d343efc387895" +dependencies = [ + "bitflags", + "cexpr", + "clang-sys", + "itertools", + "proc-macro2", + "quote", + "regex", + "rustc-hash", + "shlex", + "syn", +] + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "cexpr" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6fac387a98bb7c37292057cffc56d62ecb629900026402633ae9160df93a8766" +dependencies = [ + "nom", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "clang-sys" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b023947811758c97c59bf9d1c188fd619ad4718dcaa767947df1cadb14f39f4" +dependencies = [ + "glob", + "libc", + "libloading", +] + +[[package]] +name = "either" +version = "1.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e5e8f6c15a24b9a3ee5efec809ccd006d3b30e8b3bb63c39af737c7f87daa1d" + +[[package]] +name = "glob" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e4eba85ea1d0a966a983acd07deee566e67395d2d96b6fb39e62b5a833f1eb0b" + +[[package]] +name = "itertools" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186" +dependencies = [ + "either", +] + +[[package]] +name = "lazy_static" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" + +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "libloading" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" +dependencies = [ + "cfg-if", + "windows-link", +] + +[[package]] +name = "magnus" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b36a5b126bbe97eb0d02d07acfeb327036c6319fd816139a49824a83b7f9012" +dependencies = [ + "bytes", + "magnus-macros", + "rb-sys", + "rb-sys-env", + "seq-macro", +] + +[[package]] +name = "magnus-macros" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47607461fd8e1513cb4f2076c197d8092d921a1ea75bd08af97398f593751892" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "minimal-lexical" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a" + +[[package]] +name = "nom" +version = "7.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a" +dependencies = [ + "memchr", + "minimal-lexical", +] + +[[package]] +name = "opendal-dynamic-extension-sdk" +version = "0.0.0" + +[[package]] +name = "opendal-ruby-runtime-poc" +version = "0.0.0" +dependencies = [ + "bytes", + "libloading", + "magnus", + "opendal-dynamic-extension-sdk", + "rb-sys", + "rb-sys-env", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rb-sys" +version = "0.9.128" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "45ca28513560e56cfb79a62b1fce363c73af170a182024ce880c77ee9429920a" +dependencies = [ + "rb-sys-build", +] + +[[package]] +name = "rb-sys-build" +version = "0.9.128" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce04b2c55eff3a21aaa623fcc655d94373238e72cac6b3e1a3641ff31649f99a" +dependencies = [ + "bindgen", + "lazy_static", + "proc-macro2", + "quote", + "regex", + "shell-words", + "syn", +] + +[[package]] +name = "rb-sys-env" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cca7ad6a7e21e72151d56fe2495a259b5670e204c3adac41ee7ef676ea08117a" + +[[package]] +name = "regex" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d" +dependencies = [ + "aho-corasick", + "memchr", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "regex-automata" +version = "0.4.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fcfdb36bda0c880c5931cdc7a2bcdc8ba4556847b9d912bca70bc94708711ad" +dependencies = [ + "aho-corasick", + "memchr", + "regex-syntax", +] + +[[package]] +name = "regex-syntax" +version = "0.8.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" + +[[package]] +name = "rustc-hash" +version = "2.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d" + +[[package]] +name = "seq-macro" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc" + +[[package]] +name = "shell-words" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc6fe69c597f9c37bfeeeeeb33da3530379845f10be461a66d16d03eca2ded77" + +[[package]] +name = "shlex" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64" + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" diff --git a/bindings/ruby/dynamic-extensions/Cargo.toml b/bindings/ruby/dynamic-extensions/Cargo.toml new file mode 100644 index 000000000000..14107bea1b15 --- /dev/null +++ b/bindings/ruby/dynamic-extensions/Cargo.toml @@ -0,0 +1,25 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[workspace] +members = ["adapter"] +resolver = "2" + +[workspace.package] +edition = "2024" +license = "Apache-2.0" +publish = false +rust-version = "1.91" +version = "0.0.0" diff --git a/bindings/ruby/dynamic-extensions/NOTES.md b/bindings/ruby/dynamic-extensions/NOTES.md new file mode 100644 index 000000000000..72979021d55c --- /dev/null +++ b/bindings/ruby/dynamic-extensions/NOTES.md @@ -0,0 +1,50 @@ + + +# Prototype Notes + +Question: Can Ruby use the same shared runtime and native extension artifacts +as Python without linking a binding-private OpenDAL or Tokio graph? + +Observed result: + +- The Ruby adapter negotiates runtime protocol 1 and rejects a binding that + requires protocol 2. +- Registering the FS Ruby package does not load its native library. Constructing + the first FS operator activates it. +- Ruby writes and reads an 8 KiB payload through the runtime-owned handle. The + adapter exercises the output-buffer resize contract and rejects use after + `Operator#close`. +- The Ruby adapter links Magnus, the protocol SDK, and the native loader. It + does not link `opendal-core`, a service crate, or Tokio. +- The same physical runtime and FS artifacts complete a second FS round trip + through the Python adapter. +- The runtime, FS extension, and Ruby adapter match their explicit ELF export + allowlists. Ruby requires both `Init_opendal_ruby_poc` and + `ruby_abi_version`. + +Production gate: the current runtime protocol owns service registration and an +operation callback table, but it has no layer registration, `LayerHandle`, or +operator-layer composition function. A successful FS run proves cross-language +runtime reuse only. It does not satisfy the design requirement that arbitrary +native layers preserve `apply_service` and `apply_context` semantics. + +Decision: do not migrate the production Python and Ruby bindings yet. The +current runtime implements the Design B operation-table boundary for services, +not the selected Design C shared OpenDAL graph. Production work first needs a +runtime-owned service/layer factory interface and a real operation that proves +an independently packaged native layer composes through that graph. diff --git a/bindings/ruby/dynamic-extensions/README.md b/bindings/ruby/dynamic-extensions/README.md new file mode 100644 index 000000000000..338a587c75fb --- /dev/null +++ b/bindings/ruby/dynamic-extensions/README.md @@ -0,0 +1,40 @@ + + +# Ruby Shared Runtime Prototype + +PROTOTYPE: delete or absorb this directory after the design question is +answered. + +Question: Can a Ruby binding adapter use the exact language-neutral runtime and +FS service artifacts used by the Python dynamic-extension POC? + +The Magnus adapter depends on the extension SDK and loader only. It does not +link OpenDAL core, an OpenDAL service, or Tokio. The runner builds and stages +the existing `opendal-runtime-poc` and `fs-extension` artifacts without +recompiling them for Ruby. + +Run the experiment on Linux: + +```console +./run-ruby-linux.sh +``` + +The runner verifies protocol negotiation, lazy FS activation, real FS +write/read, runtime-owned handle destruction, rejection of a newer binding +protocol, and final ELF export allowlists. It also stages the same runtime and +FS binaries under the Python adapter and completes a second real FS round trip. diff --git a/bindings/ruby/dynamic-extensions/adapter/Cargo.toml b/bindings/ruby/dynamic-extensions/adapter/Cargo.toml new file mode 100644 index 000000000000..618c18535b49 --- /dev/null +++ b/bindings/ruby/dynamic-extensions/adapter/Cargo.toml @@ -0,0 +1,36 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +name = "opendal-ruby-runtime-poc" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] +name = "opendal_ruby_poc" + +[dependencies] +bytes = "1" +libloading = "0.8.9" +magnus = { version = "0.8", features = ["bytes"] } +opendal-dynamic-extension-sdk = { path = "../../../python/dynamic-extensions/sdk" } +rb-sys = { version = "0.9.110", default-features = false } + +[build-dependencies] +rb-sys-env = "0.2.3" diff --git a/bindings/ruby/dynamic-extensions/adapter/build.rs b/bindings/ruby/dynamic-extensions/adapter/build.rs new file mode 100644 index 000000000000..d41d49b50f90 --- /dev/null +++ b/bindings/ruby/dynamic-extensions/adapter/build.rs @@ -0,0 +1,31 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::env; +use std::path::PathBuf; + +fn main() -> Result<(), Box> { + let _ = rb_sys_env::activate()?; + if env::var("CARGO_CFG_TARGET_OS").as_deref() == Ok("linux") { + let manifest_dir = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); + let export_map = manifest_dir.join("exports.map"); + println!("cargo:rerun-if-changed={}", export_map.display()); + println!( + "cargo:rustc-cdylib-link-arg=-Wl,--version-script={}", + export_map.display() + ); + } + Ok(()) +} diff --git a/bindings/ruby/dynamic-extensions/adapter/exports.map b/bindings/ruby/dynamic-extensions/adapter/exports.map new file mode 100644 index 000000000000..f1e9b63961c7 --- /dev/null +++ b/bindings/ruby/dynamic-extensions/adapter/exports.map @@ -0,0 +1,21 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +OPENDAL_RUBY_POC_1 { + global: + Init_opendal_ruby_poc; + local: + *; +}; diff --git a/bindings/ruby/dynamic-extensions/adapter/src/lib.rs b/bindings/ruby/dynamic-extensions/adapter/src/lib.rs new file mode 100644 index 000000000000..ac83522dff86 --- /dev/null +++ b/bindings/ruby/dynamic-extensions/adapter/src/lib.rs @@ -0,0 +1,329 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::collections::HashMap; +use std::ffi::c_void; +use std::mem::size_of; +use std::path::{Path, PathBuf}; +use std::ptr; +use std::sync::OnceLock; +use std::sync::atomic::{AtomicUsize, Ordering}; + +use libloading::{Library, Symbol}; +use magnus::prelude::*; +use magnus::{Error, RString, Ruby, function, method}; +use opendal_dynamic_extension_sdk::{ + ByteSlice, KeyValue, OutputBuffer, RuntimeApiV1, RuntimeProtocolInfoV1, + STATUS_BUFFER_TOO_SMALL, STATUS_OK, ServiceRegistrationV1, +}; + +type GetRuntimeApiFn = + unsafe extern "C" fn(u32, *mut RuntimeProtocolInfoV1, *mut *const RuntimeApiV1) -> i32; + +struct NativeRuntime { + _library: Library, + api_address: usize, + library_path: PathBuf, + minimum_protocol: u32, + protocol: u32, +} + +impl NativeRuntime { + fn api(&self) -> &RuntimeApiV1 { + unsafe { &*(self.api_address as *const RuntimeApiV1) } + } +} + +static RUNTIME: OnceLock = OnceLock::new(); + +fn runtime_error(ruby: &Ruby, message: impl AsRef) -> Error { + Error::new(ruby.exception_runtime_error(), message.as_ref().to_owned()) +} + +fn output(capacity: usize) -> (OutputBuffer, Vec) { + let mut storage = vec![0; capacity]; + let buffer = OutputBuffer { + data: storage.as_mut_ptr(), + capacity: storage.len(), + len: 0, + }; + (buffer, storage) +} + +fn output_message(buffer: &OutputBuffer, storage: &[u8]) -> String { + String::from_utf8_lossy(&storage[..buffer.len.min(storage.len())]).into_owned() +} + +fn byte_slice(value: &[u8]) -> ByteSlice { + ByteSlice { + data: value.as_ptr(), + len: value.len(), + } +} + +fn require_runtime(ruby: &Ruby) -> Result<&'static NativeRuntime, Error> { + RUNTIME + .get() + .ok_or_else(|| runtime_error(ruby, "OpenDAL runtime is not loaded")) +} + +fn load_runtime(ruby: &Ruby, library_path: String, required_protocol: u32) -> Result<(), Error> { + let library_path = Path::new(&library_path) + .canonicalize() + .map_err(|error| runtime_error(ruby, error.to_string()))?; + if let Some(runtime) = RUNTIME.get() { + if runtime.library_path != library_path { + return Err(runtime_error( + ruby, + format!( + "runtime already loaded from {}", + runtime.library_path.display() + ), + )); + } + if required_protocol < runtime.minimum_protocol || required_protocol > runtime.protocol { + return Err(runtime_error(ruby, "runtime protocol is incompatible")); + } + return Ok(()); + } + + let library = unsafe { Library::new(&library_path) } + .map_err(|error| runtime_error(ruby, error.to_string()))?; + let get_api: Symbol<'_, GetRuntimeApiFn> = unsafe { + library + .get(b"opendal_runtime_get_api_v1") + .map_err(|error| runtime_error(ruby, error.to_string()))? + }; + let mut protocol = RuntimeProtocolInfoV1 { + struct_size: size_of::(), + minimum_runtime_protocol: 0, + runtime_protocol: 0, + }; + let mut api = ptr::null(); + let status = unsafe { get_api(required_protocol, &mut protocol, &mut api) }; + if status != STATUS_OK || api.is_null() { + return Err(runtime_error( + ruby, + format!( + "runtime protocol negotiation failed: required={required_protocol}, supported={}..{}", + protocol.minimum_runtime_protocol, protocol.runtime_protocol + ), + )); + } + + RUNTIME + .set(NativeRuntime { + _library: library, + api_address: api as usize, + library_path, + minimum_protocol: protocol.minimum_runtime_protocol, + protocol: protocol.runtime_protocol, + }) + .map_err(|_| runtime_error(ruby, "runtime was loaded concurrently")) +} + +fn minimum_runtime_protocol(ruby: &Ruby) -> Result { + Ok(require_runtime(ruby)?.minimum_protocol) +} + +fn runtime_protocol(ruby: &Ruby) -> Result { + Ok(require_runtime(ruby)?.protocol) +} + +fn register_service( + ruby: &Ruby, + package_id: String, + component_id: String, + entry_symbol: String, + library_path: String, + required_protocol: u32, +) -> Result<(), Error> { + let runtime = require_runtime(ruby)?; + let package = byte_slice(package_id.as_bytes()); + let component = byte_slice(component_id.as_bytes()); + let entry = byte_slice(entry_symbol.as_bytes()); + let path = Path::new(&library_path) + .canonicalize() + .map_err(|error| runtime_error(ruby, error.to_string()))?; + let path = path.to_string_lossy(); + let path_slice = byte_slice(path.as_bytes()); + let registration = ServiceRegistrationV1 { + struct_size: size_of::(), + required_runtime_protocol: required_protocol, + package_id: package, + component_id: component, + entry_symbol: entry, + library_path: path_slice, + }; + let (mut error, error_storage) = output(4096); + let status = unsafe { (runtime.api().register_service)(®istration, &mut error) }; + if status != STATUS_OK { + return Err(runtime_error(ruby, output_message(&error, &error_storage))); + } + Ok(()) +} + +#[magnus::wrap(class = "OpenDal::Operator", free_immediately)] +struct Operator { + handle_address: AtomicUsize, +} + +impl Drop for Operator { + fn drop(&mut self) { + self.close(); + } +} + +impl Operator { + fn close(&self) { + let handle_address = self.handle_address.swap(0, Ordering::AcqRel); + if handle_address == 0 { + return; + } + if let Some(runtime) = RUNTIME.get() { + unsafe { + (runtime.api().operator_destroy)(handle_address as *mut c_void); + } + } + } + + fn require_handle(&self, ruby: &Ruby) -> Result<*mut c_void, Error> { + let handle_address = self.handle_address.load(Ordering::Acquire); + if handle_address == 0 { + return Err(runtime_error(ruby, "operator is closed")); + } + Ok(handle_address as *mut c_void) + } + + fn new( + ruby: &Ruby, + scheme: String, + options: Option>, + ) -> Result { + let runtime = require_runtime(ruby)?; + let scheme = scheme.trim().to_lowercase().replace('_', "-"); + let options = options.unwrap_or_default(); + let pairs: Vec = options + .iter() + .map(|(key, value)| KeyValue { + key: byte_slice(key.as_bytes()), + value: byte_slice(value.as_bytes()), + }) + .collect(); + let mut handle = ptr::null_mut(); + let (mut error, error_storage) = output(4096); + let status = unsafe { + (runtime.api().create_operator)( + byte_slice(scheme.as_bytes()), + pairs.as_ptr(), + pairs.len(), + &mut handle, + &mut error, + ) + }; + if status != STATUS_OK || handle.is_null() { + return Err(runtime_error(ruby, output_message(&error, &error_storage))); + } + Ok(Self { + handle_address: AtomicUsize::new(handle as usize), + }) + } + + fn close_ruby(operator: &Self) { + operator.close(); + } + + fn info_json(ruby: &Ruby, operator: &Self) -> Result { + let runtime = require_runtime(ruby)?; + let (mut result, result_storage) = output(4096); + let (mut error, error_storage) = output(4096); + let status = unsafe { + (runtime.api().operator_info)(operator.require_handle(ruby)?, &mut result, &mut error) + }; + if status != STATUS_OK { + return Err(runtime_error(ruby, output_message(&error, &error_storage))); + } + Ok(output_message(&result, &result_storage)) + } + + fn write(ruby: &Ruby, operator: &Self, path: String, data: RString) -> Result<(), Error> { + let runtime = require_runtime(ruby)?; + let (mut error, error_storage) = output(4096); + let data = data.to_bytes(); + let status = unsafe { + (runtime.api().operator_write)( + operator.require_handle(ruby)?, + byte_slice(path.as_bytes()), + byte_slice(&data), + &mut error, + ) + }; + if status != STATUS_OK { + return Err(runtime_error(ruby, output_message(&error, &error_storage))); + } + Ok(()) + } + + fn read(ruby: &Ruby, operator: &Self, path: String) -> Result { + let runtime = require_runtime(ruby)?; + let (mut result, mut result_storage) = output(4096); + let (mut error, error_storage) = output(4096); + let mut status = unsafe { + (runtime.api().operator_read)( + operator.require_handle(ruby)?, + byte_slice(path.as_bytes()), + &mut result, + &mut error, + ) + }; + if status == STATUS_BUFFER_TOO_SMALL { + (result, result_storage) = output(result.len); + status = unsafe { + (runtime.api().operator_read)( + operator.require_handle(ruby)?, + byte_slice(path.as_bytes()), + &mut result, + &mut error, + ) + }; + } + if status != STATUS_OK { + return Err(runtime_error(ruby, output_message(&error, &error_storage))); + } + result_storage.truncate(result.len); + Ok(result_storage.into()) + } +} + +#[magnus::init(name = "opendal_ruby_poc")] +fn init(ruby: &Ruby) -> Result<(), Error> { + let opendal = ruby.define_module("OpenDal")?; + let runtime = opendal.define_module("Runtime")?; + runtime.define_singleton_method("load", function!(load_runtime, 2))?; + runtime.define_singleton_method( + "minimum_runtime_protocol", + function!(minimum_runtime_protocol, 0), + )?; + runtime.define_singleton_method("runtime_protocol", function!(runtime_protocol, 0))?; + runtime.define_singleton_method("register_service", function!(register_service, 5))?; + + let operator = opendal.define_class("Operator", ruby.class_object())?; + operator.define_singleton_method("new", function!(Operator::new, 2))?; + operator.define_method("close", method!(Operator::close_ruby, 0))?; + operator.define_method("info_json", method!(Operator::info_json, 0))?; + operator.define_method("write", method!(Operator::write, 2))?; + operator.define_method("read", method!(Operator::read, 1))?; + Ok(()) +} diff --git a/bindings/ruby/dynamic-extensions/ruby/example.rb b/bindings/ruby/dynamic-extensions/ruby/example.rb new file mode 100644 index 000000000000..5379ca2c5a81 --- /dev/null +++ b/bindings/ruby/dynamic-extensions/ruby/example.rb @@ -0,0 +1,28 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# frozen_string_literal: true + +require "tmpdir" +require "opendal" +require "opendal/services/fs" + +Dir.mktmpdir("opendal-ruby-runtime-poc-") do |root| + operator = OpenDal::Operator.new("fs", {"root" => root}) + operator.write("hello.txt", "Hello from Ruby through the shared runtime!") + puts operator.info + puts operator.read("hello.txt") + operator.close +end diff --git a/bindings/ruby/dynamic-extensions/ruby/main/lib/opendal.rb b/bindings/ruby/dynamic-extensions/ruby/main/lib/opendal.rb new file mode 100644 index 000000000000..c0b9f775e1ad --- /dev/null +++ b/bindings/ruby/dynamic-extensions/ruby/main/lib/opendal.rb @@ -0,0 +1,37 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# frozen_string_literal: true + +require "json" +require_relative "opendal_ruby_poc" + +module OpenDal + REQUIRED_RUNTIME_PROTOCOL = Integer( + ENV.fetch("OPENDAL_POC_REQUIRED_RUNTIME_PROTOCOL", "1"), + 10 + ) + + Runtime.load( + File.expand_path("opendal/_native/libopendal_runtime_poc.so", __dir__), + REQUIRED_RUNTIME_PROTOCOL + ) + + class Operator + def info + JSON.parse(info_json) + end + end +end diff --git a/bindings/ruby/dynamic-extensions/ruby/python_cross_check.py b/bindings/ruby/dynamic-extensions/ruby/python_cross_check.py new file mode 100644 index 000000000000..24238c8aa48a --- /dev/null +++ b/bindings/ruby/dynamic-extensions/ruby/python_cross_check.py @@ -0,0 +1,26 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from tempfile import TemporaryDirectory + +import opendal.services.fs +from opendal import Operator + +with TemporaryDirectory(prefix="opendal-shared-artifact-poc-") as root: + with Operator("fs", root=root) as operator: + operator.write("python.txt", b"same runtime and FS artifacts") + assert operator.read("python.txt") == b"same runtime and FS artifacts" + +print({"adapter": "ctypes", "same_native_artifacts": True, "fs_round_trip": True}) diff --git a/bindings/ruby/dynamic-extensions/ruby/service-fs/lib/opendal/services/fs.rb b/bindings/ruby/dynamic-extensions/ruby/service-fs/lib/opendal/services/fs.rb new file mode 100644 index 000000000000..f8b6d641404e --- /dev/null +++ b/bindings/ruby/dynamic-extensions/ruby/service-fs/lib/opendal/services/fs.rb @@ -0,0 +1,39 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# frozen_string_literal: true + +require "opendal" + +module OpenDal + module Services + module Fs + MANIFEST = { + required_runtime_protocol: 1, + package_id: "opendal-service-fs-poc", + component_id: "fs", + native_entry_symbol: "opendal_service_fs_bootstrap_v1" + }.freeze + + Runtime.register_service( + MANIFEST.fetch(:package_id), + MANIFEST.fetch(:component_id), + MANIFEST.fetch(:native_entry_symbol), + File.expand_path("fs/_native/libfs_extension.so", __dir__), + MANIFEST.fetch(:required_runtime_protocol) + ) + end + end +end diff --git a/bindings/ruby/dynamic-extensions/ruby/test_poc.rb b/bindings/ruby/dynamic-extensions/ruby/test_poc.rb new file mode 100644 index 000000000000..b34844288765 --- /dev/null +++ b/bindings/ruby/dynamic-extensions/ruby/test_poc.rb @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# frozen_string_literal: true + +require "open3" +require "rbconfig" +require "tmpdir" +require "opendal" + +def assert(condition, message) + raise message unless condition +end + +def loaded?(basename) + File.read("/proc/self/maps").include?(basename) +end + +assert(OpenDal::Runtime.minimum_runtime_protocol == 1, "unexpected minimum protocol") +assert(OpenDal::Runtime.runtime_protocol == 1, "unexpected current protocol") +assert(!loaded?("libfs_extension.so"), "FS loaded before package registration") + +begin + OpenDal::Operator.new("fs", {}) + raise "unregistered FS construction succeeded" +rescue RuntimeError => error + assert(error.message.include?("not registered"), "unexpected registration error") +end + +require "opendal/services/fs" +assert(!loaded?("libfs_extension.so"), "FS loaded during package registration") + +Dir.mktmpdir("opendal-ruby-runtime-poc-") do |root| + operator = OpenDal::Operator.new("fs", {"root" => root}) + assert(loaded?("libfs_extension.so"), "FS did not load during construction") + payload = "ruby-runtime-poc-" * 512 + operator.write("hello.txt", payload) + assert(operator.read("hello.txt") == payload, "FS round trip failed") + assert(operator.info.fetch("scheme") == "fs", "unexpected operator scheme") + operator.close + + begin + operator.read("hello.txt") + raise "closed operator remained usable" + rescue RuntimeError => error + assert(error.message == "operator is closed", "unexpected closed-handle error") + end +end + +main = ENV.fetch("OPENDAL_RUBY_POC_MAIN") +env = {"OPENDAL_POC_REQUIRED_RUNTIME_PROTOCOL" => "2"} +_, stderr, status = Open3.capture3( + env, + RbConfig.ruby, + "-I#{main}", + "-ropendal", + "-e", + "abort 'incompatible runtime unexpectedly loaded'" +) +assert(!status.success?, "newer binding protocol unexpectedly loaded") +assert(stderr.include?("runtime protocol negotiation failed"), "missing protocol diagnostic") + +puts({ + adapter: "Magnus", + runtime_protocol: OpenDal::Runtime.runtime_protocol, + fs_lazy_load: true, + fs_round_trip: true, + closed_handle_rejected: true, + newer_protocol_rejected: true +}.inspect) diff --git a/bindings/ruby/dynamic-extensions/run-ruby-linux.sh b/bindings/ruby/dynamic-extensions/run-ruby-linux.sh new file mode 100755 index 000000000000..aa84027c7f6c --- /dev/null +++ b/bindings/ruby/dynamic-extensions/run-ruby-linux.sh @@ -0,0 +1,70 @@ +#!/usr/bin/env sh +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eu + +prototype_dir=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd) +shared_dir="$prototype_dir/../../python/dynamic-extensions" +target_root="$prototype_dir/target" +runtime_target="$target_root/shared-runtime" +fs_target="$target_root/shared-fs" +adapter_target="$target_root/ruby-adapter" +stage="$target_root/ruby-stage" + +if cargo tree --locked --offline --manifest-path "$prototype_dir/Cargo.toml" \ + --package opendal-ruby-runtime-poc \ + | rg -q 'opendal-core|opendal-service-|tokio'; then + echo "Ruby adapter unexpectedly links OpenDAL core, a service, or Tokio" >&2 + exit 1 +fi + +CARGO_TARGET_DIR="$runtime_target" cargo build --release --locked --offline \ + --manifest-path "$shared_dir/Cargo.toml" --package opendal-runtime-poc +CARGO_TARGET_DIR="$fs_target" cargo build --release --locked --offline \ + --manifest-path "$shared_dir/Cargo.toml" --package fs-extension +CARGO_TARGET_DIR="$adapter_target" cargo build --release --locked --offline \ + --manifest-path "$prototype_dir/Cargo.toml" --package opendal-ruby-runtime-poc + +runtime="$runtime_target/release/libopendal_runtime_poc.so" +fs="$fs_target/release/libfs_extension.so" +adapter="$adapter_target/release/libopendal_ruby_poc.so" + +"$shared_dir/audit-elf-exports.sh" "$runtime" opendal_runtime_get_api_v1 +"$shared_dir/audit-elf-exports.sh" "$fs" opendal_service_fs_bootstrap_v1 +"$shared_dir/audit-elf-exports.sh" \ + "$adapter" Init_opendal_ruby_poc ruby_abi_version + +mkdir -p \ + "$stage/main/lib/opendal/_native" \ + "$stage/fs/lib/opendal/services/fs/_native" \ + "$stage/python/main/opendal/_native" \ + "$stage/python/fs/opendal/services/fs/_native" +cp -R "$prototype_dir/ruby/main/." "$stage/main/" +cp -R "$prototype_dir/ruby/service-fs/." "$stage/fs/" +cp "$runtime" "$stage/main/lib/opendal/_native/" +cp "$adapter" "$stage/main/lib/opendal_ruby_poc.so" +cp "$fs" "$stage/fs/lib/opendal/services/fs/_native/" +cp -R "$shared_dir/python/main/." "$stage/python/main/" +cp -R "$shared_dir/python/service-fs/." "$stage/python/fs/" +cp "$runtime" "$stage/python/main/opendal/_native/" +cp "$fs" "$stage/python/fs/opendal/services/fs/_native/" + +RUBYLIB="$stage/main/lib:$stage/fs/lib" \ +OPENDAL_RUBY_POC_MAIN="$stage/main/lib" \ + ruby "$prototype_dir/ruby/test_poc.rb" +RUBYLIB="$stage/main/lib:$stage/fs/lib" ruby "$prototype_dir/ruby/example.rb" +PYTHONPATH="$stage/python/main:$stage/python/fs" \ + python3 "$prototype_dir/ruby/python_cross_check.py" diff --git a/core/services/s3/Cargo.toml b/core/services/s3/Cargo.toml index 7512b310a530..d747fd7de648 100644 --- a/core/services/s3/Cargo.toml +++ b/core/services/s3/Cargo.toml @@ -34,7 +34,7 @@ all-features = true [dependencies] base64 = { workspace = true } bytes = { workspace = true } -crc-fast = "1.9.0" +crc-fast = { version = "1.9.0", default-features = false, features = ["std"] } http = { workspace = true } log = { workspace = true } md-5 = "0.11.0"