From dfe83c4f3f4e650c3da7a76ab2587b42afb0e285 Mon Sep 17 00:00:00 2001 From: Xuanwo Date: Fri, 24 Jul 2026 20:34:44 +0800 Subject: [PATCH 1/3] docs: propose limited read --- core/core/src/docs/rfcs/0000_limited_read.md | 270 +++++++++++++++++++ core/core/src/docs/rfcs/mod.rs | 4 + 2 files changed, 274 insertions(+) create mode 100644 core/core/src/docs/rfcs/0000_limited_read.md diff --git a/core/core/src/docs/rfcs/0000_limited_read.md b/core/core/src/docs/rfcs/0000_limited_read.md new file mode 100644 index 000000000000..3a10f5e92336 --- /dev/null +++ b/core/core/src/docs/rfcs/0000_limited_read.md @@ -0,0 +1,270 @@ +- Proposal Name: `limited_read` +- Start Date: 2026-07-24 +- RFC PR: [apache/opendal#0000](https://github.com/apache/opendal/pull/0000) +- Tracking Issue: [apache/opendal#7938](https://github.com/apache/opendal/issues/7938) + +# Summary + +Add `limit` to `Operator::read_with` and `ReadOptions`. + +`range` continues to request an exact bounded range. `limit` caps the amount of +data returned but accepts a clean end of file before the cap. This supports +small probes, such as reading up to 16 KiB from an object whose size is unknown, +without a preceding `stat`. + +The implementation reuses the existing read path. It adds no raw operation or +new planning type. Core only carries an `exact` boolean so the component that +collects a stream knows whether a clean short read is an error. + +# Motivation + +Callers often need only the beginning of an object: + +- inspect a file signature or header; +- detect a format; +- parse metadata stored near the beginning; +- sample a small object without first knowing its size. + +Today a caller can use `range(0..N)`, but a bounded range is exact. Reading an +object shorter than `N` fails even though the bytes that do exist are sufficient +for these use cases. The caller can avoid the error by calling `stat` first and +then choosing a smaller range, but that adds a request and introduces a race +between metadata lookup and data access. + +OpenDAL needs an at-most read alongside its existing exact range read. + +# Guide-level explanation + +Use `limit` when any number of bytes from zero through the limit is a successful +result: + +```rust +let header = op + .read_with("path/to/file") + .limit(16 * 1024) + .await?; +``` + +If the object contains at least 16 KiB, this returns 16 KiB. If it contains less, +this returns the whole object. OpenDAL does not perform a `stat` before reading. + +`limit` can start at an offset: + +```rust +let data = op + .read_with("path/to/file") + .range(4096..) + .limit(1024) + .await?; +``` + +This returns at most 1024 bytes starting at offset 4096. If the offset is at or +beyond the end of the object, the result is empty. + +`range` without `limit` keeps its current behavior: + +```rust +let data = op + .read_with("path/to/file") + .range(0..16 * 1024) + .await?; +``` + +This succeeds only when the complete 16 KiB range is available. A clean end of +file before the requested end remains an error. + +For a non-empty limit, `limit` changes only clean end-of-file handling. A +missing object, a failed condition, a permission failure, or a transport error +still fails the read. + +# Reference-level explanation + +## Public API + +`ReadOptions` gains one field: + +```rust,ignore +pub struct ReadOptions { + pub range: BytesRange, + pub limit: Option, + // Existing fields. +} +``` + +`FutureRead` gains the matching builder: + +```rust,ignore +pub fn limit(mut self, limit: u64) -> Self { + self.args.limit = Some(limit); + self +} +``` + +`limit` applies after `range`. Core converts the two options into one physical +`BytesRange`: + +| Options | Physical range | Completion | +| --- | --- | --- | +| no `range`, `limit(n)` | offset 0, size `n` | at most `n` | +| `range(offset..)`, `limit(n)` | offset `offset`, size `n` | at most `n` | +| `range(start..end)`, `limit(n)` | offset `start`, size `min(end - start, n)` | at most that size | +| bounded `range` without `limit` | unchanged | exact | + +The initial API rejects combining a suffix range with `limit`. A suffix is +relative to the object end, so resolving it without metadata would defeat the +no-`stat` property. + +`limit(0)` returns an empty buffer without checking the object, consistent with +an empty range. + +Combining `limit` with a suffix range, an explicit `chunk` size, or a +`concurrent` value greater than one returns `ErrorKind::ConfigInvalid` before +storage I/O. + +## Exact completion + +Core carries a private boolean named `exact` with the normalized range. It does +not introduce a public type or a new raw abstraction. + +- A regular bounded, non-suffix range sets `exact` to `true`. +- A read with `limit` sets `exact` to `false`. +- An open-ended or suffix range sets `exact` to `false`. + +The buffer stream tracks the number of bytes it yields. At clean end of file, it +compares that count with the bounded range size only when `exact` is `true`. +This preserves current exact range behavior while allowing limited reads to +finish early. + +The `exact` flag belongs to the public read execution path. It is not passed to +services because services should not decide whether a caller accepts a short +result. + +## Raw read contract + +This proposal keeps the raw `oio::Read` API unchanged: + +```rust,ignore +pub trait Read { + fn open(&self, range: BytesRange) -> ...; + fn read(&self, range: BytesRange) -> ...; +} +``` + +The two existing methods already provide the required split: + +- `open(range)` returns the bytes available inside the range, stops at the range + boundary, and treats a clean end of file as normal stream completion. +- `read(range)` remains an exact bounded read for chunked and concurrent + planning. It returns the complete range or an error. + +`PositionReadStream` therefore treats an empty positioned read as clean EOF, +while `PositionReader::read` keeps rejecting EOF before its exact bounded read +is complete. Stream-based services follow the same contract. + +When a service reports that a range starts at or beyond EOF, its `open` path +normalizes that response to an empty stream if the object is known to exist. +The core exactness check then rejects the result for an exact range and accepts +it for a limited read. A missing object remains `NotFound`. + +`CompleteLayer` also separates these responsibilities. Its `read(range)` path +continues to require the exact bounded size. Its `open(range)` path validates +the service stream rather than imposing public exactness: + +- it always rejects bytes beyond the requested range; +- when `RpRead` contains the full object length, it requires exactly the bytes + available in the requested range; +- without that metadata, it relies on the service to distinguish clean EOF from + a truncated response. + +HTTP services still validate the response body's `Content-Length`, so accepting +object EOF does not turn a truncated network response into success. + +## Execution + +A limited read opens one bounded stream and collects it. It does not split the +limit into speculative exact reads because the final chunk may legitimately +cross the object end. + +The initial implementation rejects `limit` together with an explicit `chunk` +size or `concurrent` value greater than one. This keeps their exact chunk +contract intact instead of silently changing or ignoring execution options. +Limited reads target small probes; support for parallel limited reads can be +added later if a real workload requires it. + +No capability flag is needed. Every readable service already supports the +`open(range)` path needed by this API. + +`presign_read_options` rejects `limit`. A presigned request is executed by the +caller, so OpenDAL cannot apply its completion check to the response. + +## Compatibility and validation + +Existing reads do not set `limit`, so their normalized ranges and behavior +remain unchanged. As with other additions to `ReadOptions`, callers that +construct the public struct without `..Default::default()` must initialize the +new field. + +Implementation tests should cover: + +- objects shorter than, equal to, and longer than the limit; +- `range(offset..).limit(n)` before, at, and beyond EOF; +- unchanged failure for a short exact range; +- unchanged propagation of non-EOF errors; +- rejection of suffix, chunked, and concurrent combinations; +- detection of a truncated HTTP body; +- both stream-based and positioned-read services. + +# Drawbacks + +`range` and `limit` are similar size controls with deliberately different EOF +semantics. Documentation must make the exact-versus-at-most distinction clear. + +The first implementation does not combine limited reads with suffix ranges or +parallel chunk planning. + +# Rationale and alternatives + +## Add a separate operation + +A method such as `read_up_to` would make the semantic difference obvious, but it +would duplicate all existing read options and their builder surface. `limit` +fits the current `read_with` model and composes naturally with an offset range. + +## Change bounded ranges to accept EOF + +Rejected. Exact bounded ranges are useful for validating file structure and for +safe concurrent chunk planning. Changing their behavior would weaken an +existing contract. + +## Call `stat` before a range read + +Rejected. It adds latency and cannot make the subsequent read atomic with the +metadata result. Services can already report the available range through the +read response. + +## Add a raw method or planning type + +Rejected. The existing `open` and `read` methods already distinguish streaming +from exact materialization. A local `exact` boolean expresses the only missing +decision without adding `ReadPlan`, `ReadCompletion`, or another service-facing +contract. + +# Prior art + +Rust I/O adapters commonly use a limit to cap bytes while treating EOF before +that limit as normal. This proposal applies the same expectation to OpenDAL's +read request construction. + +RFC-0090 introduced limited readers to prevent over-reading, while RFC-7660 +separated range streams from exact bounded raw reads. This proposal builds on +those boundaries and adds the missing public at-most completion semantics. + +# Unresolved questions + +None. + +# Future possibilities + +OpenDAL can support suffix ranges or parallel planning with `limit` later if it +can preserve at-most completion without a metadata request or speculative +requests beyond EOF. diff --git a/core/core/src/docs/rfcs/mod.rs b/core/core/src/docs/rfcs/mod.rs index 8437a3dde4c5..49afad0e1b73 100644 --- a/core/core/src/docs/rfcs/mod.rs +++ b/core/core/src/docs/rfcs/mod.rs @@ -21,6 +21,10 @@ #[doc = include_str!("0000_example.md")] pub mod rfc_0000_example {} +/// Limited Read +#[doc = include_str!("0000_limited_read.md")] +pub mod rfc_0000_limited_read {} + /// Rename If Not Exists #[doc = include_str!("7818_rename_if_not_exists.md")] pub mod rfc_7818_rename_if_not_exists {} From 3c40d7e10c406a5f85df00393e15cd3eb32ecd71 Mon Sep 17 00:00:00 2001 From: Xuanwo Date: Fri, 24 Jul 2026 20:35:39 +0800 Subject: [PATCH 2/3] docs: assign RFC number 7945 --- .../docs/rfcs/{0000_limited_read.md => 7945_limited_read.md} | 2 +- core/core/src/docs/rfcs/mod.rs | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) rename core/core/src/docs/rfcs/{0000_limited_read.md => 7945_limited_read.md} (99%) diff --git a/core/core/src/docs/rfcs/0000_limited_read.md b/core/core/src/docs/rfcs/7945_limited_read.md similarity index 99% rename from core/core/src/docs/rfcs/0000_limited_read.md rename to core/core/src/docs/rfcs/7945_limited_read.md index 3a10f5e92336..9b52b623896d 100644 --- a/core/core/src/docs/rfcs/0000_limited_read.md +++ b/core/core/src/docs/rfcs/7945_limited_read.md @@ -1,6 +1,6 @@ - Proposal Name: `limited_read` - Start Date: 2026-07-24 -- RFC PR: [apache/opendal#0000](https://github.com/apache/opendal/pull/0000) +- RFC PR: [apache/opendal#7945](https://github.com/apache/opendal/pull/7945) - Tracking Issue: [apache/opendal#7938](https://github.com/apache/opendal/issues/7938) # Summary diff --git a/core/core/src/docs/rfcs/mod.rs b/core/core/src/docs/rfcs/mod.rs index 49afad0e1b73..eb9a8cb4cef1 100644 --- a/core/core/src/docs/rfcs/mod.rs +++ b/core/core/src/docs/rfcs/mod.rs @@ -22,8 +22,8 @@ pub mod rfc_0000_example {} /// Limited Read -#[doc = include_str!("0000_limited_read.md")] -pub mod rfc_0000_limited_read {} +#[doc = include_str!("7945_limited_read.md")] +pub mod rfc_7945_limited_read {} /// Rename If Not Exists #[doc = include_str!("7818_rename_if_not_exists.md")] From 6fd37700fbad75e4e0a3670d823aab23dc9c3146 Mon Sep 17 00:00:00 2001 From: Xuanwo Date: Fri, 31 Jul 2026 17:39:30 +0800 Subject: [PATCH 3/3] docs: refine limited read semantics --- core/core/src/docs/rfcs/7945_limited_read.md | 226 ++++++++----------- 1 file changed, 96 insertions(+), 130 deletions(-) diff --git a/core/core/src/docs/rfcs/7945_limited_read.md b/core/core/src/docs/rfcs/7945_limited_read.md index 9b52b623896d..06796f9c3aeb 100644 --- a/core/core/src/docs/rfcs/7945_limited_read.md +++ b/core/core/src/docs/rfcs/7945_limited_read.md @@ -7,36 +7,28 @@ Add `limit` to `Operator::read_with` and `ReadOptions`. -`range` continues to request an exact bounded range. `limit` caps the amount of -data returned but accepts a clean end of file before the cap. This supports -small probes, such as reading up to 16 KiB from an object whose size is unknown, -without a preceding `stat`. +`range` continues to request an exact bounded range. `limit` caps the bytes +returned from the selected range and accepts EOF before the cap when the range +has a satisfiable starting position. A non-empty forward range that starts at or +beyond EOF remains `RangeNotSatisfied`. -The implementation reuses the existing read path. It adds no raw operation or -new planning type. Core only carries an `exact` boolean so the component that -collects a stream knows whether a clean short read is an error. +Core lowers the cap into the service range when possible and carries a private +`exact` boolean while collecting the stream. The design adds no raw operation or +service-facing completion mode. # Motivation -Callers often need only the beginning of an object: +Callers often need only an object's signature, header, or other leading metadata +to detect its format. -- inspect a file signature or header; -- detect a format; -- parse metadata stored near the beginning; -- sample a small object without first knowing its size. - -Today a caller can use `range(0..N)`, but a bounded range is exact. Reading an -object shorter than `N` fails even though the bytes that do exist are sufficient -for these use cases. The caller can avoid the error by calling `stat` first and -then choosing a smaller range, but that adds a request and introduces a race -between metadata lookup and data access. - -OpenDAL needs an at-most read alongside its existing exact range read. +Today a caller can use `range(0..N)`, but a bounded range is exact. When its +starting byte exists but its end crosses EOF, the read fails even if the +available bytes are sufficient. Calling `stat` first avoids that error but adds +a request and introduces a race between metadata lookup and data access. # Guide-level explanation -Use `limit` when any number of bytes from zero through the limit is a successful -result: +Use `limit` when fewer bytes than the cap are useful: ```rust let header = op @@ -45,8 +37,9 @@ let header = op .await?; ``` -If the object contains at least 16 KiB, this returns 16 KiB. If it contains less, -this returns the whole object. OpenDAL does not perform a `stat` before reading. +If the object contains at least 16 KiB, this returns 16 KiB. If its non-empty +content is shorter, this returns the whole object. OpenDAL does not perform a +`stat` before reading. `limit` can start at an offset: @@ -58,8 +51,20 @@ let data = op .await?; ``` -This returns at most 1024 bytes starting at offset 4096. If the offset is at or -beyond the end of the object, the result is empty. +This returns at most 1024 bytes from offset 4096. A valid starting byte followed +by EOF is successful; an offset at or beyond EOF returns `RangeNotSatisfied`. + +`limit` also applies after a suffix range: + +```rust +let data = op + .read_with("path/to/file") + .range(BytesRange::suffix(1024)) + .limit(512) + .await?; +``` + +This returns up to the first 512 bytes from the selected suffix. `range` without `limit` keeps its current behavior: @@ -70,12 +75,8 @@ let data = op .await?; ``` -This succeeds only when the complete 16 KiB range is available. A clean end of -file before the requested end remains an error. - -For a non-empty limit, `limit` changes only clean end-of-file handling. A -missing object, a failed condition, a permission failure, or a transport error -still fails the read. +This still requires the complete 16 KiB range. Missing objects, failed +conditions, permission failures, and transport errors also remain errors. # Reference-level explanation @@ -100,44 +101,41 @@ pub fn limit(mut self, limit: u64) -> Self { } ``` -`limit` applies after `range`. Core converts the two options into one physical -`BytesRange`: +`limit` applies after `range`. Core pushes the cap into the service range when +the range has an absolute start: -| Options | Physical range | Completion | +| Options | Service range | Completion | | --- | --- | --- | | no `range`, `limit(n)` | offset 0, size `n` | at most `n` | | `range(offset..)`, `limit(n)` | offset `offset`, size `n` | at most `n` | | `range(start..end)`, `limit(n)` | offset `start`, size `min(end - start, n)` | at most that size | +| `suffix(s)`, `limit(n)` | suffix `s` | first `n` bytes of the selected suffix | | bounded `range` without `limit` | unchanged | exact | -The initial API rejects combining a suffix range with `limit`. A suffix is -relative to the object end, so resolving it without metadata would defeat the -no-`stat` property. +A suffix range has no absolute start until the service knows the object length. +Core therefore opens the original suffix and caps the returned stream without a +`stat`. This preserves the public limit, but a service may transfer more than +the limit when its protocol cannot express a suffix with a relative end. `limit(0)` returns an empty buffer without checking the object, consistent with -an empty range. +an empty range. A non-zero forward limit on an empty object is +`RangeNotSatisfied` because its first byte does not exist. -Combining `limit` with a suffix range, an explicit `chunk` size, or a -`concurrent` value greater than one returns `ErrorKind::ConfigInvalid` before -storage I/O. +Combining `limit` with an explicit `chunk` size or a `concurrent` value greater +than one returns `ErrorKind::ConfigInvalid` before storage I/O. ## Exact completion -Core carries a private boolean named `exact` with the normalized range. It does -not introduce a public type or a new raw abstraction. - -- A regular bounded, non-suffix range sets `exact` to `true`. -- A read with `limit` sets `exact` to `false`. -- An open-ended or suffix range sets `exact` to `false`. - -The buffer stream tracks the number of bytes it yields. At clean end of file, it -compares that count with the bounded range size only when `exact` is `true`. -This preserves current exact range behavior while allowing limited reads to -finish early. +Core carries a private boolean named `exact`: it is `true` for a regular bounded +non-suffix range and `false` for a limited, open-ended, or suffix range. The +buffer stream tracks emitted bytes, slices the final buffer at the limit, and +stops at the cap. At EOF, it checks the bounded range size only when `exact` is +`true`. -The `exact` flag belongs to the public read execution path. It is not passed to -services because services should not decide whether a caller accepts a short -result. +Services receive the lowered `BytesRange` and the existing operation choice: +`open` for a stream or `read` for exact bounded materialization. The `exact` +flag stays in core because it changes completion acceptance, not the storage +request. ## Raw read contract @@ -152,112 +150,80 @@ pub trait Read { The two existing methods already provide the required split: -- `open(range)` returns the bytes available inside the range, stops at the range - boundary, and treats a clean end of file as normal stream completion. +- `open(range)` returns the bytes available inside a satisfiable range and never + crosses its boundary. For a non-empty bounded forward range, EOF after at + least one requested byte is clean stream completion; an offset at or beyond + EOF is `RangeNotSatisfied`. - `read(range)` remains an exact bounded read for chunked and concurrent planning. It returns the complete range or an error. -`PositionReadStream` therefore treats an empty positioned read as clean EOF, -while `PositionReader::read` keeps rejecting EOF before its exact bounded read -is complete. Stream-based services follow the same contract. - -When a service reports that a range starts at or beyond EOF, its `open` path -normalizes that response to an empty stream if the object is known to exist. -The core exactness check then rejects the result for an exact range and accepts -it for a limited read. A missing object remains `NotFound`. - -`CompleteLayer` also separates these responsibilities. Its `read(range)` path -continues to require the exact bounded size. Its `open(range)` path validates -the service stream rather than imposing public exactness: +`PositionReadStream` returns `RangeNotSatisfied` if its first read for a +non-empty bounded range returns no bytes, but treats a later empty read as clean +completion. `PositionReader::read` keeps rejecting any EOF before its exact +bounded read completes. Stream-based services follow the same rule. -- it always rejects bytes beyond the requested range; -- when `RpRead` contains the full object length, it requires exactly the bytes - available in the requested range; -- without that metadata, it relies on the service to distinguish clean EOF from - a truncated response. +`CompleteLayer::read` continues to require the exact bounded size. +`CompleteLayer::open` rejects bytes beyond the requested range and, when +`RpRead` contains the full object length, requires exactly the bytes available +in a satisfiable range. Without that metadata, it relies on the service to +distinguish clean EOF from a truncated response. HTTP services still validate the response body's `Content-Length`, so accepting object EOF does not turn a truncated network response into success. ## Execution -A limited read opens one bounded stream and collects it. It does not split the -limit into speculative exact reads because the final chunk may legitimately -cross the object end. +A limited read opens one stream and collects until the limit or EOF; it does not +issue speculative exact chunks across EOF. An explicit `chunk` size or +`concurrent` value greater than one is `ConfigInvalid`. No capability is needed +because every readable service supports `open(range)`. -The initial implementation rejects `limit` together with an explicit `chunk` -size or `concurrent` value greater than one. This keeps their exact chunk -contract intact instead of silently changing or ignoring execution options. -Limited reads target small probes; support for parallel limited reads can be -added later if a real workload requires it. - -No capability flag is needed. Every readable service already supports the -`open(range)` path needed by this API. - -`presign_read_options` rejects `limit`. A presigned request is executed by the -caller, so OpenDAL cannot apply its completion check to the response. +`presign_read_options` also rejects `limit` because OpenDAL cannot apply its +completion check to a response executed by the caller. ## Compatibility and validation -Existing reads do not set `limit`, so their normalized ranges and behavior -remain unchanged. As with other additions to `ReadOptions`, callers that -construct the public struct without `..Default::default()` must initialize the -new field. - -Implementation tests should cover: +Existing reads omit `limit` and retain their behavior. Callers that construct +`ReadOptions` without `..Default::default()` must initialize the new field. -- objects shorter than, equal to, and longer than the limit; -- `range(offset..).limit(n)` before, at, and beyond EOF; -- unchanged failure for a short exact range; -- unchanged propagation of non-EOF errors; -- rejection of suffix, chunked, and concurrent combinations; -- detection of a truncated HTTP body; -- both stream-based and positioned-read services. +Tests must cover forward and suffix ranges around the limit and EOF, including +`RangeNotSatisfied` at or beyond EOF, unchanged exact-range and non-EOF errors, +invalid chunked or concurrent combinations, and truncated HTTP bodies. Both +stream-based and positioned-read services need coverage. # Drawbacks -`range` and `limit` are similar size controls with deliberately different EOF -semantics. Documentation must make the exact-versus-at-most distinction clear. - -The first implementation does not combine limited reads with suffix ranges or -parallel chunk planning. +`range` and `limit` are similar size controls with different EOF semantics. +Suffix ranges may transfer more data than OpenDAL returns, and the initial +implementation does not support parallel chunk planning. # Rationale and alternatives ## Add a separate operation -A method such as `read_up_to` would make the semantic difference obvious, but it -would duplicate all existing read options and their builder surface. `limit` -fits the current `read_with` model and composes naturally with an offset range. +Rejected. `read_up_to` would duplicate the existing read options and builder +surface; `limit` composes with them directly. ## Change bounded ranges to accept EOF -Rejected. Exact bounded ranges are useful for validating file structure and for -safe concurrent chunk planning. Changing their behavior would weaken an -existing contract. +Rejected. Exact bounded ranges validate file structure and support safe +concurrent chunk planning. ## Call `stat` before a range read -Rejected. It adds latency and cannot make the subsequent read atomic with the -metadata result. Services can already report the available range through the -read response. +Rejected. It adds latency and cannot make the read atomic with its metadata. ## Add a raw method or planning type -Rejected. The existing `open` and `read` methods already distinguish streaming -from exact materialization. A local `exact` boolean expresses the only missing -decision without adding `ReadPlan`, `ReadCompletion`, or another service-facing -contract. +Rejected. `open`, `read`, and the lowered range already express the storage +request. A local `exact` boolean expresses the remaining core decision. # Prior art -Rust I/O adapters commonly use a limit to cap bytes while treating EOF before -that limit as normal. This proposal applies the same expectation to OpenDAL's -read request construction. - -RFC-0090 introduced limited readers to prevent over-reading, while RFC-7660 -separated range streams from exact bounded raw reads. This proposal builds on -those boundaries and adds the missing public at-most completion semantics. +Rust I/O adapters commonly cap bytes while treating EOF before the cap as +normal. RFC-0090 addressed over-reading, and RFC-7660 separated range streams +from exact bounded raw reads; this proposal adds the public at-most completion +semantics. # Unresolved questions @@ -265,6 +231,6 @@ None. # Future possibilities -OpenDAL can support suffix ranges or parallel planning with `limit` later if it -can preserve at-most completion without a metadata request or speculative -requests beyond EOF. +OpenDAL can add parallel planning for limited reads or service-specific +suffix-limit pushdown when those implementations preserve the public completion +and range-satisfaction contracts.