Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions HISTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
## UNRELEASED

- Added streaming I/O via `FileCacheMode.streaming` for S3, Azure Blob Storage, Google Cloud Storage, and HTTP/HTTPS. In this mode `CloudPath.open()` returns the standard `io` file objects (`BufferedReader`/`BufferedWriter`/`TextIOWrapper`) backed by ranged reads and multipart uploads instead of the local cache, so only the bytes you read are downloaded and written data is uploaded as you write it. Read streams are seekable; append and update modes fall back to the cache and remove the cache file on close. `buffering` has its builtin `open()` meaning and sets the ranged-request size (default 5 MiB). A new `streaming_max_concurrency` client option (default 4) bounds background read-ahead and part uploads per stream; the buffer size, concurrency, and multipart part size can also be set with the `CLOUDPATHLIB_STREAMING_BUFFER_SIZE`, `CLOUDPATHLIB_STREAMING_MAX_CONCURRENCY`, and `CLOUDPATHLIB_[<PROVIDER>_]STREAMING_PART_SIZE` environment variables. Opening a file for append or update in streaming mode warns that it falls back to the cache. `copy`/`rename`/`replace` stream between clients, `fspath` raises `CloudPathNotImplementedError`, and failures raise `cloudpathlib.exceptions` types including the new `CloudPathStreamingError`. See the [Streaming I/O](https://cloudpathlib.drivendata.org/stable/streaming_io/) docs. (PR [#535](https://github.com/drivendataorg/cloudpathlib/pull/535))
- Changed append mode (`a`, `ab`, `a+`, and `ab+`) to create missing cloud files, matching Python file behavior. Appending to existing files still preserves their content. (Issue [#585](https://github.com/drivendataorg/cloudpathlib/issues/585))

## v0.26.0 (2026-10-01)

Expand Down
4 changes: 2 additions & 2 deletions cloudpathlib/cloudpath.py
Original file line number Diff line number Diff line change
Expand Up @@ -839,9 +839,9 @@ def open(
f"Cannot open directory, only files. Tried to open ({self})"
)

if not exists_on_cloud and any(m in mode for m in ("r", "a")):
if not exists_on_cloud and "r" in mode:
raise CloudPathFileNotFoundError(
f"File opened for read or append, but it does not exist on cloud: {self}"
f"File opened for read, but it does not exist on cloud: {self}"
)

if "x" in mode and exists_on_cloud:
Expand Down
42 changes: 42 additions & 0 deletions tests/test_cloudpath_file_io.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@

from cloudpathlib.cloudpath import _ensure_local_path_within_base
from cloudpathlib.exceptions import (
CloudPathFileNotFoundError,
CloudPathLocalPathTraversalError,
CloudPathNotExistsError,
CloudPathIsADirectoryError,
Expand Down Expand Up @@ -619,6 +620,47 @@ def test_file_read_writes(rig, tmp_path):
(p / "not_exists_file").download_to(dl_file)


@pytest.mark.parametrize("mode", ["a", "ab", "a+", "ab+"])
def test_append_creates_missing_file_and_preserves_content(rig, tmp_path, mode):
p = rig.create_cloud_path("new_dir/append.txt")
assert not p.exists()
content = b"first" if "b" in mode else "first"

with p.open(mode) as f:
f.write(content)

assert p.download_to(tmp_path / "created.txt").read_bytes() == b"first"
p.clear_cache()

with p.open(mode) as f:
if "+" in mode:
f.seek(0)
assert f.read() == content
f.write(content)

assert p.download_to(tmp_path / "appended.txt").read_bytes() == b"firstfirst"


@pytest.mark.parametrize("mode", ["a", "ab", "a+", "ab+"])
def test_append_creates_empty_file_on_close(rig, tmp_path, mode):
p = rig.create_cloud_path("new_dir/empty.txt")
assert not p.exists()

with p.open(mode):
pass

assert p.exists()
assert p.download_to(tmp_path / "empty.txt").read_bytes() == b""


@pytest.mark.parametrize("mode", ["r", "rb", "r+", "rb+"])
def test_read_missing_file_still_raises(rig, mode):
p = rig.create_cloud_path("missing.txt")

with pytest.raises(CloudPathFileNotFoundError):
p.open(mode)


def test_filenames(rig):
# test that we can handle filenames with special characters
p = rig.create_cloud_path("dir_0/new_file.txt") # real extension
Expand Down
Loading