Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions HISTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

## UNRELEASED

- Delegated `AWS_ENDPOINT_URL` handling to boto3 now that the AWS SDK supports endpoint environment variables directly (Issue [#194](https://github.com/drivendataorg/cloudpathlib/issues/194)).
- Added streaming I/O via `FileCacheMode.streaming` for S3, Azure Blob Storage, Google Cloud Storage, and HTTP/HTTPS. In this mode `CloudPath.open()` returns the standard `io` file objects (`BufferedReader`/`BufferedWriter`/`TextIOWrapper`) backed by ranged reads and multipart uploads instead of the local cache, so only the bytes you read are downloaded and written data is uploaded as you write it. Read streams are seekable; append and update modes fall back to the cache and remove the cache file on close. `buffering` has its builtin `open()` meaning and sets the ranged-request size (default 5 MiB). A new `streaming_max_concurrency` client option (default 4) bounds background read-ahead and part uploads per stream; the buffer size, concurrency, and multipart part size can also be set with the `CLOUDPATHLIB_STREAMING_BUFFER_SIZE`, `CLOUDPATHLIB_STREAMING_MAX_CONCURRENCY`, and `CLOUDPATHLIB_[<PROVIDER>_]STREAMING_PART_SIZE` environment variables. Opening a file for append or update in streaming mode warns that it falls back to the cache. `copy`/`rename`/`replace` stream between clients, `fspath` raises `CloudPathNotImplementedError`, and failures raise `cloudpathlib.exceptions` types including the new `CloudPathStreamingError`. See the [Streaming I/O](https://cloudpathlib.drivendata.org/stable/streaming_io/) docs. (PR [#535](https://github.com/drivendataorg/cloudpathlib/pull/535))
- Added `ignore_errors`, `onerror`, and `onexc` options to `CloudPath.rmtree()` for suppressing deletion errors or handling them with a callback. Calls without these options still raise errors. (Issue [#270](https://github.com/drivendataorg/cloudpathlib/issues/270), PR [#604](https://github.com/drivendataorg/cloudpathlib/pull/604))
- Changed append mode (`a`, `ab`, `a+`, and `ab+`) to create missing cloud files, matching Python file behavior. Appending to existing files still preserves their content. (Issue [#585](https://github.com/drivendataorg/cloudpathlib/issues/585))
Expand Down
9 changes: 5 additions & 4 deletions tests/test_s3_specific.py
Original file line number Diff line number Diff line change
Expand Up @@ -420,12 +420,10 @@ def test_addressing_style_virtual_live(s3_rig):


def test_aws_endpoint_url_env(monkeypatch):
"""Allows setting endpoint_url from env variable
until upstream boto3 PR is merged.
https://github.com/boto/boto3/pull/2746
"""
"""Delegates AWS_ENDPOINT_URL handling to boto3."""
s3_url = "https://s3.amazonaws.com"
localstack_url = "http://localhost:4566"
explicit_url = "http://localhost:9000"

s3_client = S3Client()
assert s3_client.client.meta.endpoint_url == s3_url
Expand All @@ -434,6 +432,9 @@ def test_aws_endpoint_url_env(monkeypatch):
s3_client_custom_endpoint = S3Client()
assert s3_client_custom_endpoint.client.meta.endpoint_url == localstack_url

s3_client_explicit_endpoint = S3Client(endpoint_url=explicit_url)
assert s3_client_explicit_endpoint.client.meta.endpoint_url == explicit_url


def test_as_url_local(monkeypatch):
path = S3Path("s3://arxiv/pdf")
Expand Down
Loading