diff --git a/HISTORY.md b/HISTORY.md index d5de5419..dff43254 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -2,6 +2,7 @@ ## UNRELEASED +- Delegated `AWS_ENDPOINT_URL` handling to boto3 now that the AWS SDK supports endpoint environment variables directly (Issue [#194](https://github.com/drivendataorg/cloudpathlib/issues/194)). - Added streaming I/O via `FileCacheMode.streaming` for S3, Azure Blob Storage, Google Cloud Storage, and HTTP/HTTPS. In this mode `CloudPath.open()` returns the standard `io` file objects (`BufferedReader`/`BufferedWriter`/`TextIOWrapper`) backed by ranged reads and multipart uploads instead of the local cache, so only the bytes you read are downloaded and written data is uploaded as you write it. Read streams are seekable; append and update modes fall back to the cache and remove the cache file on close. `buffering` has its builtin `open()` meaning and sets the ranged-request size (default 5 MiB). A new `streaming_max_concurrency` client option (default 4) bounds background read-ahead and part uploads per stream; the buffer size, concurrency, and multipart part size can also be set with the `CLOUDPATHLIB_STREAMING_BUFFER_SIZE`, `CLOUDPATHLIB_STREAMING_MAX_CONCURRENCY`, and `CLOUDPATHLIB_[_]STREAMING_PART_SIZE` environment variables. Opening a file for append or update in streaming mode warns that it falls back to the cache. `copy`/`rename`/`replace` stream between clients, `fspath` raises `CloudPathNotImplementedError`, and failures raise `cloudpathlib.exceptions` types including the new `CloudPathStreamingError`. See the [Streaming I/O](https://cloudpathlib.drivendata.org/stable/streaming_io/) docs. (PR [#535](https://github.com/drivendataorg/cloudpathlib/pull/535)) - Added `ignore_errors`, `onerror`, and `onexc` options to `CloudPath.rmtree()` for suppressing deletion errors or handling them with a callback. Calls without these options still raise errors. (Issue [#270](https://github.com/drivendataorg/cloudpathlib/issues/270), PR [#604](https://github.com/drivendataorg/cloudpathlib/pull/604)) - Changed append mode (`a`, `ab`, `a+`, and `ab+`) to create missing cloud files, matching Python file behavior. Appending to existing files still preserves their content. (Issue [#585](https://github.com/drivendataorg/cloudpathlib/issues/585)) diff --git a/tests/test_s3_specific.py b/tests/test_s3_specific.py index 348c5c91..0d1e6a40 100644 --- a/tests/test_s3_specific.py +++ b/tests/test_s3_specific.py @@ -420,12 +420,10 @@ def test_addressing_style_virtual_live(s3_rig): def test_aws_endpoint_url_env(monkeypatch): - """Allows setting endpoint_url from env variable - until upstream boto3 PR is merged. - https://github.com/boto/boto3/pull/2746 - """ + """Delegates AWS_ENDPOINT_URL handling to boto3.""" s3_url = "https://s3.amazonaws.com" localstack_url = "http://localhost:4566" + explicit_url = "http://localhost:9000" s3_client = S3Client() assert s3_client.client.meta.endpoint_url == s3_url @@ -434,6 +432,9 @@ def test_aws_endpoint_url_env(monkeypatch): s3_client_custom_endpoint = S3Client() assert s3_client_custom_endpoint.client.meta.endpoint_url == localstack_url + s3_client_explicit_endpoint = S3Client(endpoint_url=explicit_url) + assert s3_client_explicit_endpoint.client.meta.endpoint_url == explicit_url + def test_as_url_local(monkeypatch): path = S3Path("s3://arxiv/pdf")