From bc309fd525fde194339f03ecaf29823990038810 Mon Sep 17 00:00:00 2001 From: terminalchai <213856599+terminalchai@users.noreply.github.com> Date: Mon, 12 Oct 2026 01:26:22 +0530 Subject: [PATCH 1/2] docs: clarify concurrent access to persistent caches --- HISTORY.md | 1 + README.md | 2 +- docs/docs/concurrent_cache_access.md | 69 ++++++++++++++++++++++++++++ docs/docs/other_client_settings.md | 8 ++++ docs/mkdocs.yml | 1 + 5 files changed, 80 insertions(+), 1 deletion(-) create mode 100644 docs/docs/concurrent_cache_access.md diff --git a/HISTORY.md b/HISTORY.md index 1ae9db02..8569510f 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -3,6 +3,7 @@ ## UNRELEASED - Added streaming I/O via `FileCacheMode.streaming` for S3, Azure Blob Storage, Google Cloud Storage, and HTTP/HTTPS. In this mode `CloudPath.open()` returns the standard `io` file objects (`BufferedReader`/`BufferedWriter`/`TextIOWrapper`) backed by ranged reads and multipart uploads instead of the local cache, so only the bytes you read are downloaded and written data is uploaded as you write it. Read streams are seekable; append and update modes fall back to the cache and remove the cache file on close. `buffering` has its builtin `open()` meaning and sets the ranged-request size (default 5 MiB). A new `streaming_max_concurrency` client option (default 4) bounds background read-ahead and part uploads per stream; the buffer size, concurrency, and multipart part size can also be set with the `CLOUDPATHLIB_STREAMING_BUFFER_SIZE`, `CLOUDPATHLIB_STREAMING_MAX_CONCURRENCY`, and `CLOUDPATHLIB_[_]STREAMING_PART_SIZE` environment variables. Opening a file for append or update in streaming mode warns that it falls back to the cache. `copy`/`rename`/`replace` stream between clients, `fspath` raises `CloudPathNotImplementedError`, and failures raise `cloudpathlib.exceptions` types including the new `CloudPathStreamingError`. See the [Streaming I/O](https://cloudpathlib.drivendata.org/stable/streaming_io/) docs. (PR [#535](https://github.com/drivendataorg/cloudpathlib/pull/535)) +- Added documentation on concurrent cache access, including the limits of shared persistent caches and an example with a separate cache for each worker. (Issue [#522](https://github.com/drivendataorg/cloudpathlib/issues/522)) - Changed append mode (`a`, `ab`, `a+`, and `ab+`) to create missing cloud files, matching Python file behavior. Appending to existing files still preserves their content. (Issue [#585](https://github.com/drivendataorg/cloudpathlib/issues/585)) - Changed test dependencies to permit newer `pytest-rerunfailures` releases while excluding version 16.0, which causes distributed-test serialization failures. (Issue [#528](https://github.com/drivendataorg/cloudpathlib/issues/528), PR [#606](https://github.com/drivendataorg/cloudpathlib/pull/606)) diff --git a/README.md b/README.md index f76eb223..cf0c2f42 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ with CloudPath("s3://bucket/filename.txt").open("w+") as f: - **Supported clouds**: AWS S3, Google Cloud Storage, and Azure Blob Storage are implemented. FTP is on the way. - **Extensible**: The base classes do most of the work generically, so implementing two small classes `MyPath` and `MyClient` is all you need to add support for a new cloud storage service. - **Read/write support**: Reading just works. Using the `write_text`, `write_bytes` or `.open('w')` methods will all upload your changes to cloud storage without any additional file management as a developer. - - **Seamless caching**: Files are downloaded locally only when necessary. You can also easily pass a persistent cache folder so that across processes and sessions you only re-download what is necessary. + - **Seamless caching**: Files are downloaded locally only when necessary. A persistent cache folder can reuse downloads across sequential processes and sessions. For multiple workers, see [concurrent cache access](https://github.com/drivendataorg/cloudpathlib/blob/master/docs/docs/concurrent_cache_access.md). - **Tested**: Comprehensive test suite and code coverage. - **Testability**: Local filesystem implementations that can be used to easily mock cloud storage in your unit tests. diff --git a/docs/docs/concurrent_cache_access.md b/docs/docs/concurrent_cache_access.md new file mode 100644 index 00000000..49f7d5f4 --- /dev/null +++ b/docs/docs/concurrent_cache_access.md @@ -0,0 +1,69 @@ +# Concurrent cache access + +A persistent cache can reuse downloaded files across sequential sessions and +processes. It does not coordinate concurrent access to the same cached file. +This applies both to `local_cache_dir` and to the +`CLOUDPATHLIB_LOCAL_CACHE_DIR` environment variable. + +## Accessing the same file from multiple workers + +Cloudpathlib does not lock cache files while downloading, reading, writing, or +uploading them. If two workers access the same cloud object through a shared +cache, one may encounter a partially downloaded file or an +`OverwriteNewerLocalError` while the other is still downloading. The behavior +can depend on the provider and operation. Setting +`force_overwrite_from_cloud=True` does not coordinate the workers. + +For independent work, create a client and cache directory inside each task. +Tasks can then read the same cloud object without sharing its local cache. +The example below creates a fresh cache for each task invocation, so repeated +reads download again. This prevents tasks from using or clearing each other's +cache files. Concurrent writes to the same cloud object still need +coordination at the application level. + +```python +from concurrent.futures import ThreadPoolExecutor +from tempfile import TemporaryDirectory + +from cloudpathlib import S3Client + + +def read_in_worker(uri): + with TemporaryDirectory() as cache_dir: + client = S3Client(local_cache_dir=cache_dir, file_cache_mode="persistent") + return client.CloudPath(uri).read_bytes() + + +if __name__ == "__main__": + paths = ["s3://bucket/first.csv", "s3://bucket/first.csv"] + with ThreadPoolExecutor(max_workers=4) as executor: + for data in executor.map(read_in_worker, paths): + print(len(data)) +``` + +The explicit `local_cache_dir` overrides `CLOUDPATHLIB_LOCAL_CACHE_DIR`, so each +task remains isolated even if that environment variable is set. The explicit +cache mode also overrides `CLOUDPATHLIB_FILE_CACHE_MODE`. The temporary +cache is removed after the read finishes. The same approach can be used with +GCS or Azure clients, or with a process pool: create the client and cache inside +the worker, and keep pool startup under the `__main__` guard. + +## Reusing a shared persistent cache + +Explicitly set `file_cache_mode="persistent"` when sharing a cache directory, +and coordinate any cache cleanup. Other cache modes can remove cached files +when a file, path, or client is closed or released. + +Workers operating on different cloud objects do not contend for the same +cached file. Use separate caches for different providers, whose bucket or +container names may otherwise map to the same local paths. + +If workers must share a cache for the same object, coordinate access outside +cloudpathlib, such as with a lock shared by all processes that every worker respects. +Hold the lock for the whole operation, including opening the file, using it, +and closing it so any upload finishes before another worker accesses the cache. +A lock that only protects client construction is insufficient. + +See [Caching](caching.ipynb) for cache modes and cleanup behavior, and +[Streaming I/O](streaming_io.md) for reads that avoid the local cache. Append and +update operations in streaming mode still fall back to the cache. diff --git a/docs/docs/other_client_settings.md b/docs/docs/other_client_settings.md index 92a33e64..8153c55c 100644 --- a/docs/docs/other_client_settings.md +++ b/docs/docs/other_client_settings.md @@ -1,5 +1,13 @@ # Other `Client` settings +## Cache directories for concurrent workers + +A persistent `local_cache_dir` can be reused across sequential sessions, but it +does not synchronize concurrent access to the same cached file. This also +applies to `CLOUDPATHLIB_LOCAL_CACHE_DIR`. See +[Concurrent cache access](concurrent_cache_access.md) for worker-specific caches +and coordination when sharing a cache. + ## Content type guessing (`content_type_method`) All of the clients support passing a `content_type_method` when they are instantiated. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 1fcbd3d7..8158d32a 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -21,6 +21,7 @@ nav: - AnyPath: "anypath-polymorphism.md" - HTTP URLs: "http.md" - Caching: "caching.ipynb" + - Concurrent cache access: "concurrent_cache_access.md" - Streaming I/O: "streaming_io.md" - Compatibility: "patching_builtins.ipynb" - Other Client settings: "other_client_settings.md" From fb1c34d4406f64c6866e661c734bbf31c3f26696 Mon Sep 17 00:00:00 2001 From: terminalchai <213856599+terminalchai@users.noreply.github.com> Date: Mon, 19 Oct 2026 02:31:34 +0530 Subject: [PATCH 2/2] docs: explain shared persistent caches by worker and object --- HISTORY.md | 2 +- README.md | 2 +- cloudpathlib/azure/azblobclient.py | 3 ++ cloudpathlib/gs/gsclient.py | 3 ++ cloudpathlib/http/httpclient.py | 3 ++ cloudpathlib/s3/s3client.py | 3 ++ docs/docs/caching.ipynb | 4 +- docs/docs/concurrent_cache_access.md | 81 +++++++++++++++++++++++----- docs/docs/other_client_settings.md | 11 ++-- docs/docs/script/caching.py | 2 + 10 files changed, 92 insertions(+), 22 deletions(-) diff --git a/HISTORY.md b/HISTORY.md index 8569510f..988068a1 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -3,7 +3,7 @@ ## UNRELEASED - Added streaming I/O via `FileCacheMode.streaming` for S3, Azure Blob Storage, Google Cloud Storage, and HTTP/HTTPS. In this mode `CloudPath.open()` returns the standard `io` file objects (`BufferedReader`/`BufferedWriter`/`TextIOWrapper`) backed by ranged reads and multipart uploads instead of the local cache, so only the bytes you read are downloaded and written data is uploaded as you write it. Read streams are seekable; append and update modes fall back to the cache and remove the cache file on close. `buffering` has its builtin `open()` meaning and sets the ranged-request size (default 5 MiB). A new `streaming_max_concurrency` client option (default 4) bounds background read-ahead and part uploads per stream; the buffer size, concurrency, and multipart part size can also be set with the `CLOUDPATHLIB_STREAMING_BUFFER_SIZE`, `CLOUDPATHLIB_STREAMING_MAX_CONCURRENCY`, and `CLOUDPATHLIB_[_]STREAMING_PART_SIZE` environment variables. Opening a file for append or update in streaming mode warns that it falls back to the cache. `copy`/`rename`/`replace` stream between clients, `fspath` raises `CloudPathNotImplementedError`, and failures raise `cloudpathlib.exceptions` types including the new `CloudPathStreamingError`. See the [Streaming I/O](https://cloudpathlib.drivendata.org/stable/streaming_io/) docs. (PR [#535](https://github.com/drivendataorg/cloudpathlib/pull/535)) -- Added documentation on concurrent cache access, including the limits of shared persistent caches and an example with a separate cache for each worker. (Issue [#522](https://github.com/drivendataorg/cloudpathlib/issues/522)) +- Added documentation on concurrent cache access, recommending shared persistent caches for workers accessing different objects and separate caches or external coordination for the same object. (Issue [#522](https://github.com/drivendataorg/cloudpathlib/issues/522), PR [#602](https://github.com/drivendataorg/cloudpathlib/pull/602)) - Changed append mode (`a`, `ab`, `a+`, and `ab+`) to create missing cloud files, matching Python file behavior. Appending to existing files still preserves their content. (Issue [#585](https://github.com/drivendataorg/cloudpathlib/issues/585)) - Changed test dependencies to permit newer `pytest-rerunfailures` releases while excluding version 16.0, which causes distributed-test serialization failures. (Issue [#528](https://github.com/drivendataorg/cloudpathlib/issues/528), PR [#606](https://github.com/drivendataorg/cloudpathlib/pull/606)) diff --git a/README.md b/README.md index cf0c2f42..b0182aa2 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ with CloudPath("s3://bucket/filename.txt").open("w+") as f: - **Supported clouds**: AWS S3, Google Cloud Storage, and Azure Blob Storage are implemented. FTP is on the way. - **Extensible**: The base classes do most of the work generically, so implementing two small classes `MyPath` and `MyClient` is all you need to add support for a new cloud storage service. - **Read/write support**: Reading just works. Using the `write_text`, `write_bytes` or `.open('w')` methods will all upload your changes to cloud storage without any additional file management as a developer. - - **Seamless caching**: Files are downloaded locally only when necessary. A persistent cache folder can reuse downloads across sequential processes and sessions. For multiple workers, see [concurrent cache access](https://github.com/drivendataorg/cloudpathlib/blob/master/docs/docs/concurrent_cache_access.md). + - **Seamless caching**: Files are downloaded locally only when necessary. A persistent cache folder can reuse downloads across sequential processes and sessions. For multiple workers, see [concurrent cache access](https://cloudpathlib.drivendata.org/stable/concurrent_cache_access/). - **Tested**: Comprehensive test suite and code coverage. - **Testability**: Local filesystem implementations that can be used to easily mock cloud storage in your unit tests. diff --git a/cloudpathlib/azure/azblobclient.py b/cloudpathlib/azure/azblobclient.py index 4f4b577b..bc514886 100644 --- a/cloudpathlib/azure/azblobclient.py +++ b/cloudpathlib/azure/azblobclient.py @@ -124,6 +124,9 @@ def __init__( local_cache_dir (Optional[Union[str, os.PathLike]]): Path to directory to use as cache for downloaded files. If None, will use a temporary directory. Default can be set with the `CLOUDPATHLIB_LOCAL_CACHE_DIR` environment variable. + Workers accessing different objects can share a persistent cache; concurrent access + to the same cached file requires separate caches or external coordination. See + [Concurrent cache access](https://cloudpathlib.drivendata.org/stable/concurrent_cache_access/). content_type_method (Optional[Callable]): Function to call to guess media type (mimetype) when writing a file to the cloud. Defaults to `mimetypes.guess_type`. Must return a tuple (content type, content encoding). streaming_max_concurrency (Optional[int]): Maximum concurrent requests per open diff --git a/cloudpathlib/gs/gsclient.py b/cloudpathlib/gs/gsclient.py index 8f967edb..e1697438 100644 --- a/cloudpathlib/gs/gsclient.py +++ b/cloudpathlib/gs/gsclient.py @@ -143,6 +143,9 @@ def __init__( local_cache_dir (Optional[Union[str, os.PathLike]]): Path to directory to use as cache for downloaded files. If None, will use a temporary directory. Default can be set with the `CLOUDPATHLIB_LOCAL_CACHE_DIR` environment variable. + Workers accessing different objects can share a persistent cache; concurrent access + to the same cached file requires separate caches or external coordination. See + [Concurrent cache access](https://cloudpathlib.drivendata.org/stable/concurrent_cache_access/). content_type_method (Optional[Callable]): Function to call to guess media type (mimetype) when writing a file to the cloud. Defaults to `mimetypes.guess_type`. Must return a tuple (content type, content encoding). download_chunks_concurrently_kwargs (Optional[Dict[str, Any]]): Keyword arguments to pass to diff --git a/cloudpathlib/http/httpclient.py b/cloudpathlib/http/httpclient.py index 5d25ee75..99cc9872 100644 --- a/cloudpathlib/http/httpclient.py +++ b/cloudpathlib/http/httpclient.py @@ -51,6 +51,9 @@ def __init__( local_cache_dir (Optional[Union[str, os.PathLike]]): Path to directory to use as cache for downloaded files. If None, will use a temporary directory. Default can be set with the `CLOUDPATHLIB_LOCAL_CACHE_DIR` environment variable. + Workers accessing different objects can share a persistent cache; concurrent access + to the same cached file requires separate caches or external coordination. See + [Concurrent cache access](https://cloudpathlib.drivendata.org/stable/concurrent_cache_access/). content_type_method (Optional[Callable]): Function to call to guess media type (mimetype) when uploading files. Defaults to `mimetypes.guess_type`. auth (Optional[urllib.request.BaseHandler]): Authentication handler to use for the client. Defaults to None, which will use the default handler. diff --git a/cloudpathlib/s3/s3client.py b/cloudpathlib/s3/s3client.py index 651b154f..b2fce030 100644 --- a/cloudpathlib/s3/s3client.py +++ b/cloudpathlib/s3/s3client.py @@ -74,6 +74,9 @@ def __init__( local_cache_dir (Optional[Union[str, os.PathLike]]): Path to directory to use as cache for downloaded files. If None, will use a temporary directory. Default can be set with the `CLOUDPATHLIB_LOCAL_CACHE_DIR` environment variable. + Workers accessing different objects can share a persistent cache; concurrent access + to the same cached file requires separate caches or external coordination. See + [Concurrent cache access](https://cloudpathlib.drivendata.org/stable/concurrent_cache_access/). endpoint_url (Optional[str]): S3 server endpoint URL to use for the constructed boto3 S3 resource and client. Parameterize it to access a customly deployed S3-compatible object store such as MinIO, Ceph or any other. addressing_style (Optional[str]): S3 addressing style to pass through to boto3's S3 config. diff --git a/docs/docs/caching.ipynb b/docs/docs/caching.ipynb index 7fdc729b..9cac9f1e 100644 --- a/docs/docs/caching.ipynb +++ b/docs/docs/caching.ipynb @@ -6,7 +6,9 @@ "source": [ "# Caching\n", "\n", - "Interacting with files on a cloud provider can mean a lot of waiting on files downloading and uploading. `cloudpathlib` provides seamless on-demand caching of cloud content that can be persistent across processes and sessions to make sure you only download or upload when you need to." + "Interacting with files on a cloud provider can mean a lot of waiting on files downloading and uploading. `cloudpathlib` provides seamless on-demand caching of cloud content that can be persistent across processes and sessions to make sure you only download or upload when you need to.\n", + "\n", + "Workers accessing different cloud objects can share a persistent cache. Give each thread its own `CloudPath` and share one client; construct a client inside each process with the same cache directory and `file_cache_mode=\"persistent\"`. Concurrent access to the same cached file requires separate caches or external coordination. See [Concurrent cache access](concurrent_cache_access.md) for examples and cache-key collision limits." ] }, { diff --git a/docs/docs/concurrent_cache_access.md b/docs/docs/concurrent_cache_access.md index 49f7d5f4..0b7b7d35 100644 --- a/docs/docs/concurrent_cache_access.md +++ b/docs/docs/concurrent_cache_access.md @@ -1,10 +1,71 @@ # Concurrent cache access -A persistent cache can reuse downloaded files across sequential sessions and -processes. It does not coordinate concurrent access to the same cached file. -This applies both to `local_cache_dir` and to the +Parallelize by cloud object. Workers that read or write different cloud objects +can share one persistent cache. Give each thread its own `CloudPath`; one +`Client` can be shared across threads. Each process should construct its own +client pointed at the same `local_cache_dir` with `file_cache_mode="persistent"`. +This also applies when setting the cache directory with the `CLOUDPATHLIB_LOCAL_CACHE_DIR` environment variable. +## Different objects with a shared persistent cache + +Create a client once for a thread pool and pass different object URIs to the +workers. Each worker creates its own path using that client: + +```python +from concurrent.futures import ThreadPoolExecutor + +from cloudpathlib import S3Client + + +def read_in_worker(uri): + return client.CloudPath(uri).read_bytes() + + +if __name__ == "__main__": + client = S3Client(local_cache_dir="cache", file_cache_mode="persistent") + paths = ["s3://bucket/first.csv", "s3://bucket/second.csv"] + with ThreadPoolExecutor(max_workers=4) as executor: + for data in executor.map(read_in_worker, paths): + print(len(data)) +``` + +For a process pool, pass URI strings and construct the client inside the worker. +The processes can reuse the same cache directory when accessing different objects: + +```python +from concurrent.futures import ProcessPoolExecutor + +from cloudpathlib import S3Client + + +def read_in_worker(uri): + client = S3Client(local_cache_dir="cache", file_cache_mode="persistent") + return client.CloudPath(uri).read_bytes() + + +if __name__ == "__main__": + paths = ["s3://bucket/first.csv", "s3://bucket/second.csv"] + with ProcessPoolExecutor(max_workers=4) as executor: + for data in executor.map(read_in_worker, paths): + print(len(data)) +``` + +Do not pass a `Client` or `CloudPath` into the process pool. Pickling a `CloudPath` +removes its `_client`, so the child uses its process's default client, which may +differ from the parent's configured client. Constructing the client in the subprocess keeps its +credentials, endpoint, and cache settings explicit. + +Explicitly set `file_cache_mode="persistent"` when sharing a cache directory, +and coordinate any cache cleanup. Other cache modes can remove cached files +when a file, path, or client is closed or released. + +Different objects must also map to different local cache paths. The cache key is +the cloud path with its scheme removed; for HTTP, it is the URL path with the +host removed. Bucket or container names and keys can therefore collide across +providers, accounts, or endpoints, and HTTP URLs on different hosts can collide +when their URL paths match. Use separate cache directories for these namespaces. + ## Accessing the same file from multiple workers Cloudpathlib does not lock cache files while downloading, reading, writing, or @@ -14,7 +75,9 @@ cache, one may encounter a partially downloaded file or an can depend on the provider and operation. Setting `force_overwrite_from_cloud=True` does not coordinate the workers. -For independent work, create a client and cache directory inside each task. +If multiple workers need the same object, use a separate cache per worker or +coordinate access with an external lock. For separate caches, create a client +and cache directory inside each task. Tasks can then read the same cloud object without sharing its local cache. The example below creates a fresh cache for each task invocation, so repeated reads download again. This prevents tasks from using or clearing each other's @@ -48,16 +111,6 @@ cache is removed after the read finishes. The same approach can be used with GCS or Azure clients, or with a process pool: create the client and cache inside the worker, and keep pool startup under the `__main__` guard. -## Reusing a shared persistent cache - -Explicitly set `file_cache_mode="persistent"` when sharing a cache directory, -and coordinate any cache cleanup. Other cache modes can remove cached files -when a file, path, or client is closed or released. - -Workers operating on different cloud objects do not contend for the same -cached file. Use separate caches for different providers, whose bucket or -container names may otherwise map to the same local paths. - If workers must share a cache for the same object, coordinate access outside cloudpathlib, such as with a lock shared by all processes that every worker respects. Hold the lock for the whole operation, including opening the file, using it, diff --git a/docs/docs/other_client_settings.md b/docs/docs/other_client_settings.md index 8153c55c..8ff7b6f9 100644 --- a/docs/docs/other_client_settings.md +++ b/docs/docs/other_client_settings.md @@ -2,11 +2,12 @@ ## Cache directories for concurrent workers -A persistent `local_cache_dir` can be reused across sequential sessions, but it -does not synchronize concurrent access to the same cached file. This also -applies to `CLOUDPATHLIB_LOCAL_CACHE_DIR`. See -[Concurrent cache access](concurrent_cache_access.md) for worker-specific caches -and coordination when sharing a cache. +Workers accessing different cloud objects can share a persistent `local_cache_dir`. +Share one client across threads, giving each thread its own `CloudPath`; construct +a client inside each process with the same cache directory. This also applies to +`CLOUDPATHLIB_LOCAL_CACHE_DIR`. See [Concurrent cache access](concurrent_cache_access.md) +for shared-cache examples and the separate caches or external coordination needed +when workers access the same object. ## Content type guessing (`content_type_method`) diff --git a/docs/docs/script/caching.py b/docs/docs/script/caching.py index 69930564..68afe31d 100644 --- a/docs/docs/script/caching.py +++ b/docs/docs/script/caching.py @@ -4,6 +4,8 @@ # # Caching # # Interacting with files on a cloud provider can mean a lot of waiting on files downloading and uploading. `cloudpathlib` provides seamless on-demand caching of cloud content that can be persistent across processes and sessions to make sure you only download or upload when you need to. +# +# Workers accessing different cloud objects can share a persistent cache. Give each thread its own `CloudPath` and share one client; construct a client inside each process with the same cache directory and `file_cache_mode="persistent"`. Concurrent access to the same cached file requires separate caches or external coordination. See [Concurrent cache access](concurrent_cache_access.md) for examples and cache-key collision limits. # ## Are we synced? #