Document concurrent access limits for persistent caches

HISTORY.md

@@ -3,6 +3,7 @@
 ## UNRELEASED
 
 - Added streaming I/O via `FileCacheMode.streaming` for S3, Azure Blob Storage, Google Cloud Storage, and HTTP/HTTPS. In this mode `CloudPath.open()` returns the standard `io` file objects (`BufferedReader`/`BufferedWriter`/`TextIOWrapper`) backed by ranged reads and multipart uploads instead of the local cache, so only the bytes you read are downloaded and written data is uploaded as you write it. Read streams are seekable; append and update modes fall back to the cache and remove the cache file on close. `buffering` has its builtin `open()` meaning and sets the ranged-request size (default 5 MiB). A new `streaming_max_concurrency` client option (default 4) bounds background read-ahead and part uploads per stream; the buffer size, concurrency, and multipart part size can also be set with the `CLOUDPATHLIB_STREAMING_BUFFER_SIZE`, `CLOUDPATHLIB_STREAMING_MAX_CONCURRENCY`, and `CLOUDPATHLIB_[<PROVIDER>_]STREAMING_PART_SIZE` environment variables. Opening a file for append or update in streaming mode warns that it falls back to the cache. `copy`/`rename`/`replace` stream between clients, `fspath` raises `CloudPathNotImplementedError`, and failures raise `cloudpathlib.exceptions` types including the new `CloudPathStreamingError`. See the [Streaming I/O](https://cloudpathlib.drivendata.org/stable/streaming_io/) docs. (PR [#535](https://github.com/drivendataorg/cloudpathlib/pull/535))
+- Added documentation on concurrent cache access, recommending shared persistent caches for workers accessing different objects and separate caches or external coordination for the same object. (Issue [#522](https://github.com/drivendataorg/cloudpathlib/issues/522), PR [#602](https://github.com/drivendataorg/cloudpathlib/pull/602))
 - Changed append mode (`a`, `ab`, `a+`, and `ab+`) to create missing cloud files, matching Python file behavior. Appending to existing files still preserves their content. (Issue [#585](https://github.com/drivendataorg/cloudpathlib/issues/585))
 - Changed test dependencies to permit newer `pytest-rerunfailures` releases while excluding version 16.0, which causes distributed-test serialization failures. (Issue [#528](https://github.com/drivendataorg/cloudpathlib/issues/528), PR [#606](https://github.com/drivendataorg/cloudpathlib/pull/606))
 

README.md

@@ -24,7 +24,7 @@ with CloudPath("s3://bucket/filename.txt").open("w+") as f:
  - **Supported clouds**: AWS S3, Google Cloud Storage, and Azure Blob Storage are implemented. FTP is on the way.
  - **Extensible**: The base classes do most of the work generically, so implementing two small classes `MyPath` and `MyClient` is all you need to add support for a new cloud storage service.
  - **Read/write support**: Reading just works. Using the `write_text`, `write_bytes` or `.open('w')` methods will all upload your changes to cloud storage without any additional file management as a developer.
- - **Seamless caching**: Files are downloaded locally only when necessary. You can also easily pass a persistent cache folder so that across processes and sessions you only re-download what is necessary.
+ - **Seamless caching**: Files are downloaded locally only when necessary. A persistent cache folder can reuse downloads across sequential processes and sessions. For multiple workers, see [concurrent cache access](https://cloudpathlib.drivendata.org/stable/concurrent_cache_access/).
  - **Tested**: Comprehensive test suite and code coverage.
  - **Testability**: Local filesystem implementations that can be used to easily mock cloud storage in your unit tests.
 

cloudpathlib/azure/azblobclient.py

@@ -124,6 +124,9 @@ def __init__(
             local_cache_dir (Optional[Union[str, os.PathLike]]): Path to directory to use as cache
                 for downloaded files. If None, will use a temporary directory. Default can be set with
                 the `CLOUDPATHLIB_LOCAL_CACHE_DIR` environment variable.
+                Workers accessing different objects can share a persistent cache; concurrent access
+                to the same cached file requires separate caches or external coordination. See
+                [Concurrent cache access](https://cloudpathlib.drivendata.org/stable/concurrent_cache_access/).
             content_type_method (Optional[Callable]): Function to call to guess media type (mimetype) when
                 writing a file to the cloud. Defaults to `mimetypes.guess_type`. Must return a tuple (content type, content encoding).
             streaming_max_concurrency (Optional[int]): Maximum concurrent requests per open

cloudpathlib/gs/gsclient.py

@@ -143,6 +143,9 @@ def __init__(
             local_cache_dir (Optional[Union[str, os.PathLike]]): Path to directory to use as cache
                 for downloaded files. If None, will use a temporary directory. Default can be set with
                 the `CLOUDPATHLIB_LOCAL_CACHE_DIR` environment variable.
+                Workers accessing different objects can share a persistent cache; concurrent access
+                to the same cached file requires separate caches or external coordination. See
+                [Concurrent cache access](https://cloudpathlib.drivendata.org/stable/concurrent_cache_access/).
             content_type_method (Optional[Callable]): Function to call to guess media type (mimetype) when
                 writing a file to the cloud. Defaults to `mimetypes.guess_type`. Must return a tuple (content type, content encoding).
             download_chunks_concurrently_kwargs (Optional[Dict[str, Any]]): Keyword arguments to pass to

cloudpathlib/http/httpclient.py

@@ -51,6 +51,9 @@ def __init__(
             local_cache_dir (Optional[Union[str, os.PathLike]]): Path to directory to use as cache
                 for downloaded files. If None, will use a temporary directory. Default can be set with
                 the `CLOUDPATHLIB_LOCAL_CACHE_DIR` environment variable.
+                Workers accessing different objects can share a persistent cache; concurrent access
+                to the same cached file requires separate caches or external coordination. See
+                [Concurrent cache access](https://cloudpathlib.drivendata.org/stable/concurrent_cache_access/).
             content_type_method (Optional[Callable]): Function to call to guess media type (mimetype) when
                 uploading files. Defaults to `mimetypes.guess_type`.
             auth (Optional[urllib.request.BaseHandler]): Authentication handler to use for the client. Defaults to None, which will use the default handler.

cloudpathlib/s3/s3client.py

@@ -74,6 +74,9 @@ def __init__(
             local_cache_dir (Optional[Union[str, os.PathLike]]): Path to directory to use as cache
                 for downloaded files. If None, will use a temporary directory. Default can be set with
                 the `CLOUDPATHLIB_LOCAL_CACHE_DIR` environment variable.
+                Workers accessing different objects can share a persistent cache; concurrent access
+                to the same cached file requires separate caches or external coordination. See
+                [Concurrent cache access](https://cloudpathlib.drivendata.org/stable/concurrent_cache_access/).
             endpoint_url (Optional[str]): S3 server endpoint URL to use for the constructed boto3 S3 resource and client.
                 Parameterize it to access a customly deployed S3-compatible object store such as MinIO, Ceph or any other.
             addressing_style (Optional[str]): S3 addressing style to pass through to boto3's S3 config.

docs/docs/caching.ipynb

@@ -6,7 +6,9 @@
    "source": [
     "# Caching\n",
     "\n",
-    "Interacting with files on a cloud provider can mean a lot of waiting on files downloading and uploading. `cloudpathlib` provides seamless on-demand caching of cloud content that can be persistent across processes and sessions to make sure you only download or upload when you need to."
+    "Interacting with files on a cloud provider can mean a lot of waiting on files downloading and uploading. `cloudpathlib` provides seamless on-demand caching of cloud content that can be persistent across processes and sessions to make sure you only download or upload when you need to.\n",
+    "\n",
+    "Workers accessing different cloud objects can share a persistent cache. Give each thread its own `CloudPath` and share one client; construct a client inside each process with the same cache directory and `file_cache_mode=\"persistent\"`. Concurrent access to the same cached file requires separate caches or external coordination. See [Concurrent cache access](concurrent_cache_access.md) for examples and cache-key collision limits."
    ]
   },
   {

docs/docs/concurrent_cache_access.md

@@ -0,0 +1,122 @@
+# Concurrent cache access
+
+Parallelize by cloud object. Workers that read or write different cloud objects
+can share one persistent cache. Give each thread its own `CloudPath`; one
+`Client` can be shared across threads. Each process should construct its own
+client pointed at the same `local_cache_dir` with `file_cache_mode="persistent"`.
+This also applies when setting the cache directory with the
+`CLOUDPATHLIB_LOCAL_CACHE_DIR` environment variable.
+
+## Different objects with a shared persistent cache
+
+Create a client once for a thread pool and pass different object URIs to the
+workers. Each worker creates its own path using that client:
+
+```python
+from concurrent.futures import ThreadPoolExecutor
+
+from cloudpathlib import S3Client
+
+
+def read_in_worker(uri):
+    return client.CloudPath(uri).read_bytes()
+
+
+if __name__ == "__main__":
+    client = S3Client(local_cache_dir="cache", file_cache_mode="persistent")
+    paths = ["s3://bucket/first.csv", "s3://bucket/second.csv"]
+    with ThreadPoolExecutor(max_workers=4) as executor:
+        for data in executor.map(read_in_worker, paths):
+            print(len(data))
+```
+
+For a process pool, pass URI strings and construct the client inside the worker.
+The processes can reuse the same cache directory when accessing different objects:
+
+```python
+from concurrent.futures import ProcessPoolExecutor
+
+from cloudpathlib import S3Client
+
+
+def read_in_worker(uri):
+    client = S3Client(local_cache_dir="cache", file_cache_mode="persistent")
+    return client.CloudPath(uri).read_bytes()
+
+
+if __name__ == "__main__":
+    paths = ["s3://bucket/first.csv", "s3://bucket/second.csv"]
+    with ProcessPoolExecutor(max_workers=4) as executor:
+        for data in executor.map(read_in_worker, paths):
+            print(len(data))
+```
+
+Do not pass a `Client` or `CloudPath` into the process pool. Pickling a `CloudPath`
+removes its `_client`, so the child uses its process's default client, which may
+differ from the parent's configured client. Constructing the client in the subprocess keeps its
+credentials, endpoint, and cache settings explicit.
+
+Explicitly set `file_cache_mode="persistent"` when sharing a cache directory,
+and coordinate any cache cleanup. Other cache modes can remove cached files
+when a file, path, or client is closed or released.
+
+Different objects must also map to different local cache paths. The cache key is
+the cloud path with its scheme removed; for HTTP, it is the URL path with the
+host removed. Bucket or container names and keys can therefore collide across
+providers, accounts, or endpoints, and HTTP URLs on different hosts can collide
+when their URL paths match. Use separate cache directories for these namespaces.
+
+## Accessing the same file from multiple workers
+
+Cloudpathlib does not lock cache files while downloading, reading, writing, or
+uploading them. If two workers access the same cloud object through a shared
+cache, one may encounter a partially downloaded file or an
+`OverwriteNewerLocalError` while the other is still downloading. The behavior
+can depend on the provider and operation. Setting
+`force_overwrite_from_cloud=True` does not coordinate the workers.
+
+If multiple workers need the same object, use a separate cache per worker or
+coordinate access with an external lock. For separate caches, create a client
+and cache directory inside each task.
+Tasks can then read the same cloud object without sharing its local cache.
+The example below creates a fresh cache for each task invocation, so repeated
+reads download again. This prevents tasks from using or clearing each other's
+cache files. Concurrent writes to the same cloud object still need
+coordination at the application level.
+
+```python
+from concurrent.futures import ThreadPoolExecutor
+from tempfile import TemporaryDirectory
+
+from cloudpathlib import S3Client
+
+
+def read_in_worker(uri):
+    with TemporaryDirectory() as cache_dir:
+        client = S3Client(local_cache_dir=cache_dir, file_cache_mode="persistent")
+        return client.CloudPath(uri).read_bytes()
+
+
+if __name__ == "__main__":
+    paths = ["s3://bucket/first.csv", "s3://bucket/first.csv"]
+    with ThreadPoolExecutor(max_workers=4) as executor:
+        for data in executor.map(read_in_worker, paths):
+            print(len(data))
+```
+
+The explicit `local_cache_dir` overrides `CLOUDPATHLIB_LOCAL_CACHE_DIR`, so each
+task remains isolated even if that environment variable is set. The explicit
+cache mode also overrides `CLOUDPATHLIB_FILE_CACHE_MODE`. The temporary
+cache is removed after the read finishes. The same approach can be used with
+GCS or Azure clients, or with a process pool: create the client and cache inside
+the worker, and keep pool startup under the `__main__` guard.
+
+If workers must share a cache for the same object, coordinate access outside
+cloudpathlib, such as with a lock shared by all processes that every worker respects.
+Hold the lock for the whole operation, including opening the file, using it,
+and closing it so any upload finishes before another worker accesses the cache.
+A lock that only protects client construction is insufficient.
+
+See [Caching](caching.ipynb) for cache modes and cleanup behavior, and
+[Streaming I/O](streaming_io.md) for reads that avoid the local cache. Append and
+update operations in streaming mode still fall back to the cache.

docs/docs/other_client_settings.md

@@ -1,5 +1,14 @@
 # Other `Client` settings
 
+## Cache directories for concurrent workers
+
+Workers accessing different cloud objects can share a persistent `local_cache_dir`.
+Share one client across threads, giving each thread its own `CloudPath`; construct
+a client inside each process with the same cache directory. This also applies to
+`CLOUDPATHLIB_LOCAL_CACHE_DIR`. See [Concurrent cache access](concurrent_cache_access.md)
+for shared-cache examples and the separate caches or external coordination needed
+when workers access the same object.
+
 ## Content type guessing (`content_type_method`)
 
 All of the clients support passing a `content_type_method` when they are instantiated.  

docs/docs/script/caching.py

@@ -4,6 +4,8 @@
 # # Caching
 # 
 # Interacting with files on a cloud provider can mean a lot of waiting on files downloading and uploading. `cloudpathlib` provides seamless on-demand caching of cloud content that can be persistent across processes and sessions to make sure you only download or upload when you need to.
+#
+# Workers accessing different cloud objects can share a persistent cache. Give each thread its own `CloudPath` and share one client; construct a client inside each process with the same cache directory and `file_cache_mode="persistent"`. Concurrent access to the same cached file requires separate caches or external coordination. See [Concurrent cache access](concurrent_cache_access.md) for examples and cache-key collision limits.
 
 # ## Are we synced?
 # 

docs/mkdocs.yml

@@ -21,6 +21,7 @@ nav:
   - AnyPath: "anypath-polymorphism.md"
   - HTTP URLs: "http.md"
   - Caching: "caching.ipynb"
+  - Concurrent cache access: "concurrent_cache_access.md"
   - Streaming I/O: "streaming_io.md"
   - Compatibility: "patching_builtins.ipynb"
   - Other Client settings: "other_client_settings.md"