Resumable HTTP uploads

config/simdb.cfg

@@ -34,3 +34,4 @@ result_backend = redis://redis:6379/0
 
 [partition]
 data = /data/simdb/partition
+http = /data/simdb/http

docker-compose.yml

@@ -20,6 +20,7 @@ services:
       - ./validation:/app/validation:ro
       - ./config:/app/config:ro
       - ./tmp/partition_data:/data/simdb/partition:ro
+      - ./tmp/http:/data/simdb/http
       - ./upload_folder:/data/simdb/simulations
     depends_on:
       redis:
@@ -38,6 +39,7 @@ services:
     volumes:
       - ./config:/app/config:ro
       - ./tmp/partition_data:/data/simdb/partition:ro
+      - ./tmp/http:/data/simdb/http
       - ./upload_folder:/data/simdb/simulations
     depends_on:
       redis:

docs/generate_cli_docs.py

@@ -39,7 +39,9 @@ def extract_sub_commands(output: str) -> list[str]:
     sub_commands = []
     for line in output.split("\n"):
         if in_commands:
-            if line:
+            # Lines indented further than the command column are continuations
+            # of the previous command's help text.
+            if line and not line.startswith("   "):
                 sub_commands.append(line.split()[0])
         if line == "Commands:":
             in_commands = True

docs/how-to/push-pull.md

@@ -42,6 +42,108 @@ simdb simulation push SIM_ID --add-watcher
 See [watchers](../explanation/concepts.md#watchers) and the
 `simdb remote watcher` commands in the [CLI reference](../reference/cli.md).
 
+### Resumable uploads
+
+Against a v1.3 remote, `push` sends the file bytes using the IETF
+[Resumable Uploads for HTTP](https://datatracker.ietf.org/doc/draft-ietf-httpbis-resumable-upload/)
+protocol (draft 11, interop version 8). No shared file system is required. The
+server stages the uploaded bytes in its `http` partition, and from there the
+flow matches `push_local`: SimDB sends the metadata, the server queues the copy
+into its upload folder as a background
+[Celery task](operate-server/run-celery-workers.md), and the CLI blocks while
+reporting the ingestion state. Against a v1.2 remote, `push` falls back to the
+earlier non-resumable transfer.
+
+Because the protocol is resumable, an interrupted upload (a lost connection,
+Ctrl-C) does not have to start over. Re-running `push` for the same simulation
+asks the server how many bytes it already holds for each file and continues from
+that offset.
+
+The chunk size is governed by the server: it advertises a maximum append size
+through the `Upload-Limit` response header and the client sizes its chunks to
+stay within that bound. The limit defaults to 8 MiB and can be tuned with
+[`server.max_append_size`](../reference/server-configuration.md#server), for
+example to fit within a reverse proxy's request body limit.
+
+Each chunk is integrity-checked with an RFC 9530 `Content-Digest` (SHA-256), so
+the server can verify a chunk before appending it. A digest mismatch is rejected
+with a `400` response and the offending bytes are not stored, so corruption in
+transit cannot be silently committed.
+
+### Stage uploads on the server
+
+`push` needs no client-side partition configuration — the paths come straight
+from the local simulation. The server, however, must define where uploaded bytes
+are staged, through a partition named `http` in its `simdb.cfg` (see
+[Server configuration](../reference/server-configuration.md#partition)):
+
+```ini
+[partition]
+http = /var/lib/simdb/http-staging
+```
+
+A file uploaded to `<sim_uuid>/file/<path>` is written to
+`<http partition>/<sim_uuid>/file/<path>` and referenced by an
+`http:///<sim_uuid>/file/<path>` URI. The background ingestion task resolves that
+URI against the `http` partition, copies the file into the simulation's upload
+folder, and finally removes the staged copy.
+
+Subfolder structure is handled just as it is for
+[`push_local`](#push-on-a-shared-file-system): files keep their relative layout,
+so multi-file IMAS datasets (HDF5, ASCII, and MDSplus backends) stay contained
+within their own directory and are reconstructed correctly on the server, while
+standalone files (such as an IMAS netCDF `.nc`) are not given a spurious
+enclosing folder.
+
+## Push on a shared file system
+
+If your machine and the server can reach the same physical file paths (as on the
+ITER network), sending large datasets over HTTP is slow and redundant. Use
+`push_local` instead:
+
+```bash
+simdb simulation push_local SIM_ID
+```
+
+`push_local` sends only the metadata and the storage paths. The server then
+
+1. validates the metadata against the active schemas,
+2. queues the file copy as a background [Celery task](operate-server/run-celery-workers.md), and
+3. completes the ingestion once the copy finishes.
+
+The command blocks and reports the ingestion state as it changes:
+
+```text
+Waiting for ingestion to complete... QUEUED -> COPYING -> COPIED -> COMPLETED
+Successfully pushed simulation UUID
+```
+
+### Configure partitions
+
+For `push_local` to resolve files on both sides, client and server must agree on
+a set of *partitions*: short logical names mapped to absolute directories. Add a
+`[partition]` section to your client configuration (see
+[Client configuration](../reference/configuration.md#partition)):
+
+```ini
+[partition]
+data = /home/user/my_simdb_data
+work = /work/imas/shared
+sdcc = /
+```
+
+Mapping `sdcc` to the system root makes any path under `/sdcc/projects/...`
+match, so `/sdcc/projects/my_run` becomes `sdcc:sdcc/projects/my_run`. When
+several partitions contain a file the most specific (deepest) path wins, so a
+catch-all mapping like this never shadows the others.
+
+When you run `push_local`, SimDB checks every input and output path against your
+partitions. A path inside a partition is rewritten to a partition-relative URI —
+`/home/user/my_simdb_data/scenarios/run1.txt` becomes
+`data:scenarios/run1.txt`. The server resolves that URI against its own
+`[partition]` configuration, so the two sides may mount the same storage at
+different absolute paths.
+
 ## Pull
 
 Pull copies a simulation's metadata into your local catalogue and downloads its

docs/reference/configuration.md

@@ -90,6 +90,17 @@ One section per configured remote server. Manage these with
 | --- | --- |
 | `file` | Path to the local SQLite catalogue. Defaults to `sim.db` in the user data directory (for example `~/.local/share/simdb/sim.db`). |
 
+### `[partition]`
+
+Maps logical partition names to absolute directories on this machine. Used by
+`simdb simulation push_local` to rewrite file paths into partition-relative URIs
+that the server can resolve (see
+[Push and pull simulations](../how-to/push-pull.md#configure-partitions)).
+
+| Option | Description |
+| --- | --- |
+| `NAME` | Directory that partition `NAME` is mounted at, for example `data = /home/user/my_simdb_data`. |
+
 ### `[development]`
 
 | Option | Description |

docs/reference/server-configuration.md

@@ -43,6 +43,7 @@ See [Set up PostgreSQL](../how-to/operate-server/set-up-postgresql.md).
 | `copy_files` | No | `True`/`False`: copy uploaded data files into the server's storage. Defaults to `True`. |
 | `copy_ids` | No | `True`/`False`: copy uploaded IMAS IDS data into the server's storage. Defaults to `True`. |
 | `user_upload_folder` | No | Optional staging directory clients upload into before ingest (returned by the `staging_dir` endpoint). Falls back to `upload_folder` if unset. |
+| `max_append_size` | No | Maximum size in bytes of a single resumable-upload chunk, advertised to clients via the `Upload-Limit` header. Defaults to `8388608` (8 MiB). Lower it to fit a reverse proxy's request body limit. |
 
 ## `[flask]`
 
@@ -174,6 +175,7 @@ Used by the optional
 | Option | Required | Description |
 | --- | --- | --- |
 | `data` | No | Directory used for partitioned data, for example `/data/simdb/partition`. |
+| `http` | For `push` | Directory where resumable HTTP uploads are staged before ingestion, for example `/var/lib/simdb/http-staging`. Required for `simdb simulation push` against a v1.3 server. |
 
 ## `[role "NAME"]`
 

docs/reference/uri-schemes.md

@@ -1,8 +1,10 @@
 # URI schemes
 
 The `inputs` and `outputs` in a [manifest](manifest-format.md) reference data
-through URIs. SimDB understands two schemes: `file` for ordinary files and
-`imas` for IMAS data entries.
+through URIs. SimDB understands two schemes you can write yourself: `file` for
+ordinary files and `imas` for IMAS data entries. Two further forms — a remote
+`imas` URI and an `http` staging URI — are produced by SimDB itself and are
+described at the end of this page.
 
 ## `file` scheme
 
@@ -64,3 +66,18 @@ data. MDSplus data must have been written with Access Layer 5 (AL5) or later;
 Access Layer 4 (AL4) data must be migrated first. See
 [Migrate AL4 MDSplus data](../how-to/migrate-al4-mdsplus.md).
 ```
+
+## `http` scheme (internal)
+
+While `simdb simulation push` uploads a simulation to a v1.3 server, each file
+is referenced by a staging URI of the form
+
+```
+http:///<sim_uuid>/file/<path>
+```
+
+The server resolves it against its `http` partition, copies the file into the
+simulation's upload folder, and removes the staged copy (see
+[Stage uploads on the server](../how-to/push-pull.md#stage-uploads-on-the-server)).
+Like the remote `imas` form above, this scheme is generated by SimDB during a
+push; you cannot use it in a manifest.

pyproject.toml

@@ -39,7 +39,7 @@ dependencies = [
     "distro>=1.8.0",
     "email-validator>=1.1",
     "imas-python>=2.0.1",
-    "netCDF4>=1.5",
+    "netCDF4>=1.7.2",
     "numpy>=1.14",
     "pydantic>=2.10.6",
     "python-dateutil>=2.6",

scripts/test_v13_ingestion.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 """Test script for v1.3 simulation ingestion against a running server."""
-from simdb.workers.tasks import _calculate_checksum
+from simdb.checksum import calculate_checksum
 from pathlib import Path
 
 import base64
@@ -37,7 +37,7 @@
 
 
 def generate_simulation_file():
-    checksum = _calculate_checksum(Path("tmp/partition_data/subdir/test_file.txt"))
+    checksum = calculate_checksum(Path("tmp/partition_data/subdir/test_file.txt"))
     return FileData(
         type="FILE",
         uri="data:///subdir/test_file.txt",
@@ -46,7 +46,7 @@ def generate_simulation_file():
     )
 
 def generate_imas_file(relative_path):
-    checksum = _calculate_checksum(Path(f"tmp/partition_data/subdir/{relative_path}"))
+    checksum = calculate_checksum(Path(f"tmp/partition_data/subdir/{relative_path}"))
     return FileData(
         type="IMAS",
         uri=f"data:///subdir/{relative_path}",

src/simdb/checksum.py

@@ -1,14 +1,42 @@
 import hashlib
 from pathlib import Path
+from typing import Callable, Optional
 
 from simdb.imas.utils import SimDBUrl
 
+#: Algorithm used for all catalog checksums.
+CHECKSUM_ALGORITHM = "sha1"
+#: Buffer size for reading files while hashing. Larger reads mean far fewer
+#: syscalls on big files, which noticeably speeds up checksumming.
+READ_CHUNK_SIZE = 1024 * 1024
 
-def sha1_checksum(uri: SimDBUrl) -> str:
-    """Generate a SHA1 checksum from the given file.
+
+def hash_file(
+    path: Path,
+    algorithm: str = CHECKSUM_ALGORITHM,
+    progress: Optional[Callable[[int], None]] = None,
+) -> str:
+    """Return the hex digest of ``path`` computed with ``algorithm``.
+
+    @param progress: optional callback invoked with the number of bytes read for
+                     each block, suitable for advancing a progress bar.
+    """
+    digest = hashlib.new(algorithm)
+    with path.open("rb") as file:
+        for chunk in iter(lambda: file.read(READ_CHUNK_SIZE), b""):
+            digest.update(chunk)
+            if progress is not None:
+                progress(len(chunk))
+    return digest.hexdigest()
+
+
+def file_checksum(uri: SimDBUrl, algorithm: str = CHECKSUM_ALGORITHM) -> str:
+    """Generate a checksum for the file at ``uri``.
+
+    Checksums use :data:`CHECKSUM_ALGORITHM` (SHA-1).
 
     :param uri: the URI of the file to checksum
-    :return: a string containing the hex representation of the computed SHA1 checksum
+    :return: a string containing the hex representation of the computed checksum
     """
     if uri.scheme != "file":
         raise ValueError(f"invalid scheme for file checksum: {uri.scheme}")
@@ -21,8 +49,4 @@ def sha1_checksum(uri: SimDBUrl) -> str:
     if not path.is_file():
         raise ValueError("File appears to be a directory")
 
-    sha1 = hashlib.sha1()
-    with path.open("rb") as file:
-        for chunk in iter(lambda: file.read(4096), b""):
-            sha1.update(chunk)
-    return sha1.hexdigest()
+    return hash_file(path, algorithm)