Offload from an edit timeline: resolve a cut against a drive, copy only what is missing

CHANGELOG.md

@@ -27,6 +27,43 @@ project uses [semantic versioning][semver].
 
 ### Fixed
 
+- **A timeline offload could call a partial cut complete.** With nothing to
+  copy it printed "every reference is already here" and exited 0 while
+  references were missing or ambiguous, and with gaps to copy it reported
+  "Verified" and exited 0 while the reports listed only the copied files. Now
+  every unresolved reference is printed, recorded on the job and listed in the
+  CSV, HTML and PDF reports; the job reads `Incomplete` and exits 4, the same
+  code `resolve` uses.
+- **One file could satisfy several different timeline references.** Two
+  recordings sharing a name (`f2/260223_009.WAV`, `f3/260223_009.WAV`) both
+  resolved to the single copy on the drive. They are now `ambiguous` unless
+  their bytes show which is which. A readable original is compared with the
+  same-named file under the roots (size, then checksum), so a "MISSING MEDIA"
+  slate no longer stands in for real footage the timeline can reach; the
+  original is copied instead, with a warning naming the slate.
+- **Offline clips and image sequences were reported as generated media.** An
+  OTIO `MissingReference`, an FCP 7 clip without a `<pathurl>` and an image
+  sequence are now reported by name as `offline` or `unsupported`, count as
+  unresolved and fail the exit code. A generator, title or matte is still
+  `generated`.
+- **A proxy could stand in without any check.** A same-stem file that yields no
+  frame count no longer satisfies a camera original. A proxy whose count
+  cannot be compared with the original's is shown in the terminal with its
+  `frames unverified` note, carried as a job warning, and keeps the job from
+  reading `Verified`.
+- **Relative timeline paths were read against the working directory.** They
+  are now read against the timeline file's folder, so an unrelated file of the
+  same relative path is never copied. Non-file URLs (`https`, `s3` and so on)
+  are reported as `external` and never matched to a local file by name.
+- **A `--control-file` cancel exited 0.** A cancelled job now exits 130, as
+  Ctrl+C does, and the summary says how many files were not attempted.
+- **`--control-file` could overwrite a source or destination file.** Starting a
+  job wrote `run` to the control path before anything was validated, so a path
+  that named a source clip (or a good copy) truncated it to five bytes, and the
+  job then copied and reported the replacement as Verified. The path is now
+  checked first: it may not be inside the source, be a selected timeline file,
+  be a destination root, or be where a copy would land, and an existing file is
+  only overwritten if it is empty or holds `run`, `pause` or `cancel`.
 - **`verify` passed a destination whose offload had failed a file.** The
   manifest listed only the files that copied, so a clip that failed was simply
   absent and `verify` said "VERIFIED - safe to erase the source". MHL and ASC
@@ -162,6 +199,73 @@ project uses [semantic versioning][semver].
 
 ### Added
 
+- **Pause, resume and cancel from the command line.** `JobControl` has existed
+  since the desktop app needed transport buttons, and is checked once per 8 MiB
+  chunk, but the CLI never passed one — so a job started in a terminal could
+  only be killed. `--control-file PATH` wires one to a file, and
+  `offloader control PATH --pause|--resume|--cancel` drives it from anywhere.
+  A file rather than a signal or a keypress because Windows has almost no
+  signal support, a detached job has no console to type into, and a file needs
+  no port or daemon, survives the terminal closing, and can be read to see what
+  a job is doing.
+
+  The failure modes are the interesting part. The file holds one word;
+  **anything else — empty, garbled, half-written, momentarily locked — is "no
+  opinion" and leaves the job in the state it is already in**, because
+  inferring `cancel` from a damaged control file would let a stray byte stop an
+  offload that is nine hours in. `offloader control` writes through a staging
+  file and renames it into place, so a job polling between chunks cannot read a
+  truncated instruction. Starting a job writes `run` to the path first, so a
+  stale `pause` from a previous job cannot silently stop the next one.
+
+  "One word" means the whole value, not its first token: `cancel pending
+  upload`, the kind of thing an editor or a sync client leaves behind, is
+  damaged content and not an instruction to stop. Contents that are not valid
+  UTF-8 are damage too — that raises a `UnicodeDecodeError` rather than an
+  `OSError`, so it used to abort the checkpoint instead of being read as no
+  opinion.
+
+- **Offloading from an edit timeline.** `offloader resolve --timeline cut.xml
+  --search-root E:\Media` reports which of a cut's media is already on the
+  drive and which is not; `offloader offload --timeline ...` copies the
+  difference, with the same verified copy, checksums and reports as a card.
+  Timelines are read with OpenTimelineIO (`pip install "offloader[timeline]"`),
+  covering FCP 7 XML, FCPXML, EDL, AAF and `.otio` — but trusted with media
+  references only, never with structure: on the file this was measured
+  against, the adapter recovered all 334 references exactly while reporting
+  29.97 fps and 12 audio tracks for a sequence that declares 24 and 23. See
+  [`docs/timeline.md`](docs/timeline.md).
+
+  The resolver **refuses to choose between two files that share a name.** On
+  that conform, 20 basenames had more than one copy under the search root:
+  seven byte-identical and harmless, thirteen genuinely different files,
+  eleven of those being "MISSING MEDIA" stand-in slates from an earlier pass
+  sitting beside the real archival footage. A relink by filename picks one at
+  random and the clip reports as online either way. Byte-identical duplicates
+  resolve normally and are not copied again, which on that job avoided 1.4 GB
+  of transfer and, more to the point, avoided manufacturing 32 filename
+  collisions on a drive that had none. A camera original may be satisfied by a
+  proxy already on the drive, but only where the frame counts agree — a proxy
+  one frame short moves every edit point after it.
+
+  A `file:` URL keeps its leading slash unless a drive letter follows it.
+  Stripping it unconditionally turns `/Volumes/Edit/a.mov` into the *relative*
+  path `Volumes/Edit/a.mov`, which still resolves by basename and so looks
+  correct on Windows, while every "is it where the timeline says?" test quietly
+  answers no. An editor cutting on a Mac addresses every clip that way, so it
+  is the common case rather than an edge; CI on macOS and Linux is what caught
+  it.
+
+- **`OffloadOptions.selection`**, an explicit file set for `engine.run` in
+  place of scanning one source root. Files may come from any number of volumes
+  and each carries its own destination-relative path, so the engine infers no
+  layout. A selection is held to a narrower safety rule than a card's: a card
+  refuses a destination inside the source, which would wrongly refuse the
+  ordinary timeline case of collecting a cut's gaps into a folder on the drive
+  they were resolved against. Instead, no file being read may sit at or beneath
+  somewhere being written, and a destination-relative path that is absolute or
+  climbs out with `..` is refused before the job starts.
+
 - **The desktop app can set `replace_existing`.** The engine refuses to replace
   an existing destination file whose content differs, and its message pointed
   at `--replace-existing`, a flag the app could not pass. Presets now carry

CONTRIBUTING.md

@@ -37,7 +37,7 @@ pip install -e ".[dev]"
 `ffmpeg` and `ffprobe` on `PATH` are optional — the suite runs without them.
 
 ```sh
-pytest                      # 713 tests, about 20s
+pytest                      # 834 tests, about 20s
 pytest --fuzz               # property tests at 3000 examples each, about 3 min
 ruff check src tests
 pytest --cov=offloader --cov-report=term-missing

README.md

@@ -91,6 +91,8 @@ offloader verify D:\video\080426\A001
 | Command | What it does |
 | --- | --- |
 | `offload` | copy and verify a source to one or more destinations |
+| `resolve` | report which of an edit timeline's media is already here, copying nothing |
+| `control` | pause, resume or cancel a running offload from another terminal |
 | `verify` | re-check an offloaded tree against its manifests |
 | `report` | regenerate paperwork for an existing tree, copying nothing |
 | `info` | show tool and environment status |
@@ -101,6 +103,7 @@ offloader verify D:\video\080426\A001
 | Flag | Meaning |
 | --- | --- |
 | `--source PATH` | card or folder to offload |
+| `--timeline PATH` | offload an edit timeline's media instead of a folder; see [From an edit timeline](#from-an-edit-timeline) |
 | `--dest PATH` | destination root; repeat for multiple copies |
 | `--hash ALGO` | `xxh3-64` (default), `xxh3-128`, `xxh64`, `xxh64be`, `md5`, `sha1`, `sha256`, `c4`, `none` |
 | `--verify MODE` | `full` (default), `source-only`, `none` |
@@ -115,6 +118,7 @@ offloader verify D:\video\080426\A001
 | `--exclude GLOB` | extra filename pattern to skip; repeatable |
 | `--flat` | do not recreate the source folder structure; refused if two files would land on one path |
 | `--skip-existing` | skip files already present at matching size |
+| `--control-file PATH` | make the job pausable from another terminal; see [Pausing a running job](#pausing-a-running-job) |
 | `--proxies-first` | copy the camera's proxy folders before the originals (default) |
 | `--originals-first` | copy in plain tree order instead |
 | `--replace-existing` | let a copy replace an existing destination file whose content differs; by default that file is kept and the copy fails (offload only; in the app, *Replace existing files whose content differs* under Options) |
@@ -124,20 +128,117 @@ offloader verify D:\video\080426\A001
 | `--no-probe` | skip ffprobe metadata and thumbnails |
 | `--quiet` | suppress progress |
 
-Exit status for `offload`:
+Exit status for `offload` (and, where they apply, `resolve`, `report` and
+`verify`):
 
 | Code | Meaning |
 | --- | --- |
 | `0` | Verified (or Copied under `--verify none`), and every report was written |
-| `1` | a file failed, or a source folder could not be listed |
-| `2` | usage or I/O error; nothing was attempted |
-| `3` | a destination was refused as unsafe; nothing was copied |
-| `4` | nothing failed, but the job is not Verified: an empty source, files skipped by `--skip-existing` (checked by size only), or a cancel |
+| `1` | failed: a copy, verify or scan failure (a file failed, or a source folder could not be listed) |
+| `2` | error: usage, environment (such as a missing timeline adapter) or an unreadable manifest; nothing was attempted |
+| `3` | refused: an unsafe destination or control file; nothing was copied |
+| `4` | incomplete: nothing failed, but not everything asked for was verified (unresolved timeline references, an empty source, files skipped by `--skip-existing` and checked by size only) |
 | `5` | the copies verified but a report or manifest could not be written |
-| `130` | interrupted |
+| `130` | cancelled: Ctrl+C or a control-file cancel |
 
 `report` exits `5` when a report could not be written and `0` otherwise.
 
+### Pausing a running job
+
+A long offload gets started detached, over ssh, or by a scheduler, and the
+person who wants it paused is rarely sitting at that terminal. Start it with a
+control file and it can be driven from anywhere:
+
+```sh
+offloader offload --source E:\ --dest D:\A001 --control-file D:\A001\job.control
+
+# from any other terminal
+offloader control D:\A001\job.control --pause
+offloader control D:\A001\job.control --resume
+offloader control D:\A001\job.control --cancel
+offloader control D:\A001\job.control            # what state is it in?
+```
+
+The job reads the file **once per 8 MiB chunk**, so a pause takes effect inside
+a second even in the middle of a 14 GB clip. Paused, it holds its place with
+the file still in flight; resumed, it carries on from that chunk rather than
+restarting the file. A cancel keeps every finished file and discards the one in
+flight, which never had its real name.
+
+The file holds one word: `run`, `pause` or `cancel`. Deleting it releases the
+job. **Anything else is "no opinion" and leaves the job exactly as it is** —
+empty, garbled, half-written, or momentarily unreadable because another process
+has it open. That asymmetry is deliberate: inferring `cancel` from a damaged
+control file would let a stray byte stop an offload that is nine hours in, and
+a small text file is precisely what a sync client rewrites in two steps.
+`offloader control` writes through a staging file and renames it into place, so
+a job polling between chunks can never read a half-written instruction.
+
+Starting a job also *claims* the path by writing `run` to it. A control file
+left saying `pause` by a previous job would otherwise stop the next one before
+it copied a byte, with nothing on screen to explain why. The path is validated
+first and claiming never truncates anything that is not a control file: a path
+inside the source, on a selected media file, on a destination root, or naming an
+existing file that is not empty or a control word is refused before anything is
+written.
+
+Why a file rather than a signal or a keypress: Windows has almost no signal
+support beyond SIGINT, a detached job has no console to press a key in, and a
+file needs no port and no daemon, survives the terminal going away, and can be
+read to see what a job is doing. The desktop app drives the same `JobControl`
+through its transport buttons.
+
+### From an edit timeline
+
+A card offload knows its source. This is the other job: a cut comes back from
+an editor, and the question is *which files does this timeline need, and are
+they all here?*
+
+```sh
+pip install "offloader[timeline]"
+
+offloader resolve --timeline "01 Chairs Row V6.xml" --search-root E:\ChairsDoc
+offloader offload --timeline "01 Chairs Row V6.xml" --search-root E:\ChairsDoc \
+                  --dest E:\ChairsDoc\RowV6_Media
+```
+
+`resolve` copies nothing and prints the answer. `offload` copies only the files
+that are not already under a `--search-root`, with the same verified copy,
+checksums and reports as a card.
+
+| Flag | Meaning |
+| --- | --- |
+| `--timeline PATH` | the cut to read: `.xml` (FCP 7), `.fcpxml`, `.edl`, `.aaf`, `.otio` |
+| `--search-root PATH` | where the media already lives; repeatable, and required |
+| `--adapter NAME` | override the OpenTimelineIO adapter chosen from the suffix |
+| `--no-proxy-substitution` | do not let a proxy on disk stand in for a camera original |
+| `--layout mirror\|flat` | keep each file's path below its volume root (default), or put everything in one folder |
+| `--csv PATH` | (`resolve`) write the full per-reference table |
+
+Every reference gets one status: `present`, `substituted`, `gap`, `ambiguous`,
+`missing`, `offline`, `unsupported`, `external` or `generated`. Only `gap` is
+copied. While anything is unresolved (every status but `present`, `gap` and
+`generated`, plus a proxy whose frame counts could not be checked), `resolve`
+exits 4 and `offload` copies the gaps, lists what it could not supply in the
+terminal and every report, calls the job `Incomplete` and exits 4. See
+[docs/timeline.md](docs/timeline.md).
+
+The part worth knowing before you trust it: **it refuses to choose between two
+files that share a name.** On the conform it was built against, 20 basenames
+had more than one copy under the search root. Seven were byte-identical and
+harmless. Thirteen were different files -- eleven of them "MISSING MEDIA"
+stand-in slates from an earlier pass, sitting beside the real archival footage
+that arrived later. Relinking by filename picks one at random, and when it
+picks a slate the clip **reports as online**. So the tool reports every
+candidate, offers the closest-path match as an explicitly labelled guess, and
+copies nothing.
+
+OpenTimelineIO is trusted with media references and nothing else. It recovered
+all 334 of that file's references exactly, including the clipitems that name
+their file by id and carry no path; on the same file it reported 29.97 fps and
+12 audio tracks for a sequence declaring 24 fps and 23 tracks. Full account and
+the adapter table in [`docs/timeline.md`](docs/timeline.md).
+
 ### `verify`
 
 ```sh
@@ -486,6 +587,7 @@ general-purpose tool reports a filename, a size, and a placeholder icon.
 | [`docs/braw.md`](docs/braw.md) | Blackmagic RAW container parsing, proxy pairing, and the interrupted-recording check |
 | [`docs/ascmhl.md`](docs/ascmhl.md) | ASC MHL v2.0, and how it was validated against the reference implementation |
 | [`docs/ixml.md`](docs/ixml.md) | Broadcast WAV chunk walking, the iXML slate, and where sound timecode comes from |
+| [`docs/timeline.md`](docs/timeline.md) | Resolving an edit timeline to its media, what OpenTimelineIO is and is not trusted with, and why ambiguity refuses |
 
 ## Library
 
@@ -517,13 +619,13 @@ what makes the report layer testable without moving bytes.
 
 ```sh
 pip install -e ".[dev]"
-pytest                      # 713 tests, ~20s
+pytest                      # 834 tests, ~20s
 pytest --fuzz               # same suite, 3000 examples per property (~3 min)
 ruff check src tests
 pytest --cov=offloader --cov-report=term-missing
 ```
 
-713 tests at 86% line coverage. They cover formatting against the reference's
+834 tests at 86% line coverage. They cover formatting against the reference's
 exact strings, checksum vectors and streaming equivalence, copy/verify
 behaviour including simulated destination corruption, pause/resume/cancel
 concurrency, retry discrimination, BRAW container parsing, ffprobe parsing,

ROADMAP.md

@@ -23,6 +23,16 @@ checksum-cost labels in every picker, a running-job summary line, the
 local-first parallel drive scan, jobs named after the volume label, and a PDF
 title that tells reports apart.
 
+Offloading from an **edit timeline** rather than a card: `offloader resolve`
+and `offload --timeline` read an NLE's interchange file with OpenTimelineIO,
+work out which of its media is already under a search root, and offload only
+what is not. See [`docs/timeline.md`](docs/timeline.md). It refuses to choose
+between two files that share a name, which is not fastidiousness: on the
+conform it was written against, thirteen names resolved to *different* files,
+eleven of them "MISSING MEDIA" stand-in slates from an earlier pass sitting
+beside the real archival footage that arrived later. A relink by filename picks
+one at random, and the clip reports as online either way.
+
 The PDF matches a real ShotPut Pro report's geometry, measured from its content
 streams. ASC MHL is diffed against the reference implementation's own worked
 example. BRAW metadata comes out of the container because ffprobe cannot read
@@ -49,7 +59,11 @@ takes something on trust, and it is listed under "What is still not protected"
 in [`docs/data-safety.md`](docs/data-safety.md) for that reason. A checksum
 variant would make it a safe option rather than a fast one.
 
-*Where:* `engine.py`, in the `skip_existing` branch.
+*Where:* `engine.py`, in the `skip_existing` branch. `timeline._same_file` is
+the comparison already written: size first, because it settles almost every
+case for a stat, then the checksum. What is missing is only the wiring into the
+engine's own skip, where the file being skipped is at the destination rather
+than under another search root.
 
 ### `previousPath`, so a rename survives a generation