feat: Windows on ARM64 (NVIDIA RTX Spark) support; replace passlib with bcrypt 5

[lstein]

Summary

Moved from invoke-ai#261 and merged up to this repository's main (torch 2.13 / cu130, ROCm 10 on Windows, the database layer). Every new upstream win32 marker is narrowed to x64 so Windows ARM64 keeps NVIDIA's torch in every extra.

Invoke could not be installed on Windows on ARM64 (NVIDIA's RTX Spark developer preview): every marker treated win32 as x64, the lockfile carried only win_amd64 wheels, and several dependencies have no ARM64 build at the pinned versions. With this PR, uv sync --frozen and uv pip install on a native ARM64 Python 3.12 produce a working install with NVIDIA's torch 2.14.0+cu134 (from https://pypi.nvidia.com/nvtorch_oot/), CI proves it on GitHub's windows-11-arm runners, and every other platform resolves exactly as before apart from one deliberate change (bcrypt 5 replaces passlib).

How it fits together:

• pyproject.toml forks the packages whose ARM64 wheels start at newer versions (numpy 2.3, onnxruntime 1.24, bitsandbytes 0.50.1, PyWavelets 1.10; main's onnx 1.21.0 and sentencepiece 0.2.2 already have ARM64 wheels, and its move to PyJWT removed the cryptography 46.0.x pin python-jose needed) behind sys_platform == 'win32' and platform_machine == 'ARM64', routes torch/torchvision to NVIDIA's index there for every extra, and sets uv's required-environments so a wheel-only package without an ARM64 wheel fails at lock time.
• OpenCV, blake3 and tiktoken have no ARM64 wheel anywhere. A new dispatch-only workflow builds them on windows-11-arm and publishes them, with a flat index.html, as assets of release win-arm64-wheels-v1 in this repository (byte-identical copies of the wheels first built in the now-archived InvokeAI-7); pyproject consumes that page as a flat index and pins the exact versions so the manual --find-links path prefers the wheels over PyPI's sdists. Pull-request runs of that workflow only build and verify. Published tags are immutable because the lockfile pins the assets by hash: a dispatch must name a new release_tag (no default), the publish job runs in the win-arm64-wheels environment, and it never replaces an asset — it uploads only names the release lacks, skips identical ones (so "Re-run failed jobs" can finish an interrupted upload) and fails on any other difference.
• scripts/check_aarch64_lock.py becomes scripts/check_platform_locks.py: linux/aarch64 still requires every extra to carry its own aarch64 torch pin, and win32/ARM64 additionally requires torch from NVIDIA's registry and walks the whole dependency closure for win_arm64 wheels (uv's required-environments cannot see packages that also publish an sdist). The closure check blocks: every dependency bump must keep a win_arm64 (or pure) wheel; .github/AGENTS.md describes the options when it fires.
• Runtime: mediapipe (no solutions API on ARM64) and umap/numba (no wheel for 3.12) are excluded there; the face nodes import mediapipe lazily and name the platform when it is the cause, the image map falls back to a PCA projection on Windows ARM64 only (elsewhere a failed umap import is a failed fit; the stored params record the method, so a PCA map is refit once umap imports), password hashing calls bcrypt directly (passlib 1.7 cannot use bcrypt >= 4.1, the first release with ARM64 wheels), the bitsandbytes guards survive a build whose native library fails to load and log the real error once (the loaders' error carries it), and startup logs the torch build and host architecture. One shim is specific to NVIDIA's build: without a CUDA device it dies with an access violation in torch.cuda.is_current_stream_capturing(), which transformers calls on every forward pass, so on such a machine the app (and the test session) make it answer False.
• Docs cover the native interpreter request (cpython-3.12-windows-aarch64-none), the install options, and the platform's feature differences.

The launcher needs a companion change (arch detection, aarch64 uv, native Python request, arm64 package build); it is prepared but not part of this PR. pins.json needs no change: the launcher's bootstrap path installs from the release's own lockfile.

Related Issues / Discussions

NVIDIA RTX Spark Developer Preview: https://forums.developer.nvidia.com/t/rtx-spark-developer-preview/377106

QA Instructions

After merging this repository's main (Linux, worktree venv):

• uv lock / uv lock --locked: resolved; against main's lock the changes are the ARM64 forks plus bcrypt 3.2.2 → 5.0.0 and passlib removed.
• check_platform_locks.py uv.lock → linux/aarch64 OK, win32/ARM64 OK (111 packages per ARM64 universe, 144 with test, none without a wheel); check_pins.py → consistent.
• pytest tests/backend/util/test_devices.py tests/app/services/auth tests/app/services/users tests/app/services/shared tests/app/util tests/test_check_pins.py tests/test_check_platform_locks.py tests/backend/util/test_rocm_sdpa_head_dim_guard.py → 1705 passed, 21 skipped. ruff check / ruff format --check clean.

After the first review round (Linux, worktree venv):

• uv lock --locked, check_pins.py and check_platform_locks.py pass; moving the wheel index changes only URLs in uv.lock (identical modulo the repository name).
• pytest -n 4 over the touched areas (image index/map, routers, devices, quantization, patches, model loading, imports, platform locks, wheel publish) → 2043 passed, 153 skipped; the image-map suites again with Windows ARM64 simulated (platform forced, umap/numba hidden) → 346 passed, 1 skipped.
• The new tests were checked against the unfixed code: each fails there (umap fallback scope, method staleness in the worker and the router, cache leak, float64 copy, shim install, bnb error text, per-extra aarch64 pin, duplicate output).

Before the move (InvokeAI-7, Linux, worktree venv):

• uv lock then uv lock --locked: resolved; the diff vs main is the ARM64 forks plus bcrypt 3.2.2 → 5.0.0 and passlib removed (verified by parsing both locks: no other package's version or wheel list changed).
• uv run --no-project --with 'packaging<26' --python 3.12 scripts/check_platform_locks.py uv.lock → linux/aarch64 OK, win32/ARM64 OK (115 packages in each ARM64 universe, none without a wheel).
• python3 scripts/check_pins.py → consistent.
• pytest tests/app/services/auth tests/app/services/image_index tests/app/routers/test_utilities.py tests/backend/image_util tests/test_imports.py tests/test_check_platform_locks.py tests/test_check_pins.py → pass.
• Full suite (pytest -n logical): 8250 passed; 13 failures are the same GPU tests that fail identically on an untouched main checkout because both GPUs on this machine were at 99%/82% use (HIP out-of-memory in test_flux2_working_memory, test_model_cache_*, test_pid_chunked_equivalence), plus the repo-lock check that passes since the lock was regenerated.
• ruff check / ruff format --check clean; pnpm -C docs build → 326 pages, no new warnings.

GitHub (InvokeAI-7):

• build-win-arm64-wheels: OpenCV, blake3 and tiktoken built and smoke-imported on windows-11-arm; release win-arm64-wheels-v1 carries the three wheels and index.html; the publish job's uv pip compile --find-links <page> verification passed. (First OpenCV attempt failed: MSVC has no __fp16, which the DNN Winograd kernel uses once NEON_FP16 is in the CPU baseline; fixed by pinning the baseline to NEON + DOTPROD dispatch.)
• PR CI on the final commit: every check green. The py3.12: windows-arm64 lane installs from the lock on a native ARM64 interpreter (platform.machine() == "ARM64" asserted), imports the self-built wheels, and runs the full suite: 14,608 passed, 239 skipped, 8 xfailed in 31 minutes (run 39166449058; 37 minutes end to end, against a 60-minute job timeout, with a 600 s per-test timeout). Its runs surfaced these ARM64-only failures, all fixed: the stream-capture crash above (which had taken an xdist worker down and left ~2,500 tests unrun); torch 2.14 rejecting an SDPA test's explicit mask combined with is_causal=True; a frame fixture that overflowed uint8 (numpy 2 no longer wraps); a MediaPipe gate test that assumed a non-ARM64 host; a leaked ffmpeg (Chocolatey's ffmpeg is a launcher shim, and cancellation killed only the shim, leaving an endless test source pinning the runner's CPU); and frame counts doubled for clips whose soundtrack outlasts the video (the self-built OpenCV has no FFmpeg backend, and Media Foundation counts frames from the container duration).

Not verified (no hardware): CUDA generation on sm_121, cuDNN SDPA priority on this torch build, cudaMallocAsync acceptance in configure_torch_cuda_allocator(), bitsandbytes 0.50.2's CUDA kernels, VRAM accounting on unified memory. The docs mark the platform as a developer preview.

Review

Material findings resolved before the move: the wheel workflow's pull-request trigger could republish non-reproducible wheels over the assets the lock pins by hash (publishing is now dispatch-only under a concurrency group, and wheels are verified before upload); a top-level import mediapipe would have failed test_imports on ARM64; the woff2 font test needs brotli, absent on ARM64; the checker's closure walk skipped a package's extras once a plain edge had reached it (fixed, with a fixture that fails against the old walk); the MediaPipe error attributed every import failure to the platform; the admin guide still generated password hashes with passlib; bitsandbytes guards catch (ImportError, OSError, RuntimeError) rather than everything; the ARM64 test lane no longer caches uv's store. An old-vs-new lock comparison across the non-ARM64 platforms showed no change beyond bcrypt/passlib.

First review round (joshistoast, Pfannkuchensack): the wheel workflow could still replace pinned assets on a dispatch with default inputs, and the wheels lived in an archived repository (both fixed as above); the PCA fallback applied on every platform, was never refit once UMAP became available, and copied the whole matrix to float64; test guards for mediapipe, numba and brotli applied on every platform; one test leaked a cached None for UMAP into later tests, and the stream-capture shim was installed for the whole Linux test session; the bitsandbytes guards dropped the real error; the aarch64 check stopped requiring per-extra fallback pins and printed lines twice. All fixed with regression tests.

After the merge, the ARM64 cryptography pin had nothing left to serve (PyJWT does not need it) and was removed. Second review round (Pfannkuchensack): the lane timeout and the merge with main (fixed as above), plus the projection-method check (now one-way, so a UMAP map survives on a host without umap), the thread hop on every /points call, legacy-hash vectors (>72 bytes, $2a$/$2y$), and test and docs nits. Remaining risks: GPU behaviour on sm_121 is unverified (see QA), and the ffmpeg probes outside run_ffmpeg_process still kill only a shim on timeout.

Compatibility / Rollout

• The published wheel's metadata now carries ARM64-only exact pins; nothing changes for other platforms' resolution.
• All platforms: passlib is replaced by direct bcrypt 5 calls for password hashing (passlib 1.7 cannot use bcrypt >= 4.1). Worth a changelog line.
• All platforms: the xpu extra now resolves bitsandbytes 0.50.2 on Linux x86_64 and Windows x64, the version the cpu, cuda and rocm extras already use (it was 0.49.2 only for xpu).
• Stored password hashes are unaffected: bcrypt verifies the $2b$12$ hashes passlib produced (covered by a regression test), and new hashes use the same cost factor.
• Bumping a self-built wheel means dispatching the workflow with a new release tag and updating the tag and pins in pyproject.toml together, then re-locking; never overwrite a tag that a tagged Invoke release's lock names.
• The launcher change ships separately; until then Windows ARM64 users install manually (documented). An existing launcher install on ARM64 runs an emulated x86_64 Python, which CPython 3.12 reports as ARM64, so it selects the ARM64 packages and cannot update to this release. The release notes should point those users to the new FAQ entry (reinstall with a native Python); ideally the launcher fix ships first.
• Risk: NVIDIA's nvtorch_oot index carries a single torch build (2.14.0+cu134); if it is withdrawn, the ARM64 lock breaks until a re-lock.
• Maintainers: add required reviewers to the win-arm64-wheels environment so a wheel dispatch cannot publish without approval.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[joshistoast]

One release-publishing defect that can invalidate existing Windows ARM64 lockfiles when a wheel tag is reused.

[Pfannkuchensack]

I reviewed the code on an x64 box; I have no ARM64 hardware. The overall approach holds up. I compared the old and new uv.lock for every non-ARM64 environment × extra, and versions, wheels and hashes are identical apart from bcrypt/passlib (and one bitsandbytes change, see below). Every win32 marker is now qualified, both new indexes are explicit, and existing passlib hashes ($2a$/$2b$/$2y$, unicode, >72 bytes) verify under bcrypt 5. Locally ruff, uv lock --locked, check_pins.py, check_platform_locks.py and the focused tests pass.

A few things I'd like addressed before merge.

Before merge

1. The wheel index lives in an archived repo. pyproject.toml:573 and manual.mdx point at invoke-ai/InvokeAI-7, which is archived. Nobody can dispatch a v2 there, and every future release would lock assets owned by that repo. Suggestion: create win-arm64-wheels-v1 in invoke-ai/InvokeAI with byte-identical copies of the three wheels. The hashes stay the same, so this is only a URL change, a regenerated index.html and a re-lock.
2. One dispatch with default inputs breaks every locked ARM64 install. build-win-arm64-wheels.yml defaults release_tag to win-arm64-wheels-v1 (L27), the tag the lock pins by hash, and uploads with --clobber (L224). The builds aren't reproducible, so the wheels get replaced. After that, every ARM64 uv sync --frozen fails on a hash mismatch, and uv lock --locked goes red on every PR. Please make the tag a required input with no default, and fail if the tag already has assets unless an explicit "resume upload" input is set. A protected environment on the publish job and a top-level permissions: contents: read would also help.

Should fix

3. x64 Python on Windows ARM64 stops installing. CPython 3.12's platform.machine() asks WMI for the physical CPU, so an emulated x86_64 interpreter reports ARM64. Markers then pick the ARM64 fork while the install tags are win_amd64. torch 2.14.0+cu134 has only a win_arm64 wheel, so the install fails. Users on the current x64 launcher (uv installs x86_64 Python there by default) would break as soon as a release contains this PR, unless the launcher fix ships first. Markers can't fix this, but the release order and a release note should account for it.
4. One resolution change isn't mentioned in the description. bitsandbytes moves 0.49.2 → 0.50.2 for the xpu extra on linux x86_64 and win x64. Either mention it and sanity-check it, or pin it.
5. The PCA fallback applies on every platform.
   • _umap_class() turns any ImportError into PCA. On x64, a broken numba/llvmlite now degrades silently instead of failing and retrying.
   • The result is stored with projection_params(), which doesn't record the method, and the cache check only compares scope_hash. So the PCA map is served even after the install is fixed, until the gallery changes.
   • The SVD on the full N×D float64 matrix costs about 25 KB per point (about 4 GB at 170k images). That path is now the default on every ARM64 install.
   • Suggestion: fall back only on win/ARM64, record the method in the params, and use eigh on the D×D covariance instead.
6. Some test guards are weakened on all platforms, not just ARM64.
   • tests/test_imports.py:25 adds mediapipe_face_common to KNOWN_IMPORT_ERRORS unconditionally.
   • test_umap_import.py:16 (importorskip("numba")) and test_utilities.py:249 (importorskip("brotli")) skip on any platform.
   • Together with the silent fallbacks, x64 CI would stay green if umap, numba, mediapipe or brotli dropped out of the x64 resolution. Please make these conditional on sys.platform == "win32" and platform.machine() == "ARM64".
7. Two test-isolation problems.
   • test_umap_missing_falls_back_to_pca calls compute_umap again at L496, after the finally: cache_clear(), while umap is still patched to None. That caches None beyond the test, so later tests in the same xdist worker silently run PCA instead of UMAP.
   • tests/conftest.py:59 installs the stream-capture shim at import time and never restores it. Linux CI runs PyPI's CUDA torch without a GPU, so the shim is live there, and test_answers_false_without_a_device_and_installs_once passes without exercising the install path (the sentinel check returns early).
8. The bitsandbytes guards drop the real error. They now catch (ImportError, OSError, RuntimeError) without logging it. When bnb is installed but its native library fails to load on x64 CUDA, users later see "Please install bitsandbytes", which is misleading. Logging the exception once and including it in the load-time error would fix that.

Minor

• Contributor runbook: With required-environments plus the closure check, every dependency bump is now gated on win_arm64 wheel availability. A short section in .github/AGENTS.md on what to do when that gate fires (fork pin, marker exclusion, self-built wheel), and how to bump a self-built wheel, would save contributors a lot of guessing. It's also worth deciding whether the closure check should block or only warn while the platform is a developer preview.
• NVIDIA index: nvtorch_oot carries exactly one torch version (2.14.0+cu134). If NVIDIA pulls it, the lock breaks. Worth noting as a risk.
• Changelog: Replacing passlib with bcrypt 5 affects authentication on every platform. It deserves a mention in the title or changelog, not just under "ARM64 support".
• aarch64 check: check_platform_locks.py merges the base edges into every extra, so the linux/aarch64 check no longer catches a missing per-extra fallback pin. The old script did; removing the cpu extra's aarch64 torch edges fails the old check and passes the new one. That matches what uv installs, but the description says the check is unchanged. The aarch64 lines are also printed twice now.
• Stale text:
  • The pyproject.toml comment on onnx-cuda/onnx-directml says required-environments rejects them, but the markers just drop them.
  • The checker's docstring still says "platforms no test job installs on".
  • The test fixtures still model the removed cryptography/python-jose pin.
• patchmatch: pypatchmatch logs ERROR patchmatch failed to load or compile on every ARM64 startup. A sentence in patchmatch.mdx would keep that from being reported as a bug.
• Commit trailers: The commits carry Co-Authored-By: Claude and Claude-Session: trailers, which AGENTS.md asks us not to include. Please strip them when squashing.

CI: py3.12: windows-arm64 was still running when I finished, and it's the first run of the merged code in this repo, so its result is still open. Everything else is green.

[lstein]

@Pfannkuchensack thanks for the thorough review. I confirmed each point against the code before fixing it. Everything is in 3752285..9a2823d, and each fix has a regression test that fails against the old code.

Before merge

1. Archived repo: fixed. win-arm64-wheels-v1 now exists in this repo. It holds byte-identical copies of the three wheels (same SHA-256) and an index.html regenerated for this repo. pyproject.toml, manual.mdx and uv.lock point at it. With the repo name normalized, the lock diff is identical, so only URLs changed.

2. Republishing over the pinned tag: fixed as you suggested. Details are in the reply to joshistoast's comment:

   • release_tag is required and has no default;
   • assets are never replaced, but identical ones are skipped, so "Re-run failed jobs" can still finish an interrupted upload;
   • there's a top-level permissions: contents: read, and the publish job runs in a win-arm64-wheels environment.

   I went with "refuse anything that differs" instead of a separate "resume upload" input. A rerun of only the publish job reuses the same build artifacts, so their digests match and the rerun goes through; any fresh build won't match. Once this merges, required reviewers still have to be added to that environment in the repo settings.

Should fix

3. x64 Python on Windows ARM64: confirmed, and markers can't fix it. I added a FAQ entry, "Updating fails on Windows on ARM64…", which says to reinstall with a native Python, and linked it from the system requirements. The PR description's rollout section now says the release notes must point to it and that the launcher fix should ideally ship first.
4. bitsandbytes for xpu: confirmed. It's a side effect of the fork unification: the xpu extra now gets 0.50.2 on Linux x86_64 and Windows x64, the version cpu, cuda and rocm already resolve there. I kept 0.50.2 rather than pinning back to 0.49.2, since x64 CI already imports that version, and noted the change in the description. I have no XPU hardware to check its kernels on.
5. PCA fallback: all three sub-points fixed.
   • _umap_class() falls back only on Windows ARM64. Elsewhere the ImportError propagates, so the existing failed-fit and retry path handles it. functools.cache doesn't cache the raise, so a repaired install is picked up.
   • projection_params() records method. Both the worker's cache check and /points treat a stored PCA map as stale once umap imports, even when the scope is unchanged. Rows written before this change are read as umap, since main has no fallback. Maps of 3 points or fewer, and failed (empty) maps, are exempt, so the retry budget and refit loops behave as before.
   • PCA now streams a D×D float64 scatter matrix in 8192-row chunks and uses eigh, with a fixed sign. A tracemalloc test asserts that peak memory stays under half of an N×D float64 copy.
6. Weakened test guards: fixed. Allowing mediapipe_face_common to fail import, and the numba and brotli skips, now apply only when is_windows_arm64() is true. That's a new helper in invokeai/backend/util/host_platform.py, which the mediapipe gate uses too. With numba or brotli hidden on Linux, those tests now fail instead of skipping.
7. Test isolation: both problems fixed.
   • The umap-hiding fixture clears _umap_class's cache on exit as well as on entry. A test checks that nothing cached survives the block.
   • tests/conftest.py installs the stream-capture shim only on Windows ARM64. The shim test now installs over a fresh, unshimmed stand-in and asserts the function was actually replaced.
   • Our own review found one more leak: test_apply_monkeypatches_installs_the_guard left the shim installed on a CUDA build without a GPU. That test now restores the original.
8. bitsandbytes errors: fixed. A new invokeai/backend/quantization/bnb_import.py records the first failure from any guard. It logs once, unless bitsandbytes simply isn't installed, as on macOS. The loaders' error now reads "bitsandbytes is installed but failed to load: …" with the real cause.

Minor

• Contributor runbook: added as "Windows ARM64 lock gate" in .github/AGENTS.md. It covers pinning the last version with a wheel, dropping the package by marker with a degraded path, or self-building it under a new tag. On block versus warn: I'm keeping the closure check blocking. A warning would let a release ship a lock that fails partway through installing on every ARM64 machine, and the contributor doing the bump is the person best placed to choose one of those options.
• NVIDIA index: noted as a risk in the description.
• Changelog: the PR title now names the bcrypt 5 change, and the compatibility section flags that it affects every platform.
• aarch64 check: you were right on both counts.
  • The base torch==2.7.1 edge happened to satisfy every extra, so removing the cpu extra's aarch64 edges passed the check. linux/aarch64 now requires each extra to carry its own pin, and your mutation fails again.
  • The duplicate lines came from that same base-plus-extra edge pair, and win_arm64 printed them twice as well. Each package now prints once per extra.
• Stale text: fixed the onnx extras comment, the checker docstring and --help, and the fixtures. The fixtures now model the blake3 index fork and transformers→tokenizers in place of cryptography and python-jose.
• patchmatch: patchmatch.mdx now says the patchmatch failed to load or compile error at startup is expected on ARM64.
• Commit trailers: I'll strip them when squashing.

Checks:

• uv lock --locked, check_pins.py and check_platform_locks.py are green.
• The touched suites pass on Linux.
• The image-map suites also pass with Windows ARM64 simulated, with the platform forced on and umap and numba hidden. That's the lane where our own review caught four tests that had assumed x64.

[Pfannkuchensack]

Thanks for the quick turnaround. I re-reviewed head 9a2823d. Almost everything from my first pass is resolved:

• Wheels: they're hosted here now. win-arm64-wheels-v1 digests match the lock, and only URLs changed.
• Publish workflow: it requires a new tag and --clobber is gone. plan_release_upload.py refuses on any difference, and the protected win-arm64-wheels environment and permissions: contents: read are in place.
• Image map: the PCA fallback is ARM64-only, records the method in the params, refits once UMAP imports, and uses the D×D scatter matrix. That's about 0.17 GB extra at 300k points, down from 2.5 GB at 100k before.
• bitsandbytes: guards now log the real failure and carry it into the loader error.
• Tests and checker: x64 skips are confined to ARM64, the conftest shim only installs there, the PCA cache leak is gone, and the aarch64 check again requires a per-extra pin.
• Disclosure and docs: the bitsandbytes xpu change is disclosed, and the FAQ and patchmatch docs are added.

Locally, ruff, uv lock --locked, check_platform_locks.py and check_pins.py pass, along with the focused tests (also with bcrypt 5). The old-vs-new lock comparison again differs only in bcrypt/passlib and bitsandbytes for the xpu extra.

Still blocking

1. The py3.12: windows-arm64 lane times out on both heads. Runs 38663611897 and 38676871057 were both cancelled at the 60-minute job timeout at about 74 %. Both show the same 8 failures on the 73 % progress line (...FFF.FFF.......FF...), and the progress has 5–8 minute gaps, so something is hanging or very slow. Because the job is cancelled before pytest prints its summary, the failing tests can't be identified from the log. Adding -rf, a junit report, or a per-test --timeout would surface them. The lane numbers in the description ("every check green", 13 min) come from InvokeAI-7, not from this repo.
2. The merge with current main breaks the test suite. The textual conflict is only in .github/AGENTS.md (after #9863). The semantic one is in tests/conftest.py: since #9859, main's _cheap_password_hashing and production_password_hashing patch password_utils.pwd_context, which this PR removes. After the merge, collection fails. Your hash_password reads BCRYPT_ROUNDS on each call, so the fix is simple: patch password_utils.BCRYPT_ROUNDS to 4 in the autouse fixture and restore 12 in production_password_hashing. Also keep test_hashing_cost_stays_at_a_defensible_work_factor on production_password_hashing. When resolving python-tests.yml, don't copy windows-cpu's not platform_independent marker to the ARM64 entry; ARM64 has its own dependency set, so it should run those tests.

Minor

• .github/AGENTS.md placement: since #9863 it's a terse policy file. One or two policy lines (the gate blocks; the options are a fork pin, a marker exclusion with a system-requirements entry, or a self-built wheel) would fit. The dispatch/retag/relock procedure is better kept in the workflow header or in docs/.../development/.
• Two-way method check: projection_method_is_current treats any mismatch as stale. On ARM64, a valid UMAP map (a database copied from x64, or a legacy row without the field) gets replaced by PCA. Treating only stored pca with UMAP available as stale avoids that.
• Thread hop: image_map.py now calls asyncio.to_thread(projection_method_is_current, …) on every /points call on every platform. Doing it inline unless is_windows_arm64() would skip that.
• Partial uploads: the workflow header says "re-run failed jobs" completes an interrupted upload. An upload that dies mid-asset can leave a digest-less asset, which the planner (correctly) refuses, so a manual delete is needed. A note in the header would help.
• Still open from last time:
  • test vectors for >72-byte and $2a$/$2y$ legacy passwords;
  • the "protobuf mismatch" comment in mediapipe_face/__init__.py (that failure raises TypeError, not ImportError);
  • the mediapipe gate test patching global sys.platform (patching mediapipe_face.is_windows_arm64 would be narrower);
  • the production shim scope and docstring;
  • the tab labels in manual.mdx;
  • ${{ matrix.* }} interpolated into a run: step in the wheel workflow.

[lstein]

Thanks for the second pass. Here is where each item stands; head is 44b63e6876.

Blocking

1. The windows-arm64 lane timeout. I added a per-test --timeout and -v for diagnosis, then sampled the runner's CPU per process. Memory stayed at about 10 GB free throughout.
   • About 20 minutes in, an orphaned ffmpeg -f lavfi -i anullsrc -f null - held the 4-vCPU runner at 100% for the rest of the job. It came from test_cancellation_kills_a_running_ffmpeg.
   • The cause: on ARM64, imageio_ffmpeg.get_ffmpeg_exe() resolves to the ffmpeg on PATH. Chocolatey's ffmpeg is a launcher shim that runs the real binary as a child, and run_ffmpeg_process killed only the shim.
   • The 8 failures (the sqlite cancel/retry tests) and the 300 s timeouts (the import-boundary subprocess tests, which take 14 s on an idle runner) all followed from the starved CPU. The 5–8 minute gaps in progress did too.
   • Users hit the same bug: any ARM64 install with a Chocolatey or Scoop ffmpeg leaked one on every cancel or timeout. 71b93a424a now suspends the shim, kills the process tree, and routes the upload conversion through the same runner. A regression test runs a Python launcher shim, and that test fails against the old kill.
   • With the leak gone, the suite finished in 23 minutes with 4 real failures, all in test_video_frame_extract_range_audio_splice.py. The self-built OpenCV wheel has no FFmpeg backend, and Media Foundation derives CAP_PROP_FRAME_COUNT from the container duration. A soundtrack longer than the video therefore doubled the frame count. 54dc0e6464 makes the decode worker count video packets with ffmpeg (-c copy -f framecrc) whenever getBackendName() != "FFMPEG". Other platforms are unchanged.
   • The lane now keeps --timeout=600, so a hang becomes a traceback and the summary still prints. Its job timeout is 60 minutes, about 1.6× the slowest measured job (37 min). Run 39166449058 on 88f951826e (the last commit only raises that timeout): 14,608 passed, 239 skipped, 8 xfailed in 31 min; every other check is green.
2. The merge with main. 8fa82de6dd merges current main:
   • The conftest.py fixtures patch password_utils.BCRYPT_ROUNDS (4 for the autouse fixture, the production value in production_password_hashing).
   • test_hashing_cost_stays_at_a_defensible_work_factor stays on production_password_hashing.
   • The ARM64 matrix entry has no pytest-markers, so it runs the platform_independent tests.
   • I took main's uv-lock-checks.yml setup-uv pin and kept the ARM64 skip in test_umap_import.py.

Minor

• .github/AGENTS.md: it's main's terse file plus one policy line: the gate blocks; the options are a fork pin, a marker exclusion with a system-requirements.mdx entry, or a self-built wheel. The procedure lives in the build-win-arm64-wheels.yml header.
• Two-way method check: projection_method_is_current is now one-way. Only a stored pca map where UMAP imports counts as stale, so a UMAP map, or a legacy row without the field, survives on a host without umap. There is a test for both cases.
• Thread hop: image_map.py calls the check inline, and uses asyncio.to_thread only when is_windows_arm64().
• Partial uploads: the workflow header now says an upload that dies mid-asset can leave a digest-less asset, which has to be deleted by hand before re-running.
• Still open from last time:
  • Legacy-hash vectors made with passlib 1.7.4 + bcrypt 3.2.2: over 72 bytes (ASCII, and UTF-8 split at the boundary), $2a$ and $2y$. Each one checks that the password verifies and that a one-character change does not.
  • The mediapipe_face/__init__.py comment now says "a missing native library".
  • The gate test patches mediapipe_face.is_windows_arm64.
  • The stream-capture shim returns early unless is_windows_arm64(). Its docstring says NVIDIA's ARM64 build crashes where others raise. A test covers the non-ARM64 no-op.
  • The manual.mdx tab is "Windows on ARM64", matching step 8.
  • The wheel workflow passes matrix.* values through env:. Every setup-uv use is pinned to the hash.

Known gap

The two ffmpeg probe calls (probe_media_streams, _probe_audio_sample_rate) and the LTX-2 conditioning helper still use subprocess.run(timeout=...), so behind a shim a timeout leaves the real ffmpeg running. These are short probes that only time out on hostile input. I'd rather fix them in a follow-up than widen this PR. I'll also update the description's lane numbers to this repo's run.

[Pfannkuchensack]

Thanks, that all checks out on 44b63e6876:

• The windows-arm64 lane is green (36 min against a 60 min job timeout).
• The conflict resolution looks right.
• The packet-count fix for the 4 video failures is sound, and its test is a good regression test.
• Leaving the probe timeouts for a follow-up is fine with me.

A few small things left, none of them big:

1. The shim regression test can pass without exercising the kill. In test_cancellation_kills_the_ffmpeg_a_launcher_shim_started, Popen is patched before _ffmpeg_exe() runs. When the test is the first ffmpeg lookup in its worker, imageio_ffmpeg's lru_cached get_ffmpeg_exe() validates the binary with a -version subprocess, and that call goes through the fake shim too. The shim writes the probe's PID, is_canceled() fires on the first poll, and the test then finds that PID already gone and passes. With the cache cold, it also passes against the old proc.kill(); proc.wait(). It only fails once the cache is warm, so whether CI actually checks the fix depends on xdist ordering. Calling video_audio._ffmpeg_exe() before the patch fixes it. Alternatively, the shim could record the PID only for the anullsrc command.
2. The new process-tree kill duplicates an existing helper. _kill_process_tree (video_audio.py:49) duplicates video_thumbnails._terminate_process_tree. Yours is the better one, since it suspends the parent before listing the children, so I'd fold the two together. One detail: suspend() and children() share a try, so if suspend() raises (for example AccessDenied), the child list is dropped and the real ffmpeg survives, even though children() alone would have worked.
3. The new packet count has no timeout of its own. _ffmpeg_packet_count calls subprocess.run with no timeout. That's fine in practice, because the worker's 30 s budget and _terminate_process_tree bound it and kill the descendants. Just note that on ARM64 it now demuxes the whole file inside that budget, so a very large upload on slow storage could hit it.
4. History cleanup before merge: squash away the two temporary diagnostics commits, and drop the Co-Authored-By: Claude / Claude-Session: trailers from the squash message.

Separately, for the launcher: OSSign/invoke-ai-launcher#3 was merged at 0a9ff2d without the VS 2022 Build Tools step or npm_config_msvs_version that launcher #159 needed on windows-11-arm. Its build-arm64 job also runs npm ci before the "does this ref support ARM64" check. So I'd expect the next launcher tag, even one without #159/#160, to fail in the arm64 job, which means release never runs and no Windows release is published, x64 included. It's worth porting the toolchain fix (and moving the support check right after checkout) before the next launcher tag.