Feat: Remote Workers Panel

[mickr777]

Summary

This PR adds optional multi-host rendering to InvokeAI v7. A primary InvokeAI instance can continue rendering locally while additional compatible InvokeAI instances on the network act as Remote Workers.

Remote work is coordinated by a backend worker pool and remains tied to InvokeAI's normal queue items, progress events, destinations, cancellation state, and result handling. Worker reachability and job claiming are backend-owned; browser health is display-only. No custom node pack is required; the primary and workers should run the same compatible InvokeAI build.

Implementation

• Remote Workers panel: Configure additional InvokeAI instances, custom worker names, individual enable/disable controls, per-user authentication, and display-only worker availability.
• Distributed mode: Local and enabled Remote Workers share the normal InvokeAI queue. The backend verifies reachability/authentication immediately before a worker claims an item, so unavailable workers do not block Local or other workers. An offline worker remains configured and can resume claiming already-pending eligible work when it recovers.
• Remote Only mode: Eligible items are reserved for Remote Workers and are not rendered locally. If Local dequeues the helper first, it hands the item back to the backend worker-pool path instead of running the workflow locally.
• Native queue integration: Remote execution is tied to the original backend queue item rather than browser-created synthetic queue sessions. Remote progress and completion use the normal InvokeAI event flow.
• Backend-owned availability: Online, Offline, Login required, and Paused badges are presentation only. The per-worker power toggle controls whether a worker is included in new jobs; actual reachability and claim decisions happen in the backend worker pool.
• Per-user settings and credentials: Remote Worker settings and saved credentials are persisted on the primary per user using cross-platform AES-256-GCM authenticated encryption. Passwords are not stored in browser settings or workflow graphs.
• Previews and results: Live Remote Worker progress is shown through the normal Gallery, Canvas, Preview, and queue progress paths. Completed remote image and video outputs are imported back into the primary instance and routed through the original queue item's normal result handling.
• Canvas result routing: Imported remote outputs preserve their real source output node IDs (for example canvas_output / video_output) and are persisted before terminal completion, so normal Canvas staging/result filtering continues to work after remote completion and refresh.
• Gallery/Canvas routing: The selected Gallery board or Canvas destination is captured when the workflow is queued and applied when remote results are imported.
• Queue visibility: Remote jobs remain represented by their normal primary queue item. The queue UI can display the Remote Worker name as the execution device.
• Cancellation: Native queue cancellation is authoritative. The backend worker pool observes the local queue item's canceled state, cancels the corresponding remote queue item, and also stops in-progress model-transfer work for that generation.
• Missing-model transfer: Supported missing single-file models and complete Diffusers model directories can be transferred automatically to a worker. Transfers include progress notifications, cancellation, installed-model hash verification, and singleflight protection so concurrent generations do not duplicate the same worker/model transfer.
• Remote cleanup: Successfully imported remote queue items and media can be removed from the worker when remote copies are not requested.
• Optional and lightweight: When Remote Workers are disabled, remote dispatch and availability polling are inactive and InvokeAI continues through its normal local rendering path.

Related Issues / Discussions

Feedback is welcome on whether this style of multi-host rendering belongs in InvokeAI core and whether the backend worker-pool / normal-queue integration fits the project's preferred architecture.

QA Instructions

Manual test environment

• Windows primary InvokeAI v7 instance, also rendering locally.
• Separate Unraid-hosted InvokeAI v7 Linux Docker worker.
• Separate Windows systems running InvokeAI v7 as Remote Workers.
• Separate Linux systems running InvokeAI v7 as Remote Workers.
• Up to three Remote Workers tested concurrently with the primary.
• Primary and workers running the same compatible InvokeAI v7 build.

Setup and checks

1. Run the primary and workers on the same compatible InvokeAI v7 build.
2. Ensure each worker is reachable from the primary. For a typical direct LAN setup this may require host: 0.0.0.0 in the worker's invoke.yaml.
3. Configure workers in the Remote Workers panel. If a worker has multi-user authentication enabled, save that user's worker credentials on the primary.
4. Test Distributed mode with Local plus multiple enabled workers and verify that free workers continue claiming work independently.
5. Test Remote Only mode and verify the primary does not render the workflow locally.
6. Queue work with one enabled worker offline, then bring that worker back online while eligible items are still pending. Verify it claims pending work without deleting/re-adding the worker or submitting another job.
7. Disable and re-enable individual workers and verify Paused workers remain configured but receive no new work while disabled.
8. Test Gallery and Canvas destinations, board routing, live previews, returned image/video results, batching, queue device labels, and Canvas staging selection after remote completion.
9. Test cancellation while a remote generation is running and while a missing-model transfer is in progress.
10. For automatic LAN model transfer, add allow_private_download_urls: true to each receiving worker's invoke.yaml where required by InvokeAI's model installer.
11. Test both supported single-file transfer and Diffusers directory transfer, including progress, cancellation, hash verification, and reuse of a concurrent transfer for the same worker/model.

Automated checks run

• Focused Remote Worker backend pytest coverage, including worker-pool dispatch, backend availability/retry behavior, result persistence/routing, model-transfer cancellation/singleflight, queue cleanup, video import, and Diffusers transfer.
• Ruff format/lint checks on touched backend code.
• Frontend Remote Worker graph/health tests.
• Frontend oxfmt --check.
• Frontend tsc --noEmit.
• Frontend production build and architecture checks.

Known limitations / considerations

• The impact on a primary instance using multiple local GPUs has not been tested.
• Remote workers must have compatible versions of any custom nodes required by the dispatched workflow. Custom node packs are not transferred or installed automatically.
• Workers must have compatible models available unless the missing model is supported by automatic model transfer and transfer is enabled.
• The frontend still injects a small internal Remote Worker helper into the submitted graph to snapshot per-job dispatch settings and the selected Gallery/Canvas destination. The saved Canvas/workflow graph is not modified.
• Native API-token support in InvokeAI could provide a cleaner future authentication option than saved user login credentials.

Review

The earlier browser-driven Mirror / round-robin / auto-balance dispatch implementation was removed. The final implementation uses two modes only—Distributed and Remote Only—and routes work through a backend worker pool tied to normal InvokeAI queue items.

Worker availability and job eligibility have also been moved out of the core queue runtime. Browser health is now display-only; the backend probes workers before claim and can resume an enabled worker against already-pending eligible work after it recovers. The queue runtime retains only the submitted-graph helper injection needed to snapshot per-job destination/board settings.

Legacy synthetic Remote Worker queue/progress sessions, bridge replay state, custom Remote Worker cancellation endpoints, and obsolete Mirror collector code were removed during cleanup.

Compatibility / Rollout

• Remote Workers are opt-in and can be disabled without changing the normal local rendering path.
• Primary and workers should run the same compatible InvokeAI build.
• No database migration is required for Remote Worker configuration.
• Saved Remote Worker settings and credentials are primary-side, per-user, and encrypted.
• Workers receiving LAN model transfers may require allow_private_download_urls: true.
• Workers must listen on a network-reachable address such as host: 0.0.0.0 when accessed from another machine.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

[lstein]

This is a big feature that has security implications. It will need a thorough review by multiple people. I am therefore marking it as DO NOT MERGE to avoid accidents!

[mickr777]

Quick update: this PR has had a substantial refactor. The total code change has been reduced by nearly half, with most Remote Worker code moved into the backend and the older frontend dispatch/recovery machinery removed. Remote workers now integrate much more directly with InvokeAI’s normal queue and execution flow

[mickr777]

Superseded by invoke-ai#9642 after the InvokeAI-7 repository was merged back into invoke-ai/InvokeAI.