|
| 1 | +# MCP Server for diffusers-workflow — Scoping Doc |
| 2 | + |
| 3 | +## Background |
| 4 | + |
| 5 | +`diffusers-workflow` (dkackman/diffusers-workflow) is a declarative JSON |
| 6 | +workflow engine for Hugging Face Diffusers, with a local HTTP server |
| 7 | +(`dw.serve`, default port 8765) that exposes a full REST API: workflow |
| 8 | +CRUD, a job queue with SSE progress streaming, model management, |
| 9 | +introspection of diffusers pipeline signatures, and a gallery of past |
| 10 | +generations with embedded metadata. |
| 11 | + |
| 12 | +Today, workflow authoring and debugging, for the project developer, |
| 13 | +can happen via Claude Code in VS Code, |
| 14 | +with direct repo/filesystem access: "create a workflow that does X," or |
| 15 | +"workflow Y produces bad output, find out why" — Claude Code runs the |
| 16 | +workflow, reads logs, checks GPU memory, and proposes fixes to code or |
| 17 | +workflow config. It has even gone so far as to identify bad RAM causing |
| 18 | +intermittent generation failures. |
| 19 | + |
| 20 | +That flow works well for the developer (me), but it assumes a dev |
| 21 | +environment: a repo checkout, a terminal, VS Code, comfort reading stack |
| 22 | +traces. The next audience is **non-developer end users** of the packaged |
| 23 | +product, who won't have any of that. They'll have the app (web UI / |
| 24 | +`dw.serve`, or an embedded product surface), not a coding agent with shell |
| 25 | +access. |
| 26 | + |
| 27 | +## Goal |
| 28 | + |
| 29 | +Turn this from a dev-only workflow authoring/debugging tool into a **non-dev-friendly MCP server** that exposes the same "author a workflow / run it / diagnose why it |
| 30 | +failed or produced bad output" capability, but in a structured, bounded way that |
| 31 | +doesn't require shell access or a dev environment. The MCP server will wrap the existing `dw.serve` HTTP API, exposing a constrained tool surface for workflow authoring and debugging, suitable for non-developer users. |
| 32 | + |
| 33 | +The modes the underlying workflow can be authored and run today are:: |
| 34 | + |
| 35 | +- a text editor and CLI `python -m dw.run workflow.json` |
| 36 | +- with command line REPL `python -m dw.repl` |
| 37 | +- via the HTTP API `dw.serve` and existing web UI |
| 38 | + |
| 39 | +So this adds a fourth mode, which would be cool if we turned this into an AI first product: an MCP server that can be driven by a Claude Code agent, or eventually a local LLM-backed agent, to author and run workflows without deep |
| 40 | +knowledge of diffusers, my workflow engine, or any of that complexity. The MCP server would expose a structured tool surface for workflow authoring and debugging, suitable for non-developer users. |
| 41 | + |
| 42 | +- via the MCP server (new) — structured tool surface |
| 43 | + |
| 44 | +Expose the same "author a workflow / run it / diagnose why it failed or |
| 45 | +produced bad output" capability as an **MCP server**, wrapping the existing |
| 46 | +`dw.serve` HTTP API. This lets any MCP-compatible client — Claude Code |
| 47 | +first, later Claude Desktop/claude.ai, and eventually a local-LLM-backed |
| 48 | +client — drive the engine through a bounded, structured tool surface |
| 49 | +instead of raw shell/API access. |
| 50 | + |
| 51 | +## Why MCP (not just keep using Claude Code) |
| 52 | + |
| 53 | +- Claude Code assumes a dev environment; end users won't have one. |
| 54 | +- MCP tools are a defined, constrained surface (structured in/out, no |
| 55 | + arbitrary code execution) — appropriate once non-developers are the |
| 56 | + audience, and safer than raw shell access. |
| 57 | +- The underlying HTTP API already exists and is already defensive |
| 58 | + (path validation, traversal blocking, `Origin` header checks, JSON size |
| 59 | + limits) — MCP adds a protocol layer in front of it, not new attack |
| 60 | + surface. |
| 61 | +- Protocol-agnostic: same server can later be pointed at Claude Code, a |
| 62 | + future embedded chat UI in the product, or a local-LLM-backed client. |
| 63 | + |
| 64 | +## Rollout sequence (decided) |
| 65 | + |
| 66 | +1. **Start with MCP server + Claude Code** as the client, for continued dev |
| 67 | + use and to validate the tool surface. |
| 68 | +2. **Move to embedded-in-app** later, once the product is packaged for |
| 69 | + non-developer users. This later phase will need its own auth story, |
| 70 | + since the server currently binds to `127.0.0.1` with no authentication |
| 71 | + (see Security, below) — fine for local/trusted-LAN use, not fine for a |
| 72 | + hosted/remote product. |
| 73 | +3. **Local LLM support** stays a goal throughout — keep the option open |
| 74 | + for users who won't have/want a paid Claude subscription. MCP is |
| 75 | + protocol-agnostic, so this is a separate, later concern from the MCP |
| 76 | + server itself, not a blocker to starting. |
| 77 | + |
| 78 | +## Proposed MCP tool set |
| 79 | + |
| 80 | +Mapped to existing `dw.serve` routes — no new server-side endpoints needed |
| 81 | +for the initial build. |
| 82 | + |
| 83 | +### Read-only / low-cost (build first) |
| 84 | + |
| 85 | +| Tool | Route(s) | Notes | |
| 86 | +| --- | --- | --- | |
| 87 | +| `list_workflows` | `GET /api/workflows` | descriptions, output kinds, variable counts | |
| 88 | +| `get_workflow(name)` | `GET /api/workflows/{name}` | full JSON | |
| 89 | +| `validate_workflow(json)` | `POST /api/validate` | schema + signature-level argument checks; no GPU cost — good pre-flight tool | |
| 90 | +| `get_schema` | `GET /api/schema` | workflow JSON schema | |
| 91 | +| `get_pipeline_signature(name)` | `GET /api/pipelines/{name}` | lets the agent check real pipeline arguments before proposing a config fix | |
| 92 | +| `list_classes(kind)` / `get_class(name)` | `GET /api/classes?kind=...`, `GET /api/classes/{name}` | broader introspection beyond pipelines | |
| 93 | +| `list_tasks` / `get_task(command)` | `GET /api/tasks`, `GET /api/tasks/{command}` | task-step argument schemas | |
| 94 | +| `list_models` | `GET /api/models` | hub cache inventory: sizes, revisions, last-used | |
| 95 | +| `get_memory` | `GET /api/memory` | worker VRAM/RAM stats — answers the GPU-diagnosis need directly, no new endpoint required | |
| 96 | +| `get_health` | `GET /api/health` | liveness | |
| 97 | +| `get_gallery_item(name)` | `GET /api/gallery/{name}/metadata` | pull back a prior run's exact workflow + seed, e.g. to reproduce a failure | |
| 98 | + |
| 99 | +### Diagnostic loop (build second) |
| 100 | + |
| 101 | +| Tool | Route(s) | Notes | |
| 102 | +| --- | --- | --- | |
| 103 | +| `run_workflow(workflow_path \| inline_workflow, arguments)` | `POST /api/jobs` | **Costs GPU time; one job runs at a time.** See confirm-gating below. | |
| 104 | +| `get_job(job_id)` | `GET /api/jobs/{id}` | full detail: spec, events, manifest, error — primary diagnosis payload | |
| 105 | +| `stream_job_events(job_id, after?)` | `GET /api/jobs/{id}/events` | SSE; `job_status`/`phase`/`memory`/`log` events cover exactly the "instrument the process" need (phase transitions: loading → generating → decoding → saving) | |
| 106 | +| `cancel_job(job_id)` | `POST /api/jobs/{id}/cancel` | cooperative cancel | |
| 107 | +| `rerun_job(job_id)` | `POST /api/jobs/{id}/rerun` | re-queue after a fix | |
| 108 | +| `move_job(job_id, direction)` | `POST /api/jobs/{id}/move` | reorder queued jobs | |
| 109 | + |
| 110 | +### Deferred / optional |
| 111 | + |
| 112 | +- `POST /api/models/download`, `DELETE /api/models`, `/api/system/diffusers*` |
| 113 | + — model and dependency management; useful eventually, not needed for the |
| 114 | + first author/run/diagnose loop. |
| 115 | +- Prompt library endpoints (`/api/prompts*`, `/api/enhance`) — relevant if |
| 116 | + the agent should also help build/enhance prompts, not core to diagnosis. |
| 117 | + |
| 118 | +## Key design decisions |
| 119 | + |
| 120 | +1. **`run_workflow` is confirm-gated.** A run costs real GPU time (and |
| 121 | + possibly money) and the engine is single-GPU/single-job-at-a-time. MCP |
| 122 | + has no built-in elicitation/confirmation primitive, so the confirm step |
| 123 | + should live in the client/agent behavior (state what's about to run and |
| 124 | + get explicit go-ahead) rather than the tool silently firing. |
| 125 | + `validate_workflow` is free and can run without confirmation first. |
| 126 | + |
| 127 | +2. **Security model carries over, doesn't change.** The server already |
| 128 | + validates every path (`dw/security.py`), blocks traversal, rejects |
| 129 | + non-local `Origin` headers, and confines file access to the configured |
| 130 | + workflow/output/prompt directories. MCP wrapping this doesn't add new |
| 131 | + risk for the local/Claude-Code phase. It does **not** solve remote |
| 132 | + auth — the server binds to `127.0.0.1` by default with no |
| 133 | + authentication; `--host 0.0.0.0` exists but is explicitly documented as |
| 134 | + trusted-LAN-only. The embedded-in-app phase needs a real auth story |
| 135 | + before any hosted/remote exposure. |
| 136 | + |
| 137 | +3. **Build order:** read-only tools first (workflow authoring assistant |
| 138 | + usable almost immediately), then the run/diagnose loop once the |
| 139 | + confirm-gating pattern for `run_workflow` is settled. |
| 140 | + |
| 141 | +## Open questions for implementation |
| 142 | + |
| 143 | +- Which MCP transport/SDK — Python `mcp` SDK (project is already Python) |
| 144 | + talking to `http://127.0.0.1:8765/api/...` via `httpx`, run as a stdio |
| 145 | + server for Claude Code. |
| 146 | +- Exact confirmation UX for `run_workflow` — client-side prompt text, |
| 147 | + and whether to surface estimated cost/time if the API can supply it. |
| 148 | +- Whether GPU/worker diagnostics need anything beyond `/api/memory`, or |
| 149 | + whether that's sufficient for the diagnostic loop as scoped. |
0 commit comments