Repository navigation
Text-to-speech, MCP capability guides, and a two-tree workflow catalog - #41
Merged
Merged
Conversation
Design notes only. Records why dialogue audio should not be generated ahead of H3 (follow-supplied-audio lip sync is the weak pathway), where a text-to-speech task does earn its place (voice timbre reference), and the larger missing piece: script -> Context-IR shot list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Removed the MCP proposal document as it is no longer needed. - Added a new task for speech generation using Hugging Face transformers, allowing the generation of audio from text with a specified voice preset. - Introduced AudioTrack class to encapsulate generated audio and its sample rate. - Updated media_arguments to handle AudioTrack and AudioVideo types. - Created workflows for generating speech and integrating it into video generation, ensuring that the audio sample rate is preserved. - Added tests for the speech generation functionality to ensure correct behavior and integration.
Separates the catalog into templates (teach a pattern) and model configs (record a hardware fact), which have been sharing one flat list of 121 entries. Measured: 54 of 84 non-archive workflows share a feature signature with another, and the clusters are checkpoint iterations rather than feature variations. Records the compatibility constraints a move runs into - the step cache survives on id, run-directory identity does not survive a path change - and defers the derived shape index until after the restructure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Backward compatibility is not maintained: existing outputs may be orphaned and references into them may break, and no alias or migration shim is carried for a one-time move. What replaces it is stricter - when this lands, every reference in the repo resolves. Enumerates the reference classes with counts and what enforces each, which surfaced one gap: 18 distinct prompt: references across 20 workflows that nothing checks, and that collapsing will rewrite. Closing that gap is now part of the work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven tasks: close the prompt: reference gap first as a safety net, inventory all 121 workflows and stop for approval, then build templates/ and models/, prune, repoint the docs, and prove every reference class resolves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sub-workflow paths, type names and constants each have a test that they resolve; prompt: references had none, and they fail on the GPU rather than in CI. Added ahead of the catalog restructure, which rewrites the 20 workflows carrying them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first draft argued from a crude feature signature and overstated the redundancy. Reading found it concentrated in archive/, which is a pure model gallery, while minimax/ and ltx2/ are a curriculum where each file demonstrates a conditioning mode or mechanism nothing else does - those stay. gyre/ is a test project's production history; only the mechanisms that generalize earn templates. Also records two ids used by two workflows each, which collide in the step cache today and are resolved by emptying archive/. Expected scale is now 45-50 deletions rather than a halving. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A verdict per file - template, model config, collapse target or delete - with the id each surviving file must keep. Awaiting approval; nothing is moved or deleted until it has it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One approved exception to the keep-ids-verbatim constraint: the text-to-image template takes the id 'text-to-image' rather than the 'test_job' it would inherit from sd15.json. Costs one workflow's step-cache entries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One file per capability, with six checkpoint variations collapsed into argument sets in each template's description where the pipeline class matched. Every moved file keeps its id, so the step cache survives. Two deliberate id deviations, both narrow. The text-to-image template takes 'text-to-image' rather than sd15.json's 'test_job' (approved). The three templates replacing GYRE's editing passes are authored fresh rather than moved and take new ids: the originals referenced workflows/gyre/assets/, which was never committed, so they could not run as written and have no cache entries to preserve. The rest of the GYRE project goes with them - it is a test project's production history, not a set of examples. Doc links are updated in a later commit; test_docs_links fails until then. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each names the template it configures, so a tuned 24GB config is findable as an instance of a pattern rather than as another undifferentiated catalog entry. All nine configure templates/text-to-image: the collapse rule stops at a pipeline class boundary, and FluxPipeline, Flux2Pipeline, Krea2Pipeline and ZImagePipeline are four of them. The ltx2/ and minimax/ READMEs move with the families they document. Ids preserved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every removed file was a further checkpoint through a pattern the tree already demonstrates. Git history is the archive, so archive/ stops being a graveyard that costs listing space. 121 workflows become 73. compose-workflows.json named two archive workflows as its defaults, which would have broken silently - it now points at surviving files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All 66 broken workflow links, plus 75 whose text still named the old filename, plus the REPL examples that loaded flux/FluxDev. Fourteen links whose targets were deleted needed judgement rather than a mapping. In RECIPES_24GB the per-model VRAM prose is knowledge no template carries, so it stays and only the dead example links go, replaced by a pointer to the template to author from. The IP-Adapter, LoRA and quantization examples move to the surviving template or model config. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
templates/ teaches a pattern, models/ records what makes a checkpoint fit and names the template it configures. The workflows guide leads with the layout, and the MCP guide summary says so - that is the first thing an agent matching a request's shape needs. Also corrects the plan's own id check, which excluded only DELETE rows and so would fail on the six COLLAPSE rows whose files were folded away. Every reference class resolves: markdown links, sub-workflow paths, type and constant references, and prompt: references. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The restructure regressed browsing without touching the UI. FolderGroups grouped by the first path segment only, so the new layout collapsed to two groups - templates (64) and models (9) - hiding the ltx2 and minimax families entirely, where the old layout showed six. Gallery entries regressed the same way: their names are <workflow identity>/<file> and the identity is now itself a path, so every run grouped under 'templates'. Grouping by everything but the last segment restores the structure and fixes both pages, so it lives in one shared helper rather than being recomputed per page. The listing also carries 'configures' now, and a model config's card says which template it is a tuned instance of - without it the two trees look identical in the one place a person actually browses them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The follow-on the restructure deferred, written up as a proposal rather than left in the spec's 'what this unblocks' paragraph. Derive a small shape vocabulary in workflow_details, which already parses and caches every workflow, with an optional declared override where the derivation is wrong. Report observed runtime from the jobs table. Records the join problem as confirmed rather than assumed: jobs.workflow stores the workflow's id, not its catalog path, and ids are not unique by construction - two shared 'sd35' until archive/ was emptied. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The field was added to WorkflowsPage's local copy of the detail shape but not to the api.ts type it is assigned from, so npm run check failed on the incompatible index signature. Optional there, matching prompt_refs: the UI can be talking to a server older than this field. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Playwright suite drives the real catalog by name, so the renames broke five specs: ZImage, flux/FluxDev, tasks/ImageToText and FluxFill no longer exist. Routes now name the new paths, headings expect the full catalog name and breadcrumbs the leaf. The filter assertion is anchored as /^inpaint / rather than /inpaint/, which matches both the inpaint template and segment-and-inpaint. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…confined An unconfined CLI run (workflow_from_file with no workflow_dir) had only the '..' regex standing between a relative sub-workflow path and the filesystem above it. Normalising the path before validation (previous commit) collapsed that regex's usefulness away, so a path like '../../../../etc/passwd.json' would resolve and pass. Confine such a path to the nearest ancestor directory literally named 'workflows', reusing the approach dw.runs.workflow_identity uses for output naming (dw.workflow.catalog_root_dir), falling back to the referencing file's own directory when there is no such ancestor. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…httpx Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Audio-only pipelines now record their vocoder or VAE rate the way LTX-2 already did, and get_artifact_list wraps each item so a workflow need not declare one. Follow-up: a modular pipeline's dict output (music.json) still carries no rate of its own. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…o is saved as audio Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… drops Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ixes - black/prettier the flagged files (CI gate) - correct generate_speech compose wording and add the voice_preset refusal note in docs/TASKS.md - fix the WORKFLOW_GUIDE.md claim that audio never carries its own rate, now false for AudioLDM2/StableAudio/generate_speech - generalise the batched-audio-save recursion to any artifact carrying a sample_rate, not just AudioTrack, with a test - stop consulting a video pipeline's VAE for an audio sample rate on .audio outputs (only .audios), with tests - assert the "no component reports its sample rate" warning fires - remove the unreachable `or os.sep` fallback in catalog_root_dir - document that a models/ catalog entry carries configures / configures_missing Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three pieces of work, each building on the last. They started from a design
question about scripted dialogue shorts and ended in a restructured catalog.
1.
generate_speechA HuggingFace
text-to-speechtask (dw/tasks/speech_generation.py),modelled on
text_generationand defaulting tosuno/bark-small— its voicepresets give distinct speakers, which is what two characters in a scene need.
The role it is built for is voice timbre reference, not the track a mouth
follows: MiniMax H3 lip-syncs well when it generates the speech itself and
poorly when it must follow supplied audio.
templates/minimax/voice-timbre-reference.jsonshows the clip going in as
<Audio 1>viafrom_previous_result.It returns a new
AudioTrack— a waveform carrying the rate it was generatedat — rather than the bare waveform the other audio tasks return. Those rates
are a property of the workflow; a TTS model's rate is a property of the model,
and every one differs, so a declared 44100 against a 24 kHz model plays the
speech fast and low without ever failing.
normalize_audioandmedia_argumentsnow read.audio/.sample_rateoff whatever they are handed,the way the rest of the audio plumbing already did.
Verified beyond the mocks: bark-small on MPS wrote a 24000 Hz, 3.59 s wav with
nothing declared in the result, and
voice_presetwas confirmed to reach theprocessor as
history_promptagainst the real library.2. MCP capability guides
list_guides/get_guide(dw_mcp/guides.py) serve ninecurated docs, read by path so the no-
dw.*-import boundary holds, fromdw/docs/when installed and the repo'sdocs/from a checkout.The gap they close: a request names a subject ("a lego movie trailer set in
the marvel universe") while the catalog is written in shapes. The index is
~830 tokens — small enough to read every session — and a guide can be fetched
one section at a time, 3.3k chars against 51.7k for the whole of TASKS.md. The
server instructions now say to decide the deliverable's shape first.
3. Two-tree workflow catalog
121 workflows → 73: 64 templates, 9 model configs.
templates/teaches a pattern,models/records what makes a checkpoint fitreal hardware and names the template it configures via a new
configuresfield. Previously nothing in a catalog entry distinguished the two, so the
eight files that answer a multi-shot video request were indistinguishable from
a hundred checkpoint iterations.
archive/(a pure model gallery) andgyre/(a test project's productionhistory) are gone;
qr_code,sdxlandtxt2img2vidgraduatedminimax/andltx2/are kept intact — each file demonstrates aconditioning mode or mechanism nothing else does
descriptionBackward compatibility is deliberately not maintained: run-directory identity
changes with the path, so existing outputs are orphaned. What replaces it is
stricter — every reference in the repo resolves: 66 markdown links (plus 75
whose text still named the old filename), 6 workflow→workflow paths, type and
constant references, and the 18
prompt:references that nothing checkedbefore this branch added a guard for them.
Workflow
ids are preserved (70 of them) so the step cache survives, with oneapproved exception. The duplicate
sd35andtest_jobids, which collided inthe step cache, are resolved by emptying
archive/.Notes
The design spec's original premise — that most of the catalog was redundant —
was wrong, and is corrected in
d5693bfafter reading all 121 workflows.The redundancy was concentrated in
archive/. The inventory indocs/superpowers/plans/2026-09-06-catalog-inventory.mdrecords a verdict forevery file.
Tests: 2544 passed, 5 skipped.
🤖 Generated with Claude Code