From 6e5730fc96c6e2a4969c705adb5da07aad87e862 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Sun, 30 Aug 2026 18:04:40 +0000 Subject: [PATCH 01/15] Add new MiniMax examples and enhance README documentation - Introduced MiniMaxH3Ref2VAVideo.json: A reference video and image combination for generating a video clip. - Added MiniMaxH3SitcomShort.json: A structured sitcom short with multiple shots, showcasing character consistency and scene cuts. - Created MiniMaxMusic.json: An audio-only generation example demonstrating a minimal modular pipeline for music output. - Updated README.md to include descriptions of new examples and their functionalities. - Enhanced test coverage in test_arguments.py, test_text_sections.py, and test_workflow.py to validate new audio reference handling and ensure proper referencing in workflows. --- docs/ACCELERATION.md | 2 +- docs/QUANTIZATION.md | 2 +- docs/RECIPES_24GB.md | 13 +- docs/WORKFLOW_GUIDE.md | 24 +- dw/arguments.py | 12 +- dw/workflow.py | 14 +- examples/{ => ltx2}/LTX2.json | 0 examples/{ => ltx2}/LTX2Extend.json | 0 examples/{ => ltx2}/LTX2I2V.json | 0 examples/{ => ltx2}/LTX2I2VChained.json | 0 examples/{ => ltx2}/LTX2I2VEnhancePrompt.json | 0 examples/{ => ltx2}/LTX2ICLora.json | 0 examples/{ => ltx2}/LTX2Keyframes.json | 0 examples/{ => ltx2}/LTX2TwoStage.json | 0 examples/ltx2/README.md | 36 ++ examples/{ => minimax}/MiniMaxH3.json | 0 .../{ => minimax}/MiniMaxH3EnhancePrompt.json | 0 examples/{ => minimax}/MiniMaxH3FL2VA.json | 0 examples/{ => minimax}/MiniMaxH3I2V.json | 0 .../{ => minimax}/MiniMaxH3I2VChained.json | 0 .../MiniMaxH3I2VEnhancePrompt.json | 0 examples/{ => minimax}/MiniMaxH3L2V.json | 0 examples/minimax/MiniMaxH3MusicVideo.json | 416 ++++++++++++++++++ examples/{ => minimax}/MiniMaxH3Ref2VA.json | 0 .../{ => minimax}/MiniMaxH3Ref2VAChained.json | 0 .../MiniMaxH3Ref2VAChainedAligned.json | 0 .../MiniMaxH3Ref2VAChainedVideo.json | 0 .../MiniMaxH3Ref2VAGeneratedSubject.json | 6 +- .../{ => minimax}/MiniMaxH3Ref2VAVideo.json | 0 examples/minimax/MiniMaxH3SitcomShort.json | 350 +++++++++++++++ examples/{ => minimax}/MiniMaxMusic.json | 0 examples/minimax/README.md | 81 ++++ tests/test_arguments.py | 19 + tests/test_text_sections.py | 2 +- tests/test_workflow.py | 20 + 35 files changed, 976 insertions(+), 21 deletions(-) rename examples/{ => ltx2}/LTX2.json (100%) rename examples/{ => ltx2}/LTX2Extend.json (100%) rename examples/{ => ltx2}/LTX2I2V.json (100%) rename examples/{ => ltx2}/LTX2I2VChained.json (100%) rename examples/{ => ltx2}/LTX2I2VEnhancePrompt.json (100%) rename examples/{ => ltx2}/LTX2ICLora.json (100%) rename examples/{ => ltx2}/LTX2Keyframes.json (100%) rename examples/{ => ltx2}/LTX2TwoStage.json (100%) create mode 100644 examples/ltx2/README.md rename examples/{ => minimax}/MiniMaxH3.json (100%) rename examples/{ => minimax}/MiniMaxH3EnhancePrompt.json (100%) rename examples/{ => minimax}/MiniMaxH3FL2VA.json (100%) rename examples/{ => minimax}/MiniMaxH3I2V.json (100%) rename examples/{ => minimax}/MiniMaxH3I2VChained.json (100%) rename examples/{ => minimax}/MiniMaxH3I2VEnhancePrompt.json (100%) rename examples/{ => minimax}/MiniMaxH3L2V.json (100%) create mode 100644 examples/minimax/MiniMaxH3MusicVideo.json rename examples/{ => minimax}/MiniMaxH3Ref2VA.json (100%) rename examples/{ => minimax}/MiniMaxH3Ref2VAChained.json (100%) rename examples/{ => minimax}/MiniMaxH3Ref2VAChainedAligned.json (100%) rename examples/{ => minimax}/MiniMaxH3Ref2VAChainedVideo.json (100%) rename examples/{ => minimax}/MiniMaxH3Ref2VAGeneratedSubject.json (97%) rename examples/{ => minimax}/MiniMaxH3Ref2VAVideo.json (100%) create mode 100644 examples/minimax/MiniMaxH3SitcomShort.json rename examples/{ => minimax}/MiniMaxMusic.json (100%) create mode 100644 examples/minimax/README.md diff --git a/docs/ACCELERATION.md b/docs/ACCELERATION.md index 4ce1ac9e..f05a25f4 100644 --- a/docs/ACCELERATION.md +++ b/docs/ACCELERATION.md @@ -277,7 +277,7 @@ Both `storage_dtype` and `compute_dtype` are required. Applied via the component `"residency": "on_demand"` on a component is the cheap case of the same trade: the model rests in system memory and is moved to the device whole around each of its own calls. That is a bad deal for anything called once per step, and a good one for a VAE called twice a run - it frees the VAE's VRAM for the denoise loop at the cost of two transfers, where group offloading the same VAE would restream it once per decode tile. See [On-demand components](WORKFLOW_GUIDE.md#on-demand-components). -**Example:** [FluxDev.json](../examples/flux/FluxDev.json) (`"offload": "model"`), [ZImage.json](../examples/ZImage.json) (`"offload": "sequential"`), [MiniMaxH3.json](../examples/MiniMaxH3.json) (`group_offload` per component), [MiniMaxH3Ref2VA.json](../examples/MiniMaxH3Ref2VA.json) (`group_offload` for the transformer, `on_demand` for the VAEs) +**Example:** [FluxDev.json](../examples/flux/FluxDev.json) (`"offload": "model"`), [ZImage.json](../examples/ZImage.json) (`"offload": "sequential"`), [MiniMaxH3.json](../examples/minimax/MiniMaxH3.json) (`group_offload` per component), [MiniMaxH3Ref2VA.json](../examples/minimax/MiniMaxH3Ref2VA.json) (`group_offload` for the transformer, `on_demand` for the VAEs) ## TF32 and cuDNN diff --git a/docs/QUANTIZATION.md b/docs/QUANTIZATION.md index 270ad15c..1f528123 100644 --- a/docs/QUANTIZATION.md +++ b/docs/QUANTIZATION.md @@ -146,7 +146,7 @@ examples use for their large transformers and text encoders: - `quantization_device` / `return_device` — where the quantization pass runs and where the finished component lands; quantizing on `cuda` is much faster than on CPU - `use_quantized_matmul` — quantized matmul kernels (CUDA/XPU only) -**Examples:** [LTX2.json](../examples/LTX2.json), [MiniMaxH3.json](../examples/MiniMaxH3.json) +**Examples:** [LTX2.json](../examples/LTX2.json), [MiniMaxH3.json](../examples/minimax/MiniMaxH3.json) ### Pre-quantized models diff --git a/docs/RECIPES_24GB.md b/docs/RECIPES_24GB.md index 2c44753a..d2e112d6 100644 --- a/docs/RECIPES_24GB.md +++ b/docs/RECIPES_24GB.md @@ -81,7 +81,18 @@ encoder's leaf offload was backed out with it for the same host-memory reason. O faster GPU (or a host with more RAM) both are worth re-testing; the step budget there may actually expose the transfer time. -**Examples:** [MiniMaxH3Ref2VA.json](../examples/MiniMaxH3Ref2VA.json), [MiniMaxH3Ref2VAChained.json](../examples/MiniMaxH3Ref2VAChained.json), [MiniMaxH3I2V.json](../examples/MiniMaxH3I2V.json) +Length costs VRAM but the configuration holds to the model's full range: a single +345-frame take (14.4s, the `17n+5` maximum) peaks at 23.6GiB reserved at 960x544 - +inside 24GB with nothing to spare - and denoises in ~13 minutes on a 3090 with the +9-step turbo schedule (~85-100s a step once warm, against ~15s at 124 frames). + +Host RAM is the tighter budget than VRAM on a 64GiB box. Loading H3 peaks around +59GiB RSS and a running ref2va shot sits at 45-53GiB, so a workflow that ran another +model first (Z-Image drawing a subject, Music3 writing a song) must free it with +`release_pipeline` before H3 loads - with it, the multi-model digital-short +workflows below fit; without it, the load is an OOM kill, not a slowdown. + +**Examples:** [MiniMaxH3Ref2VA.json](../examples/minimax/MiniMaxH3Ref2VA.json), [MiniMaxH3Ref2VAChained.json](../examples/minimax/MiniMaxH3Ref2VAChained.json), [MiniMaxH3I2V.json](../examples/minimax/MiniMaxH3I2V.json), [MiniMaxH3SitcomShort.json](../examples/minimax/MiniMaxH3SitcomShort.json) (five ref2va shots + two Z-Image portraits in ~35 minutes end to end) ## LTX-2.5 (22B, video + audio) diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index cfe52272..2be4824c 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -396,8 +396,8 @@ cache and succeeds on the retry - but each one is a synchronising stall, and a r close to the limit fails outright on any workload that needs slightly more. Every MiniMax H3 example uses on-demand VAEs for this reason. -**Example:** [MiniMaxH3Ref2VA.json](../examples/MiniMaxH3Ref2VA.json), -[MiniMaxH3I2V.json](../examples/MiniMaxH3I2V.json) +**Example:** [MiniMaxH3Ref2VA.json](../examples/minimax/MiniMaxH3Ref2VA.json), +[MiniMaxH3I2V.json](../examples/minimax/MiniMaxH3I2V.json) #### Releasing a pipeline mid-workflow @@ -439,7 +439,7 @@ The flag applies to any step type, and on a `workflow` step it fires once the wh sub-workflow has finished. It clears every cached task model, not only this step's, and a later step needing one of them reloads it. -**Example:** [MiniMaxH3EnhancePrompt.json](../examples/MiniMaxH3EnhancePrompt.json) +**Example:** [MiniMaxH3EnhancePrompt.json](../examples/minimax/MiniMaxH3EnhancePrompt.json) ### VAE Options @@ -675,11 +675,11 @@ downloaded and loaded: A task is chosen by the arguments the step passes, so one `workflow` name can cover more than one of them: MiniMax-H3's `fl2va` takes an `image`, a `last_image`, or both. Given only a `last_image` it generates *up to* that frame, inventing everything that leads to -it — see [examples/MiniMaxH3L2V.json](../examples/MiniMaxH3L2V.json) beside -[examples/MiniMaxH3FL2VA.json](../examples/MiniMaxH3FL2VA.json). +it — see [examples/minimax/MiniMaxH3L2V.json](../examples/minimax/MiniMaxH3L2V.json) beside +[examples/minimax/MiniMaxH3FL2VA.json](../examples/minimax/MiniMaxH3FL2VA.json). -See [examples/MiniMaxMusic.json](../examples/MiniMaxMusic.json) and -[examples/MiniMaxH3.json](../examples/MiniMaxH3.json) for full examples. +See [examples/minimax/MiniMaxMusic.json](../examples/minimax/MiniMaxMusic.json) and +[examples/minimax/MiniMaxH3.json](../examples/minimax/MiniMaxH3.json) for full examples. ### Chained Video Generation @@ -756,8 +756,8 @@ it is single-frame conditioning; `last_segment` continuity exists for exactly th where the pipeline can take a video reference. See [examples/LTX2I2VChained.json](../examples/LTX2I2VChained.json), -[examples/MiniMaxH3I2VChained.json](../examples/MiniMaxH3I2VChained.json), and -[examples/MiniMaxH3Ref2VAChained.json](../examples/MiniMaxH3Ref2VAChained.json) +[examples/minimax/MiniMaxH3I2VChained.json](../examples/minimax/MiniMaxH3I2VChained.json), and +[examples/minimax/MiniMaxH3Ref2VAChained.json](../examples/minimax/MiniMaxH3Ref2VAChained.json) (audio-matched lip-sync of arbitrary length). ## Schedulers @@ -897,7 +897,7 @@ its own 24 fps, so a wrong rate is a request conditioned at the wrong speed — motion and camera alone. A name that is neither an argument of `from_file()` nor a field of the object raises, with the fields it does have. -See [examples/MiniMaxH3Ref2VA.json](../examples/MiniMaxH3Ref2VA.json) for a full example. +See [examples/minimax/MiniMaxH3Ref2VA.json](../examples/minimax/MiniMaxH3Ref2VA.json) for a full example. ### Objects Built From an Earlier Step @@ -926,14 +926,14 @@ lands in comes from the type's own `kind`: | ------- | ------------------------------------------------------------------------- | | `image` | The generated image | | `video` | The generated frames, and the soundtrack generated with them if there was one | -| `audio` | The generated soundtrack | +| `audio` | The generated soundtrack - or, for a step that produced audio alone (a music pipeline, a `slice_audio` task), the waveform itself, which carries no rate of its own, so declare `sample_rate` beside `from_previous_result` | Any other key is a field of the object and wins over what the media carried — `"fps": 30.0` where the producing pipeline generated at a rate the consuming one does not share, for instance. A step that produced several artifacts fans out the same way every `previous_result` reference does: four images in, four videos out. -See [examples/MiniMaxH3Ref2VAGeneratedSubject.json](../examples/MiniMaxH3Ref2VAGeneratedSubject.json) +See [examples/minimax/MiniMaxH3Ref2VAGeneratedSubject.json](../examples/minimax/MiniMaxH3Ref2VAGeneratedSubject.json) for a full example. ### Objects Built From Named Arguments diff --git a/dw/arguments.py b/dw/arguments.py index 4c8a61ce..d8c2cd74 100644 --- a/dw/arguments.py +++ b/dw/arguments.py @@ -703,12 +703,22 @@ def media_arguments(object_type, artifact): arguments["sample_rate"] = sample_rate return arguments + if audio is None and hasattr(artifact, "ndim") and artifact.ndim <= 3: + # A step that generates audio alone - a music pipeline, a slice_audio + # task - produces the waveform itself rather than an AudioVideo. It + # carries no rate of its own, so the reference declares 'sample_rate' + # alongside 'from_previous_result' + audio = torch.as_tensor(as_channels_samples(artifact)) + if audio is None: raise ValueError( f"{object_type.__name__} holds audio, but the step it names produced " f"none - reference a step that generates a soundtrack" ) - return {"audio": audio, "sample_rate": sample_rate} + arguments = {"audio": audio} + if sample_rate is not None: + arguments["sample_rate"] = sample_rate + return arguments def validate_media_location(location, base_dir=None): diff --git a/dw/workflow.py b/dw/workflow.py index 336ddbf7..7dc3fff8 100644 --- a/dw/workflow.py +++ b/dw/workflow.py @@ -107,7 +107,9 @@ def referenced_result_names(steps): """Every previous_result reference the given steps make, as full names. Scans nested dicts and lists, so references inside pipeline arguments, - task arguments and sub-workflow argument maps are all found. + task arguments and sub-workflow argument maps are all found - including a + constructed object's 'from_previous_result', which names a step without + the 'previous_result:' prefix. """ prefix = "previous_result:" names = set() @@ -116,6 +118,9 @@ def scan(value): if isinstance(value, str) and value.startswith(prefix): names.add(value[len(prefix) :]) elif isinstance(value, dict): + reference = value.get("from_previous_result") + if isinstance(reference, str): + names.add(reference) for item in value.values(): scan(item) elif isinstance(value, list): @@ -338,6 +343,13 @@ def run( logger.info(f"Releasing pipeline for step: {step.name}") pipelines.pop(self._pipeline_keys_by_step.get(step.name), None) + # The loop's own locals are the last references to this step's + # action and result - a released pipeline would otherwise stay + # resident through the next step's load, which is exactly when + # both models would be in memory at once + step_action = None + result = None + # Task models are cached for the life of the process - the cache # exists so a step's cartesian product loads its model once, and # nothing else evicts it. A prompt-expanding language model diff --git a/examples/LTX2.json b/examples/ltx2/LTX2.json similarity index 100% rename from examples/LTX2.json rename to examples/ltx2/LTX2.json diff --git a/examples/LTX2Extend.json b/examples/ltx2/LTX2Extend.json similarity index 100% rename from examples/LTX2Extend.json rename to examples/ltx2/LTX2Extend.json diff --git a/examples/LTX2I2V.json b/examples/ltx2/LTX2I2V.json similarity index 100% rename from examples/LTX2I2V.json rename to examples/ltx2/LTX2I2V.json diff --git a/examples/LTX2I2VChained.json b/examples/ltx2/LTX2I2VChained.json similarity index 100% rename from examples/LTX2I2VChained.json rename to examples/ltx2/LTX2I2VChained.json diff --git a/examples/LTX2I2VEnhancePrompt.json b/examples/ltx2/LTX2I2VEnhancePrompt.json similarity index 100% rename from examples/LTX2I2VEnhancePrompt.json rename to examples/ltx2/LTX2I2VEnhancePrompt.json diff --git a/examples/LTX2ICLora.json b/examples/ltx2/LTX2ICLora.json similarity index 100% rename from examples/LTX2ICLora.json rename to examples/ltx2/LTX2ICLora.json diff --git a/examples/LTX2Keyframes.json b/examples/ltx2/LTX2Keyframes.json similarity index 100% rename from examples/LTX2Keyframes.json rename to examples/ltx2/LTX2Keyframes.json diff --git a/examples/LTX2TwoStage.json b/examples/ltx2/LTX2TwoStage.json similarity index 100% rename from examples/LTX2TwoStage.json rename to examples/ltx2/LTX2TwoStage.json diff --git a/examples/ltx2/README.md b/examples/ltx2/README.md new file mode 100644 index 00000000..d5a616e2 --- /dev/null +++ b/examples/ltx2/README.md @@ -0,0 +1,36 @@ +# LTX-2.5 examples + +Text- and image-to-video with a generated soundtrack, using +[LTX-2.5](https://huggingface.co/Lightricks/LTX-Video) fitted onto a single +24GB consumer GPU. The memory configuration the examples share is explained in +[docs/RECIPES_24GB.md](../../docs/RECIPES_24GB.md). + +Read them in this order and each introduces one new idea on top of the last. + +## The basics + +| Example | What it introduces | +| ------- | ------------------ | +| [LTX2.json](LTX2.json) | The baseline: text to video plus soundtrack on a fixed eight-step distilled schedule, quantized per component | +| [LTX2I2V.json](LTX2I2V.json) | A supplied still becomes the first frame, with VAE tiling for the longer clip | +| [LTX2Keyframes.json](LTX2Keyframes.json) | Pinning the first and last frames as conditions, each with the latent index it lands on | + +## Writing the prompt with a model + +| Example | What it introduces | +| ------- | ------------------ | +| [LTX2I2VEnhancePrompt.json](LTX2I2VEnhancePrompt.json) | LTX-2.5's own enhancer rewrites a one-line idea into a trained-format prompt, conditioned on the reference frame | + +## Quality and scale + +| Example | What it introduces | +| ------- | ------------------ | +| [LTX2TwoStage.json](LTX2TwoStage.json) | The recommended quality flow: render small, double in latent space, pair the original soundtrack back on | +| [LTX2ICLora.json](LTX2ICLora.json) | A generative 2x upscale: an in-context LoRA re-renders a clip at twice the size, inventing detail | + +## Going long + +| Example | What it introduces | +| ------- | ------------------ | +| [LTX2Extend.json](LTX2Extend.json) | Continuing a clip by conditioning on it in full, both steps sharing one loaded model | +| [LTX2I2VChained.json](LTX2I2VChained.json) | A chain re-runs the pipeline per segment on the previous last frame and stitches frames and audio back together | diff --git a/examples/MiniMaxH3.json b/examples/minimax/MiniMaxH3.json similarity index 100% rename from examples/MiniMaxH3.json rename to examples/minimax/MiniMaxH3.json diff --git a/examples/MiniMaxH3EnhancePrompt.json b/examples/minimax/MiniMaxH3EnhancePrompt.json similarity index 100% rename from examples/MiniMaxH3EnhancePrompt.json rename to examples/minimax/MiniMaxH3EnhancePrompt.json diff --git a/examples/MiniMaxH3FL2VA.json b/examples/minimax/MiniMaxH3FL2VA.json similarity index 100% rename from examples/MiniMaxH3FL2VA.json rename to examples/minimax/MiniMaxH3FL2VA.json diff --git a/examples/MiniMaxH3I2V.json b/examples/minimax/MiniMaxH3I2V.json similarity index 100% rename from examples/MiniMaxH3I2V.json rename to examples/minimax/MiniMaxH3I2V.json diff --git a/examples/MiniMaxH3I2VChained.json b/examples/minimax/MiniMaxH3I2VChained.json similarity index 100% rename from examples/MiniMaxH3I2VChained.json rename to examples/minimax/MiniMaxH3I2VChained.json diff --git a/examples/MiniMaxH3I2VEnhancePrompt.json b/examples/minimax/MiniMaxH3I2VEnhancePrompt.json similarity index 100% rename from examples/MiniMaxH3I2VEnhancePrompt.json rename to examples/minimax/MiniMaxH3I2VEnhancePrompt.json diff --git a/examples/MiniMaxH3L2V.json b/examples/minimax/MiniMaxH3L2V.json similarity index 100% rename from examples/MiniMaxH3L2V.json rename to examples/minimax/MiniMaxH3L2V.json diff --git a/examples/minimax/MiniMaxH3MusicVideo.json b/examples/minimax/MiniMaxH3MusicVideo.json new file mode 100644 index 00000000..cdb456cb --- /dev/null +++ b/examples/minimax/MiniMaxH3MusicVideo.json @@ -0,0 +1,416 @@ +{ + "id": "MiniMaxH3MusicVideo", + "description": "A music video built from cuts, sung to a soundtrack that never touches a chain. The long-take way to film a song - one chained generation lip-synced end to end - degrades with every carried segment and can let the sync slip. This builds the video the way music television does instead: MiniMax-Music3 writes the song, 'slice_audio' cuts it into frame-exact pieces (124 frames at 24 fps each), and every shot is generated fresh from the same Z-Image portrait plus its own slice, lip-synced to just those five seconds. No shot conditions on another shot's output, so the last cut is as clean as the first. 'concat_videos' splices the shots, and because each one covered exactly its slice's frames, the edit is sample-accurate by construction: 'pair_audio' drops the original, unbroken song over the whole cut and the mouths line up in every shot. The generation models pass through one at a time - each is released before the next loads - so the workflow peaks no higher than its largest single model.", + "variables": { + "singer_portrait_prompt": "a 3/4-length portrait of an anthropomorphic river otter lounge singer in a burgundy velvet dinner jacket and black bow tie, standing at a vintage chrome ribbon microphone on a small nightclub stage, a single warm spotlight from above, deep blue darkness behind, photographic, sharp focus", + "song_prompt": "Genre: retro soul jingle, doo-wop tinged. BPM: 92. Key: E-flat major. Emotional progression: warm and wry throughout, swelling on the hook. Listening scenario: a late-night variety show number. Production profile: vintage, close-mic, plate reverb, slightly warm tape saturation. Vocals: smooth male crooner lead, velvety and unhurried, small melisma on the hook. Arrangement: upright bass and brushed drums under electric piano, a two-note baritone sax riff answering each line, finger snaps on the backbeat. The hook lands twice and the last line is held.", + "song_lyrics": "[verse]\nI've been waiting on you darling like a render overnight\nEvery frame of my devotion coming slowly into light\n[chorus]\nOoh, progress bar of love\nOoh, don't you stall on me now", + "audio_duration": 21, + "shot_1_wide_open": "subject_definitions:\n is the anthropomorphic river otter lounge singer in , in a burgundy velvet dinner jacket and black bow tie, whose face, fur, jacket and proportions define his appearance for the whole video.\n