From 7bd942ccee1e493ae3662b9957ddd42121808b7e Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Sun, 6 Sep 2026 13:22:34 +0000 Subject: [PATCH 01/32] Proposal: scripted dialogue shorts and where TTS belongs Design notes only. Records why dialogue audio should not be generated ahead of H3 (follow-supplied-audio lip sync is the weak pathway), where a text-to-speech task does earn its place (voice timbre reference), and the larger missing piece: script -> Context-IR shot list. Co-Authored-By: Claude Opus 5 (1M context) --- docs/proposals/scripted-dialogue-and-tts.md | 109 ++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 docs/proposals/scripted-dialogue-and-tts.md diff --git a/docs/proposals/scripted-dialogue-and-tts.md b/docs/proposals/scripted-dialogue-and-tts.md new file mode 100644 index 00000000..fbac2a1b --- /dev/null +++ b/docs/proposals/scripted-dialogue-and-tts.md @@ -0,0 +1,109 @@ +# Proposal: scripted dialogue shorts, and where text-to-speech belongs + +Status: design only, no code changes. Records the conclusion of a design +discussion about supporting a flow like: *write a script of two people talking +→ give each a consistent voice and face → generate H3 clips of each speaking +their lines → cut between them into a finished dialogue scene.* + +## The question + +Should text-to-speech become a task (or a workflow shape), generating each +speaker's lines as audio up front, which H3 then follows? Or does predefined +textual content belong inside H3 itself? + +## Conclusion: do not generate the dialogue audio first + +The text being predefined is not an argument for TTS. H3 already takes +predefined dialogue — it speaks the exact line written inline in the prompt: + +``` + (S1), a fast, nasal, high-pitched male tenor voice, tight with +disbelief, says [English] Okay. Who finished the pistachio? +``` + +Generating the audio first and asking H3 to follow it moves every line of +dialogue onto the model's weak pathway. H3 lip-syncs well when it *generates* +the speech (speech and mouth come out of one process, sync is free) and poorly +when it must *follow* an audio reference. Investigated 2026-09-04 on the +MarmotCroons rebuild; ruled out with a run each: conditioning-window choice, +prompt language, LoRA family, 30 steps without the turbo LoRA, and dw's own +plumbing (verified end to end). Nothing recovered it. Conditioning on a +centre-extracted vocal isolate was a partial win, not a fix. + +So: script → TTS → `match_audio` trades away the one thing that works for free. + +## The workflow already largely exists + +[`workflows/minimax/MiniMaxH3SitcomShort.json`](../../workflows/minimax/MiniMaxH3SitcomShort.json) +is nearly the described flow already: + +- two Z-Image steps draw the cast portraits (the second reuses the first's + loaded pipeline; `release_pipeline` frees it before the video model loads) +- the first shot loads H3 once; the remaining shots are `pipeline_reference` + steps rerunning that loaded model with a new prompt, references and length +- `concat_videos` splices the episode with hard cuts, and `audio_bleed_ms` + rings each shot's room tone over the silent head of the next, so only the + picture cuts hard + +Consistency in this cut-based shape comes from repetition, not from a shared +audio track: + +| What stays consistent | How | +| --- | --- | +| Identity | reference the same portrait in every shot | +| Voice | repeat the character's voice description **verbatim** in every prompt | +| Ambience | repeat `overall_soundscape` verbatim; `audio_bleed_ms` carries it across the cut | +| Quality | each shot is generated fresh from the portraits, so shot 12 is as clean as shot 1 | + +Constraints to design around: a shot is 5.17–14.4 s (`num_frames` of the form +`17n + 5`, 124–345, at 24 fps), so a long line must be split across shots; and +roughly 13 shots is the ceiling for a single `dw.run` on a 24 GB box — beyond +that, batch and stitch. + +## Where a TTS task does earn its place + +There is no TTS in the repo today. It is worth adding, in these roles — none of +which is "the track the mouth follows": + +1. **Voice *timbre* reference.** `ref2va` takes an `