Activity

1. dkackman

   added

   enhancementNew feature or requestNew feature or request

   owner:donParked for human decisionParked for human decision

   ideaAn idea for the researcher agent to assessAn idea for the researcher agent to assess

   backend:shared

   owner:leadFeature lead's turn (harnest R11)Feature lead's turn (harnest R11)

   priority:2Backlog rank: solid ROI, moderate scopeBacklog rank: solid ROI, moderate scope

   featureWork bigger than a fix: designed with Don, built in stages (harnest R11)Work bigger than a fix: designed with Don, built in stages (harnest R11)

   and removed

   owner:donParked for human decisionParked for human decision

   ideaAn idea for the researcher agent to assessAn idea for the researcher agent to assess

   on Oct 5, 2026
2. dkackman commented on Oct 5, 2026

   

   dkackman

   on Oct 5, 2026

   OwnerAuthor

   More actions

   Approved by Don (2026-10-05) for a full build: plan both pieces, audio-hold in the H3 denoise loop and the refine pass after the latent upscale, as one feature. No A/B gate before planning. The lem A/B against upscale-only (and #612's LMS upscaler, if it lands first) belongs in the plan's acceptance cases. Handing to the feature lead for a plan.
3. dkackman commented on Oct 5, 2026

   

   dkackman

   on Oct 5, 2026

   Last edited by dkackman

   OwnerAuthor

   More actions

   Plan v4: H3 audio-hold + refine pass after latent upscale (#598)

   Written by model claude-opus-5-5 via provider anthropic. v1 was a design session; v2 a decompose session (folds in Don's approval of v1); v3 a design session (2026-10-06), which folded in Don's choice of refine schedule (B) from #620, recorded what stages 1 and 4 shipped, and corrected stage 2 against installed diffusers. v4 is a decompose session (2026-10-06): it folds in Don's approval of v3, turning Q1 and Q2 into decisions D7 and D8. Scope is unchanged: no new surface, and the remaining stages are the same two.

   Status of the stages

   • #598 stage 1: H3 audio-hold blocks in the denoise pipeline (hold_audio) #618, stage 1 (hold blocks, hold_audio): shipped and verified.
   • #598 stage 4: hold_audio in music-video and chain-matched templates + lip-sync A/B #619, stage 4 (templates + lip-sync A/B): shipped and verified, on the fallback. The A/B (one seed, sung track) went against hold: the mouth was open at ~3/6 onsets with hold, against 6/6 with the reference, at 19.8 vs 18.3 min. So the plan's named fallback applies: the three templates keep the per-shot AudioReference, hold_audio is opt-in engine surface, and the guide, TASKS.md and SKILL say so with the numbers. D4 is superseded, and D7 makes the fallback final (see Decisions).
   • #598 stage 2: H3 refine schedule after latent upscale (refine_strength) #620, stage 2 (refine schedule): not built. Plan v2's schedule was impossible (below). v3 replaced it.
   • #598 stage 3: templates/minimax/upscale-refine + refine A/B gate #621, stage 3 (upscale-refine + A/B gate): not built.

   Verdict: build the remaining two stages, with the refine template still gated on its A/B

   • Value. The hold half has shipped, and its lip-sync value did not appear on the one measured seed. That doesn't touch refine. Refine needs hold only to keep the base pass's own audio, which was generated jointly with that video and already matches it. Refine's value is unchanged from v2: closing #471 stage 2: minimax/upscale-preview template + skill line (gated on Don's stage-1 comparison) #500's "close but soft". Its evidence is still the weaker half. Don's gate timed the upscale preview at 7.8 min and native 768p at 12.7 min, and the eval: H3 community LoRAs - Realism People stacked wins; 4-step v1.2 viable draft; Acc/HyperFlow/FastH3 need loaders #585 Realism LoRA on the native turbo path raised the bar. So the template (stage 3) stays gated on the A/B. The engine half (stage 2) is cheap now that the hold seam exists.
   • Build cost. Stages 1 and 4 are spent. About $11-16 remains: stage 2 $6-9, stage 3 $5-7.
   • Inertia. As v2 said, minus what has already landed:
     • one more block in h3_blocks.py, the one place dw modifies a modular pipeline (D1), covered by the anchor-pin test;
     • one call argument, refine_strength. There is no MCP tool, so SURFACE_BUDGET is untouched;
     • one template if the gate passes. COMPACT_BUDGET (tests/test_catalog_structure.py:431, 9_550) has about 90 tokens of headroom and needs a raise of about 100;
     • a second owner of an H3 sigma schedule. Option (B) computes the shift-spaced tail in dw, because set_timesteps(sigmas=...) applies no shift. It reads the shift from the loaded scheduler and holds no constants.
   • Reversibility. refine_strength is opt-in. A run that doesn't name it is unchanged, and the no-op insertion keeps the cache key. If the gate fails, the template goes and the engine stays as guide surface, as with #471 stage 1: upscale_h3_latents + decode_h3_latents tasks, no refine (the measurement) #499.
   • Cheaper alternatives.
     • (a) Stop here. Hold has shipped, and there is no refine. #471 stage 2: minimax/upscale-preview template + skill line (gated on Don's stage-1 comparison) #500's softness stays.
     • (b) A template-only refine: still impossible. PrepareLatentsStep treats a passed latents as pure noise, and SetTimesteps takes no strength.
     • (c) Schedules (A) and (C) from #598 stage 2: H3 refine schedule after latent upscale (refine_strength) #620: rejected by Don in favour of (B). (A) is a single evaluation at 0.2. (C) is ComfyUI's grid-fraction denoise, which starts near σ 0.6 at shift 6 and re-noises far more than detail.
   • Overlap.
     • idea (parked): H3 latent-space chunk continuation - blocked on multi-frame keyframes in diffusers #611: its stage 1, #611 stage 1: H3 multi-frame video guides (guides argument, layout step) #648, is now blocked by #598 stage 2: H3 refine schedule after latent upscale (refine_strength) #620, because both edit h3_blocks.py and the anchor-pin test (Don's ordering). If #598 stage 2: H3 refine schedule after latent upscale (refine_strength) #620 is declined, #611 stage 1: H3 multi-frame video guides (guides argument, layout step) #648 goes ahead on #598 stage 1: H3 audio-hold blocks in the denoise pipeline (hold_audio) #618's seam alone.
     • idea: trial Alissonerdx H3 LoRAs + LMS sharpness latent upscaler (aligned-guide LoRAs blocked on #611) #612 (LMS upscaler) adds an A/B arm to stage 3 only if it has landed. It is not a dependency.

   Proposed follow-up (Don decides): none.

   Problem and non-goals

   Problem. H3 can't re-denoise an upscaled latent from a low sigma. (Generating to a fixed soundtrack is solved: #618.)

   Non-goals:

   • no 3-pass (2 MP) variant;
   • no LTX-2 changes;
   • no persisted-latents result type;
   • no change to downloads_required;
   • no re-measure of hold's lip sync (D7);
   • nothing taken from vrgamegirl19's repo.

   Design (stage 2, corrected)

   Correction (v3): plan v2's schedule could not run. v2 said the refine runs "the grid points below σ₀", about 4-5 evaluations at 9 steps, shift 6, strength 0.2. Against the installed MiniMaxH3Scheduler, the 9-point shift-6 grid is 1, .977, .947, .909, .857, .783, .667, .462, 0. No point lies in (0, 0.2], so v2's schedule gave zero evaluations (#620, build session).

   The schedule (D6, Don's option B). Start at exactly σ₀ = refine_strength. Take num_inference_steps points from σ₀ down to 0, spaced by the scheduler's shift: linear in the unshifted coordinate u, from u₀ = σ₀ / (shift − (shift−1)·σ₀) down to 0, mapped through σ = shift·u / (1 + (shift−1)·u). That gives num_inference_steps − 1 evaluations. Some values:

   shift	num_inference_steps	σ list	evaluations
   6	5	0.2, .157, .109, .057, 0	4
   6	6	0.2, .166, .129, .089, .046, 0	5

   What the installed diffusers requires (from the v3 sweep):

   • set_timesteps(sigmas=...) applies no shift and appends nothing. It raises unless the σ list is strictly decreasing and ends at exactly 0.0 (scheduling_minimax_h3.py:160-167; timesteps = 1 − σ[:-1]). So the block applies the shift itself, appends the 0, and refuses a list that collapses (num_inference_steps < 2).
   • The audio scheduler needs a matching list. denoise.py:222-241 calls audio_scheduler.step even when every audio row is held. It needs audio_timesteps the same length as the video's, and an exact index_for_timestep match. The block therefore resets the audio scheduler with the same σ list. The values don't matter, because no audio row is stepped, and the block asserts that every audio row is a held condition row before the loop runs.
   • What it rebuilds: SetTimesteps writes timesteps, audio_timesteps and row_timestep_plan (before_denoise.py:1245-1266). The block rebuilds all three, the plan through build_row_timesteps (:1197-1236), so condition rows keep max(t, keyframe_noise_aug).
   • Re-noise only the generated rows. FL2VA and Ref2VA put condition rows at the front of latents (before_denoise.py:1030, 1109-1121). The block re-noises only latents[num_condition_video_rows:]: x = (1−σ₀)·x₀ + σ₀·ε, with ε from the step's generator so a run is seeded.
   • Passed latents: a (1, 24, F, H, W) tensor skips the noise draw (before_denoise.py:865-879). That matches upscale_h3_latents' output, provided the step's width and height equal the upscale target.

   Placement. The block runs after set_timesteps in each of the three core-denoise sequences, so it goes in just before denoise. insert_audio_hold (h3_blocks.py:325-354) inserts before an anchor, so the refine block anchors on the denoise block. Three things change with it:

   • blocks() (:283-288) returns three blocks and caches them, and :333 changes from unpacking two;
   • the "already inserted" check (:339) covers the refine block too;
   • introspection.py:361-380 and components.py:634-636 already call insert_audio_hold, so get_pipeline_signature shows refine_strength with no further change.

   It does nothing unless refine_strength is set (D2).

   Validation. The static checks sit beside hold's, in dw/hold_audio.py:68-91 and dw/validation.py:315-317; the runtime check sits beside _with_held_audio in pipeline.py:615-639. A step is refused when:

   • refine_strength is not a number in (0, 1);
   • it has no latents;
   • it has no hold_audio;
   • it is not an H3 step;
   • num_inference_steps is below 2.

   Cost (corrected from v2). Refine time scales with num_inference_steps, not independently of it. dw/plan.py:398-409 resets a catalog cost to unknown only for a declared cost driver. So the stage 3 template declares refine_strength in its cost_drivers, along with num_frames, num_inference_steps, width and height. Otherwise a changed strength would be quoted at the template's measured cost. Progress needs no change: the total comes from len(timesteps) (progress.py:87-144), and N points report N−1 steps, which is what the docs already say.

   Surfaces decided here:

   • Engine: one more no-op block.
   • Call argument: refine_strength (D3).
   • REST, MCP and the workflow schema: none.
   • Trust gate: not widened.
   • docs/ARCHITECTURE.md:95 changes "two blocks go in" to three.

   Stages remaining

   Build order: 2 → 3. Each stage is blocked by the one before it. No cross-feature blockers. #620 blocks #611's #648.

   Stage 2 (#620): refine schedule (engine), est. $6-9, deploy: server

   • Builds:
     • DwH3RefineScheduleStep as designed above, inserted before denoise in all three core-denoise sequences;
     • the matched audio σ list;
     • the held-audio assertion;
     • re-noising of the generated rows only;
     • validation;
     • the ARCHITECTURE.md row.
   • Unit tests:
     • the σ list for (shift 6, 5 points, 0.2) equals 0.2, .157, .109, .057, 0 to 3 dp, and is strictly decreasing and ends at 0;
     • the schedule survives set_timesteps (the block runs after it);
     • the audio timesteps match the video's length;
     • the assertion fires when an audio row isn't held;
     • condition rows are untouched by the re-noise (FL2VA and Ref2VA shapes);
     • the re-noise formula, and a seeded draw that reproduces;
     • the anchor-pin test covers the third block in all three sequences;
     • the refusals;
     • no refine_strength leaves outputs identical (CPU tiny-tensor).
   • Docs: rewrite WORKFLOW_GUIDE.md Promoting an H3 take to 768p in latent space:
     • drop the "no refine pass" lines (:1646, :1675);
     • add the refine step: base → upscale_h3_latents → H3 at 1344x768 with latents: previous_result:up, refine_strength: 0.2, num_inference_steps: 5, hold_audio: previous_result:base.audio;
     • put the new JSON after the existing example, because test_modular_output_properties.py:96-102 reads the first JSON block after the heading;
     • keep upscale_h3_latents, pair_audio, previous_result:base.videos and "fps": 24 in the section, for test_server_guides.py:263-278;
     • add a sentence where hold is called opt-in (:1632-1639): refine uses hold to keep the base pass's own audio, which is a different case from lip sync to supplied audio;
     • add the cost note: refine evaluations = num_inference_steps − 1.
   • Acceptance intent (MCP, lem). v3 changes the evaluation counts in M-F084:
     1. Runs the guide's workflow. get_guide serves it, and it validates and completes at 1344x768 with 124 frames. The output audio is the base pass's track. get_job_events show 4 denoise steps in the refine step at num_inference_steps: 5, and 5 at 6.
     2. Strength changes how far it starts, not how many steps it runs. At 0.4 and 5 points it still runs 4 evaluations, and moves the picture further from the upscale-only decode than 0.2 does. (v2's "0.4 runs more evaluations than 0.2" no longer holds under (B).)
     3. Changes only detail. At 0.2 it keeps the subject, layout and motion of the upscale-only decode of the same seed, judged side by side with get_output_frames.
     4. Refusals at validate:
        • refine_strength of 0, 1.0, −0.2, 1.5 or "0.2";
        • no latents;
        • no hold_audio;
        • on SD 1.5;
        • num_inference_steps: 1 with refine_strength.

   Stage 3 (#621): templates/minimax/upscale-refine + the A/B gate, est. $5-7, deploy: server (catalog) + plugin

   • Builds:

     • the template: 544p base with the turbo LoRA, then upscale_h3_latents, then refine at 1344x768 with the 768p LoRA at video_shift 6, num_inference_steps 5 and refine_strength 0.2 (D8), holding the base audio;
     • compliance with test_h3_schedule.py: scheduler.shift = variable:video_shift (12 or 6), audio_scheduler.shift = variable:audio_shift (3), LoRA alpha = variable:lora_alpha defaulting to null, and no 20 steps;
     • cost_drivers including refine_strength;
     • measured cost and vram_estimate, from arm (b);
     • COMPACT_BUDGET raised by about 100;
     • a short minimax-h3 SKILL row. The SKILL is at 10,872 of 12,288 bytes.

   • Acceptance intent: the A/B, unchanged from v2. The #471 stage 2: minimax/upscale-preview template + skill line (gated on Don's stage-1 comparison) #500 crowd-faces prompt, seed 42, 124 frames, on lem, with these arms:

     • (a) upscale-only;
     • (b) this template;
     • (c) native video-with-audio-768p;
     • (d) idea: trial Alissonerdx H3 LoRAs + LMS sharpness latent upscaler (aligned-guide LoRAs blocked on #611) #612's LMS upscaler, if it has landed.

     For each arm the tester records minutes and face crops. (b)'s minutes must be within 50% of the template's cost.

     The gate is Don's call. If he says no, the template and SKILL row are reverted before close-out.

   Risks

   • Held clean audio against a re-noised video. In #598 stage 4: hold_audio in music-video and chain-matched templates + lip-sync A/B #619, held audio drove the picture worse than a reference. Here the audio is the base take's own, and the video starts at σ 0.2 from a picture that already matches it, so the risk is low. Stage 2's acceptance item 3 catches a drift.
   • Audio and video lists of equal length despite different shifts. The block uses one σ list for both, and the assertion guards the case where an audio row would be stepped.
   • Anchor drift with diffusers: covered by the pin test, now with three blocks.
   • Refine shows no gain over native 768p: stage 3's gate catches it.
   • Audio VAE VRAM at 768p: measured in stage 3.

   Decisions

   • D1 (2026-10-05): engine H3 blocks, with an ARCHITECTURE.md row naming h3_blocks.py.
   • D2: the blocks are inserted always, as no-ops.
   • D3: the arguments are hold_audio and refine_strength.
   • D4: superseded (2026-10-06). The decision was that hold replaces the per-shot reference. #598 stage 4: hold_audio in music-video and chain-matched templates + lip-sync A/B #619's A/B went against hold, so the plan's named fallback shipped: the templates keep the reference and hold is opt-in.
   • D5: build order 1 → 4 → 2 → 3. Stages 1 and 4 are done.
   • D6 (Don, 2026-10-06, on #598 stage 2: H3 refine schedule after latent upscale (refine_strength) #620): refine schedule (B). σ₀ = refine_strength, then num_inference_steps shift-spaced points to 0, which is num_inference_steps − 1 evaluations. Refine cost scales with num_inference_steps.
   • D7 (Don, 2026-10-06, v3 Q1): the hold fallback from #598 stage 4: hold_audio in music-video and chain-matched templates + lip-sync A/B #619 is final. The templates keep the reference path and hold_audio is opt-in. A re-measure of hold is out of this feature; it would come back as its own idea if a field report shows reference lip sync failing where hold might help.
   • D8 (Don, 2026-10-06, v3 Q2): the stage 3 template uses num_inference_steps 5 (4 evaluations) and refine_strength 0.2. The guide example matches.

   Questions

   None open. #621's template stays gated on its A/B, and that gate is Don's call.
4. dkackman

   added

   owner:donParked for human decisionParked for human decision

   status:plan-reviewA feature plan is posted and waiting for DonA feature plan is posted and waiting for Don

   and removed

   owner:leadFeature lead's turn (harnest R11)Feature lead's turn (harnest R11)

   on Oct 5, 2026
5. dkackman commented on Oct 5, 2026

   

   dkackman

   on Oct 5, 2026

   OwnerAuthor

   More actions

   Plan v1 approved by Don (2026-10-05).

   • Q1-Q4: the defaults are accepted:
     • engine H3 blocks, with an ARCHITECTURE.md row;
     • insert them always, as no-ops;
     • the arguments are named hold_audio and refine_strength;
     • hold replaces the per-shot AudioReference.
   • Q5: the order is 1 → 4 → 2 → 3. Hold in the audio-driven templates comes before the refine engine. The stage 4 lip-sync A/B is the bigger value and tests the "lip-syncs poorly when following supplied audio" risk before any money goes into refine.

   Two corrections from checking installed diffusers (modular_pipelines/minimax_h3):

   1. Hold insertion point. prepare_latents → set_timesteps exist only inside the three core-denoise sequences: MiniMaxH3CoreDenoiseStep, MiniMaxH3FL2VACoreDenoiseStep and MiniMaxH3Ref2VACoreDenoiseStep, the branches of MiniMaxH3AutoDenoiseStep. The hold block has to go into each of them, between prepare_latents and set_timesteps, not onto MiniMaxH3Blocks. The anchor-pin test should cover all three. Ref2VA's prepare-latents also runs a reference-row count check (before_denoise.py:1113). Extend the count after that check, never before it.
   2. Refine block placement. MiniMaxH3SetTimestepsStep.__call__ calls scheduler.set_timesteps(num_inference_steps) itself (before_denoise.py:~1245), so a tail schedule set before it is overwritten. The refine block must run after set_timesteps. It has to:
      • reset the video scheduler with sigmas=;
      • rebuild timesteps and row_timestep_plan;
      • deal with the zip(timesteps, audio_timesteps) pairing. The shortened video schedule pairs with the head of the full audio schedule. That is harmless only while every audio row is a held condition row, so assert it rather than relying on it.

   Confirmed while checking: target audio rows at t=1.0 are never stepped (denoise.py:235), and the checkpoint is guidance-distilled with no unconditional pass to drop them, so the "no replacement loop" correction stands.
6. 56 remaining items

   Load more
7. dkackman

   added 2 commits that reference this issue on Oct 7, 2026

   Merge branch 'docs/598-record-fix-forward' into develop (#598)

   caf6338

   Merge branch 'docs/598-record-fix-forward' into develop (#598)

   6ec9a77
8. dkackman commented on Oct 7, 2026

   

   dkackman

   on Oct 7, 2026

   OwnerAuthor

   More actions

   Close-out: fix-forward #664 closed; handing the parent to the tester for the final check. Written by model claude-opus-5-5 via provider anthropic (close-out session, 2026-10-07).

   Closed: stages #618, #619, #620 and #621, plus fix-forward #664 (the upscale-refine template was reverted in 125818e7, verified 2026-10-07). The engine half of refine (refine_strength) stays as opt-in surface.

   Design record: docs/proposals/complete/h3-audio-hold-refine-complete.md, which now has a Fix-forward: #664 section, and todo.md is updated. On develop @ 6ec9a77. These are docs only, with no deploy.

   verify: run every case whose source: names one of this feature's stages, and close the parent if they all pass. Per Don's decision 2 (2026-10-06), the GPU cases are credited from their stage verifies and are not re-run. The final check is free only:

   • templates/minimax/upscale-refine is gone from list_workflows;
   • the minimax-h3 skill no longer names it;
   • the guide's Refining the upscaled latents section still validates, with no reference to the template.

   Non-blocking, per Don: M-F077 (no fixture), the M-F079(a) fixture ruling, harnest#62/#66, the M-F086 wording request, and retiring M-F085/M-F086/M-F099.
9. dkackman

   added

   owner:testerTester's turn to actTester's turn to act

   status:fixed-pending-verifyImplementer fixed, awaiting tester verificationImplementer fixed, awaiting tester verification

   and removed

   owner:leadFeature lead's turn (harnest R11)Feature lead's turn (harnest R11)

   status:plan-approvedDon approved the feature plan (Don only)Don approved the feature plan (Don only)

   on Oct 7, 2026
10. dkackman commented on Oct 7, 2026

    

    dkackman

    on Oct 7, 2026

    OwnerAuthor

    More actions

    Verified (parent final check). Tester: claude-opus-5-5 via anthropic, against lem develop @ 7f524c4. Free calls only, following Don's decision 2: the GPU cases count from their stage verifies (#618, #619, #620, #621) and were not re-run.

    • upscale-refine is gone from the catalog. list_workflows(shape="shot", traits="has-audio") lists 34 workflows, and none is templates/minimax/upscale-refine. The minimax entries that remain are the existing templates, including video-with-audio and video-with-audio-768p.
    • The minimax-h3 skill no longer names it. I loaded the skill from the plugin tree at develop. It doesn't mention upscale-refine or a refine template anywhere. hold_audio is still described as opt-in in its Hard rules.
    • The guide's refine section still works. get_guide("workflows", section="Refining the upscaled latents") returns "Refining the upscaled latents: refine_strength", and it doesn't reference the template. I passed its JSON example to validate_workflow unchanged: valid: true, no errors, no warnings, 3 steps, estimate basis unknown. The engine surface stays as Don ruled: refine_strength, hold_audio: previous_result:base.audio, and the refusal list.
    • The M-F085/M-F086/M-F099 retirement is already filed. It is harnest#76, now closed (the curator applied it).

    Not blocking (named by Don): M-F077 (no fixture), the M-F079(a) fixture ruling, harnest#62/#66, and the M-F086 wording request.

    Side note, filed separately: the guide's prose doesn't match its own example. The prose says "5 points at shift 6 and strength 0.2 are sigma 0.2, 0.157, 0.109, 0.057, 0". But the example's refine step takes scheduler.shift: variable:video_shift, which is 12.0, with the fl2v 8-step LoRA. That reads like a leftover from the reverted template, which used the 768p LoRA at shift 6. It's a doc nit and doesn't block this close.
11. dkackman

    added

    status:verifiedTester confirmed the fix via a real MCP callTester confirmed the fix via a real MCP call

    and removed

    status:fixed-pending-verifyImplementer fixed, awaiting tester verificationImplementer fixed, awaiting tester verification

    on Oct 7, 2026
12. dkackman

    closed this as completedon Oct 7, 2026
13. nikhilash6

    added 4 commits that reference this issue on Oct 10, 2026

    docs(proposals): dkackman#598 close-out - design record for H3 audio-…

    ...

    42d04c4

    docs(proposals): dkackman#598 - record Don's no on the refine gate; t…

    ...

    789a294

    docs(proposals): dkackman#598 - record the dkackman#664 fix-forward r…

    ...

    3f35e44

    docs(proposals): dkackman#598 - name the reverted template by catalog…

    ...

    06bd54f