LTX-2.5 Integration - #1683
Open
mi804 wants to merge 33 commits into
Open
Conversation
…ference examples - Merge LTX-2.5 into LTX2AudioVideoPipeline (drop separate ltx25_audio_video.py) - Register ltx25 DiT / Gemma4 text encoder / feature extractor + connectors / video VAE (DiffVAE + ConvVAE) / audio VAE / vocoder / duration head, BF16 and INT8 ConvRot variants - Flatten DiffVAE into diffsynth/models/ltx25_diffusion_video_vae.py with a single decode interface and auto tiling - Add keyframes abs pos embedding and tokenwise AV cross-attention scale/shift to ltx2_dit - Add stage1 distilled ancestral schedule unit, audio-only (T2A) and A2V freeze support, auto_duration in one pipe call - VRAM management maps for new modules; low-VRAM and standard inference examples for T2AV/I2AV/A2V/T2A/Retake/IC-LoRA/INT8
- Resolve aliased/preprocessor module references through the owner in ltx2_dit so VRAM-wrapped modules are used under CPU offload - Move STFT/mel buffers and DiffVAE raw parameters (scale_shift_table, fused QKV) to the input device/dtype at the use site - Read functional projection weights through the VRAM wrapper computation path; reduce the DiffVAE module map to leaf-level entries since the decoder calls block methods directly - Treat missing generate_video input param as True so training caches video positions (T2A keeps generate_video=False) - Add LTX-2.5 T2AV split training scripts (LoRA/full/debug), validate scripts and a series-local zero3 accelerate config - Rewrite LTX-2.5 docs for the unified pipeline, training and unsupported features; add README news and model table rows - Drop the PR leftover LTX-2.5-Keyframe-Interpolation low-VRAM script (keyframes are covered by I2AV)
ZeRO-3 is incompatible with the Gemma4 text encoder: DiffSynth constructs models inside deepspeed.zero.Init, where transformers' _init_weights indexes weight[padding_idx] on an empty sharded embedding and raises IndexError. Replace the series-local zero3 config with the ZeRO-2 + CPU optimizer/param offload config used by the LTX-2.3 full scripts.
- Return the original input audio (resampled and trimmed) when the audio modality is fully frozen, matching the upstream A2V pipeline which skips the VAE/vocoder round trip to preserve fidelity (mel corr vs input 0.928 -> 0.998) - Run the second stage without classifier-free guidance, as upstream uses a simple denoiser there; the stage-1 cfg scale previously leaked into stage 2 - Align the stage-2 distilled LoRA strength default with upstream (1.0 instead of 0.8), which removes residual dithering in dev two-stage outputs
Drop unreferenced helpers, tiling constructors and the NATTEN attention fallback that the portable eager path never selects. Verified bit-identical outputs (max abs pixel diff 0) for the distilled T2AV and I2AV keyframe examples before/after.
The Lightricks/LTX-2.3 repo no longer ships ltx-2.3-spatial-upscaler-x2-1.0.safetensors; the registered ltx2_latent_upsampler hash matches the x2-1.1 file, so update the example scripts, docs and the registration example comment accordingly.
The decoder selector validated the loaded decoder component at unit execution time, which breaks split training for LTX-2/2.3 repackaged checkpoints whose stage 1 loads only the VAE encoder. Caching stages never decode, so skip the validation when the scheduler is in training mode; inference behavior is unchanged.
Remove 192 docstring blocks (~1000 lines) from the portable DiffVAE implementation; the module keeps its inline WHY comments. Distilled T2AV output remains bit-identical (max abs pixel diff 0) to the pre-change baseline. File size 6391 -> 5036 lines.
The dev weights are the general-purpose checkpoint, so they now cover the full feature set: OneStage/TwoStage T2AV and I2AV examples are added in both the standard and the low-VRAM directories. The distilled and INT8 checkpoints keep a single T2AV example each, so the distilled I2AV scripts are removed; IC-LoRA stays a distilled exception because upstream runs both of its stages on the distilled weights. Docs and README list the resulting ten-row matrix. LTX-2.5 also ships a longer default negative prompt than LTX-2.3: upstream prefixes the shared tag list with has_subtitles, has_blurbox, transition from black, transition to black and speech_ending_short. Add an "LTX-2.5" entry to pipe.default_negative_prompt and switch every 2.5 example, low-VRAM example and validation script to it (the T2A scripts previously used the placeholder "noise").
LTX25_GEMMA_CONFIG was a raw pprint dump with single quotes and hanging indentation. Restyle it to match the other text encoder configs (double quotes, four-space indent, one key per line, inline leaf dicts) and collapse the 48-entry layer_types list into its repeating five-sliding-plus-one-full pattern. Formatting only: the resolved Gemma4UnifiedConfig.to_dict() is byte-identical to the previous one.
Moving _init_preprocessors into __init__ made the argument preprocessors capture the plain modules before VRAM management replaces them with wrappers, so under CPU offload the preprocessors called stale modules whose weights never onload. PR modelscope#1602 built the preprocessors at the start of every forward, after wrapping; restore that timing and drop the OwnerModuleProxy workaround it had made unnecessary. Also restore the upstream stage2_lora_strength default of 0.8, which the LTX-2/2.3 two-stage examples rely on, and pass 1.0 explicitly in the LTX-2.5 two-stage examples whose distilled stage-2 LoRA needs full strength.
… the TE file The VRAM module maps and the audio/video VAE modules carried changes that only existed to support our own earlier fine-grained map experiment: PerChannelStatistics, Snake and sibling vocoder entries, the LTX2AudioEncoder entry, the LTXModel block-level entry and the use-site STFT/mel casts. With the maps back to the upstream layout the vocoder and audio encoder fall back to the whole-model wrap exactly as LTX-2.3 runs today, so those code changes are reverted too, together with the string-to-Enum coercions in the video VAE whose only consumers were string extra_kwargs we added for the 2.5 entries. The only remaining map delta is the LTX-2.5 DiffusionVideoDecoder entry, whose flat class path and leaf-level wrapping the 2.5 decoder requires. LTX25GemmaTokenizer moves verbatim into ltx25_text_encoder.py, mirroring how the LTX-2.3 tokenizer lives inside its TE file, and the 2.5 registry entries are regrouped by model hash with concrete dev-file example comments.
…checkpoint Register the LTX-2.5 components under the same model names as LTX-2/2.3 so from_pretrained fetches every version through one shared path; only the tokenizer construction stays version-specific. The text encoder post modules are the single component whose weights are scattered across two upstream files (feature extractor in the TE checkpoint, connectors in the transformer checkpoint), so they are packed into DiffSynth-Studio/LTX-2.5-Repackage/text_encoder_post_modules.safetensors in target key layout, needing no state dict converter, and the packed Gemma4 tokenizer assets are unpacked into an HF-style tokenizer directory so tokenizer_config works like LTX-2.3. Pipeline cleanups along the way: the video decoder is selected by availability (conv decoder first, diffusion decoder as fallback), gemma_path and load_duration_head are gone, and the upstream section comments, import layout and upsampler fetch position are restored.
The Gemma3Processor was assigned but never read anywhere in the repository, so remove it together with its attribute and imports. Tokenizer construction collapses to picking the version-specific class and instantiating it with tokenizer_path, which is now the parameter name for LTX25GemmaTokenizer as well.
The decoder selector unit is gone: the pipeline keeps a single video_vae_decoder attribute, filled with the ConvVAE decoder when loaded and the DiffVAE decoder otherwise, and __call__ passes the tiling and seed arguments straight to decode(). Both decoders accept **kwargs so the uniform argument set needs no branching; the DiffVAE decoder takes seed/rand_device and builds its sampling generator internally, offset by 42 from the denoise seed, and tiles automatically when tiled is set, matching the official pipeline default. Auto duration validation moves into the AutoDuration unit, which now declares onload_model_names and clamps the requested bounds into (0, 20] instead of raising.
…audio_only NoiseInitializer goes back to the upstream process_stage/process structure; the only additions are the generate-video guard and a build_video_keyframes_mask helper that returns the LTX-2.5 first-frame marker (None for other versions), so the mask is rebuilt at every stage resolution automatically. The token-layout noise, unpatchify bridges and ancestral transforms are gone: noise is drawn in the 5D latent layout like LTX-2.3. The generate_video flag is renamed to audio_only with inverted polarity, which removes the None-normalization trick: the unit runner passes None for missing keys, and a falsy value now correctly means "generate video" during training caching stages.
The decode path accepts tile_size_in_pixels / tile_size_in_frames (overlaps stay halo-derived, since DiffVAE ramps need complementary masks) and validates them; any invalid configuration falls back to the automatic tiling instead of raising. The automatic path now returns the minimum legal tile (spatial floor 512 px, temporal 80 frames), which keeps the decode peak around 7 GB at 1024x1536x121 instead of the recommender's spare-VRAM-hungry choice (~68 GB measured). Inference examples pass tile_size_in_frames=80 and carry a comment pointing at the conv vae decoder as the low-VRAM alternative.
…iations - Move the rectified-flow ancestral Euler step into AncestralFlowMatchScheduler (flow_match.py base classes stay byte-identical to upstream); installed in __call__ for LTX-2.5 distilled runs, stage 2 restores the base scheduler. - Revert base_pipeline.py to upstream: drop the dead generate_noise generator parameter and handle disabled-modality predictions with a 0 placeholder at the model_fn_ltx2 return site instead of a CFG None guard. - Remove LTX-2.5 special-casing from denoise_stage (timestep dtype/sigmas switch, cfg_scale read, timestep_scale), the PipelineChecker raises, and cosmetic reformats; denoise_stage and PipelineChecker now match upstream exactly. - Point LTX25TextEncoder.forward at the inner hidden-states pass so both text encoder generations share one call site in the prompt embedder unit. - Run LTX-2.5 one-stage examples at half resolution, matching LTX-2/2.3. - Ignore *.mp4 and *.wav test outputs.
- Drop the wrapper layout: the class now inherits the transformers model, the config literal lives in __init__ (no deepcopy, no module-level constant), and the forward override is gone since the inherited forward returns hidden states. - Delete reset_non_persistent_buffers: rope values match transformers' own init bit-for-bit and the embed_scale difference cancels in RMSNorm, verified by end-to-end bit-identical encodings with and without it. - Update the state dict converter key prefixes for the inherited layout (model.language_model.*, model.embed_vision.*, model.embed_audio.*, lm_head).
Upstream extends the keyframes mask with zeros for given-content conditioning (keyframe_cond.py:86, reference_video_cond.py:102-104: "Reference tokens are never keyframes"); we concatenated ones, which added the learned keyframe embedding to tokens that must not receive it. Verified against the target library with per-channel probes (appended tokens stay unmarked, first latent frame stays marked unconditionally).
The audio_fully_frozen shortcut returned the resampled input waveform and skipped the audio decoder when the retake mask was all-zero; upstream always decodes the (frozen) audio latent (retake.py:326). Restore the unconditional decode path and drop the now-unused resample_waveform import.
The *.mp4 / *.wav ignore rules were local-only convenience for test outputs and must not ship in the integration branch.
…r generation
- Treat LTX-2.5 as a version update: fold its content into docs/{zh,en}/Model_Details/LTX-2.md,
delete the standalone LTX-2.5 pages, point README news/links at the merged page and drop the
duplicated doc link entry.
- Keep video positions in fp32 for LTX-2.5 and cast them to the model dtype only for LTX-2/2.3;
bf16 time coordinates made the 2.5 outputs flicker while fp32 breaks the older generations.
- Drop the dead ff_bias plumbing from LTX2TextEncoder and the redundant prose comments in the
LTX-2.5 example scripts.
…fault The LTX2AudioVideoPipeline defaults cfg_scale to 3.0. Remove the explicit cfg_scale=3.0 from the LTX-2.5 inference and validate scripts (no behavior change) and remove cfg_scale=4.0 from the T2AV validate scripts so they also fall back to 3.0. Keep cfg_scale=1.0 in the distilled / INT8-ConvRot / IC-LoRA-Pixel-Spatial-Upscaler scripts, where CFG is intentionally disabled.
The LTX-2.5 T2AV split-training scripts downloaded and read from ltx2/LTX-2.3-T2AV-splited. Give LTX-2.5 a dedicated example dataset (ltx2/LTX-2.5-T2AV-splited), matching the I2AV scripts, so each model version references its own dataset.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LTX-2.5 Integration
Summary
Integrate Lightricks/LTX-2.5 into DiffSynth-Studio by extending the existing
LTX2AudioVideoPipeline(shared with LTX-2 / LTX-2.3), supporting:auto_duration=True, a singlepipe(...)call via the Duration Head) and CFG disabledinput_images/input_images_indexesaudio_only=Truein_context_videosint8_w8a8, ConvRot) quantized DiT and Gemma4 text encoder checkpointsuse_diffusion_vae=False)vram_limit) for every exampleexamples/ltx2/model_training/train.pyThis branch is built on top of PR #1602 (its commit is kept as an ancestor so the original PR stays mergeable); the integration was then completed and corrected on top of it (unified pipeline instead of a separate
LTX25AudioVideoPipeline, flat DiffVAE module, VRAM management, training, docs).Model Components
diffsynth/models/ltx2_dit.pydiffsynth/models/ltx25_text_encoder.pydecode()entry)diffsynth/models/ltx25_diffusion_video_vae.pydiffsynth/models/ltx2_video_vae.pydiffsynth/models/ltx2_audio_vae.pydiffsynth/models/ltx25_duration_head.pydiffsynth/models/ltx2_upsampler.pyState dict converters:
diffsynth/utils/state_dict_converters/ltx25_diffusion_video_vae.py,ltx25_text_encoder.py,ltx25_duration_head.py,ltx2_video_vae.py,ltx2_audio_vae.py,ltx2_dit.py. Registrations (BF16 and INT8-ConvRot variants) live indiffsynth/configs/model_configs.py; VRAM management maps indiffsynth/configs/vram_management_module_maps.py.Examples
Inference (each with a matching low-VRAM variant under
examples/ltx2/model_inference_low_vram/):examples/ltx2/model_inference/)LTX-2.5-T2AV-OneStage.pyLTX-2.5-T2AV-TwoStage.pyauto_duration=True, CFG disabled)LTX-2.5-T2AV-DistilledPipeline.pyLTX-2.5-T2AV-INT8-ConvRot.pyLTX-2.5-I2AV-OneStage.pyLTX-2.5-I2AV-TwoStage.pyLTX-2.5-A2V-TwoStage.pyLTX-2.5-T2AV-TwoStage-Retake.pyLTX-2.5-IC-LoRA-Pixel-Spatial-Upscaler.pyaudio_only=True)LTX-2.5-T2A.pyTraining (split / two-stage;
data_processthentrain), all underexamples/ltx2/model_training/:full/LTX-2.5-T2AV-splited.shlora/LTX-2.5-T2AV-splited.shvalidate_full/LTX-2.5-T2AV.py,validate_lora/LTX-2.5-T2AV.pyfull/LTX-2.5-I2AV-splited.shlora/LTX-2.5-I2AV-splited.shvalidate_full/LTX-2.5-I2AV.py,validate_lora/LTX-2.5-I2AV.pyDocumentation: LTX-2.5 is documented within the existing LTX-2 page —
docs/en/Model_Details/LTX-2.mdanddocs/zh/Model_Details/LTX-2.md— plus README news entries and model-table rows (README.md,README_zh.md).Dependencies Added
pyproject.toml. LTX-2.5 text encoding requires atransformersrelease that ships thegemma4_unifiedmodeling code (>=5.8,<5.15; verified with 5.14.1). Audio/video I/O uses the existing[audio]extra (av,torchaudio,librosa). All other components run on plain PyTorch (noltx_core, NATTEN, Triton, or ltx-kernels).