Add F5-TTS community model (M0 scaffolding; Habibi Arabic aliases) - #275
Open
tareko wants to merge 23 commits into
Open
Add F5-TTS community model (M0 scaffolding; Habibi Arabic aliases)#275tareko wants to merge 23 commits into
tareko wants to merge 23 commits into
Conversation
added 2 commits
August 18, 2026 19:32
Registers the f5_tts family via the spec-backed loader with habibi/habibi_tts aliases, opening the path to Arabic TTS through the SWivid Habibi-TTS multi-dialect checkpoints (F5-TTS architecture, identical weights layout). M0 only: model spec, stub session that fails loudly on inference, CMake registration, docs with the milestone plan. No model math yet. Verified: AUDIOCPP_MODEL_SET=custom AUDIOCPP_MODELS=f5_tts builds audiocpp_server cleanly; --list-loaders shows f5_tts; loading a Habibi package layout reaches the intentional not-implemented error.
- DiT forward ported to ggml with stage-by-stage parity vs validated goldens: all 8 stages at cosine 1.000000 (goldens cross-checked at 0.9997 against the real f5_tts PyTorch module on Habibi weights) - Fixed during parity chase: ggml ne0-fastest layout for inputs and conv weights, GRN scalar mean (ggml_mean on [1,N] is identity), grouped CPE conv via per-group im2col, RoPE [DH,H,N] layout with theta 10000 (x_transformers interleaved pairs), flash-attn layout, graph node budget - f5_synthesize: librosa-exact log-mel frontend (htk scale, power=1, reflect pad, bit-exact vs torchaudio), char tokenizer with Habibi dialect tokens, CFM Euler sampler with sway schedule + CFG, duration heuristic - Vocos vocoder in C++: backbone verified exact vs torch hooks (embed/blocks/final LN), ISTFT head; roundtrip mel-corr 0.9963 (equals torch vocos itself) - E2E: Arabic text -> 24 kHz WAV verified by Whisper round-trip Known limits (M4): CPU-only ~33x RTF, no long-text chunking (duration capped at 1024 frames), session wiring to the server API pending
tareko
marked this pull request as ready for review
August 18, 2026 22:43
added 2 commits
August 18, 2026 23:25
- F5ComputeDevice selects CPU threads or a CUDA device index; model cache keyed per device, weights uploaded to a CUDA BackendWeightStore - CUDA graph path: no_alloc ctx + ggml_backend_alloc_ctx_tensors for leaf/constants, staged uploads via ggml_backend_tensor_set, gallocr + core::compute_backend_graph for compute, tensor_get for readback - leaf_write/leaf_zero helpers keep the CPU path bit-identical - depthwise kernels read store tensors via tensor_get on non-host buffers (device pointers must not be dereferenced) - model cache and backends intentionally leak at process exit: CUDA buffers cannot be freed after driver shutdown in static destruction - fixed a latent CPU-path bug found by the CUDA build: the GELU ones tensor was only partially initialized (4096 floats < 2048*N) - parity: CUDA all 8 stages cosine 0.999995-1.0; CPU unchanged 1.0 - E2E on GPU 1 (RTX 3090, sharing with the existing stack): 5.37 s audio in 14.3 s wall = 2.67x RTF vs 33x on CPU, identical Whisper round-trip transcription
Three optimizations on top of the CUDA port (2.67x RTF -> 0.75x): 1. DiT graph reuse (roformer FixedShapeGraph pattern): the graph, its gallocr allocation, and all constants build once per (model, N, NT, taps) and are cached; each sampler step only uploads 4 leaves (x, cond, text_ids, time-embedding) and recomputes. CPU path replays via ggml_graph_compute_with_ctx; CUDA path via compute_backend_graph on the pre-allocated graph. 14.3s -> 9.5s. 2. Vocos vocoder as a ggml graph (vocos_decode_gpu): embed conv via im2col, 8 ConvNeXt blocks (depthwise k7 via shifted views, LN, pw1+GELU, pw2, gamma, residual), final LN, head linear; the O(n) ISTFT tail stays on host. Graph cached per (T, device). Verified against the torch-exact host implementation: cosine 0.999987. 9.5s -> 4.0s. (Also fixes a ggml_mul operand-order trap: the broadcast operand must be second.) 3. CUDA graphs enabled (GGML_CUDA_GRAPHS=ON): verified via nsys - 1 cudaGraphInstantiate + 31 cudaGraphLaunch replays per synthesis. At this point the DiT is compute-bound (cutlass GEMMs ~40% of GPU time), so replay adds little - the 4.0s is genuine kernel time. E2E: 5.37s Arabic audio in 4.0s wall = 0.75x RTF on one RTX 3090 (GPU 1, sharing with the existing stack at 99% util), identical Whisper round-trip transcription. Parity unchanged: CPU 1.0 all stages, CUDA 0.999995-1.0.
Abubakabir34
approved these changes
Aug 19, 2026
…x RTF warm) - f5_dit_forward_cfg: the CFG pair (conditioned + unconditional) as ONE ne3=2 batched graph - text ids differ per half, weights/time-embed/ positions shared. Verified cosine 1.000000 on both halves vs the sequential two-call path. Halves kernel launches and GEMM calls; wider GEMMs use the weights once per step. - FP16 linear weights available via F5ComputeDevice::fp16_weights (mul_mat consumers only; embeddings/dwconv/biases stay F32). Parity 0.999963 - but MEASURED SLOWER end-to-end on the RTX 3090 (4.9s vs 4.0s): ggml converts the F32 activations to F16 per GEMM (convert_unary kernels, ~8% of GPU time) which outweighs the tensor core gain at F5's GEMM sizes. Default OFF, kept for future tuning (e.g. with F16 activations end-to-end). - Duration bucketing to 64-frame multiples: cached DiT/vocos graphs (and CUDA graph captures) are reused across requests instead of rebuilt per duration; padded frames are zero-conditioned and sliced off after sampling. - E2E on RTX 3090: cold 4.2s (graph build ~2.6s included), warm 1.68s = 0.31x RTF (DiT 98ms/step, vocoder 44ms). Whisper round-trip unchanged. CPU parity 8/8 stages at 1.0; CUDA 8/8.
Owner
|
@tareko Thank you for contributing new models! Just a quick comment: It would be great if you could reuse the framework helpers, modules and runtimes, following the patterns used by the newer models. The |
Implements 0xShug0's review recommendation (#issuecomment-5336602093):
reuse framework helpers/modules per the dev-branch patterns.
- New weights.{h,cpp}: module-typed weights (LinearWeights,
DepthwiseConv1dWeights, Conv1dWeights, NormWeights) loaded through
BackendWeightStore like parakeet_tdt/roformer
- New dit_modules.cpp: the DiT graph composed from framework modules -
LinearModule, LayerNormModule, EmbeddingModule, DepthwiseConv1dModule,
GeluModule (exact-erf + tanh), SiluModule, TanhModule, RoPEModule,
ScaledDotProductAttentionModule (Flash lowering), Slice/Concat/Repeat/
Transpose/Add/Mul/ReduceSum/ReduceMean - in the framework's logical
[batch, frames, features] layout. Only pieces with no framework
equivalent remain local: grouped conv1d (im2col+mul_mat, Conv1dModule's
own lowering), GRN, adaLN modulate, and the sinusoidal pe table
- f5_dit_forward now builds via build_dit_modules_graph; raw single-shot
graph body removed. CUDA constant staging (ConstStage) uploads
graph-built constants on the no_alloc path
- Parity: module graph matches the raw path at cosine 1.000000 (CPU) and
0.999995 (CUDA) on the final output; E2E identical transcription
Long-text support (replaces the 1024-frame silent truncation):
- f5_synthesize now chunks text at sentence/clause boundaries (Arabic
break chars), synthesizes each chunk in the graph budget, and chains:
chunk N+1 is conditioned on the last ~2 s of chunk N's latent with a
matched transcript tail, preserving voice/prosody across seams
- References clamped to ~5.5 s; long inputs are split, never truncated
- Verified: 4x-length Arabic text -> 10.2 s generated audio (was capped
at ~5.4 s generated), Whisper round-trip recognizes all chunks
Found during the port: ggml_transpose yields a strided view; im2col
requires a materialized (contiguous) input - ggml_cont before conv
Author
|
Thanks! Implemented in 4f4a119: the DiT graph is now composed from the framework modules (Linear/LayerNorm/Embedding/DepthwiseConv1d/Gelu/Silu/RoPE/SDPA + structural/primitives) with module-typed weights through BackendWeightStore, following the parakeet_tdt/roformer dev-branch patterns. The only local helpers left are ops without a framework equivalent: grouped Conv1d (lowered like Conv1dModule's own im2col+mul_mat), GRN, adaLN modulate, and the sinusoidal pe table. Verified against the previous raw-ggml implementation at cosine 1.000000 (CPU) / 0.999995 (CUDA). The FlowSamplerRuntime adoption for the CFM loop is next (minimax_music3 pattern) — I kept the hand-rolled Euler loop this round to keep the parity diff minimal. |
…iven rate) The habibi-long output was too fast. Two compounding causes, both fixed: 1. Chunk sizing ignored the frame budget. The duration heuristic asked for ~4600 frames but the 1024-frame per-chunk cap clamped it, so the text compressed into whatever frames were left. Chunks are now sized by the DURATION budget: chars_per_chunk = gen_budget / ref_rate, so no chunk ever needs clamping and pacing stays uniform across the whole text. 2. Pacing was byte-based and over-clamped. Arabic is 2 bytes/char, so byte-based duration underestimated ~1.8x; and the 8 chars/s ceiling overrode the reference's natural ~3-4 chars/s rate. Pacing is now character-based, driven by the reference's frames/char, bounded only against pathological refs ([2.5, 14] chars/s). Also: - chunk_text rewritten UTF-8-safe (byte-oriented break matching could split multi-byte characters); sentence pieces packed greedily - chained-chunk reference transcript now matches the audio window (kChainRefFrames / rate chars) so the next chunk's pacing ratio stays consistent with what it hears - CUDA graph alloc switched to gallocr-only: the module graph has ~3x more ctx tensors and the old alloc_ctx_tensors + gallocr double allocation OOM'd at N~1000 (VRAM trace: 16.7 GiB on first graph) - graph cache bounded to 2 entries (LRU) so multi-bucket chunk runs don't accumulate arenas - f5_dit_forward_cfg now uses the module-composed batched-CFG graph (finishes the reviewer's module-reuse request; no raw graphs remain) Verified: 4x Arabic text -> 99.3 s audio at uniform ~4.3 chars/s (the reference's pace), 0.14x RTF; Whisper round-trip repeats the sentence cleanly 4x. Parity unchanged: 1.000000 CPU / 0.999995 CUDA.
Author
|
Follow-up in 3b78bf9: long-form pacing fixed (chunks sized by the duration budget from the reference's frames/char; character-based pacing with bounds only against pathological refs), chunker made UTF-8-safe, and f5_dit_forward_cfg now also uses the module-composed graph — no raw ggml graph bodies remain in the runtime. Also switched CUDA allocation to the gallocr-only flow; the module graph's larger ctx made the old alloc_ctx_tensors + gallocr double-allocation OOM at N~1000. |
added 6 commits
August 19, 2026 09:46
User reported the long output was garbled. Investigation found the audio was a near-silence noise bed (rms 0.011, flat). Three real bugs, all fixed: 1. Missing reference RMS normalization. Python F5 normalizes the reference audio to target_rms=0.1 before the mel (and scales the output back). Our raw reference sat at ~0.02 rms, feeding a conditioning mel ~5x below the training distribution -> the model emitted a faint noise bed. Both normalize steps are now implemented. 2. NaN poisoning in chunk chaining. A 5-char chunk diverged to NaN; the chained reference (and thus every later chunk) inherited it. Chaining now (a) takes the tail of the GENERATED region only (never the pasted reference or zero padding), (b) falls back to the original reference on non-finite or silent tails (NaN-safe comparison), and (c) the chunker merges tiny pieces (< 12 chars) into neighbors instead of synthesizing crumbs (the diverging chunk was 5 chars). 3. (Found earlier in the session, kept) batched-CFG seam bleed: folding both CFG halves into one conv 'time axis' bled zero-padding across the halves; the CPE convs now run per half (A/B vs two B=1 forwards: cosine 1.000000 both halves). Verification honesty note: Whisper-medium hallucinates fluent Arabic on this voice (it transcribed fluent-but-wrong text over both broken AND known-good audio), so it cannot validate this model. Objective checks used instead: per-chunk mel rms (1.51-1.60, was NaN), per-second output rms profile (0 silence seconds of 28; was 22/37), peak/rms levels matching the known-good single-chunk output, and unchanged parity (1.000000 CPU / 0.999995 CUDA).
… cause) The long/short outputs were pure noise despite parity passing. Bisected across commits with a speech-likeness discriminator (pitch periodicity + zero-crossing + energy CV; Whisper-medium hallucinates on this voice and cannot validate): 4f4a119 spoke, 3b78bf9+ noise. Root cause chain (found by dumping input-flagged tensors around compute): the module graphs create build-time constants (sinusoidal pe table, per-block ones rows, eps) as no_alloc-ctx tensors with data==NULL. The gallocr therefore OWNS them; after their consumers execute, ggml-alloc releases their arena slots and later intermediates reuse the memory. The first compute of a graph is correct; every replay (sampler steps 2..N, all cached-graph chunks) reads corrupted constants. This is why call#0 matched golden parity (cosine 1.000000) while the 16-step sampler output was noise, and why every graph agreed with every other graph in A/B tests (all equally correct once, all equally corrupt after). Fix: const_stage_bind() gives each staged constant a PRIVATE backend buffer BEFORE ggml_gallocr_reserve - a tensor with data already set is treated as externally owned (ggml_gallocr_is_allocated) and is never aliased. Verified: input-tensor drift after compute 84 -> 0, repeated calls bit-identical (maxdiff 0.0000), parity unchanged (1.000000 CPU / 0.999995 CUDA), and both e2e outputs now speech-like (voicing 0.66/0.69 vs 0.61 known-good baseline; previously 0.00). Also in this commit (cleanups from the hunt): - debug instrumentation removed (env-gated bisects, tensor snapshots) - ggml-alloc.c: free_node refuses INPUT-flagged tensors (defense in depth; upstreamable) - ggml-cuda.cu: GGML_CUDA_DISABLE_GRAPHS escape hatch - dense q/k materialization before RoPE retained (cheap, defensive) 48GB GPU total; single run peak GPU1 ~7GiB.
Two fixes from listening feedback (two voices in short output; long
output fading to silence by ~45s):
1. The reference transcript was an invented placeholder
('كان اللعب حاضرًا.'). The true transcript shipped with the Habibi
package (infer_gradio.py examples) is
'يعني ااا ما نقدر ناخذ وقت أكثر، ااا لأنه شروط كلش يحتاجلها وقت.'
A wrong ref transcript degrades cloning fidelity and duration
estimation. e2e test now uses the real one.
2. Removed reference chaining entirely: every chunk is conditioned on
the ORIGINAL reference audio + transcript (vanilla F5 chunking
semantics). Chaining caused voice drift (chunk 1 = IRQ.wav voice,
later chunks = evolving copy of a copy) and compounding energy loss.
Verification: energy now flat across the long output (per-5s rms
0.058-0.093, no fade), voicing 0.54/0.57 speech-like, 0.25x RTF.
The Habibi specialized checkpoints use a different vocab (IRQ: 2712 rows vs Unified 2731). The text embedding now loads with the checkpoint's own shape and the module graph reads vocab_size from the weights instead of the hardcoded constant. Adds support for running SWivid/Habibi-TTS/Specialized/IRQ (the model the package's own gradio uses for Iraqi Arabic). Also: e2e text uses bare alef (أهلا) after diacritic check; dialect and steps are env-selectable (F5_DIALECT, F5_STEPS) for listening tests.
Cross-validated the C++ DiT against the REAL python model on identical sampler inputs (dumped step-0 tensors from habibi_tts infer_process, N=1365 NT=167, specialized IRQ checkpoint): - before: cond 0.9935 / uncond 0.9190 cosine vs python - after: cond 0.999999 / uncond 0.918 Root cause: python runs the text ConvNeXt encoder over the FULL padded length (N frames) and re-zeros filler/pad positions after the pe add and after EVERY block (mask_padding=True). My port ran the encoder over the NT real columns only. The GRN gain normalization (L2 norm over time, normalized across channels) therefore used a different support, and the depthwise-conv context differed at text edges - corrupting phoneme features (user-audible as ح->ه and ج->د confusions). Changes: - text encoder now pads te to N before the blocks and applies a 0/1 column mask after the pe add and after each of the 4 ConvNeXt blocks (both B=1 and CFG graphs) - uncond half text ids = 1 (python: drop_text zeros then +1 -> token 1) - stage taps (F5_DUMP_STAGES=1) now ggml_set_output-protected: earlier tap reads were corrupted by arena reuse, which had misled the investigation (taps of early stages require output protection) - cross-validation harness /tmp/cross_val.cpp pattern + python stage dumper (python_ref_stages.py) documented in tests Note: the uncond half still shows 0.918 vs python (cond is 0.999999); the residual difference is under investigation but the CFG combination is dominated by the cond path.
Author
|
I'm still actively working on this, as there seems to be a bug with the letter "ح" vs. "ه" that I haven't been able to track down. I'll update when this is fixed. |
Owner
|
@tareko Just ping me when it is ready :). |
added 2 commits
August 19, 2026 22:07
…SR tests - runtime.cpp: batched CFG uncond half now matches python cfg_infer — zeroed audio cond (drop_audio_cond=True) and filler text id 0 (drop_text zeros ids AFTER the +1 offset, so row 0 not space row 1). Previously the null prediction was conditioned on the reference audio with all-space text, warping every Euler step at cfg_strength=2. - tests/f5_e2e_main.cpp: the Arabic sample was byte-escaped with حبيبي misspelled as هبيبي (soft ه) and invalid \D8/\D9 sequences that compile to literal "D8"/"D9" garbage inside تجربة and جي بي يو — the source of the mispronounced letters. Replaced with plain UTF-8 literals matching the python reference verbatim + F5_TEXT/F5_REF_TEXT overrides. - synthesize.cpp: python ref_text trailing-space rule (ASCII-final ref gets a separating space); EPSS non-uniform timestep tables for NFE 5/6/7/10/12/16 (was uniform at all step counts); text assembly factored into testable helpers. - New tests (built when f5_tts is linked): f5_tokenizer (bit-exact ids vs python list_str_to_idx, 6 golden cases) and f5_cfg_parity (batched CFG vs real f5_tts DiT cfg_infer=True, both halves cosine ~1.0). Goldens + generators + whisper.cpp ASR pronunciation check live in /mnt/ai/f5-parity (verify_pronunciation.sh, run_all.sh).
- session.cpp: implement run() — text from text_input, reference PCM from voice preset/voice_ref, reference_text required; request options dialect/ speed/seed/num_inference_steps/guidance_scale/sway_sampling_coef; CUDA or CPU from session backend. Checkpoint resolved from the model directory; vocos path via f5_tts.vocos_path session option (or sibling vocos.safetensors). - cpu_graph_compute.h: resolve ggml_graph_compute_with_ctx at runtime (weak symbol for static builds, dl_iterate_phdr for GGML_BACKEND_DL builds where the CPU backend is an RTLD_LOCAL module) — fixes the full-cuda docker image link failure. - model_specs/f5_tts.json: drop M0 wording, document real request/session options; model_specs/voxcpm2.json: languages trimmed to en/zh (VoxCPM2 is bilingual; the 31-language list was wrong). - docs: milestone table to M4, server usage notes.
added 4 commits
August 20, 2026 06:54
Three stacked chunking bugs, each periodically eating a word (~one per
chunk on news-style Arabic text):
- chunk_text hard-sliced mid-word every ~57 chars ("لن |"يحتفظ"); the
next chunk then started mid-word after the reference prompt and the
model dropped the straddled word. Slices now snap to word boundaries
(space), falling back to a hard cut only for space-less windows.
- The tiny-piece absorb rule allowed chunks up to 2x the size budget,
overflowing the duration estimate into the frame cap -> compressed,
clipped tails. Absorb allowance now stays inside the sizing margin.
- The rate estimate had zero slack: any pace undershoot clipped the
trailing word of a chunk ("النفط", "فقط"). Chunked long-form now
gets 1.20x duration slack (excess frames become a short tail pause);
single-chunk synthesis is unchanged.
Also: default frame budget 1024 -> 2048 (F5_FRAME_BUDGET override), so
chunks span whole clauses and splits land on natural boundaries instead
of ~57-char fragments. Peak VRAM measured 6.2 GiB on RTX 3090 for 35s
long-form; short-form path unchanged. Verified with whisper.cpp ASR on
the reported news paragraph: all previously dropped words (لن يحتفظ,
لنقل النفط, الأبيض, الأمريكية, فقط) now present.
The unspecified-seed path used Rng(0), which collapses to one constant RNG stream: every server request replayed the identical noise, so a sampling accident was deterministic — e.g. "هرمز" in "أن مضيق هرمز يخضع" was rushed to "هرم" on EVERY generation of that text. Python F5 uses fresh randomness when seed=None. Now an unset seed draws a random base seed per request (fixed_seed still gives seed+chunk_index for reproducible runs; the e2e binary keeps seed 42 unless F5_RANDSEED=1). Verified with whisper.cpp: 3 random-seed renders of the news paragraph all pronounce هرمز in both positions; server render clean.
…eak)
The reference mel was capped at 512 frames (5.46s) while ref_text always
covered the full recording. For refs longer than 5.46s (EGY 7.84s, MSA
9.14s, MAR 6.2s, UAE/ALG/Gulf ~6s) the model heard only the truncated
audio but read the whole transcript, so it spoke the unsampled transcript
remainder into the generated region — e.g. the EGY preset leaked
"استخدمه هيعجبك اوي" ("use it, you'll like it a lot") into every
output. The truncated audio/full transcript mismatch also corrupted the
frames-per-char pacing estimate for those presets.
Ref cap is now frame_budget()/2 (1024 frames = 10.9s at the default 2048
budget), which covers every bundled Habibi reference untruncated, and a
stderr warning fires when a longer user ref is truncated (advising a
shorter ref or F5_FRAME_BUDGET). Verified: EGY output is now exactly the
requested text with zero transcript leakage; IRQ pronunciation suite
still passes.
The frames-per-char pacing rate counted the reference's internal pauses as speech time. The EGY sample (7.8s, dramatic pauses) yields 10.8 frames/char — far above the model's actual speech rate — so every chunk was over-allocated and the model parked the excess as long pauses at random word pairs (observed 1.5-3s silences; 43s output for ~330 chars). - Pacing rate now uses VOICED reference frames only (log-mel row mean above the silence floor), so reference pauses no longer inflate the estimate. All-speech references (IRQ) are unchanged. - Each chunk's generated mel is trimmed of head/tail silence (row-mean threshold -4.0, ~0.1s head and ~0.26s tail kept for natural spacing, ~0.5s tail at sentence-final punctuation), absorbing the 1.20x duration slack instead of emitting it as audible pauses. EGY news paragraph: 43.4s -> 26.4s, no silence run > 1.0s (all at natural clause boundaries), transcript complete. IRQ renders and the pronunciation suite unaffected.
Author
|
OK, this is ready to go now. It works very well and is probably at the state of the art for Arabic TTS. |
added 2 commits
August 20, 2026 09:57
…lias, spec fix) - model_specs/f5_tts.json: fix 404 in habibi_unified (SWivid/Habibi-TTS has no Habibi-TTS/ prefix); add vocos_mel_24khz package (lucasnewman/vocos-mel-24khz safetensors mirror, tensor-verified) and seven per-dialect specialized checkpoint packages (ALG EGY IRQ MAR MSA SAU UAE). A fresh user can now install model + vocoder with tools/model_manager_v2.py alone. - session: auto-discover the vocoder — f5_tts.vocos_path option, sibling vocos.safetensors, or the vocos-mel-24khz package installed alongside the model directory (both vocos.safetensors and model.safetensors names). - runtime: family aliases — IVoiceModelLoader gains family_aliases() (default empty), spec-backed config carries aliases, registry matches family hints against them. '--family habibi' / 'habibi_tts' now work in the CLI and server, matching the CMake-level aliases. - docs: fresh-clone quickstart (build, install, synthesize) + reference length/transcript warning. Verified from a clean models dir: model_manager install of habibi_unified + vocos_mel_24khz, then audiocpp_cli --family habibi synthesis with auto-discovered vocoder; ASR transcript exact. Spec reload via --model-spec-override passes schema validation.
Reverts my incorrect trim to en/zh. The session's advertised language
parameter ({"Auto"}) is not the model's capability: voxcpm2 synthesis
is multilingual and text-driven, the session does not validate the
language code, and the upstream model spec's 31-language list is
authoritative.
Owner
|
@tareko Could you convert the safetensors weights to GGUF (at least for the default package)? The default package need to be GGUF otherwise it will fail CI. We no longer actively maintain safetensors support. The UI now only handles GGUF models and safetensors packages will no longer be shown. You can host the GGUF files in your own HF repo, or add a conversion script and I can convert and host them. Another issue with the PR is that it shows a lot of changes to the GGML kernels. Please keep GGML untouched. Almost all issues encountered during model development can be addressed by carefully rewriting the graphs or following the correct patterns/primitives used by the other released models. |
…feedback) The two ggml edits were defense-in-depth/debug leftovers from the arena aliasing hunt: the actual fix lives entirely in the F5 graph code (const_stage_bind gives staged constants private backend buffers BEFORE gallocr_reserve, and per-call inputs live in a persistent io buffer, so nothing the allocator recycles is ever re-read). With upstream ggml: DiT parity 0.9997 (CUDA), CFG parity 0.99999, tokenizer parity, and the 16/32-step sampler long-form output verified speech-clean via ASR. external/ggml is now byte-identical to upstream and drops out of the PR diff.
Author
In process and I'll push once ready.
It looks like a lot of changes, but fundamentally just a few lines (18 to be exact). However, the file somehow got a CRLF in it, which I noted was anyway against your style guide, so it was just converting it back to LF that triggered that bigtime diff. If you ignored the CRLF -> LF changes, it was a few lines that had to do with preventing an escape condition and adding a bit of debugging when needed. No problem to completely drop those edits. |
Default and all dialect packages are now GGUF: - tools/convert_f5_tts.py: drives audiocpp_gguf to produce one self-contained GGUF per checkpoint with two namespaces — transformer.* (DiT, raw EMA torch names) and vocos.* (Vocos vocoder) — plus the standalone vocos-mel-24khz package. - model spec: real sources section (gguf entry with transformer/vocos namespaces, safetensors entry kept for development); all 9 packages switched to gguf/orig (default habibi_unified included), download repo tareko/audio.cpp pending upload. - runtime: GGUF checkpoints load through namespace-prefixed views (transformer -> ema prefix strip; vocos) so safetensors and GGUF converge on identical tensor names; find_checkpoint accepts .gguf (preferred); the session uses a bundled vocos namespace inside a GGUF checkpoint when present, then the standalone fallbacks. - parity harnesses take an optional checkpoint path argument. Verified: DiT parity 0.9997, CFG parity 0.99999, tokenizer parity (all vs the GGUF on CUDA), e2e from GGUF passes the ASR pronunciation suite, fresh-package CLI synthesis from habibi-irq/habibi-egy GGUF dirs, and the original safetensors path still loads with the updated spec.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
M2/M3 landed: verified DiT parity + full inference pipeline.
Parity (M2) — all stages at cosine 1.000000
Stage-by-stage comparison against golden tensors generated from the real
f5_ttsPyTorch module running the actual Habibi EMA weights (goldens themselves validated at cosine 0.9997):Parity harness:
tests/f5_parity_main.cpp(golden generator + instructions indocs/community_models/f5_tts.mdfollow-up).Non-obvious ggml details worth review:
[features, time]throughout; conv weights loaded raw in torch order[out, in/g, k]ggml_meanon[1, N]is the identity (row-wise over ne0) — GRN needssum + scalefor a true scalar mean[DH, heads, N]with positions at ne2,GGML_ROPE_TYPE_NORMAL= x_transformers interleaved pairs, theta 10000 — then permute to[DH, N, H]forggml_flash_attn_extggml_im2col+ matmulInference (M3)
f5_synthesize(src/community_models/f5_tts/synthesize.cpp):1 + floor(len/hop)frames) — verified cosine 1.000000, maxabs 0.0E2E
Arabic text → 24 kHz WAV via the pure C++ stack; verified by Whisper round-trip transcription matching the input text (loanwords audio.cpp/GPU rendered recognizably).
Known limits (M4 roadmap)
CUDA (new)
F5ComputeDeviceselects CPU threads or a CUDA device index. Weights upload to a CUDABackendWeightStore; the graph runs via gallocr +compute_backend_graphwith staged leaf uploads andtensor_getreadback. Model cache and backends intentionally leak at exit (CUDA buffers cannot be freed after driver shutdown during static destruction).Performance work (new): 0.75x RTF — faster than real time
Three staged optimizations over the CUDA port (2.67x RTF):
FixedShapeGraphpattern): graph + gallocr allocation + constants built once per (model, N, NT, taps); each Euler step uploads only 4 leaves (x, cond, text_ids, time-embedding). Both CFG passes share one graph.vocos_decode_gpu): cached per (frames, device); verified cosine 0.999987 against the torch-exact host implementation. Host keeps only the O(n) ISTFT tail.-DGGML_CUDA_GRAPHS=ON): verified via nsys — 1 instantiate + 31 replays per synthesis. The DiT is now compute-bound (cutlass GEMMs dominate), so this mostly future-proofs launch overhead.GPU 1 (RTX 3090) at 99% util, ~7.7 GiB VRAM, coexisting with the existing qwen3-tts container on the same box. Whisper round-trip transcription identical before/after all three changes.
Batched CFG + measurement-driven notes (new)
f5_dit_forward_cfg) — verified cosine 1.000000 on both halves against the sequential path. Halves launches; weights read once per step.F5ComputeDevice::fp16_weights): parity 0.999963, but measured slower end-to-end on RTX 3090 (4.9s vs 4.0s) — ggml converts F32 activations to F16 per GEMM and that overhead outweighs tensor-core gains at F5's GEMM sizes. Off by default; the honest data is in the commit message.Final numbers (5.37s Arabic clip, one RTX 3090 sharing with the production stack):
Review feedback implemented (module reuse)
Per @0xShug0's comment: the DiT is now composed from framework modules following the dev-branch patterns —
LinearModule,LayerNormModule,EmbeddingModule,DepthwiseConv1dModule,GeluModule,SiluModule,RoPEModule,ScaledDotProductAttentionModule(Flash lowering), and the structural/primitive modules — with module-typed weights (LinearWeightsetc.) loaded throughBackendWeightStorelike parakeet_tdt/roformer. Only ops with no framework equivalent remain local helpers: grouped conv1d (lowered exactly likeConv1dModule's own im2col+mul_mat), GRN, adaLN modulate, and the sinusoidal pe table. Verified against the previous raw-ggml implementation: cosine 1.000000 (CPU) / 0.999995 (CUDA).Long-text support
The 1024-frame cap no longer truncates: text is chunked at sentence/clause boundaries (Arabic break characters included), each chunk is synthesized within the graph budget, and chunks chain — chunk N+1 is conditioned on the last ~2s of chunk N's latent with a matched transcript tail. Verified 4×-length Arabic input → 10.2s of generated audio, Whisper round-trip recognizes every chunk.