Add Echo-TTS (community model)

CMakeLists.txt

@@ -625,6 +625,24 @@ audiocpp_add_model(neutts
         engine::models::neutts::make_neutts_loader
 )
 
+audiocpp_add_model(echo_tts
+    SOURCES
+        src/community_models/echo_tts/session.cpp
+        src/community_models/echo_tts/tokenizer.cpp
+        src/community_models/echo_tts/latent_post.cpp
+        src/community_models/echo_tts/sampler.cpp
+        src/community_models/echo_tts/dit.cpp
+    INCLUDES
+        engine/community_models/echo_tts/session.h
+        engine/community_models/echo_tts/config.h
+        engine/community_models/echo_tts/tokenizer.h
+        engine/community_models/echo_tts/latent_post.h
+        engine/community_models/echo_tts/sampler.h
+        engine/community_models/echo_tts/dit.h
+    LOADERS
+        engine::models::echo_tts::make_echo_tts_loader
+)
+
 if (MSVC)
     set_source_files_properties(
         src/community_models/inflect_v2/frontend.cpp
@@ -1753,6 +1771,17 @@ if (ENGINE_BUILD_TESTS)
 
     add_engine_unittest(dots_tts_vocoder_parity tests/dots_tts/dots_tts_vocoder_parity.cpp)
 
+    # Needs the GGUF and a PyTorch reference dump, so it is driven by hand
+    # rather than registered with add_test -- same as dots_tts_vocoder_parity.
+    add_engine_unittest(echo_tts_dit_parity tests/echo_tts/echo_tts_dit_parity.cpp)
+
+    add_engine_unittest(echo_tts_host_units tests/echo_tts/echo_tts_host_units.cpp)
+
+    add_test(
+        NAME echo_tts_host_units
+        COMMAND echo_tts_host_units
+    )
+
     add_engine_unittest(midi_file_test tests/unittests/test_midi_file.cpp)
 
     add_test(

docs/community_models/echo_tts_autoencoder_reuse.md

@@ -0,0 +1,129 @@
+# Echo-TTS: autoencoder reuse
+
+Status: verified against checkpoint sizes and upstream source. No weights were
+downloaded to reach these conclusions; every number below is reproducible from
+`autoencoder.py` plus the file sizes Hugging Face reports.
+
+## Summary
+
+Echo-TTS depends on the Fish S1-DAC autoencoder, and **audio.cpp already
+implements that exact autoencoder** for the `fish_audio` family in
+`src/models/fish_audio/codec.cpp`. The Echo port does not need a new decoder,
+encoder, quantiser, or window-limited transformer. It needs a `z_q` seam on the
+existing one.
+
+This changes the cost of milestones M1 and M2 substantially relative to the
+original PR plan, which scoped "Fish decode" and "native speaker encoding
+(Fish encoder + RVQ)" as separate pieces of work.
+
+## Evidence
+
+### Configuration
+
+`jordand/fish-s1-dac-min/config.json` reports:
+
+    sample_rate 44100, encoder_dim 64, encoder_rates [2,4,8,8], latent_dim 1024,
+    decoder_dim 1536, decoder_rates [8,8,4,2], n_codebooks 9, codebook_size 1024,
+    codebook_dim 8, semantic_codebook_size 4096, causal true
+
+Every one of these matches the constants already compiled into
+`fish_audio/codec.cpp`: `kCodecDim` 1024, semantic codebook 4096, nine residual
+quantisers of 1024, codebook dim 8, a final decoder snake at 96 channels
+(= 1536 / 2^4), and causal convolutions throughout.
+
+### Parameter budget
+
+Deriving the parameter count from `autoencoder.py` and comparing against the
+1.87 GB Hugging Face reports for `pytorch_model.safetensors`:
+
+| Component | Parameters |
+| --- | ---: |
+| Encoder | 76,851,328 |
+| Decoder | 54,102,722 |
+| Quantiser (incl. pre/post transformers) | 260,475,040 |
+| **Total** | **391,429,090** |
+
+At F32 that is 1.566 GB. The `Transformer` base class registers two buffers per
+instance — a `freqs_cis` table and a `block_size^2` boolean `causal_mask` — which
+for the three surviving transformer instances (encoder block 3 at block_size
+16384, quantiser pre/post at 4096) comes to 305 MB. Together: **1.871 GB**,
+against the 1.87 GB reported. This also reproduces the 303.6 MB
+"regenerable buffers" figure noted on the PR.
+
+The match only holds once the decoder is counted **without** a transformer, which
+leads to the next point.
+
+### The decoder has no transformer
+
+`build_ae` passes `decoder_transformer_layers=[4, 0, 0, 0]`, which reads as though
+decoder block 0 carries a 4-layer transformer. It does not. `DecoderBlock.__init__`
+constructs `transformer_module` into a local variable and then builds
+`self.block = nn.Sequential(Snake1d, conv_trans, ResidualUnit x3)` without it.
+The module is never assigned to `self`, so it is not a submodule, has no
+parameters, and is absent from the checkpoint. `EncoderBlock`, by contrast, does
+include `transformer_module` in its `Sequential`.
+
+Two independent checks agree:
+
+1. The 1.87 GB file size only reconciles when the decoder transformer is excluded
+   (including it predicts 2.05 GB, and adds a second 16384x16384 mask buffer that
+   would break the 303.6 MB figure).
+2. `fish_audio/codec.cpp` already loads the encoder transformer conditionally at
+   `block_index == 3` and loads no transformer anywhere in the decoder path.
+
+The C++ was evidently written against the real checkpoint, and it agrees with
+the source reading. Worth knowing before anyone "fixes" the apparent omission.
+
+## Integration seam
+
+Echo needs continuous `z_q` where `fish_audio` uses discrete codes. Both seams
+sit at existing boundaries in `codec.cpp`:
+
+**Decode.** `DAC.decode_zq` is `post_module -> upsample -> decoder`.
+`build_decode_quantizer` already performs exactly that chain; it just derives its
+input by looking up codebook entries first:
+
+    latent = build_quantizer_out(semantic) + sum(build_quantizer_out(residual_i))
+    latent = build_window_transformer(..., post_module, 128)   <- Echo enters here
+    for stage in upsample: ...
+
+Echo supplies `latent` directly from the PCA inverse and runs from the
+`post_module` line onward. The refactor is to split the code-lookup prefix from
+the `post_module`-onward suffix so both families can call the suffix.
+
+**Encode.** `DAC.encode_zq` quantises and then sums the dequantised results:
+`z_q = z_q_semantic + z_q_residual`. `build_encode_quantizer` already computes
+each `quantized` term internally on the way to emitting code indices; the sum is
+available at that point and is currently discarded. Exposing it gives native
+speaker encoding without new model code, which is most of milestone M2.
+
+## Consequences for packaging
+
+The *implementation* is shared; the *weights* are not.
+
+`fish_audio` ships Fish Audio S2 Pro. Echo is trained against the Fish S1 DAC
+(`jordand/fish-s1-dac-min`, a mirror of `fishaudio/openaudio-s1-mini`), and its
+PCA basis is fitted to that codec's latent space. The S2 technical report says S2
+retains S1's RVQ codec, and the shapes line up (10 codebooks, ~21 Hz), but
+"retains the codec" in a report can mean the architecture rather than identical
+weights -- and a retrained-but-isomorphic codec would yield wrong audio with no
+error raised anywhere. That is not a risk worth taking to save a download.
+
+`convert_echo_tts.py` therefore packages the S1 codec into Echo's GGUF under the
+`codec_weights` prefix, folding weight normalisation and dropping the 305 MB of
+regenerable buffers. Echo constructs a minimal `FishAudioAssets` around that
+tensor source: only four config fields (`sample_rate`, `frame_length`,
+`total_codebooks`, `quantizer_codebooks`) ever reach the codec graphs, and their
+defaults already describe S1-DAC.
+
+Verified against the real checkpoint manifest: the folded output supplies all 220
+tensor names `codec.cpp` loads, and the 541 stored tensors resolve to 455 after
+folding and buffer removal.
+
+## Caveat
+
+Everything above is derived from source reading plus file-size arithmetic. The
+parameter total agreeing with the reported size to three significant figures is
+strong evidence, but it is not the same as having loaded the tensors. The
+tensor-name check in `convert_echo_tts.py` and a parity run against
+`echo_tts_reference.py` remain the gates before any of this is claimed as done.

docs/community_models/echo_tts_dit_status.md

@@ -0,0 +1,133 @@
+# Echo-TTS DiT: implementation status
+
+## What exists
+
+| Component | State | Verification |
+| --- | --- | --- |
+| Byte tokenizer + WhisperD normalisation | complete | executed, output checked by hand |
+| PCA forward / inverse | complete | executed, cross-checked against numpy |
+| Flattening-point crop | complete | executed |
+| Euler dual-CFG sampler | complete | executed, matches a numpy transcription of `inference.py` to 6.6e-07 |
+| Timestep embedding | complete | matches `model.py` exactly (0.0 diff) |
+| Attention mask construction | complete | layout checked against upstream `cat()` semantics and ggml constraints |
+| DiT graph (encoders, joint attention, adaLN, blocks) | written | compiles against real headers; **never executed** |
+| Weight loading (1,117 tensors) | written | compiles; **tensor names unconfirmed against a real checkpoint** |
+| Conditioning / denoiser graph execution | written | compiles; **never executed** |
+| Fish codec `z_q` seam | not started | — |
+| Session integration | not started | — |
+
+The distinction in that last column is the important one. Everything above the
+line was run and compared against a reference. Everything below it has only been
+type-checked. A clean compile here means the framework APIs are used correctly;
+it says nothing about whether the numbers are right.
+
+## Design decisions worth reviewing
+
+### Flash attention in the DiT joint attention
+
+`joint_attention` uses `ggml_flash_attn_ext`, which never materialises the
+`(lanes, heads, seq, keys)` scores tensor. That tensor was the largest
+per-request allocation in the model:
+
+| Case | Keys | Scores tensor removed |
+| --- | ---: | ---: |
+| Typical (64 text bytes, 10 s reference) | 793 | 97 MB per attention |
+| Long text, 30 s reference | 1569 | 193 MB per attention |
+| Worst case (768 text, 5 min reference) | 3008 | 370 MB per attention |
+
+Live across 24 blocks with `ggml_gallocr` reuse, the practical saving is a few
+hundred MB to over a gigabyte, and flash attention is also faster.
+
+This was initially written with the explicit lowering on the belief that the
+speaker-unconditional CFG lane produces fully masked rows, which would make
+`-inf` softmax to NaN. That was wrong: `make_denoiser_mask` leaves the self block
+of every row unmasked, so a query always attends to at least its own 640
+positions and no row can be fully masked.
+
+Two details the flash path requires. The mask must be F16, so the masked value
+is `-65000` rather than `-1e9`; the latter converts to `-inf` in F16, which would
+reintroduce exactly the NaN hazard the explicit path was chosen to avoid. And
+`q->ne[2] % mask->ne[2]` and `q->ne[3] % mask->ne[3]` must both be zero, which
+holds because the mask carries a singleton head axis and matches the lane count.
+
+Set `AUDIOCPP_ECHO_TTS_NO_FLASH=1` to fall back to the explicit lowering and F32
+mask, for A/B comparison without a rebuild.
+
+The two encoders still use the explicit lowering. Their sequences are short (a
+few hundred tokens at most) so the scores tensors are small, and the speaker
+encoder is causal with no explicit mask, which the flash path rejects.
+
+### Speaker references are encoded in chunks
+
+`encode_speaker` splits the reference into ~29.7 s chunks (640 latents x 2048
+samples), zero-pads the last one, and concatenates the per-chunk latents,
+following `inference.py::get_speaker_latent_and_mask`. Upstream's comment calls
+that the longest chunk seen in training, so this is a fidelity matter as much as
+a memory one -- encoding several minutes in a single pass is a different
+computation from what the model saw.
+
+The memory difference is large, because the Fish encoder's first stages run at
+the full 44.1 kHz rate. A single 64-channel activation is 0.34 GB for one chunk
+against 3.04 GB for a 4m29s reference encoded in one pass, and several such
+tensors are live at once. Fixed-size chunks also mean one encode graph is built
+and reused across all chunks.
+
+After chunking, the dominant per-request allocation at long reference lengths is
+the persistent KV cache: 0.59 GB at 4m29s and 0.65 GB at the 297 s maximum,
+stored F32. Halving it to F16 is the obvious next step if that ever matters.
+
+### KV cache as a separate backend buffer
+
+The conditioning encoders and the denoiser are separate graphs so the encoders
+run once per request rather than once per sampler step. They share the cached
+projections through tensors allocated in their own `ggml_context` and backend
+buffer, referenced as leaves by both graphs. `ggml_gallocr` leaves
+already-allocated tensors alone, so the conditioning graph writes into them with
+`ggml_cpy` and the denoiser graph reads them directly.
+
+Consequence: changing text length or speaker length invalidates the cache and
+every graph built against it. `prepare_conditioning` tears all of it down and
+rebuilds, which is correct but means a request with new conditioning pays full
+graph construction. Acceptable given that a 40-step sample dominates.
+
+### Speaker KV scaling round-trips through the host
+
+`scale_speaker_kv` reads the cached tensors back, scales, and re-uploads, because
+the cache has no graph attached. This runs at most twice per request (once to
+apply, once to undo at the threshold) and touches at most 24 layers x 2 tensors.
+It is not on the per-step path. If it ever shows up in a profile, the fix is a
+tiny scaling graph rather than a host round trip.
+
+## Things most likely to be wrong
+
+Listed in rough order of how much damage they would do and how hard they would
+be to spot without a parity run:
+
+1. **Tensor names.** Derived from `model.py`'s module structure, corroborated by
+   a parameter count matching the published file size to ten digits, but never
+   resolved against an actual checkpoint. `convert_echo_tts.py --model-dir ...`
+   settles this in seconds and prints exactly what is wrong if anything is.
+2. **Half-head RoPE.** Heads 0-7 rotate, 8-15 do not. Implemented as
+   slice/rope/concat on the head axis. Wrong here means plausible-sounding but
+   incorrect audio, with no shape error.
+3. **Rotary pairing convention.** `GGML_ROPE_TYPE_NORMAL` (interleaved), matching
+   upstream's complex view of adjacent pairs. The in-tree `rf_dit.cpp` uses NEOX,
+   so copying from it would be wrong.
+4. **Speaker patchify reshape.** Folding `patch_size` frames into the feature
+   axis assumes row-major frame-then-channel ordering. A transposed reading would
+   still produce correct shapes.
+5. **adaLN chunk order.** `shift, scale, gate` from `cond_embed.chunk(3, -1)`.
+   A permutation here is silent.
+
+Items 2-5 are all caught by the per-block parity dumps from
+`tools/community_models/echo_tts_reference.py`, which is why that script dumps
+per-block activations at a fixed timestep rather than only the final output.
+
+## Next steps
+
+1. Run `convert_echo_tts.py` against the real checkpoint to confirm item 1.
+2. Build on a machine with a GPU and run the parity comparison per block.
+3. Split `fish_audio/codec.cpp`'s `build_decode_quantizer` at the `post_module`
+   boundary and expose the summed `quantized` term from
+   `build_encode_quantizer`, giving Echo decode and native speaker encoding.
+4. Wire the session: tokenize, encode speaker, sample, PCA inverse, decode, crop.

docs/community_models/echo_tts_parity_run1.md

@@ -0,0 +1,91 @@
+# Echo-TTS parity run 1: findings
+
+Source: `echo_ref.npz`, 146 arrays, generated from `audio_prompts/musk1.wav`,
+seed 0, 40 steps, sequence_length 640, model dtype bfloat16.
+
+## Components now verified against real data
+
+| Component | Result |
+| --- | --- |
+| Byte tokenizer + normalisation | **exact** — all 140 ids and the normalised string match byte-for-byte |
+| PCA orientation | **confirmed** — `pca.components` is `(80, 1024)`, as assumed |
+| `pca_unproject` (C++) | max abs 5.7e-06 against numpy on the real basis (z_q range 11.97) |
+| PCA round trip (C++) | max abs 5.3e-06 against the real speaker latent (range 2.51) |
+| Reconstructed z_q vs `ae.encode_zq` | min -10.2110 / max +11.9744 vs reference -10.2115 / +11.9740 |
+| `find_flattening_point` (C++) | **exact** — 140 of 640 frames, matching the reference heuristic |
+| `latent_scale` | float32(1/18) exactly |
+
+The flattening-point match is worth calling out: it ran on the real generated
+latent, not a synthetic one, and 140 frames is 6.502 s of audio from a 29.72 s
+window. Getting this wrong changes the output duration silently, and it is
+sensitive to the variance convention — I checked that `ddof=0` also lands on 140
+here, so this particular case would not have caught a wrong choice. The
+implementation uses the unbiased estimator to match `torch.std`, which is right
+for the general case regardless.
+
+## RNG: same stream, not bit-exact
+
+`generate_torch_cuda_randn(51200, 0)` was compared against
+`sampler.initial_noise`:
+
+    cosine                1.000000000000   (1 - cos = 3.1e-14)
+    median error          2 ULP
+    p99 error             149 ULP
+    correlation           1.000000000000 to 12 digits
+    rank agreement        99.64%
+
+Same Philox stream and same normal transform; the residual is CUDA-vs-host libm
+precision in the transcendental calls. **Seeded parity will be near-identical but
+not bit-exact**, so a 40-step trajectory will diverge slightly from the
+reference. Against the PR's cosine >= 0.999 gate this is irrelevant (the noise
+alone passes with ~3e10 margin), but any test written to expect bit-equality
+would fail for reasons that are not bugs. Write the gates as cosine plus
+max-abs-error, not equality.
+
+Also confirmed: `dit.x_input` is bitwise identical to `sampler.initial_noise`,
+so the fixed-timestep probe and the sampler share a starting draw.
+
+## Massive activations in the late blocks
+
+Activation magnitude grows monotonically through the stack:
+
+| Block | std | max |
+| ---: | ---: | ---: |
+| 0 | 0.266 | 5.69 |
+| 12 | 0.393 | 14.88 |
+| 20 | 1.147 | 38.75 |
+| 22 | 2.619 | 97.50 |
+| 23 | 5.893 | 187.00 |
+
+std grows 22x and max 33x from first block to last, with most of it in the final
+four blocks. Separately, the layer-23 key caches for **both** text and speaker
+peak at exactly 510.0 while layers 0 and 12 peak near 8-10. The two paths share
+one `k_norm` weight per layer and carry unrelated inputs, so an identical maximum
+points at a large element in that weight rather than at the data — the standard
+massive-activation / attention-sink pattern.
+
+Consequences:
+
+1. **F16 activations are safe.** 510 and 187 are far below the 65504 F16 ceiling.
+   No overflow risk in the planned conversion.
+2. **The converter's decision to keep norm weights in F32 was right for a reason
+   that was not known when it was made.** `KEEP_F32_SUBSTRINGS` already covers
+   `q_norm` and `k_norm`. Quantising a weight with a ~510 outlier to a
+   block-scaled int8 would destroy the small elements sharing its block.
+3. **Q8_0 (milestone M3) needs care in the last four blocks.** A per-block int8
+   scale resolves roughly 1/127 of the block maximum, which at block 23 is ~1.5
+   absolute against a std of 5.89. Mixed precision — leaving blocks 20-23 at F16
+   — is the obvious first thing to try if Q8_0 degrades quality.
+
+## Still unverified
+
+The DiT graph itself. Every check above exercises host-side code; nothing has run
+the ggml graph, because that needs a build. The per-block dumps in this file are
+exactly what the block-by-block comparison will consume once it can run, and the
+growth table above doubles as a smoke test: a port that gets the residual stream
+right should reproduce that monotone 22x growth, and one that gets adaLN gating
+or the half-head RoPE wrong will not.
+
+Useful next request, if another run is cheap: `--full-blocks`, which dumps every
+block activation in full rather than stats plus a 64-value prefix. Not needed
+until there is a build to compare against.