From ad2baa7539d6256642bd231a09180ab6d8811187 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Sun, 16 Aug 2026 16:49:28 +0200 Subject: [PATCH 01/23] =?UTF-8?q?[WIP]=20Add=20VoxCPM=20v1=20=E2=80=94=20l?= =?UTF-8?q?ightweight=20VoxCPM=20TTS=20support=20(0.5B=20/=201.5B)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CMakeLists.txt | 15 + docs/reports/voxcpm1_pr.md | 243 ++++++++++ docs/tts.md | 36 ++ include/engine/models/voxcpm2/assets.h | 3 +- include/engine/models/voxcpm2/loader.h | 1 + model_specs/voxcpm1.json | 118 +++++ src/models/voxcpm2/assets.cpp | 647 ++++++++++++++++++++++++- src/models/voxcpm2/generator.cpp | 65 ++- src/models/voxcpm2/loader.cpp | 115 ++++- src/models/voxcpm2/minicpm.cpp | 16 +- 10 files changed, 1220 insertions(+), 39 deletions(-) create mode 100644 docs/reports/voxcpm1_pr.md create mode 100644 model_specs/voxcpm1.json diff --git a/CMakeLists.txt b/CMakeLists.txt index 80dcdd5b..01c8964a 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -696,6 +696,21 @@ audiocpp_add_model(voxcpm2 engine::models::voxcpm2::make_voxcpm2_loader ) +audiocpp_add_model(voxcpm1 + SOURCES + src/models/voxcpm2/assets.cpp + src/models/voxcpm2/audiovae.cpp + src/models/voxcpm2/generator.cpp + src/models/voxcpm2/loader.cpp + src/models/voxcpm2/minicpm.cpp + src/models/voxcpm2/session.cpp + src/models/voxcpm2/tokenizer_text.cpp + INCLUDES + engine/models/voxcpm2/loader.h + LOADERS + engine::models::voxcpm2::make_voxcpm1_loader +) + audiocpp_add_model(vibevoice SOURCES src/models/vibevoice/assets.cpp diff --git a/docs/reports/voxcpm1_pr.md b/docs/reports/voxcpm1_pr.md new file mode 100644 index 00000000..a5d3b70e --- /dev/null +++ b/docs/reports/voxcpm1_pr.md @@ -0,0 +1,243 @@ +# PR: VoxCPM1 — lightweight VoxCPM TTS support (0.5B / 1.5B) + +> **Status: first porting attempt — runtime works end-to-end, output quality NOT yet acceptable** +> +> The port successfully loads and runs all three VoxCPM v1 GGUF variants (anchors pass, graphs +> execute, WAV files are produced at the correct sample rates/durations with active signal). +> **Known issue:** the generated audio is almost pure noise with only a faint trace of human +> voice. The pipeline is correct mechanically, but output quality requires further debugging +> (hypotheses and investigation plan in [Known issue](#known-issue-noisy-output)). + +--- + +## 1. Overview + +This PR adds support for the **OpenBMB VoxCPM v1** family of lightweight TTS models to +audio.cpp, reusing the existing and already-released `voxcpm2` model tree: + +| Model | Params | Output sample rate | GGUF file | +|---|---|---|---| +| VoxCPM-0.5B | 0.5B | **16 kHz** | `voxcpm-0.5b-q8_0-audiovae-f16.gguf` | +| VoxCPM-1.5B | 1.5B | **44.1 kHz** | `voxcpm1.5-q8_0.gguf` | +| VoxCPM-1.5B | 1.5B | **44.1 kHz** | `voxcpm1.5-q4_k-audiovae-f16.gguf` | + +The three models are architecturally **different variants** (they cannot share one config): + +- **0.5B:** VAE encoder 128 / decoder 1536, encoder_rates `[2,5,8,8]`, decoder_rates + `[8,8,5,2]`, patch_size 2, residual_lm 6 layers, encoder/dit 4 layers, 16 kHz, max_len 4096. +- **1.5B:** VAE encoder 64 / decoder 2048, encoder_rates `[2,3,6,7,7]`, decoder_rates + `[7,7,6,3,2]`, patch_size 4, residual_lm 8 layers, encoder/dit 8 layers, 44.1 kHz, max_len 8192. + +Since the v1 GGUFs store a different tensor convention than v2 (folded AudioVAE weights, no +`weight_v`/`weight_g` split, no `sr_cond_model` tensors, `voxcpm` architecture name), the port +wraps the v2 loader with a GGUF tensor-adaptation layer and adds `config.v1`-guarded branches +in the generator, mirroring the reference implementation (`VoxCPM.cpp`). + +--- + +## 2. Porting activities + +1. **Regenerated the 0.5B `config.json` from the GGUF metadata** — the previously shipped + sidecar was wrong on ~8 axes (patch, residual_lm/encoder/dit layer counts, VAE dims and + rates, sample rate 44.1 kHz vs the actual 16 kHz, max_len). +2. **Diagnosed the v1 GGUF conventions** (tensor dump + reference converter analysis): + - AudioVAE conv weights are stored **already folded** (weight-norm folded), with no + `weight_v`/`weight_g` decomposition and no `sr_cond_model.*` tensors. + - GGUF file dims == ggml `ne` order; the v1 GGUFs carry **no** `audiocpp.tensor_shapes` + override metadata (v2 does), so the adapter must present shapes itself. + - The 1.5B **Q8_0** file stores VAE conv weights **2D-flattened** (`{out, in·k}`, kernel + folded into dim1) while Q4_K and 0.5B store 3D `{out, in, k}` — both must load. +3. **Designed the identity-fold adapter** (see §4) so the existing `load_vae_weights` loader + works unchanged against folded v1 weights byte-for-byte. +4. **Mirrored the reference generator math** for the no-fusion (no `fusion_concat_proj`) + case: elementwise-add fusion inputs, elementwise-add dit-mu, and a real residual_lm + autoregressive step. +5. **Set up per-variant model directories** (`VoxCPM1-GGUF/` for 0.5B, `VoxCPM1.5-GGUF/` + for 1.5B) each with a config regenerated from its own GGUF metadata + tokenizer sidecars, + and updated `model_specs/voxcpm1.json` package targets accordingly. +6. **Verified end-to-end runs** for all three GGUFs on the CPU backend (see + [Validation](#6-validation-performed)). + +--- + +## 3. Changes per file + +| File | Change | +|---|---| +| `CMakeLists.txt` | Added `audiocpp_add_model(voxcpm1 ...)` reusing the 7 voxcpm2 sources; registers `engine::models::voxcpm2::make_voxcpm1_loader`. | +| `include/engine/models/voxcpm2/loader.h` | Declared `make_voxcpm1_loader()`. | +| `include/engine/models/voxcpm2/assets.h` | Added `VoxCPM2Config::v1 = false`; `load_voxcpm2_assets()` now takes `bool is_v1`. | +| `src/models/voxcpm2/loader.cpp` | Added `VoxCPM1Loader` (family `"voxcpm1"`), `load_voxcpm1_model()`, `make_voxcpm1_loader()`, `metadata_v1` / `capabilities_v1` / `cli_v1`. Offline-only TTS + speaker-reference clone, `text_prefix` policy, GGUF via `load_voxcpm2_assets(path, is_v1=true)`. | +| `src/models/voxcpm2/assets.cpp` | Added `TransformingTensorSource` v1 adapter (biggest chunk):
• v1→v2 tensor-name rename map (`token_embd.weight`→`base_lm.embed_tokens.weight`, gguf `blk.N.*`→`base_lm.layers.N.*` / `feat_encoder.encoder.layers.*` / `feat_decoder.estimator.decoder.layers.*` / `residual_lm.layers.*`, `attn_norm`→`input_layernorm`, `ffn_norm`→`post_attention_layernorm`, `attn_*`→`self_attn.*_proj`, `ffn_*`→`mlp.*_proj`, `time_mlp.*` (preserving `.linear_N`), `output_norm.weight`→`base_lm.norm.weight`, projection/fsq/stop mappings)
• **Folded weight-norm synthesis**: for every `audio_vae.*.weight` conv, `X.weight_v` → folded tensor data as-is, `X.weight_g` → per-row L2 norms (identity fold, see §4)
• Identity `decoder.sr_cond_model.{2..5}.scale_embed.weight` (ones) / `.bias_embed.weight` (zeros) since v1 GGUFs carry no SR-conditioning tensors
• Synthesized missing v1 tensors (`feat_encoder.scale_embed/bias_embed`, `feat_encoder.fc_logvar`, `feat_encoder.diag`, `feat_encoder.merge`, `token_embd.extra_bias`, `fusion_concat_proj.weight/bias`, `stop_proj.weight`, `stop_head.weight`)
• Rank-tolerant `require_f32` (accept element-count-equal, shape-different fetches — handles 2D-flattened convs and `{C,1}` alphas) + relaxed-rank VAE weight_v anchors for v1
• `has_tensor` / `require_metadata` / `require_tensor_data` folded + synthesized lookups
• **Anchor fix:** `encoder.fc_mu.weight_v` now uses computed encoder-in (`encoder_dim << #rates` = 2048), not `decoder_dim` (1536) | +| `src/models/voxcpm2/generator.cpp` | • v1 fusion guard: residual input = `AddModule(lm_hidden, current_embed)` / `AddModule(fsq, current_embed)` instead of concat+linear (matches reference `build_residual_fusion_input`)
• Added `add_dit_mu()` helper; v1 `mu` = elementwise add of `current_lm_dit_hidden + residual_dit_hidden` (matches reference `build_dit_mu`, `mu_dim = hidden·(fusion?2:1)`, v1 → hidden)
• CFM `mu` size check is now v1-aware (`hidden_dim * (v1 ? 1 : 2)`)
• v1 decode loop runs `residual_lm_.run_step(next_projected.residual_input).hidden` (the earlier `fsq_lm_dit_hidden` shortcut removed — v1 GGUFs have 6/8 residual_lm layers) | +| `src/models/voxcpm2/minicpm.cpp` | Prompt-prefill graph: v1 `residual_input` = `AddModule(lm_hidden, masked_current)` instead of concat+linear; residual_lm always runs (previously the concat path would have produced a wrong-dimension residual input for v1). | +| `model_specs/voxcpm1.json` | Package targets: `voxcpm1_0.5b_q8_0` → `VoxCPM1-GGUF`; `voxcpm1_1.5b_q4_k` and `voxcpm1_1.5b_q8_0` → `VoxCPM1.5-GGUF` (per-variant config/tokenizer). | +| `docs/tts.md` | Added VoxCPM1 section + TOC entry (usage, options, sample-rate notes). | +| `README.md` | Added `voxcpm1` row to the supported-model table. | +| `docs/reports/voxcpm1_port_status.md` | Port status log (analysis, decisions, timestamps, remaining tasks). | +| `models/VoxCPM1-GGUF/config.json` | **Regenerated** from 0.5B GGUF metadata. | +| `models/VoxCPM1.5-GGUF/config.json` | **New**, regenerated from 1.5B GGUF metadata. | +| `models/VoxCPM1.5-GGUF/tokenizer.json` (+config/special tokens) | Copied from 0.5B dir (same 73,448-vocab BPE tokenizer). | + +--- + +## 4. Key design: the identity-fold adapter + +The v1 GGUF (OpenBMB reference converter) stores AudioVAE conv weights **already folded** +(`weight = weight_g · weight_v / ‖weight_v‖`), with no `weight_v`/`weight_g` split, while +`audiovae.cpp` requests the decomposed names directly via `require_f32`. The adapter solves +this without touching the VAE loader: + +``` +X.weight_v := folded GGUF tensor data (as-is) +X.weight_g := per-row L2 norms of the folded tensor, + computed with the loader's own row grouping + (groups = expected_shape.front(), inner = elements/groups) +``` + +Because `fold_weight_norm` multiplies row `d0` by `weight_g[d0] / ‖row d0‖ = 1`, the loader +output equals the GGUF data **byte-for-byte** — an exact identity, with no layout drift +relative to the reference runtime's consumption of the same bytes. The same mechanism works +for 3D `{out, in, k}` and 2D-flattened `{out, in·k}` conversions (element counts must match; +ranks may differ, covered by rank-tolerant `require_f32` + relaxed-rank anchors). + +--- + +## 5. Usage + +### Build + +```bash +scripts/build_linux.sh --backend cpu --target audiocpp_cli +# or, with the standard full model set: +cmake -S . -B build/linux-cpu-release -DCMAKE_BUILD_TYPE=Release +cmake --build build/linux-cpu-release --target audiocpp_cli -j 8 +``` + +### Run — 0.5B (16 kHz output) + +```bash +build/linux-cpu-release/bin/audiocpp_cli \ + --task tts --family voxcpm1 \ + --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf \ + --backend cpu --text "Hello from VoxCPM1." --out out.wav +``` + +### Run — 1.5B (44.1 kHz output) + +```bash +build/linux-cpu-release/bin/audiocpp_cli \ + --task tts --family voxcpm1 \ + --model models/VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf \ + --backend cpu --text "Hello from VoxCPM1." --out out.wav +``` + +### Options + +| Option | Values | Default | Meaning | +|---|---:|---:|---| +| `--task` | `tts` | required | Task kind. | +| `--family` | `voxcpm1` | auto-detect | Selects the v1 loader. | +| `--backend` | `cpu`, `cuda`, `vulkan`, `metal`, `hip`, `best` | `best` | Backend. | +| `--voice-ref` | WAV path | not set | Reference speaker audio (clone). | +| `--max-tokens` | integer | `4096` | Maximum generated AR tokens. | +| `--num-inference-steps` | integer | `10` | Flow-matching steps. | +| `--guidance-scale` | float | `2.0` | CFG strength. | +| `--session-option voxcpm1.mem_saver=true\|false` | bool | `false` | Tighter graph workspaces + release request graphs after completion. | +| `--session-option voxcpm1.prompt_cache_slots=` | integer | `1` | Prompt/prompt-audio embedding cache slots. | +| `--text-chunk-mode` | `default`, `tag_aware`, `japanese`, `endline` | `tag_aware` | Long-form chunking mode. | + +--- + +## 6. Validation performed + +- **Load + anchors:** all three GGUFs pass `validate_weight_anchors` and + `load_vae_weights`/`load_model_weights` on CPU. This includes the 0.5B 3D convs, the 1.5B + Q4_K 3D convs, and the 1.5B Q8_0 2D-flattened convs. +- **End-to-end:** `--task tts` completes for all three models; outputs are written as WAV at + the correct sample rate (16 kHz for 0.5B, 44.1 kHz for 1.5B) with active signal and + speech-plausible duration/envelope. +- **Regression:** the released voxcpm2 path is untouched (guard style `config.v1`, v2 default + `false`); voxcpm2 was not re-benchmarked but the changed code paths are v1-gated or + v1/v2-neutral. + +> ⚠️ **Quality caveat:** "end-to-end completes" does **not** mean the output is usable yet. +> See the known issue below — the audio is predominantly noise. + +--- + +## 7. Supported modes + +| Mode | Supported | Notes | +|---|---|---| +| **Offline TTS** | ✅ implemented | Default and only advertised mode. | +| **Streaming** | ❌ not implemented for v1 | `voxcpm1` advertises offline-only. Streaming is a v2 capability; it has not been validated (or enabled) for v1. | +| **Voice clone** | ⚠️ surface present | Speaker-reference options are advertised (`--voice-ref`), but quality is gated on the same known issue as plain TTS. | + +--- + +## Known issue: noisy output + +**Symptom.** Generated v1 voices are almost pure noise with a little human voice mixed in — +the signal is dominated by broadband/noise content. This affects all three GGUFs. + +**What is confirmed working.** Model loading, tensor adaptation, anchor validation, graph +construction, graph execution, and WAV output plumbing are all correct (no crashes, no +shape/size errors, correct sample rates and durations). The failure is therefore in the +**numerics of synthesis**, i.e. the audio content itself. + +**Most likely causes (in rough priority order).** + +1. **Weight data interpretation** — the identity fold preserves bytes, but if some AudioVAE + layer's storage layout (depthwise vs pointwise handling, 2D-flattened Q8_0, transposed + decoder `{in,out,k}` conventions, per-group row ordering of `weight_g`) differs from what + `ggml_conv_1d` / `conv_transpose` expects, the VAE decoder outputs garbage while loading + still "succeeds" (element counts match). +2. **Synthesized tensor semantics** — `feat_encoder.scale_embed/bias_embed`, `merge`, + `diag`, `extra_bias`, `fusion_concat_proj`, `stop_*`, and the identity `sr_cond_model` + tensors were synthesized with plausible but unverified semantics; if any is required to be + learned/zero-`scale` (or absent entirely in the reference runtime), the feature stream + feeding the LM/CFM is wrong. +3. **Graph parity vs the reference** — fusion = add and dit-mu = add were taken from + reference `build_residual_fusion_input`/`build_dit_mu`, but adjacent details (masking, + slice indices, position ids, prompt handling, FSQ rounding, CFM conditioning inputs, + ordering of `nn.Module` sub-blocks in the residual_lm stack) may differ. +4. **Sample-rate/codec mismatch** — 0.5B output asserted 16 kHz but the reference may expect + a specific internal feature rate; patch_size/feat_dim interplay (2·64 vs 4·64) feeding the + CFM estimator could be off by a constant factor, producing frozen-then-noisy patches. +5. **Quantization path** — the 1.5B Q8_0 GGUF quantizes the VAE itself (2D-flattened); + dequantized values feed `require_f32`, but a transpose or block-order mismatch would + corrupt every activation. + +**Debugging plan (next iteration).** + +- [ ] Port a small deterministic parity harness: run the same prompt through the reference + `VoxCPM.cpp` and audio.cpp, dump intermediate tensors (lm hidden, residual hidden, + CFM mu, VAE latent, decoder output) at each major stage, and diff numerically. +- [ ] Verify `encoder.fc_mu` / `decoder.model.{0,1,N}` folded data against the Python + reference weights with a strict per-element comparison on non-quantized tensors + (f16 VAE files), including row-grouping of `weight_g`. +- [ ] Check whether the reference runtime actually instantiates `sr_cond_model` and + `feat_encoder` synthesizable blocks for v1; remove or zero-scale any block the + reference does not run. +- [ ] Experimentally force one suspected block to a no-op (e.g. sr_cond identity, merge + zeros, scale_embed 0/1) and measure whether noise level drops. +- [ ] Validate CFM mu dimension/conditioning against the reference expectation for + `patch=2` (0.5B) and `patch=4` (1.5B). +- [ ] After the numerics match, run a human listening + loudness/spectral sanity check + (the current output has a spectral envelope consistent with noise + faint voice). + +--- + +## 8. Remaining tasks + +- [x] Loader registration, tensor adaptation, generator v1 branches, configs, model spec +- [x] End-to-end execution for 0.5B Q8_0, 1.5B Q4_K, 1.5B Q8_0 +- [ ] **Fix noisy output (known issue above) — top priority** +- [ ] Numerical parity harness vs `VoxCPM.cpp` reference (stage-by-stage tensor diff) +- [ ] `tests/voxcpm1/` automated path tests mirroring `tests/voxcpm2/` +- [ ] WebUI catalog entry (`webui/configs/models_catalog.json`) +- [ ] `docs/gguf.md` support-table entry +- [ ] CUDA-backend verification + RTF measurement (expect voxcpm2-like speedups) +- [ ] Streaming support for v1 (only meaningful after numerics are fixed) +- [ ] Commit + release packaging for `audio.cpp-gguf` (0.5B and 1.5B packages) \ No newline at end of file diff --git a/docs/tts.md b/docs/tts.md index 770b6f6f..e77ccd8f 100644 --- a/docs/tts.md +++ b/docs/tts.md @@ -14,6 +14,7 @@ | NeuTTS | `neutts` | `tts` | [NeuTTS](#neutts) | | OmniVoice | `omnivoice` | `tts` | [OmniVoice](#omnivoice), [full guide](models/omnivoice.md) | | PocketTTS | `pocket_tts` | `tts` | [PocketTTS](#pockettts) | +| VoxCPM1 | `voxcpm1` | `tts` | [VoxCPM1](#voxcpm1) | | VoxCPM2 | `voxcpm2` | `tts`, `vdes` | [VoxCPM2](#voxcpm2) | | Higgs Audio v3 TTS | `higgs_audio_tts` | `tts` | [Higgs Audio v3 TTS](#higgs-audio-v3-tts) | | Fish Audio S2 Pro | `fish_audio` | `tts` | [Fish Audio S2 Pro](#fish-audio-s2-pro) | @@ -399,6 +400,41 @@ audiocpp_cli --task tts --family pocket_tts --model models/pocket-tts --backend | `--text-chunk-size` | integer chars | `256` | Long-form chunk size. | | `--session-option pocket_tts.voice_state_cache_slots=` | integer slots | `4` | Prepared voice-state cache slots; set `0` to disable reuse. | +## VoxCPM1 + +VoxCPM1 supports offline TTS. It reuses the VoxCPM2 runtime tree with a GGUF tensor-adaptation layer that understands the OpenBMB folded AudioVAE weights, so the same `--family voxcpm1` path serves both the 16 kHz 0.5B model and the 44.1 kHz 1.5B variants. + +| Field | Value | +|---|---| +| Family | `voxcpm1` | +| Model directory | `models/VoxCPM1-GGUF` (0.5B), `models/VoxCPM1.5-GGUF` (1.5B) | +| Task | `tts` | +| Modes | `offline` | +| Languages | Model auto-handles supported languages | +| Voice input | Optional reference WAV | +| Built-in voices | Not exposed | + +Text to speech: + +```bash +audiocpp_cli --task tts --family voxcpm1 --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf --backend cpu --text "Hello from VoxCPM1." --out out.wav +``` + +1.5B variant (44.1 kHz output): + +```bash +audiocpp_cli --task tts --family voxcpm1 --model models/VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf --backend cpu --text "Hello from VoxCPM1." --out out.wav +``` + +| Option | Values | Default | Meaning | +|---|---:|---:|---| +| `--voice-ref` | WAV path | not set | Reference speaker audio. | +| `--session-option voxcpm1.mem_saver=true\|false` | bool | `false` | Use tighter graph workspaces and release MiniCPM/AudioVAE request graphs after completion to reduce resident VRAM. | +| `--session-option voxcpm1.prompt_cache_slots=` | integer | `1` | Prompt and prompt-audio embedding cache slots. Set to `0` to disable prompt caching. | +| `--max-tokens` | integer | `4096` | Maximum generated AR tokens. | +| `--num-inference-steps` | integer | `10` | Flow matching steps. | +| `--guidance-scale` | float | `2.0` | CFG strength. | + ## VoxCPM2 VoxCPM2 supports plain TTS, voice design, controllable voice cloning, and an ultimate-clone style that uses both prompt audio and transcript. The CLI expresses voice design with the same text convention as the upstream examples: put the voice/style description in parentheses at the start of `--text`. diff --git a/include/engine/models/voxcpm2/assets.h b/include/engine/models/voxcpm2/assets.h index 89fe156b..19581ed3 100644 --- a/include/engine/models/voxcpm2/assets.h +++ b/include/engine/models/voxcpm2/assets.h @@ -85,6 +85,7 @@ struct VoxCPM2Config { int64_t max_length = 8192; std::string device = "cuda"; std::string dtype = "bfloat16"; + bool v1 = false; }; struct VoxCPM2Assets { @@ -94,6 +95,6 @@ struct VoxCPM2Assets { std::shared_ptr audiovae_weights; }; -std::shared_ptr load_voxcpm2_assets(const std::filesystem::path & model_path); +std::shared_ptr load_voxcpm2_assets(const std::filesystem::path & model_path, bool is_v1); } // namespace engine::models::voxcpm2 diff --git a/include/engine/models/voxcpm2/loader.h b/include/engine/models/voxcpm2/loader.h index 4c588482..72f7f3c5 100644 --- a/include/engine/models/voxcpm2/loader.h +++ b/include/engine/models/voxcpm2/loader.h @@ -29,5 +29,6 @@ class VoxCPM2LoadedModel final : public runtime::ILoadedVoiceModel { std::unique_ptr load_voxcpm2_model(const std::filesystem::path &model_path); std::shared_ptr make_voxcpm2_loader(); +std::shared_ptr make_voxcpm1_loader(); } // namespace engine::models::voxcpm2 diff --git a/model_specs/voxcpm1.json b/model_specs/voxcpm1.json new file mode 100644 index 00000000..905bd1df --- /dev/null +++ b/model_specs/voxcpm1.json @@ -0,0 +1,118 @@ +{ + "family": "voxcpm1", + "display_name": "VoxCPM1", + "description": "OpenBMB VoxCPM 0.5B and 1.5B tokenizer-free TTS models with 24kHz output.", + "category": "tts", + "status": "supported", + "tasks": [ + "tts", + "clone" + ], + "modes": [ + "offline" + ], + "languages": [ + "zh", + "en", + "ja", + "ko" + ], + "capabilities": { + "clone": [ + "speaker_reference" + ] + }, + "runtime": { + "tags": [ + "gguf" + ] + }, + "ui": { + "recommended_package": "voxcpm1_0.5b_q8_0", + "tags": [ + "TTS", + "Clone", + "GGUF" + ], + "docs": [ + "docs/tts.md", + "docs/gguf.md" + ] + }, + "package_defaults": { + "download": { + "kind": "huggingface_snapshot", + "repo": "audio-cpp/audio.cpp-gguf", + "revision": "main", + "gated": false + } + }, + "packages": [ + { + "id": "voxcpm1_0.5b_q8_0", + "display_name": "VoxCPM 0.5B Q8_0 GGUF", + "default": true, + "format": "gguf", + "precision": "q8_0", + "target_directory": "VoxCPM1-GGUF", + "files": [ + "VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf", + "VoxCPM1-GGUF/config.json", + "VoxCPM1-GGUF/tokenizer.json", + "VoxCPM1-GGUF/tokenizer_config.json" + ], + "strip_prefix": "VoxCPM1-GGUF" + }, + { + "id": "voxcpm1_1.5b_q4_k", + "display_name": "VoxCPM 1.5B Q4_K GGUF", + "format": "gguf", + "precision": "q4_k", + "target_directory": "VoxCPM1.5-GGUF", + "files": [ + "VoxCPM1.5-GGUF/voxcpm1.5-q4_k-audiovae-f16.gguf", + "VoxCPM1.5-GGUF/config.json", + "VoxCPM1.5-GGUF/tokenizer.json", + "VoxCPM1.5-GGUF/tokenizer_config.json" + ], + "strip_prefix": "VoxCPM1.5-GGUF" + }, + { + "id": "voxcpm1_1.5b_q8_0", + "display_name": "VoxCPM 1.5B Q8_0 GGUF", + "format": "gguf", + "precision": "q8_0", + "target_directory": "VoxCPM1.5-GGUF", + "files": [ + "VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf", + "VoxCPM1.5-GGUF/config.json", + "VoxCPM1.5-GGUF/tokenizer.json", + "VoxCPM1.5-GGUF/tokenizer_config.json" + ], + "strip_prefix": "VoxCPM1.5-GGUF" + } + ], + "sources": [ + { + "format": "gguf", + "roots": { + "model": ".", + "weights": "$gguf" + }, + "files": { + "config": "model:config.json", + "tokenizer_config": "model:tokenizer_config.json", + "tokenizer_json": "model:tokenizer.json", + "special_tokens_map": "model:special_tokens_map.json" + }, +"tensors": { + "weights": { + "source": "weights:" + }, + "audiovae_weights": { + "source": "weights:" + } + } + } + ] +} diff --git a/src/models/voxcpm2/assets.cpp b/src/models/voxcpm2/assets.cpp index 7d4ae2c4..997f54ee 100644 --- a/src/models/voxcpm2/assets.cpp +++ b/src/models/voxcpm2/assets.cpp @@ -2,11 +2,18 @@ #include "engine/framework/model_spec/package.h" #include "engine/framework/assets/resource_bundle.h" +#include "engine/framework/assets/tensor_source.h" #include "engine/framework/io/config.h" #include "engine/framework/io/json.h" #include #include +#include +#include +#include +#include +#include +#include namespace engine::models::voxcpm2 { namespace json = engine::io::json; @@ -138,8 +145,8 @@ VoxCPM2Config parse_config(const assets::ResourceBundle & resources) { const auto root = resources.parse_json("config"); VoxCPM2Config config; config.architecture = json::require_string(root, "architecture"); - if (config.architecture != "voxcpm2") { - throw std::runtime_error("VoxCPM2 config architecture mismatch: " + config.architecture); + if (config.architecture != "voxcpm2" && config.architecture != "voxcpm") { + throw std::runtime_error("VoxCPM config architecture mismatch: " + config.architecture); } config.lm = parse_lm_config(root.require("lm_config")); config.patch_size = json::optional_i64(root, "patch_size", config.patch_size); @@ -172,6 +179,616 @@ VoxCPM2Config parse_config(const assets::ResourceBundle & resources) { return config; } +namespace assets = engine::assets; + +namespace { +core::TensorShape make_tensor_shape(const std::vector & dims) { + if (dims.empty() || dims.size() > core::kMaxTensorRank) { + throw std::runtime_error("tensor rank must be between 1 and 4"); + } + switch (dims.size()) { + case 1: + return core::TensorShape::from_dims({dims[0]}); + case 2: + return core::TensorShape::from_dims({dims[0], dims[1]}); + case 3: + return core::TensorShape::from_dims({dims[0], dims[1], dims[2]}); + case 4: + return core::TensorShape::from_dims({dims[0], dims[1], dims[2], dims[3]}); + default: + throw std::runtime_error("unsupported tensor rank"); + } +} +} // namespace + +class TransformingTensorSource final : public assets::TensorSource { +public: + TransformingTensorSource( + std::shared_ptr source, + const VoxCPM2Config & config, + bool is_v1) + : source_(std::move(source)), config_(config), is_v1_(is_v1) { + build_routes(); + } + + const std::filesystem::path & source_path() const noexcept override { + return source_->source_path(); + } + + bool has_tensor(std::string_view name) const noexcept override { + const std::string key{std::string(name)}; + if (routes_.find(key) != routes_.end() || + synthesized_tensors_.find(key) != synthesized_tensors_.end()) { + return true; + } + if (is_v1_) { + // v1 GGUF stores folded AudioVAE conv weights; the loader asks for + // decomposed weight_v/weight_g names which we synthesize from the + // folded tensors on demand. + const auto base = folded_base_name(key); + if (!base.empty() && folded_convs_.find(base) != folded_convs_.end()) { + return true; + } + } + return false; + } + + assets::TensorMetadata require_metadata(std::string_view name) const override { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it != synthesized_tensors_.end()) { + return it->second; + } + const std::string key{std::string(name)}; + if (is_v1_) { + const auto base = folded_base_name(key); + if (!base.empty()) { + const auto folded_it = folded_convs_.find(base); + if (folded_it != folded_convs_.end()) { + auto metadata = source_->require_metadata(folded_it->second); + metadata.name = key; + if (has_suffix(key, ".weight_g") && !metadata.shape.empty()) { + metadata.shape = {metadata.shape.front(), 1, 1}; + } + return metadata; + } + } + } + const auto route_it = routes_.find(key); + if (route_it == routes_.end()) { + throw std::runtime_error("missing tensor: " + std::string(name)); + } + auto metadata = source_->require_metadata(route_it->second); + metadata.name = key; + // Apply shape transformations if needed + if (reshape_map_.find(key) != reshape_map_.end()) { + metadata.shape = reshape_map_.at(key); + } + return metadata; + } + + std::vector tensors() const override { + std::vector out; + out.reserve(routes_.size() + synthesized_tensors_.size()); + for (const auto & [name, route] : routes_) { + out.push_back(require_metadata(name)); + } + for (const auto & [name, metadata] : synthesized_tensors_) { + out.push_back(metadata); + } + std::sort(out.begin(), out.end(), + [](const assets::TensorMetadata & lhs, const assets::TensorMetadata & rhs) { + return lhs.name < rhs.name; + }); + return out; + } + + void release_storage() const override { source_->release_storage(); } + + assets::RawTensorData require_tensor_data(std::string_view name) const override { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it != synthesized_tensors_.end()) { + return generate_synthesized_tensor(name); + } + const std::string key{std::string(name)}; + if (is_v1_) { + const auto base = folded_base_name(key); + if (!base.empty() && folded_convs_.find(base) != folded_convs_.end()) { + auto data = source_->require_tensor_data(folded_convs_.at(base)); + data.metadata.name = key; + return data; + } + } + const auto route_it = routes_.find(key); + if (route_it == routes_.end()) { + throw std::runtime_error("missing tensor: " + std::string(name)); + } + auto data = source_->require_tensor_data(route_it->second); + data.metadata.name = key; + // Apply transformations + if (reshape_map_.find(key) != reshape_map_.end()) { + const auto & target_shape = reshape_map_.at(key); + if (data.metadata.shape != target_shape) { + // Reshape the data + data = reshape_tensor_data(data, target_shape); + } + } + return data; + } + + std::vector require_f32( + std::string_view name, + const std::optional> & expected_shape) const override { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it != synthesized_tensors_.end()) { + return generate_synthesized_f32(name); + } + const std::string key{std::string(name)}; + if (is_v1_) { + const auto base = folded_base_name(key); + if (!base.empty()) { + const auto folded_it = folded_convs_.find(base); + if (folded_it != folded_convs_.end()) { + const auto folded = source_->require_f32(folded_it->second, std::nullopt); + if (has_suffix(key, ".weight_g")) { + return folded_weight_g(folded, folded_it->second, expected_shape); + } + return folded; + } + } + } + const auto route_it = routes_.find(key); + if (route_it == routes_.end()) { + throw std::runtime_error("missing tensor: " + std::string(name)); + } + if (is_v1_ && expected_shape.has_value()) { + const auto meta = source_->require_metadata(route_it->second); + const int64_t expected_elems = checked_element_count("expected", *expected_shape); + const int64_t actual_elems = checked_element_count(route_it->second, meta.shape); + if (expected_elems == actual_elems && meta.shape != *expected_shape) { + return source_->require_f32(route_it->second, std::nullopt); + } + } + // Check if we need to reshape + if (reshape_map_.find(key) != reshape_map_.end()) { + const auto & target_shape = reshape_map_.at(key); + if (expected_shape.has_value() && *expected_shape != target_shape) { + // We'll fetch with target shape and then it will be validated + } + return source_->require_f32(route_it->second, target_shape); + } + return source_->require_f32(route_it->second, expected_shape); + } + + std::optional> optional_f32( + std::string_view name, + const std::optional> & expected_shape) const override { + if (!has_tensor(name)) return std::nullopt; + return require_f32(name, expected_shape); + } + + void set_backend_tensor( + ggml_tensor * tensor, + std::string_view name, + assets::TensorStorageType storage_type, + const std::vector & expected_shape) const override { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it != synthesized_tensors_.end()) { + const auto values = generate_synthesized_f32(name); + engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, values, + make_tensor_shape(expected_shape), + engine::assets::ggml_type_for_tensor_storage(storage_type)); + return; + } + const auto route_it = routes_.find(std::string(name)); + if (route_it == routes_.end()) { + throw std::runtime_error("missing tensor: " + std::string(name)); + } + // Check for weight norm decomposition (weight_v + weight_g) + const std::string logical_name = std::string(name); + if (weight_norm_map_.find(logical_name) != weight_norm_map_.end()) { + const auto & wn = weight_norm_map_.at(logical_name); + const auto weight_v = source_->require_f32(wn.weight_v_name, wn.weight_v_shape); + const auto weight_g = source_->require_f32(wn.weight_g_name, wn.weight_g_shape); + const auto folded = fold_weight_norm(weight_v, weight_g, wn.out_channels, wn.in_channels, wn.kernel_size); + const auto shape = make_tensor_shape(expected_shape); + const ggml_type type = engine::assets::ggml_type_for_tensor_storage( + engine::assets::resolve_tensor_storage_type(*this, name, storage_type)); + engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, folded, shape, type); + return; + } + // Check for reshape + if (reshape_map_.find(logical_name) != reshape_map_.end()) { + const auto & target_shape = reshape_map_.at(logical_name); + const auto values = source_->require_f32(route_it->second, target_shape); + const auto shape = make_tensor_shape(expected_shape); + const ggml_type type = engine::assets::ggml_type_for_tensor_storage( + engine::assets::resolve_tensor_storage_type(*this, name, storage_type)); + engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, values, shape, type); + return; + } + source_->set_backend_tensor(tensor, route_it->second, storage_type, expected_shape); + } + + void set_backend_f32_tensor( + ggml_tensor * tensor, + std::string_view name, + const std::vector & expected_shape) const override { + set_backend_tensor(tensor, name, assets::TensorStorageType::F32, expected_shape); + } + + int64_t require_i64_scalar(std::string_view name) const override { + return source_->require_i64_scalar(name); + } + +private: + struct WeightNormInfo { + std::string weight_v_name; + std::string weight_g_name; + std::vector weight_v_shape; + std::vector weight_g_shape; + int64_t out_channels = 0; + int64_t in_channels = 0; + int64_t kernel_size = 0; + }; + + void build_routes() { + // V1 -> V2 tensor name mapping + std::unordered_map rename_map = { + // LM embeddings + {"token_embd.weight", "base_lm.embed_tokens.weight"}, + // LM blocks + {"blk.", "base_lm.layers."}, + {"attn_q.weight", "self_attn.q_proj.weight"}, + {"attn_k.weight", "self_attn.k_proj.weight"}, + {"attn_v.weight", "self_attn.v_proj.weight"}, + {"attn_norm.weight", "input_layernorm.weight"}, + {"attn_output.weight", "self_attn.o_proj.weight"}, + {"ffn_norm.weight", "post_attention_layernorm.weight"}, + {"ffn_gate.weight", "mlp.gate_proj.weight"}, + {"ffn_up.weight", "mlp.up_proj.weight"}, + {"ffn_down.weight", "mlp.down_proj.weight"}, + // Output norm + {"output_norm.weight", "base_lm.norm.weight"}, + // Residual LM + {"residual_lm.blk.", "residual_lm.layers."}, + {"residual_lm.output_norm.weight", "residual_lm.norm.weight"}, + // Local encoder (feat_encoder) + {"locenc.in_proj.weight", "feat_encoder.in_proj.weight"}, + {"locenc.in_proj.bias", "feat_encoder.in_proj.bias"}, + {"locenc.special_token", "feat_encoder.special_token"}, + {"locenc.blk.", "feat_encoder.encoder.layers."}, + {"locenc.output_norm.weight", "feat_encoder.encoder.norm.weight"}, + // Local DiT (feat_decoder) + {"locdit.in_proj.weight", "feat_decoder.estimator.in_proj.weight"}, + {"locdit.in_proj.bias", "feat_decoder.estimator.in_proj.bias"}, + {"locdit.cond_proj.weight", "feat_decoder.estimator.cond_proj.weight"}, + {"locdit.cond_proj.bias", "feat_decoder.estimator.cond_proj.bias"}, + {"locdit.out_proj.weight", "feat_decoder.estimator.out_proj.weight"}, + {"locdit.out_proj.bias", "feat_decoder.estimator.out_proj.bias"}, + {"locdit.time_mlp.linear_1.weight", "feat_decoder.estimator.time_mlp.linear_1.weight"}, + {"locdit.time_mlp.linear_1.bias", "feat_decoder.estimator.time_mlp.linear_1.bias"}, + {"locdit.time_mlp.linear_2.weight", "feat_decoder.estimator.time_mlp.linear_2.weight"}, + {"locdit.time_mlp.linear_2.bias", "feat_decoder.estimator.time_mlp.linear_2.bias"}, + {"locdit.delta_time_mlp.linear_1.weight", "feat_decoder.estimator.delta_time_mlp.linear_1.weight"}, + {"locdit.delta_time_mlp.linear_1.bias", "feat_decoder.estimator.delta_time_mlp.linear_1.bias"}, + {"locdit.delta_time_mlp.linear_2.weight", "feat_decoder.estimator.delta_time_mlp.linear_2.weight"}, + {"locdit.delta_time_mlp.linear_2.bias", "feat_decoder.estimator.delta_time_mlp.linear_2.bias"}, + {"locdit.output_norm.weight", "feat_decoder.estimator.decoder.norm.weight"}, + {"locdit.blk.", "feat_decoder.estimator.decoder.layers."}, + // Projections + {"proj.enc_to_lm.weight", "enc_to_lm_proj.weight"}, + {"proj.enc_to_lm.bias", "enc_to_lm_proj.bias"}, + {"proj.lm_to_dit.weight", "lm_to_dit_proj.weight"}, + {"proj.lm_to_dit.bias", "lm_to_dit_proj.bias"}, + {"proj.res_to_dit.weight", "res_to_dit_proj.weight"}, + {"proj.res_to_dit.bias", "res_to_dit_proj.bias"}, + {"fusion_concat_proj.weight", "fusion_concat_proj.weight"}, + {"stop.stop_proj.weight", "stop_proj.weight"}, + {"stop.stop_proj.bias", "stop_proj.bias"}, + {"stop.stop_head.weight", "stop_head.weight"}, + // FSQ + {"fsq.in_proj.weight", "fsq_layer.in_proj.weight"}, + {"fsq.in_proj.bias", "fsq_layer.in_proj.bias"}, + {"fsq.out_proj.weight", "fsq_layer.out_proj.weight"}, + {"fsq.out_proj.bias", "fsq_layer.out_proj.bias"}, + // Audio VAE (prefixed with audio_vae.) + {"audio_vae.encoder.block.", "encoder.block."}, + {"audio_vae.encoder.fc_mu", "encoder.fc_mu"}, + {"audio_vae.decoder.model.", "decoder.model."}, + {"audio_vae.decoder.sr_cond_model.", "decoder.sr_cond_model."}, + }; + + // Build routes by scanning source tensors + for (const auto & tensor : source_->tensors()) { + std::string v1_name = tensor.name; + std::string v2_name = v1_name; + + // Apply prefix replacements + for (const auto & [from, to] : rename_map) { + if (v2_name.rfind(from, 0) == 0) { + v2_name = to + v2_name.substr(from.size()); + break; + } + } + + // Handle blk.N.* -> layers.N.* (base LM, residual LM, locenc, locdit) + constexpr std::string_view kBlk = "blk."; + const size_t blk_pos = v1_name.find(kBlk); + if (blk_pos != std::string::npos) { + const size_t layer_start = blk_pos + kBlk.size(); + const size_t dot = v1_name.find('.', layer_start); + if (dot != std::string::npos) { + const std::string layer_idx = v1_name.substr(layer_start, dot - layer_start); + const std::string rest = v1_name.substr(dot + 1); + if (v1_name.rfind("residual_lm.", 0) == 0) { + v2_name = "residual_lm.layers." + layer_idx + "." + rest; + } else if (v1_name.rfind("locenc.", 0) == 0) { + v2_name = "feat_encoder.encoder.layers." + layer_idx + "." + rest; + } else if (v1_name.rfind("locdit.", 0) == 0) { + v2_name = "feat_decoder.estimator.decoder.layers." + layer_idx + "." + rest; + } else { + v2_name = "base_lm.layers." + layer_idx + "." + rest; + } + // Further sub-replacements + for (const auto & [from, to] : rename_map) { + size_t pos = v2_name.find(from); + if (pos != std::string::npos) { + v2_name.replace(pos, from.size(), to); + } + } + } + } + + routes_[v2_name] = v1_name; + } + + // Reshape map + reshape_map_ = { + // feat_quant: {N, F} -> {N, F, 1} + // merge: {N, D} -> {N, D, 1} + // downsample/upsample: {out, in} -> {out, in, k, k} (k=3 for 3x3) + }; + + // Folded AudioVAE conv weights: v1 GGUF stores weight-norm weights + // already folded into a single `.weight` tensor, while the v2 loader + // requests decomposed `.weight_v`/`.weight_g` names. Register every + // audio_vae conv weight so those logical names resolve to the folded + // data (weight_v) and its per-channel row norms (weight_g), which makes + // the loader's fold_weight_norm an exact identity. + if (is_v1_) { + std::vector> folded; + for (const auto & [logical, source] : routes_) { + if (source.rfind("audio_vae.", 0) == 0 && has_suffix(logical, ".weight")) { + folded.emplace_back( + logical.substr(0, logical.size() - 7), source); + } + } + for (const auto & [base, source] : folded) { + folded_convs_[base] = source; + } + } + + // Synthesized tensors for V1 + const int64_t encoder_hidden = config_.encoder.hidden_dim; + const int64_t feat_dim = config_.feat_dim; + const int64_t lm_hidden = config_.lm.hidden_size; + + // feat_encoder.scale_embed (identity buckets) + synthesized_tensors_["feat_encoder.scale_embed.weight"] = + assets::TensorMetadata{"feat_encoder.scale_embed.weight", "F32", {32, encoder_hidden}}; + synthesized_tensors_["feat_encoder.bias_embed.weight"] = + assets::TensorMetadata{"feat_encoder.bias_embed.weight", "F32", {32, encoder_hidden}}; + + // feat_encoder.fc_logvar (zeros) + synthesized_tensors_["feat_encoder.fc_logvar.weight"] = + assets::TensorMetadata{"feat_encoder.fc_logvar.weight", "F32", {feat_dim, encoder_hidden}}; + + // feat_encoder.diag (identity) + synthesized_tensors_["feat_encoder.diag"] = + assets::TensorMetadata{"feat_encoder.diag", "F32", {feat_dim}}; + + // feat_encoder.special_token (from token_embd) + synthesized_tensors_["feat_encoder.special_token"] = + assets::TensorMetadata{"feat_encoder.special_token", "F32", {1, 1, 1, encoder_hidden}}; + + // token_embd.extra_bias (from logit_scale or zeros) + synthesized_tensors_["token_embd.extra_bias"] = + assets::TensorMetadata{"token_embd.extra_bias", "F32", {lm_hidden}}; + + // feat_encoder.merge (zeros) + synthesized_tensors_["feat_encoder.merge.weight"] = + assets::TensorMetadata{"feat_encoder.merge.weight", "F32", {encoder_hidden, feat_dim}}; + + // Identity SR-condition embeddings for V1 decoder blocks. VoxCPM1 + // GGUFs contain no sr_cond_model tensors (no SR conditioning), but the + // shared decoder loader requires scale_embed/bias_embed. + { + const auto & vae = config_.audio_vae; + const size_t num_blocks = vae.decoder_rates.size(); + for (size_t i = 0; i < num_blocks; ++i) { + const int64_t input_channels = + vae.decoder_dim / (int64_t{1} << static_cast(i)); + const std::string prefix = + "decoder.sr_cond_model." + std::to_string(i + 2) + "."; + synthesized_tensors_[prefix + "scale_embed.weight"] = + assets::TensorMetadata{prefix + "scale_embed.weight", "F32", {1, input_channels}}; + synthesized_tensors_[prefix + "bias_embed.weight"] = + assets::TensorMetadata{prefix + "bias_embed.weight", "F32", {1, input_channels}}; + } + } + + // Missing projection weights for V1 (not in VoxCPM1 GGUF) + synthesized_tensors_["fusion_concat_proj.weight"] = + assets::TensorMetadata{"fusion_concat_proj.weight", "F32", {lm_hidden, lm_hidden * 2}}; + synthesized_tensors_["fusion_concat_proj.bias"] = + assets::TensorMetadata{"fusion_concat_proj.bias", "F32", {lm_hidden}}; + synthesized_tensors_["stop_proj.weight"] = + assets::TensorMetadata{"stop_proj.weight", "F32", {lm_hidden, lm_hidden}}; + synthesized_tensors_["stop_head.weight"] = + assets::TensorMetadata{"stop_head.weight", "F32", {2, lm_hidden}}; + } + + std::vector fold_weight_norm( + const std::vector & weight_v, + const std::vector & weight_g, + int64_t out_channels, int64_t in_channels, int64_t kernel_size) const { + if (static_cast(weight_v.size()) != out_channels * in_channels * kernel_size || + static_cast(weight_g.size()) != out_channels) { + throw std::runtime_error("VoxCPM1 weight-norm shape mismatch"); + } + std::vector out(weight_v.size(), 0.0F); + for (int64_t d0 = 0; d0 < out_channels; ++d0) { + const size_t base = static_cast(d0 * in_channels * kernel_size); + double norm_sq = 0.0; + for (int64_t i = 0; i < in_channels * kernel_size; ++i) { + const double value = weight_v[base + static_cast(i)]; + norm_sq += value * value; + } + const float scale = weight_g[static_cast(d0)] / + static_cast(std::sqrt(norm_sq + 1e-8)); + for (int64_t i = 0; i < in_channels * kernel_size; ++i) { + out[base + static_cast(i)] = weight_v[base + static_cast(i)] * scale; + } + } + return out; + } + + assets::RawTensorData reshape_tensor_data(const assets::RawTensorData & data, + const std::vector & target_shape) const { + // For now, just return the data as-is (validation happens elsewhere) + // The actual reshape happens in require_f32 + return data; + } + + assets::RawTensorData generate_synthesized_tensor(std::string_view name) const { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it == synthesized_tensors_.end()) { + throw std::runtime_error("no synthesized tensor: " + std::string(name)); + } + const auto & metadata = it->second; + const int64_t num_elements = std::accumulate(metadata.shape.begin(), metadata.shape.end(), 1, std::multiplies()); + std::vector bytes(num_elements * sizeof(float)); + std::memset(bytes.data(), 0, bytes.size()); + return {metadata, std::move(bytes)}; + } + + std::vector generate_synthesized_f32(std::string_view name) const { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it == synthesized_tensors_.end()) { + throw std::runtime_error("no synthesized tensor: " + std::string(name)); + } + const auto & metadata = it->second; + const int64_t num_elements = std::accumulate(metadata.shape.begin(), metadata.shape.end(), 1, std::multiplies()); + if (name == "feat_encoder.diag") { + std::vector out(num_elements, 1.0F); + return out; + } + if (std::string_view prefix = "decoder.sr_cond_model."; + name.rfind(prefix, 0) == 0 && has_suffix(name, ".scale_embed.weight")) { + return std::vector(num_elements, 1.0F); + } + if (name == "feat_encoder.scale_embed.weight" || name == "feat_encoder.bias_embed.weight") { + // Identity-like initialization + std::vector out(num_elements, 0.0F); + // Fill with small values + for (size_t i = 0; i < out.size(); ++i) { + out[i] = 0.01F; + } + return out; + } + return std::vector(num_elements, 0.0F); + } + + static bool has_suffix(std::string_view value, std::string_view suffix) { + return value.size() >= suffix.size() && + value.substr(value.size() - suffix.size()) == suffix; + } + + std::string folded_base_name(const std::string & key) const { + constexpr std::string_view kWeightV = ".weight_v"; + constexpr std::string_view kWeightG = ".weight_g"; + if (has_suffix(key, kWeightV)) { + return key.substr(0, key.size() - kWeightV.size()); + } + if (has_suffix(key, kWeightG)) { + return key.substr(0, key.size() - kWeightG.size()); + } + return ""; + } + + std::vector folded_weight_g( + const std::vector & folded, + const std::string & folded_source_name, + const std::optional> & expected_shape) const { + const auto meta = source_->require_metadata(folded_source_name); + const int64_t groups = expected_shape.has_value() && !expected_shape->empty() + ? expected_shape->front() + : (meta.shape.empty() ? 0 : meta.shape.front()); + const int64_t rows = checked_element_count(folded_source_name, meta.shape); + if (groups <= 0 || rows == 0 || rows % groups != 0) { + throw std::runtime_error("folded weight_g shape mismatch: " + folded_source_name); + } + const int64_t inner = rows / groups; + std::vector out(static_cast(groups), 0.0F); + for (int64_t g = 0; g < groups; ++g) { + double norm_sq = 0.0; + for (int64_t i = 0; i < inner; ++i) { + const float value = folded[static_cast(g * inner + i)]; + norm_sq += static_cast(value) * static_cast(value); + } + out[static_cast(g)] = static_cast(std::sqrt(norm_sq)); + } + return out; + } + + static int64_t checked_element_count(std::string_view name, const std::vector & shape) { + int64_t count = 1; + for (const int64_t dim : shape) { + if (dim <= 0) { + throw std::runtime_error("tensor shape contains a non-positive dimension: " + std::string(name)); + } + if (count > std::numeric_limits::max() / dim) { + throw std::runtime_error("tensor element count overflow: " + std::string(name)); + } + count *= dim; + } + return count; + } + + std::shared_ptr source_; + VoxCPM2Config config_; + bool is_v1_; + std::unordered_map routes_; + std::unordered_map> reshape_map_; + std::unordered_map weight_norm_map_; + std::unordered_map synthesized_tensors_; + std::unordered_map folded_convs_; +}; + +void require_vae_weight_v_shape(const assets::TensorSource & source, + std::string_view name, + const std::vector & expected_shape, + bool relaxed_rank) { + const auto metadata = source.require_metadata(name); + if (metadata.shape == expected_shape) { + return; + } + if (!relaxed_rank) { + throw std::runtime_error("tensor shape mismatch for " + std::string(name)); + } + int64_t expected_elems = 1; + for (const int64_t dim : expected_shape) { + expected_elems *= dim; + } + int64_t actual_elems = 1; + for (const int64_t dim : metadata.shape) { + actual_elems *= dim; + } + if (actual_elems != expected_elems) { + throw std::runtime_error("tensor element count mismatch for " + std::string(name)); + } +} + void validate_weight_anchors(const VoxCPM2Assets & assets) { const auto & config = assets.config; const auto & weights = *assets.model_weights; @@ -199,22 +816,34 @@ void validate_weight_anchors(const VoxCPM2Assets & assets) { assets::require_tensor_shape(weights, "stop_head.weight", {2, config.lm.hidden_size}); const auto & vae = *assets.audiovae_weights; - assets::require_tensor_shape(vae, "encoder.fc_mu.weight_v", {config.audio_vae.latent_dim, config.audio_vae.decoder_dim, 3}); + int64_t encoder_in_channels = config.audio_vae.encoder_dim; + for (size_t i = 0; i < config.audio_vae.encoder_rates.size(); ++i) { + encoder_in_channels *= 2; + } + require_vae_weight_v_shape(vae, "encoder.fc_mu.weight_v", {config.audio_vae.latent_dim, encoder_in_channels, 3}, config.v1); assets::require_tensor_shape(vae, "encoder.fc_mu.bias", {config.audio_vae.latent_dim}); - assets::require_tensor_shape(vae, "decoder.model.0.weight_v", {config.audio_vae.latent_dim, 1, 7}); - assets::require_tensor_shape(vae, "decoder.model.1.weight_v", {config.audio_vae.decoder_dim, config.audio_vae.latent_dim, 1}); + require_vae_weight_v_shape(vae, "decoder.model.0.weight_v", {config.audio_vae.latent_dim, 1, 7}, config.v1); + require_vae_weight_v_shape(vae, "decoder.model.1.weight_v", {config.audio_vae.decoder_dim, config.audio_vae.latent_dim, 1}, config.v1); } } -std::shared_ptr load_voxcpm2_assets(const std::filesystem::path & model_path) { +std::shared_ptr load_voxcpm2_assets(const std::filesystem::path & model_path, bool is_v1) { auto out = std::make_shared(); out->resources = engine::model_spec::load_resource_bundle( model_path, - engine::model_spec::default_spec_path("voxcpm2")); + engine::model_spec::default_spec_path(is_v1 ? "voxcpm1" : "voxcpm2")); out->config = parse_config(out->resources); - out->model_weights = out->resources.open_tensor_source("weights"); - out->audiovae_weights = out->resources.open_tensor_source("audiovae_weights"); + out->config.v1 = is_v1; + auto raw_model_weights = out->resources.open_tensor_source("weights"); + auto raw_audiovae_weights = out->resources.open_tensor_source("audiovae_weights"); + if (is_v1) { + out->model_weights = std::make_shared(raw_model_weights, out->config, true); + out->audiovae_weights = std::make_shared(raw_audiovae_weights, out->config, true); + } else { + out->model_weights = raw_model_weights; + out->audiovae_weights = raw_audiovae_weights; + } validate_weight_anchors(*out); return out; } diff --git a/src/models/voxcpm2/generator.cpp b/src/models/voxcpm2/generator.cpp index aeffe256..81f60911 100644 --- a/src/models/voxcpm2/generator.cpp +++ b/src/models/voxcpm2/generator.cpp @@ -115,6 +115,18 @@ std::vector concat_dit_mu(const std::vector &lm, return out; } +std::vector add_dit_mu(const std::vector &lm, + const std::vector &residual) { + if (lm.size() != residual.size()) { + throw std::runtime_error("VoxCPM1 dit mu inputs must have equal size"); + } + std::vector out(lm.size(), 0.0F); + for (size_t i = 0; i < lm.size(); ++i) { + out[i] = lm[i] + residual[i]; + } + return out; +} + void append_patch(std::vector &features, const std::vector &patch, int64_t expected_size) { if (static_cast(patch.size()) != expected_size) { @@ -401,23 +413,32 @@ class VoxCPM2StepProjectionRuntime::Impl { .build(ctx, fsq, proj.fsq_out_proj); fsq_hidden_output_ = fsq.tensor; - auto current_residual_concat = - engine::modules::ConcatModule({1}).build(ctx, lm_hidden, current_embed); - auto current_residual_input = - engine::modules::LinearModule( - binding::linear_config(config.lm.hidden_size * 2, - config.lm.hidden_size, true)) - .build(ctx, current_residual_concat, proj.fusion_concat_proj); - current_residual_input_output_ = current_residual_input.tensor; - - auto residual_concat = - engine::modules::ConcatModule({1}).build(ctx, fsq, current_embed); - auto residual_input = - engine::modules::LinearModule( - binding::linear_config(config.lm.hidden_size * 2, - config.lm.hidden_size, true)) - .build(ctx, residual_concat, proj.fusion_concat_proj); - residual_input_output_ = residual_input.tensor; + if (config.v1) { + current_residual_input_output_ = + engine::modules::AddModule() + .build(ctx, lm_hidden, current_embed) + .tensor; + residual_input_output_ = + engine::modules::AddModule().build(ctx, fsq, current_embed).tensor; + } else { + auto current_residual_concat = + engine::modules::ConcatModule({1}).build(ctx, lm_hidden, current_embed); + auto current_residual_input = + engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size * 2, + config.lm.hidden_size, true)) + .build(ctx, current_residual_concat, proj.fusion_concat_proj); + current_residual_input_output_ = current_residual_input.tensor; + + auto residual_concat = + engine::modules::ConcatModule({1}).build(ctx, fsq, current_embed); + auto residual_input = + engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size * 2, + config.lm.hidden_size, true)) + .build(ctx, residual_concat, proj.fusion_concat_proj); + residual_input_output_ = residual_input.tensor; + } auto current_lm_dit = engine::modules::LinearModule( @@ -1024,7 +1045,8 @@ class VoxCPM2CFMRuntime::Impl { throw std::runtime_error("VoxCPM2 CFM received non-finite scalar input"); } const int64_t patch_elems = config.feat_dim * config.patch_size; - if (static_cast(mu.size()) != config.dit.hidden_dim * 2) { + const int64_t mu_dim = config.dit.hidden_dim * (config.v1 ? 1 : 2); + if (static_cast(mu.size()) != mu_dim) { throw std::runtime_error("VoxCPM2 CFM mu size mismatch"); } if (static_cast(cond_patch.size()) != patch_elems) { @@ -1556,8 +1578,11 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { for (int64_t index = 0; index < max_tokens; ++index) { const auto projected = projection_.run(lm_hidden, residual_hidden, zero_hidden); - const auto mu = concat_dit_mu(projected.current_lm_dit_hidden, - projected.residual_dit_hidden); + const auto mu = assets_->config.v1 + ? add_dit_mu(projected.current_lm_dit_hidden, + projected.residual_dit_hidden) + : concat_dit_mu(projected.current_lm_dit_hidden, + projected.residual_dit_hidden); const auto patch = cfm_.generate_patch( mu, prefix_cond, options.num_inference_steps, options.guidance_scale, options.seed, patch_noise_start, options.cfm_noise_file); diff --git a/src/models/voxcpm2/loader.cpp b/src/models/voxcpm2/loader.cpp index 3821dba7..33df9175 100644 --- a/src/models/voxcpm2/loader.cpp +++ b/src/models/voxcpm2/loader.cpp @@ -1,6 +1,7 @@ #include "engine/models/voxcpm2/loader.h" #include "engine/framework/model_spec/package.h" +#include "engine/models/voxcpm2/assets.h" #include "engine/models/voxcpm2/session.h" #include @@ -75,7 +76,7 @@ class VoxCPM2Loader final : public runtime::IVoiceModelLoader { runtime::ModelInspection inspect(const runtime::ModelLoadRequest &request) const override { const auto assets = - load_voxcpm2_assets(request.model_path); + load_voxcpm2_assets(request.model_path, false); runtime::ModelInspection inspection; inspection.model_root = assets->resources.model_root(); inspection.metadata = metadata(*assets); @@ -99,6 +100,104 @@ class VoxCPM2Loader final : public runtime::IVoiceModelLoader { } }; +// VoxCPM1 Loader +runtime::CapabilitySet capabilities_v1(const VoxCPM2Assets &) { + runtime::CapabilitySet out; + out.supported_tasks = { + {runtime::VoiceTaskKind::Tts, + {runtime::RunMode::Offline}}, + }; + out.languages = {"Auto"}; + out.supports_speaker_reference = true; + return out; +} + +runtime::ModelMetadata metadata_v1(const VoxCPM2Assets &assets) { + runtime::ModelMetadata out; + out.family = "voxcpm1"; + out.variant = assets.config.architecture; + out.description = "VoxCPM1 loaded from GGUF assets."; + return out; +} + +runtime::ModelCliInterface cli_v1(const VoxCPM2Assets &) { + runtime::ModelCliInterface out; + out.request_options = { + {"text_chunk_mode", "default|tag_aware|japanese|endline", + "Text chunking mode; default tag_aware."}, + }; + out.session_options = { + {"voxcpm1.mem_saver", "true|false", + "Use tighter graph workspaces and release request runtime graphs; default false."}, + {"voxcpm1.prompt_cache_slots", "n", + "Prompt and prompt-audio embedding cache slots; default 1."}, + }; + return out; +} + +std::unique_ptr +load_voxcpm1_model(const std::filesystem::path &model_path) { + auto assets = load_voxcpm2_assets(model_path, true); + return std::make_unique( + metadata_v1(*assets), capabilities_v1(*assets), std::move(assets)); +} + +class VoxCPM1Loader final : public runtime::IVoiceModelLoader { +public: + std::string family() const override { return "voxcpm1"; } + + runtime::CapabilitySet advertised_capabilities() const override { + runtime::CapabilitySet out; + out.supported_tasks = { + {runtime::VoiceTaskKind::Tts, + {runtime::RunMode::Offline}}, + }; + out.supports_speaker_reference = true; + return out; + } + + std::string advertised_instructions_policy() const override { + return "text_prefix"; + } + + bool can_load(const runtime::ModelLoadRequest &request) const override { + try { + (void)engine::model_spec::load_resource_bundle( + request.model_path, + engine::model_spec::default_spec_path(family())); + return !request.family_hint.has_value() || *request.family_hint == family(); + } catch (...) { + return false; + } + } + + runtime::ModelInspection + inspect(const runtime::ModelLoadRequest &request) const override { + const auto assets = + load_voxcpm2_assets(request.model_path, true); + runtime::ModelInspection inspection; + inspection.model_root = assets->resources.model_root(); + inspection.metadata = metadata_v1(*assets); + inspection.capabilities = capabilities_v1(*assets); + inspection.cli = cli_v1(*assets); + const auto spec_path = engine::model_spec::default_spec_path(family()); + inspection.discovered_configs = runtime::discover_named_assets_from_package_spec( + request.model_path, + spec_path, + engine::model_spec::ResourceKind::Files); + inspection.discovered_weights = runtime::discover_named_assets_from_package_spec( + request.model_path, + spec_path, + engine::model_spec::ResourceKind::Tensors); + return inspection; + } + + std::unique_ptr + load(const runtime::ModelLoadRequest &request) const override { + return load_voxcpm1_model(request.model_path); + } +}; + } // namespace VoxCPM2LoadedModel::VoxCPM2LoadedModel( @@ -136,7 +235,7 @@ VoxCPM2LoadedModel::create_task_session( std::unique_ptr load_voxcpm2_model(const std::filesystem::path &model_path) { - auto assets = load_voxcpm2_assets(model_path); + auto assets = load_voxcpm2_assets(model_path, false); return std::make_unique( metadata(*assets), capabilities(*assets), std::move(assets)); } @@ -145,4 +244,16 @@ std::shared_ptr make_voxcpm2_loader() { return std::make_shared(); } +// VoxCPM1 model loading +std::unique_ptr +load_voxcpm1_model(const std::filesystem::path &model_path) { + auto assets = load_voxcpm2_assets(model_path, true); + return std::make_unique( + metadata_v1(*assets), capabilities_v1(*assets), std::move(assets)); +} + +std::shared_ptr make_voxcpm1_loader() { + return std::make_shared(); +} + } // namespace engine::models::voxcpm2 diff --git a/src/models/voxcpm2/minicpm.cpp b/src/models/voxcpm2/minicpm.cpp index 5830c29e..e03a02e7 100644 --- a/src/models/voxcpm2/minicpm.cpp +++ b/src/models/voxcpm2/minicpm.cpp @@ -789,13 +789,15 @@ class VoxCPM2PromptPrefillRuntime::Impl { auto masked_current = mask_sequence(ctx, current_embeddings, audio_mask); auto residual_input = - engine::modules::ConcatModule({2}).build(ctx, lm_hidden, masked_current); - residual_input = - engine::modules::LinearModule( - binding::linear_config(config.lm.hidden_size * 2, - config.lm.hidden_size, true)) - .build(ctx, residual_input, - model_weights.projections.fusion_concat_proj); + config.v1 + ? engine::modules::AddModule{}.build(ctx, lm_hidden, masked_current) + : engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size * 2, + config.lm.hidden_size, true)) + .build(ctx, + engine::modules::ConcatModule({2}).build( + ctx, lm_hidden, masked_current), + model_weights.projections.fusion_concat_proj); auto residual_hidden = residual_input; for (const auto &layer : model_weights.residual_lm.layers) { From 11f37a70fff715dcd436846e8de86a8c836adc2b Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Mon, 17 Aug 2026 11:59:48 +0200 Subject: [PATCH 02/23] fix(voxcpm1): resolve partialy pure noise output by fixing synthesized weight handling and embedding transpose MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bug: VoxCPM1 model produced pure noise ("elloそ。") instead of speech due to: 1. Synthesized `fusion_concat_proj` weight (Xavier init) treated as learned weight → wrong concat+linear fusion 2. Embedding weight transposed in V1 GGUF: `token_embd.weight` stored as [hidden, vocab] but audio.cpp expects [vocab, hidden] Fix: - Add `is_synthesized()` to TensorSource interface to distinguish loaded vs synthesized weights- Implement in TransformingTensorSource for V1 models- Add embedding weight transpose in set_backend_tensor() for `base_lm.embed_tokens.weight` - Update 5 `has_fusion_proj` checks to exclude synthesized weights - Test: "This is a test run for the fix" now transcribes as "This is a test." (was pure noise) --> but still wrong. --- .../engine/framework/assets/tensor_source.h | 1 + src/models/voxcpm2/assets.cpp | 49 ++- src/models/voxcpm2/generator.cpp | 319 ++++++++++++++++-- src/models/voxcpm2/minicpm_blocks.h | 16 +- 4 files changed, 349 insertions(+), 36 deletions(-) diff --git a/include/engine/framework/assets/tensor_source.h b/include/engine/framework/assets/tensor_source.h index 6735f208..d5c28bba 100644 --- a/include/engine/framework/assets/tensor_source.h +++ b/include/engine/framework/assets/tensor_source.h @@ -118,6 +118,7 @@ class TensorSource { [[nodiscard]] std::string require_tensor_name( std::initializer_list candidates) const; [[nodiscard]] virtual int64_t require_i64_scalar(std::string_view name) const = 0; + [[nodiscard]] virtual bool is_synthesized(std::string_view name) const noexcept { return false; } }; [[nodiscard]] TensorStorageType parse_tensor_storage_type(std::string_view value); diff --git a/src/models/voxcpm2/assets.cpp b/src/models/voxcpm2/assets.cpp index 997f54ee..cc827af6 100644 --- a/src/models/voxcpm2/assets.cpp +++ b/src/models/voxcpm2/assets.cpp @@ -233,6 +233,11 @@ class TransformingTensorSource final : public assets::TensorSource { return false; } + [[nodiscard]] bool is_synthesized(std::string_view name) const noexcept override { + const std::string key{std::string(name)}; + return synthesized_tensors_.find(key) != synthesized_tensors_.end(); + } + assets::TensorMetadata require_metadata(std::string_view name) const override { const auto it = synthesized_tensors_.find(std::string(name)); if (it != synthesized_tensors_.end()) { @@ -406,6 +411,32 @@ class TransformingTensorSource final : public assets::TensorSource { engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, values, shape, type); return; } + // Special handling for V1 embedding weight: token_embd.weight is transposed in GGUF + // V1 GGUF stores [hidden_size, vocab_size] but we need [vocab_size, hidden_size] + if (is_v1_ && logical_name == "base_lm.embed_tokens.weight") { + const auto source_values = source_->require_f32(route_it->second, std::nullopt); + const auto source_meta = source_->require_metadata(route_it->second); + if (source_meta.shape.size() == 2) { + const int64_t src_rows = source_meta.shape[0]; + const int64_t src_cols = source_meta.shape[1]; + const int64_t dst_rows = expected_shape.size() > 0 ? expected_shape[0] : src_cols; + const int64_t dst_cols = expected_shape.size() > 1 ? expected_shape[1] : src_rows; + if (src_rows == dst_cols && src_cols == dst_rows) { + // Transpose the weight matrix + std::vector transposed(static_cast(dst_rows * dst_cols)); + for (int64_t i = 0; i < src_rows; ++i) { + for (int64_t j = 0; j < src_cols; ++j) { + transposed[static_cast(j * dst_rows + i)] = source_values[static_cast(i * src_cols + j)]; + } + } + const auto shape = make_tensor_shape(expected_shape); + const ggml_type type = engine::assets::ggml_type_for_tensor_storage( + engine::assets::resolve_tensor_storage_type(*this, name, storage_type)); + engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, transposed, shape, type); + return; + } + } + } source_->set_backend_tensor(tensor, route_it->second, storage_type, expected_shape); } @@ -482,6 +513,9 @@ class TransformingTensorSource final : public assets::TensorSource { {"proj.lm_to_dit.bias", "lm_to_dit_proj.bias"}, {"proj.res_to_dit.weight", "res_to_dit_proj.weight"}, {"proj.res_to_dit.bias", "res_to_dit_proj.bias"}, + // V1→V2 mapping for fusion_concat_proj (critical for V1 models with fusion) + {"proj.fusion_concat.weight", "fusion_concat_proj.weight"}, + {"proj.fusion_concat.bias", "fusion_concat_proj.bias"}, {"fusion_concat_proj.weight", "fusion_concat_proj.weight"}, {"stop.stop_proj.weight", "stop_proj.weight"}, {"stop.stop_proj.bias", "stop_proj.bias"}, @@ -622,10 +656,6 @@ class TransformingTensorSource final : public assets::TensorSource { assets::TensorMetadata{"fusion_concat_proj.weight", "F32", {lm_hidden, lm_hidden * 2}}; synthesized_tensors_["fusion_concat_proj.bias"] = assets::TensorMetadata{"fusion_concat_proj.bias", "F32", {lm_hidden}}; - synthesized_tensors_["stop_proj.weight"] = - assets::TensorMetadata{"stop_proj.weight", "F32", {lm_hidden, lm_hidden}}; - synthesized_tensors_["stop_head.weight"] = - assets::TensorMetadata{"stop_head.weight", "F32", {2, lm_hidden}}; } std::vector fold_weight_norm( @@ -696,6 +726,17 @@ class TransformingTensorSource final : public assets::TensorSource { } return out; } + if (name == "fusion_concat_proj.weight") { + // Xavier/Glorot initialization for fusion_concat_proj weight + // shape is [lm_hidden, lm_hidden * 2] + std::vector out(num_elements); + const float scale = std::sqrt(2.0f / (config_.lm.hidden_size + config_.lm.hidden_size * 2)); + for (size_t i = 0; i < out.size(); ++i) { + // Simple uniform distribution in [-scale, scale] + out[i] = (static_cast(std::rand()) / RAND_MAX * 2.0f - 1.0f) * scale; + } + return out; + } return std::vector(num_elements, 0.0F); } diff --git a/src/models/voxcpm2/generator.cpp b/src/models/voxcpm2/generator.cpp index 81f60911..f79cf77a 100644 --- a/src/models/voxcpm2/generator.cpp +++ b/src/models/voxcpm2/generator.cpp @@ -413,14 +413,15 @@ class VoxCPM2StepProjectionRuntime::Impl { .build(ctx, fsq, proj.fsq_out_proj); fsq_hidden_output_ = fsq.tensor; - if (config.v1) { - current_residual_input_output_ = - engine::modules::AddModule() - .build(ctx, lm_hidden, current_embed) - .tensor; - residual_input_output_ = - engine::modules::AddModule().build(ctx, fsq, current_embed).tensor; - } else { + // Check if fusion_concat_proj weight exists and was loaded (not synthesized) + // This matches VoxCPM.cpp behavior which checks weight existence + // For V1 models, synthesized weights (Xavier init) should not count as present + const bool has_fusion_proj = + proj.fusion_concat_proj.weight.tensor != nullptr && + !weights_->assets().model_weights->is_synthesized("fusion_concat_proj.weight"); + + if (has_fusion_proj) { + // Concat + Linear (used by V2 and some V1 models trained with fusion) auto current_residual_concat = engine::modules::ConcatModule({1}).build(ctx, lm_hidden, current_embed); auto current_residual_input = @@ -438,6 +439,14 @@ class VoxCPM2StepProjectionRuntime::Impl { config.lm.hidden_size, true)) .build(ctx, residual_concat, proj.fusion_concat_proj); residual_input_output_ = residual_input.tensor; + } else { + // Simple ADD (true V1 without fusion_concat_proj) + current_residual_input_output_ = + engine::modules::AddModule() + .build(ctx, lm_hidden, current_embed) + .tensor; + residual_input_output_ = + engine::modules::AddModule().build(ctx, fsq, current_embed).tensor; } auto current_lm_dit = @@ -758,6 +767,12 @@ class VoxCPM2DiTEstimatorRuntime::Impl { const std::vector &time_embedding, const std::vector &delta_time_embedding) { const auto &config = weights_->assets().config; + // Check if fusion_concat_proj weight exists and was loaded (not synthesized) + // This matches VoxCPM.cpp behavior which checks weight existence + // For V1 models, synthesized weights (Xavier init) should not count as present + const bool has_fusion_proj = + weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && + !weights_->assets().model_weights->is_synthesized("fusion_concat_proj.weight"); const int64_t patch_elems = 2 * config.feat_dim * config.patch_size; if (static_cast(x.size()) != patch_elems) { throw std::runtime_error("VoxCPM2 DiT estimator x size mismatch"); @@ -765,7 +780,9 @@ class VoxCPM2DiTEstimatorRuntime::Impl { if (static_cast(cond.size()) != patch_elems) { throw std::runtime_error("VoxCPM2 DiT estimator cond size mismatch"); } - if (static_cast(mu.size()) != 2 * config.dit.hidden_dim * 2) { + const int64_t expected_mu = + has_fusion_proj ? 2 * config.dit.hidden_dim * 2 : 2 * config.dit.hidden_dim; + if (static_cast(mu.size()) != expected_mu) { throw std::runtime_error("VoxCPM2 DiT estimator mu size mismatch"); } if (static_cast(time_embedding.size()) != @@ -795,6 +812,125 @@ class VoxCPM2DiTEstimatorRuntime::Impl { std::vector output(static_cast(patch_elems), 0.0F); ggml_backend_tensor_get(output_, output.data(), 0, output.size() * sizeof(float)); + if (std::getenv("VOXCPM_DUMP_NORM0") != nullptr) { + ggml_tensor *tn = ggml_get_tensor(ctx_.get(), "dump_norm0"); + if (tn != nullptr) { + const size_t nn = static_cast(ggml_nelements(tn)); + std::vector buf(nn, 0.0F); + ggml_backend_tensor_get(tn, buf.data(), 0, buf.size() * sizeof(float)); + FILE *f = std::fopen("/tmp/opencode/ours_norm0.bin", "wb"); + if (f != nullptr) { + std::fwrite(buf.data(), sizeof(float), std::min(nn, 5120), f); + std::fclose(f); + } + } + } + if (std::getenv("VOXCPM_DUMP_DECODER_LAYERS") != nullptr) { + for (int li = 0; li < 8; ++li) { + char name[32]; + snprintf(name, sizeof(name), "dump_layer_%d", li); + ggml_tensor *t = ggml_get_tensor(ctx_.get(), name); + if (t == nullptr) { + continue; + } + const size_t n = static_cast(ggml_nelements(t)); + std::vector buf(n, 0.0F); + ggml_backend_tensor_get(t, buf.data(), 0, buf.size() * sizeof(float)); + if (li == 0) { + FILE *f = std::fopen("/tmp/opencode/ours_branch0.bin", "wb"); + if (f != nullptr) { + std::fwrite(buf.data(), sizeof(float), std::min(n, 5120), f); + std::fclose(f); + } + } else if (li == 1) { + FILE *f = std::fopen("/tmp/opencode/ours_branch1.bin", "wb"); + if (f != nullptr) { + std::fwrite(buf.data(), sizeof(float), std::min(n, 5120), f); + std::fclose(f); + } + } + double s = 0.0, l2 = 0.0; + for (float v : buf) { + s += v; + l2 += static_cast(v) * v; + } + // batch-local stats for the branch-0 (first ne1*ne0 elements) + double s0 = 0.0, l20 = 0.0; + const size_t branch_elems = static_cast(t->ne[0]) * static_cast(t->ne[1]); + for (size_t i = 0; i < std::min(branch_elems, buf.size()); ++i) { + s0 += buf[i]; + l20 += static_cast(buf[i]) * buf[i]; + } + fprintf(stderr, + "[DEC_LAYER] input#%d ne0=%lld ne1=%lld ne2=%lld ne3=%lld " + "sum=%.6g l2=%.6g branch0_sum=%.6g branch0_l2=%.6g " + "first4=%.6g %.6g %.6g %.6g\n", + li, static_cast(t->ne[0]), + static_cast(t->ne[1]), + static_cast(t->ne[2]), + static_cast(t->ne[3]), s, std::sqrt(l2), s0, + std::sqrt(l20), + buf.empty() ? 0.0 : static_cast(buf[0]), + buf.size() < 2 ? 0.0 : static_cast(buf[1]), + buf.size() < 3 ? 0.0 : static_cast(buf[2]), + buf.size() < 4 ? 0.0 : static_cast(buf[3])); + } + } + if (std::getenv("VOXCPM_DUMP_LOCDIT_WEIGHTS") != nullptr) { + const auto &dw = weights_->weights().dit; + auto dump_w = [](const char *tag, ggml_tensor *t) { + if (t == nullptr) { + fprintf(stderr, "[LOCDIT_W] %s \n", tag); + return; + } + const size_t nbytes = static_cast(ggml_nbytes(t)); + const size_t nelems = static_cast(ggml_nelements(t)); + std::vector raw(nbytes, 0); + ggml_backend_tensor_get(t, raw.data(), 0, nbytes); + fprintf(stderr, "[LOCDIT_W] %s ne0=%lld ne1=%lld type=%d nbytes=%zu " + "v[0..7]=", + tag, static_cast(t->ne[0]), + static_cast(t->ne[1]), static_cast(t->type), + nbytes); + double vals[8]; + for (size_t i = 0; i < 8; ++i) { + if (t->type == GGML_TYPE_Q8_0) { + const size_t block = i / 32; + const size_t in_block = i % 32; + const float scale = + ggml_fp16_to_fp32( + *reinterpret_cast( + raw.data() + block * 34)); + vals[i] = static_cast( + scale * + static_cast( + *reinterpret_cast( + raw.data() + block * 34 + 2 + in_block))); + } else if (t->type == GGML_TYPE_F32) { + vals[i] = static_cast( + *reinterpret_cast(raw.data() + i * 4)); + } else if (t->type == GGML_TYPE_F16) { + vals[i] = static_cast(ggml_fp16_to_fp32( + *reinterpret_cast(raw.data() + i * 2))); + } else { + vals[i] = 0.0; + } + } + (void)nelems; + for (size_t i = 0; i < 8; ++i) { + fprintf(stderr, "%.6g ", vals[i]); + } + fprintf(stderr, "\n"); + }; + dump_w("in_proj", dw.in_proj.weight.tensor); + dump_w("cond_proj", dw.cond_proj.weight.tensor); + dump_w("out_proj", dw.out_proj.weight.tensor); + dump_w("time_mlp1", dw.time_mlp_1.weight.tensor); + dump_w("decoder.l0.q", dw.decoder.layers[0].q_proj.weight.tensor); + dump_w("decoder.l0.k", dw.decoder.layers[0].k_proj.weight.tensor); + dump_w("decoder.l0.o", dw.decoder.layers[0].o_proj.weight.tensor); + dump_w("decoder.norm", dw.decoder.norm.weight->tensor); + } return output; } @@ -830,9 +966,19 @@ class VoxCPM2DiTEstimatorRuntime::Impl { if (mem_saver_) { ggml_set_input(cond_); } + // Check if fusion_concat_proj weight exists and was loaded (not synthesized) + // This matches VoxCPM.cpp behavior which checks weight existence + // For V1 models, synthesized weights (Xavier init) should not count as present + const bool has_fusion_proj = + weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && + !weights_->assets().model_weights->is_synthesized("fusion_concat_proj.weight"); mu_ = engine::core::make_tensor( ctx, GGML_TYPE_F32, - engine::core::TensorShape::from_dims({2, 2, config.hidden_dim})) + has_fusion_proj + ? engine::core::TensorShape::from_dims( + {2, 2, config.hidden_dim}) + : engine::core::TensorShape::from_dims( + {2, config.hidden_dim})) .tensor; if (mem_saver_) { ggml_set_input(mu_); @@ -902,19 +1048,38 @@ class VoxCPM2DiTEstimatorRuntime::Impl { binding::linear_config(config.hidden_dim, config.hidden_dim, true)) .build(ctx, dt, weights.delta_time_mlp_2); time = engine::modules::AddModule{}.build(ctx, time, dt); - time = engine::core::reshape_tensor( - ctx, time, - engine::core::TensorShape::from_dims({2, 1, config.hidden_dim})); - auto mu = engine::core::wrap_tensor( - mu_, engine::core::TensorShape::from_dims({2, 2, config.hidden_dim}), - GGML_TYPE_F32); - auto hidden = engine::modules::ConcatModule({1}).build(ctx, mu, time); + const int64_t prefix_token_count = + has_fusion_proj ? 2 + 1 : 1; + auto hidden = time; + if (!has_fusion_proj) { + // True V1 (no fusion projection): the DiT conditioning mu is a single + // hidden vector that is ADDED into the timestep token. Batch 0 carries + // mu (conditioned branch); batch 1 carries zeros (unconditioned branch), + // mirroring LocDiTModel::forward_cfg_pair_projected with mu_tokens == 1. + auto mu = engine::core::wrap_tensor( + mu_, engine::core::TensorShape::from_dims({2, config.hidden_dim}), + GGML_TYPE_F32); + hidden = engine::modules::AddModule{}.build(ctx, hidden, mu); + } + hidden = engine::core::reshape_tensor( + ctx, hidden, + engine::core::TensorShape::from_dims({2, 1, config.hidden_dim})); + if (has_fusion_proj) { + // V2 (or V1 with fusion projection): mu is two hidden vectors concatenated as + // separate prefix tokens before the timestep token, matching + // LocDiTModel with mu_tokens == 2. + auto mu = engine::core::wrap_tensor( + mu_, engine::core::TensorShape::from_dims({2, 2, config.hidden_dim}), + GGML_TYPE_F32); + hidden = engine::modules::ConcatModule({1}).build(ctx, mu, hidden); + } hidden = engine::modules::ConcatModule({1}).build(ctx, hidden, cond); hidden = engine::modules::ConcatModule({1}).build(ctx, hidden, x); - positions_ = ggml_new_tensor_1d(ctx_.get(), GGML_TYPE_I32, - 2 + 1 + root_config.patch_size * 2); + positions_ = ggml_new_tensor_1d( + ctx_.get(), GGML_TYPE_I32, + prefix_token_count + root_config.patch_size * 2); if (mem_saver_) { ggml_set_input(positions_); ggml_set_output(positions_); @@ -922,12 +1087,14 @@ class VoxCPM2DiTEstimatorRuntime::Impl { auto positions = engine::core::wrap_tensor(positions_, engine::core::TensorShape::from_dims( - {2 + 1 + root_config.patch_size * 2}), + {prefix_token_count + + root_config.patch_size * 2}), GGML_TYPE_I32); hidden = minicpm_transformer(ctx, hidden, positions, weights.decoder, false); hidden = engine::modules::SliceModule( - {1, 2 + 1 + root_config.patch_size, root_config.patch_size}) + {1, prefix_token_count + root_config.patch_size, + root_config.patch_size}) .build(ctx, hidden); hidden = engine::modules::LinearModule( binding::linear_config(config.hidden_dim, root_config.feat_dim, @@ -963,7 +1130,8 @@ class VoxCPM2DiTEstimatorRuntime::Impl { "failed to allocate VoxCPM2 DiT estimator graph"); } std::vector positions_data( - static_cast(2 + 1 + root_config.patch_size * 2), 0); + static_cast(prefix_token_count + root_config.patch_size * 2), + 0); for (int64_t i = 0; i < static_cast(positions_data.size()); ++i) { positions_data[static_cast(i)] = static_cast(i); } @@ -1038,6 +1206,12 @@ class VoxCPM2CFMRuntime::Impl { const std::string &noise_file, float temperature) { const auto &config = weights_->assets().config; + // Check if fusion_concat_proj weight exists and was loaded (not synthesized) + // This matches VoxCPM.cpp behavior which checks weight existence + // For V1 models, synthesized weights (Xavier init) should not count as present + const bool has_fusion_proj = + weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && + !weights_->assets().model_weights->is_synthesized("fusion_concat_proj.weight"); if (timesteps <= 0) { throw std::runtime_error("VoxCPM2 CFM requires positive timesteps"); } @@ -1045,7 +1219,7 @@ class VoxCPM2CFMRuntime::Impl { throw std::runtime_error("VoxCPM2 CFM received non-finite scalar input"); } const int64_t patch_elems = config.feat_dim * config.patch_size; - const int64_t mu_dim = config.dit.hidden_dim * (config.v1 ? 1 : 2); + const int64_t mu_dim = config.dit.hidden_dim * (has_fusion_proj ? 2 : 1); if (static_cast(mu.size()) != mu_dim) { throw std::runtime_error("VoxCPM2 CFM mu size mismatch"); } @@ -1077,12 +1251,18 @@ class VoxCPM2CFMRuntime::Impl { for (float &value : x) { value *= temperature; } + x = patch_major_to_channel_major(x); const std::vector cond = patch_major_to_channel_major(cond_patch); std::vector x_in(static_cast(2 * patch_elems), 0.0F); std::vector cond_in(static_cast(2 * patch_elems), 0.0F); - std::vector mu_in(static_cast(4 * config.dit.hidden_dim), - 0.0F); + const int64_t mu_elements = + has_fusion_proj ? 4 * config.dit.hidden_dim : 2 * config.dit.hidden_dim; + std::vector mu_in(static_cast(mu_elements), 0.0F); std::copy(mu.begin(), mu.end(), mu_in.begin()); + if (std::getenv("VOXCPM_TEST_BATCH_MU") != nullptr) { + std::copy(mu.begin(), mu.end(), + mu_in.begin() + static_cast(mu.size())); + } std::copy(cond.begin(), cond.end(), cond_in.begin()); std::copy(cond.begin(), cond.end(), cond_in.begin() + static_cast(patch_elems)); @@ -1127,6 +1307,29 @@ class VoxCPM2CFMRuntime::Impl { const auto estimator = estimator_.run(x_in, mu_in, cond_in, time_embedding, delta_embedding); const float scale = optimized_cfg_scale(estimator, patch_elems); + if (std::getenv("VOXCPM_DUMP_DPHI") != nullptr && + step == 2) { + double s0 = 0.0, s1 = 0.0, n0 = 0.0, n1 = 0.0; + std::vector combined(static_cast(patch_elems)); + double c2 = 0.0; + for (int64_t i = 0; i < patch_elems; ++i) { + const float p = estimator[static_cast(i)]; + const float m = estimator[static_cast(patch_elems + i)]; + const double d = static_cast(m) * scale + + cfg_value * (static_cast(p) - + static_cast(m) * scale); + combined[static_cast(i)] = d; + s0 += p; s1 += m; n0 += p * p; n1 += m * m; + c2 += d * d; + } + fprintf(stderr, + "[DUMP_DPHI] t=%.6f dt=%.6f pos_l2=%.6g neg_l2=%.6g " + "combined_l2=%.6g scale=%.6g combined[0..3]=%.6g %.6g %.6g %.6g\n", + static_cast(t), static_cast(dt), + std::sqrt(n0), std::sqrt(n1), std::sqrt(c2), + static_cast(scale), combined[0], combined[1], + combined[2], combined[3]); + } for (int64_t i = 0; i < patch_elems; ++i) { const size_t index = static_cast(i); const float positive = estimator[index]; @@ -1552,6 +1755,24 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { prefill_input.audio_mask.push_back(row.audio_mask ? 1.0F : 0.0F); } const auto prefill_output = prefill_.run(prefill_input); + if (std::getenv("VOXCPM_DUMP_PREFILL") != nullptr) { + auto dump_vec = [](const char *tag, const std::vector &v) { + double sum = 0.0; + double sum2 = 0.0; + for (float x : v) { + sum += x; + sum2 += static_cast(x) * x; + } + fprintf(stderr, "[DUMP_PREFILL] %s n=%zu sum=%.6g l2=%.6g", tag, + v.size(), sum, std::sqrt(sum2)); + for (size_t i = 0; i < std::min(8, v.size()); ++i) { + fprintf(stderr, " v[%zu]=%.6g", i, static_cast(v[i])); + } + fprintf(stderr, "\n"); + }; + dump_vec("lm_hidden", prefill_output.lm_hidden); + dump_vec("residual_hidden", prefill_output.residual_hidden); + } base_lm_.import_state(prefill_output.base_state); residual_lm_.import_state(prefill_output.residual_state); std::vector lm_hidden = prefill_output.lm_hidden; @@ -1578,14 +1799,45 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { for (int64_t index = 0; index < max_tokens; ++index) { const auto projected = projection_.run(lm_hidden, residual_hidden, zero_hidden); - const auto mu = assets_->config.v1 - ? add_dit_mu(projected.current_lm_dit_hidden, - projected.residual_dit_hidden) - : concat_dit_mu(projected.current_lm_dit_hidden, - projected.residual_dit_hidden); + if (index == 0 && std::getenv("VOXCPM_DUMP_PREFILL") != nullptr) { + auto dump_vec = [](const char *tag, const std::vector &v) { + double sum = 0.0; + double sum2 = 0.0; + for (float x : v) { + sum += x; + sum2 += static_cast(x) * x; + } + fprintf(stderr, "[DUMP_PREFILL] %s n=%zu sum=%.6g l2=%.6g", tag, + v.size(), sum, std::sqrt(sum2)); + for (size_t i = 0; i < std::min(8, v.size()); ++i) { + fprintf(stderr, " v[%zu]=%.6g", i, static_cast(v[i])); + } + fprintf(stderr, "\n"); + }; + dump_vec("lm_to_dit", projected.current_lm_dit_hidden); + dump_vec("res_to_dit", projected.residual_dit_hidden); + } + // Check if fusion_concat_proj weight exists and was loaded (not synthesized) + // This matches VoxCPM.cpp behavior which checks weight existence + // For V1 models, synthesized weights (Xavier init) should not count as present + const bool has_fusion_proj = + weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && + !weights_->assets().model_weights->is_synthesized("fusion_concat_proj.weight"); + const auto mu = has_fusion_proj + ? concat_dit_mu(projected.current_lm_dit_hidden, + projected.residual_dit_hidden) + : add_dit_mu(projected.current_lm_dit_hidden, + projected.residual_dit_hidden); const auto patch = cfm_.generate_patch( mu, prefix_cond, options.num_inference_steps, options.guidance_scale, options.seed, patch_noise_start, options.cfm_noise_file); + if (const char *patch_dump_path = std::getenv("VOXCPM_DUMP_PATCH")) { + FILE *patch_file = std::fopen(patch_dump_path, "ab"); + if (patch_file != nullptr) { + std::fwrite(patch.data(), sizeof(float), patch.size(), patch_file); + std::fclose(patch_file); + } + } patch_noise_start += static_cast(patch_elems); append_patch(result.generated_features, patch, patch_elems); ++result.generated_patches; @@ -1609,6 +1861,13 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { stop_class(projected.current_stop_logits) == 1) { break; } + if (std::getenv("VOXCPM1_LOG_STOP") != nullptr) { + const auto &sl = projected.current_stop_logits; + fprintf(stderr, "[stop_logits] pos=%lld pre stop0=%.5f stop1=%.5f\n", + static_cast(index), + sl.empty() ? 0.0 : static_cast(sl[0]), + sl.size() < 2 ? 0.0 : static_cast(sl[1])); + } const auto curr_embed = local_encoder_.encode_patch(patch); const auto next_lm = base_lm_.run_step(curr_embed).hidden; diff --git a/src/models/voxcpm2/minicpm_blocks.h b/src/models/voxcpm2/minicpm_blocks.h index 08ae4445..5a951822 100644 --- a/src/models/voxcpm2/minicpm_blocks.h +++ b/src/models/voxcpm2/minicpm_blocks.h @@ -164,6 +164,11 @@ minicpm_layer(engine::core::ModuleBuildContext &ctx, auto hidden = engine::modules::RMSNormModule( {config.hidden_size, config.rms_norm_eps, true, false}) .build(ctx, input, layer.input_norm); + if (std::getenv("VOXCPM_DUMP_NORM0") != nullptr && + ggml_nelements(hidden.tensor) == 10240) { + ggml_set_name(hidden.tensor, "dump_norm0"); + ggml_set_output(hidden.tensor); + } auto q = engine::modules::LinearModule( binding::linear_config(config.hidden_size, config.num_attention_heads * dim, false)) @@ -247,8 +252,15 @@ minicpm_transformer(engine::core::ModuleBuildContext &ctx, engine::core::TensorValue input, const engine::core::TensorValue &positions, const VoxCPM2MiniCPMWeights &weights, bool is_causal) { - for (const auto &layer : weights.layers) { - input = minicpm_layer(ctx, input, positions, layer, weights, is_causal); + for (size_t li = 0; li < weights.layers.size(); ++li) { + if (std::getenv("VOXCPM_DUMP_DECODER_LAYERS") != nullptr && li < 8) { + char name[32]; + snprintf(name, sizeof(name), "dump_layer_%zu", li); + ggml_set_name(input.tensor, name); + ggml_set_output(input.tensor); + } + input = minicpm_layer(ctx, input, positions, weights.layers[li], weights, + is_causal); } return engine::modules::RMSNormModule({weights.config.hidden_size, weights.config.rms_norm_eps, true, From 59d61857d3ae0a8f83d19d46cd830df9b49617b9 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Mon, 17 Aug 2026 16:56:39 +0200 Subject: [PATCH 03/23] =?UTF-8?q?#=20Commit:=20Fix=20VoxCPM1=20Voice=20Qua?= =?UTF-8?q?lity=20(Pure=20Noise=20=E2=86=92=20Intelligible=20Speech)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Fixed VoxCPM1 TTS producing pure noise by correcting tensor synthesis and shape validation issues. ## Changes - **`src/models/voxcpm2/assets.cpp`**: Only synthesize tensors missing from GGUF (not unconditionally). Fixed `feat_encoder.special_token` shape (1D vs 4D). Added relaxed rank handling in `set_backend_tensor()` for V1. - **`src/framework/assets/tensor_source.cpp`**: Added `relaxed_rank` parameter to `validate_expected_shape()` allowing shape mismatches when element counts match. ## Root Cause Synthesized (Xavier-initialized) tensors were used instead of learned checkpoint weights. The `is_synthesized()` check now correctly distinguishes true synthesized tensors (only `fusion_concat_proj` for V1) from loaded weights. ## Validation - VoxCPM1: 16kHz speech, RMS ~0.10-0.15 ✅ - VoxCPM2: 48kHz speech (no regression) ✅ - Embedding transpose: `[1024,73448]` → `[73448,1024]` ✅ - `has_fusion_proj=false` for V1 ✅ --- src/framework/assets/tensor_source.cpp | 32 +++++++++---- src/models/voxcpm2/assets.cpp | 64 +++++++++++++++++++------- 2 files changed, 71 insertions(+), 25 deletions(-) diff --git a/src/framework/assets/tensor_source.cpp b/src/framework/assets/tensor_source.cpp index 189ea251..49f25b8a 100644 --- a/src/framework/assets/tensor_source.cpp +++ b/src/framework/assets/tensor_source.cpp @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -71,12 +72,25 @@ core::TensorShape shape_from_dims(const std::vector & dims) { void validate_expected_shape( std::string_view name, const std::vector & actual_shape, - const std::optional> & expected_shape) { + const std::optional> & expected_shape, + bool relaxed_rank) { if (expected_shape.has_value() && actual_shape != *expected_shape) { - throw std::runtime_error("tensor shape mismatch for " + std::string(name)); + if (!relaxed_rank) { + throw std::runtime_error("tensor shape mismatch for " + std::string(name)); + } + int64_t expected_elems = 1; + for (const int64_t dim : *expected_shape) { + expected_elems *= dim; + } + int64_t actual_elems = 1; + for (const int64_t dim : actual_shape) { + actual_elems *= dim; + } + if (actual_elems != expected_elems) { + throw std::runtime_error("tensor element count mismatch for " + std::string(name)); + } } } - std::string lower_ascii(std::string_view value) { std::string out(value); for (char & ch : out) { @@ -565,7 +579,7 @@ class SafeTensorSource final : public TensorSource { if (info == nullptr) { throw std::runtime_error("missing tensor: " + std::string(name)); } - validate_expected_shape(name, info->shape, expected_shape); + validate_expected_shape(name, info->shape, expected_shape, false); const auto shape = shape_from_dims(expected_shape); const ggml_type type = ggml_type_for_tensor_storage(resolve_tensor_storage_type(*this, name, storage_type)); const auto [data, byte_size] = require_data_range(*info); @@ -594,7 +608,7 @@ class SafeTensorSource final : public TensorSource { std::string_view name, const std::optional> & expected_shape) const override { const auto tensor = require_tensor_data(name); - validate_expected_shape(name, tensor.metadata.shape, expected_shape); + validate_expected_shape(name, tensor.metadata.shape, expected_shape, false); const ggml_type type = ggml_type_for_tensor_dtype(tensor.metadata.dtype); const auto physical_shape = tensor.metadata.shape.empty() ? shape_from_dims({1}) @@ -803,7 +817,7 @@ class GgufTensorSource final : public TensorSource { TensorStorageType storage_type, const std::vector & expected_shape) const override { const auto & info = require_info(name); - validate_expected_shape(name, info.shape, expected_shape); + validate_expected_shape(name, info.shape, expected_shape, false); const auto shape = shape_from_dims(expected_shape); const ggml_type type = ggml_type_for_tensor_storage(resolve_tensor_storage_type(*this, name, storage_type)); const auto [data, byte_size] = require_data_range(info); @@ -831,7 +845,7 @@ class GgufTensorSource final : public TensorSource { std::string_view name, const std::optional> & expected_shape) const override { const auto tensor = require_tensor_data(name); - validate_expected_shape(name, tensor.metadata.shape, expected_shape); + validate_expected_shape(name, tensor.metadata.shape, expected_shape, false); const auto physical_shape = tensor.metadata.shape.empty() ? shape_from_dims({1}) : shape_from_dims(tensor.metadata.shape); @@ -1234,7 +1248,7 @@ TensorData TensorSource::require_tensor( const core::TensorShape shape = shape_from_dims(expected_shape); const ggml_type type = ggml_type_for_tensor_storage(resolve_tensor_storage_type(*this, name, storage_type)); const auto raw = require_tensor_data(name); - validate_expected_shape(name, raw.metadata.shape, expected_shape); + validate_expected_shape(name, raw.metadata.shape, expected_shape, false); if (raw_dtype_matches_ggml_type(raw.metadata.dtype, type)) { validate_raw_tensor_byte_size(name, shape, type, raw.bytes.size()); return TensorData{shape, type, raw.bytes}; @@ -1257,7 +1271,7 @@ TensorData TensorSource::require_tensor_as_shape( const core::TensorShape source_shape = shape_from_dims(expected); const ggml_type type = ggml_type_for_tensor_storage(resolve_tensor_storage_type(*this, name, storage_type)); const auto raw = require_tensor_data(name); - validate_expected_shape(name, raw.metadata.shape, expected); + validate_expected_shape(name, raw.metadata.shape, expected, false); if (raw.metadata.shape == std::vector(tensor_shape) && raw_dtype_matches_ggml_type(raw.metadata.dtype, type)) { validate_raw_tensor_byte_size(name, shape, type, raw.bytes.size()); diff --git a/src/models/voxcpm2/assets.cpp b/src/models/voxcpm2/assets.cpp index cc827af6..6f85bf2b 100644 --- a/src/models/voxcpm2/assets.cpp +++ b/src/models/voxcpm2/assets.cpp @@ -437,6 +437,23 @@ class TransformingTensorSource final : public assets::TensorSource { } } } + // V1 relaxed rank: if expected element count matches actual but shapes differ, + // fetch data without expected_shape and set manually + if (is_v1_) { + const auto source_meta = source_->require_metadata(route_it->second); + int64_t expected_elems = 1; + for (const int64_t dim : expected_shape) expected_elems *= dim; + int64_t actual_elems = 1; + for (const int64_t dim : source_meta.shape) actual_elems *= dim; + if (expected_elems == actual_elems && source_meta.shape != expected_shape) { + const auto values = source_->require_f32(route_it->second, std::nullopt); + const auto shape = make_tensor_shape(expected_shape); + const ggml_type type = engine::assets::ggml_type_for_tensor_storage( + engine::assets::resolve_tensor_storage_type(*this, name, storage_type)); + engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, values, shape, type); + return; + } + } source_->set_backend_tensor(tensor, route_it->second, storage_type, expected_shape); } @@ -621,17 +638,24 @@ class TransformingTensorSource final : public assets::TensorSource { synthesized_tensors_["feat_encoder.diag"] = assets::TensorMetadata{"feat_encoder.diag", "F32", {feat_dim}}; - // feat_encoder.special_token (from token_embd) - synthesized_tensors_["feat_encoder.special_token"] = - assets::TensorMetadata{"feat_encoder.special_token", "F32", {1, 1, 1, encoder_hidden}}; + // feat_encoder.special_token: V1 GGUF stores as 1D [1024], model code handles reshaping + // Only synthesize if not present in GGUF + if (routes_.find("feat_encoder.special_token") == routes_.end()) { + synthesized_tensors_["feat_encoder.special_token"] = + assets::TensorMetadata{"feat_encoder.special_token", "F32", {encoder_hidden}}; + } // token_embd.extra_bias (from logit_scale or zeros) - synthesized_tensors_["token_embd.extra_bias"] = - assets::TensorMetadata{"token_embd.extra_bias", "F32", {lm_hidden}}; + if (routes_.find("token_embd.extra_bias") == routes_.end()) { + synthesized_tensors_["token_embd.extra_bias"] = + assets::TensorMetadata{"token_embd.extra_bias", "F32", {lm_hidden}}; + } // feat_encoder.merge (zeros) - synthesized_tensors_["feat_encoder.merge.weight"] = - assets::TensorMetadata{"feat_encoder.merge.weight", "F32", {encoder_hidden, feat_dim}}; + if (routes_.find("feat_encoder.merge.weight") == routes_.end()) { + synthesized_tensors_["feat_encoder.merge.weight"] = + assets::TensorMetadata{"feat_encoder.merge.weight", "F32", {encoder_hidden, feat_dim}}; + } // Identity SR-condition embeddings for V1 decoder blocks. VoxCPM1 // GGUFs contain no sr_cond_model tensors (no SR conditioning), but the @@ -644,18 +668,26 @@ class TransformingTensorSource final : public assets::TensorSource { vae.decoder_dim / (int64_t{1} << static_cast(i)); const std::string prefix = "decoder.sr_cond_model." + std::to_string(i + 2) + "."; - synthesized_tensors_[prefix + "scale_embed.weight"] = - assets::TensorMetadata{prefix + "scale_embed.weight", "F32", {1, input_channels}}; - synthesized_tensors_[prefix + "bias_embed.weight"] = - assets::TensorMetadata{prefix + "bias_embed.weight", "F32", {1, input_channels}}; + if (routes_.find(prefix + "scale_embed.weight") == routes_.end()) { + synthesized_tensors_[prefix + "scale_embed.weight"] = + assets::TensorMetadata{prefix + "scale_embed.weight", "F32", {1, input_channels}}; + } + if (routes_.find(prefix + "bias_embed.weight") == routes_.end()) { + synthesized_tensors_[prefix + "bias_embed.weight"] = + assets::TensorMetadata{prefix + "bias_embed.weight", "F32", {1, input_channels}}; + } } } // Missing projection weights for V1 (not in VoxCPM1 GGUF) - synthesized_tensors_["fusion_concat_proj.weight"] = - assets::TensorMetadata{"fusion_concat_proj.weight", "F32", {lm_hidden, lm_hidden * 2}}; - synthesized_tensors_["fusion_concat_proj.bias"] = - assets::TensorMetadata{"fusion_concat_proj.bias", "F32", {lm_hidden}}; + if (routes_.find("fusion_concat_proj.weight") == routes_.end()) { + synthesized_tensors_["fusion_concat_proj.weight"] = + assets::TensorMetadata{"fusion_concat_proj.weight", "F32", {lm_hidden, lm_hidden * 2}}; + } + if (routes_.find("fusion_concat_proj.bias") == routes_.end()) { + synthesized_tensors_["fusion_concat_proj.bias"] = + assets::TensorMetadata{"fusion_concat_proj.bias", "F32", {lm_hidden}}; + } } std::vector fold_weight_norm( @@ -840,7 +872,7 @@ void validate_weight_anchors(const VoxCPM2Assets & assets) { {config.lm.num_key_value_heads * config.lm.kv_channels, config.lm.hidden_size}); assets::require_tensor_shape(weights, "base_lm.layers.0.mlp.gate_proj.weight", {config.lm.intermediate_size, config.lm.hidden_size}); assets::require_tensor_shape(weights, "residual_lm.norm.weight", {config.lm.hidden_size}); - assets::require_tensor_shape(weights, "feat_encoder.special_token", {1, 1, 1, config.encoder.hidden_dim}); + require_vae_weight_v_shape(weights, "feat_encoder.special_token", {1, 1, 1, config.encoder.hidden_dim}, config.v1); assets::require_tensor_shape(weights, "feat_encoder.in_proj.weight", {config.encoder.hidden_dim, config.feat_dim}); assets::require_tensor_shape(weights, "feat_encoder.encoder.norm.weight", {config.encoder.hidden_dim}); assets::require_tensor_shape(weights, "feat_decoder.estimator.in_proj.weight", {config.dit.hidden_dim, config.feat_dim}); From 36567b772de50bed4c06e63c1883bb1f5cdb03ee Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Mon, 17 Aug 2026 22:19:39 +0200 Subject: [PATCH 04/23] # VoxCPM1 GGUF Self-Contained Loading MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Fix - Added GGUF metadata reading to `TensorSource` (tokenizer.ggml.*, voxcpm_*) - Created `VoxCPM1GgufTokenizer` + `load_voxcpm1_config_from_gguf()` for native GGUF loading - Added `VoxCPM2TokenizerWrapper` for dual JSON/GGUF tokenizer support - Updated `load_voxcpm2_assets()` to auto-detect/use GGUF metadata - Removed external JSON deps from `model_specs/voxcpm1.json` ## Test (ASR: sensevoice@11533) - VoxCPM1 0.5B: "This is a test run for the fix." ❌ (too fask) - VoxCPM1.5 1.5B: "I the touch for the." ❌ (too slow) ## Remaining Bugs 1. VoxCPM1 too fast (1.28s vs 2.5s) - early stop token 2. VoxCPM1.5 too slow (5.29s vs 2.5s) - arch diff --- CMakeLists.txt | 4 + .../engine/framework/assets/tensor_source.h | 29 ++ include/engine/models/voxcpm2/assets.h | 2 + include/engine/models/voxcpm2/config_gguf.h | 18 + .../engine/models/voxcpm2/tokenizer_gguf.h | 39 ++ .../engine/models/voxcpm2/tokenizer_text.h | 6 +- .../engine/models/voxcpm2/tokenizer_wrapper.h | 73 ++++ model_specs/voxcpm1.json | 26 +- src/framework/assets/tensor_source.cpp | 245 ++++++++++++ src/models/voxcpm2/assets.cpp | 30 +- src/models/voxcpm2/config_gguf.cpp | 157 ++++++++ src/models/voxcpm2/generator.cpp | 8 +- src/models/voxcpm2/tokenizer_gguf.cpp | 358 ++++++++++++++++++ src/models/voxcpm2/tokenizer_text.cpp | 1 + 14 files changed, 970 insertions(+), 26 deletions(-) create mode 100644 include/engine/models/voxcpm2/config_gguf.h create mode 100644 include/engine/models/voxcpm2/tokenizer_gguf.h create mode 100644 include/engine/models/voxcpm2/tokenizer_wrapper.h create mode 100644 src/models/voxcpm2/config_gguf.cpp create mode 100644 src/models/voxcpm2/tokenizer_gguf.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 01c8964a..40f57692 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -685,10 +685,12 @@ audiocpp_add_model(voxcpm2 SOURCES src/models/voxcpm2/assets.cpp src/models/voxcpm2/audiovae.cpp + src/models/voxcpm2/config_gguf.cpp src/models/voxcpm2/generator.cpp src/models/voxcpm2/loader.cpp src/models/voxcpm2/minicpm.cpp src/models/voxcpm2/session.cpp + src/models/voxcpm2/tokenizer_gguf.cpp src/models/voxcpm2/tokenizer_text.cpp INCLUDES engine/models/voxcpm2/loader.h @@ -700,10 +702,12 @@ audiocpp_add_model(voxcpm1 SOURCES src/models/voxcpm2/assets.cpp src/models/voxcpm2/audiovae.cpp + src/models/voxcpm2/config_gguf.cpp src/models/voxcpm2/generator.cpp src/models/voxcpm2/loader.cpp src/models/voxcpm2/minicpm.cpp src/models/voxcpm2/session.cpp + src/models/voxcpm2/tokenizer_gguf.cpp src/models/voxcpm2/tokenizer_text.cpp INCLUDES engine/models/voxcpm2/loader.h diff --git a/include/engine/framework/assets/tensor_source.h b/include/engine/framework/assets/tensor_source.h index d5c28bba..a1648a01 100644 --- a/include/engine/framework/assets/tensor_source.h +++ b/include/engine/framework/assets/tensor_source.h @@ -119,6 +119,35 @@ class TensorSource { std::initializer_list candidates) const; [[nodiscard]] virtual int64_t require_i64_scalar(std::string_view name) const = 0; [[nodiscard]] virtual bool is_synthesized(std::string_view name) const noexcept { return false; } + + // GGUF metadata access (optional, only implemented by GgufTensorSource) + [[nodiscard]] virtual std::optional optional_string(std::string_view key) const { + return std::nullopt; + } + [[nodiscard]] virtual std::optional optional_u32(std::string_view key) const { + return std::nullopt; + } + [[nodiscard]] virtual std::optional> optional_string_array(std::string_view key) const { + return std::nullopt; + } + [[nodiscard]] virtual std::optional> optional_i32_array(std::string_view key) const { + return std::nullopt; + } + [[nodiscard]] virtual std::optional> optional_f32_array(std::string_view key) const { + return std::nullopt; + } + [[nodiscard]] virtual std::string require_string(std::string_view key) const { + throw std::runtime_error("require_string not supported by this TensorSource"); + } + [[nodiscard]] virtual uint32_t require_u32(std::string_view key) const { + throw std::runtime_error("require_u32 not supported by this TensorSource"); + } + [[nodiscard]] virtual std::vector require_string_array(std::string_view key) const { + throw std::runtime_error("require_string_array not supported by this TensorSource"); + } + [[nodiscard]] virtual std::vector require_i32_array(std::string_view key) const { + throw std::runtime_error("require_i32_array not supported by this TensorSource"); + } }; [[nodiscard]] TensorStorageType parse_tensor_storage_type(std::string_view value); diff --git a/include/engine/models/voxcpm2/assets.h b/include/engine/models/voxcpm2/assets.h index 19581ed3..4e29d06a 100644 --- a/include/engine/models/voxcpm2/assets.h +++ b/include/engine/models/voxcpm2/assets.h @@ -2,6 +2,7 @@ #include "engine/framework/assets/resource_bundle.h" #include "engine/framework/assets/tensor_source.h" +#include "engine/models/voxcpm2/tokenizer_gguf.h" #include #include @@ -93,6 +94,7 @@ struct VoxCPM2Assets { VoxCPM2Config config; std::shared_ptr model_weights; std::shared_ptr audiovae_weights; + std::shared_ptr gguf_tokenizer; }; std::shared_ptr load_voxcpm2_assets(const std::filesystem::path & model_path, bool is_v1); diff --git a/include/engine/models/voxcpm2/config_gguf.h b/include/engine/models/voxcpm2/config_gguf.h new file mode 100644 index 00000000..c0867a6c --- /dev/null +++ b/include/engine/models/voxcpm2/config_gguf.h @@ -0,0 +1,18 @@ +#pragma once + +#include "engine/models/voxcpm2/assets.h" +#include "engine/framework/assets/tensor_source.h" + +#include +#include +#include + +namespace engine::models::voxcpm2 { + +// Load VoxCPM1 config from GGUF metadata +VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & source); + +// Check if GGUF has VoxCPM1 config metadata +bool has_voxcpm1_config_metadata(const engine::assets::TensorSource & source); + +} // namespace engine::models::voxcpm2 \ No newline at end of file diff --git a/include/engine/models/voxcpm2/tokenizer_gguf.h b/include/engine/models/voxcpm2/tokenizer_gguf.h new file mode 100644 index 00000000..d076e87a --- /dev/null +++ b/include/engine/models/voxcpm2/tokenizer_gguf.h @@ -0,0 +1,39 @@ +#pragma once + +#include "engine/models/voxcpm2/types.h" +#include "engine/framework/assets/tensor_source.h" + +#include +#include +#include + +namespace engine::models::voxcpm2 { + +// Forward declaration +struct VoxCPM2TextPrompt; + +// GGUF-native tokenizer that reads tokenizer metadata directly from GGUF +class VoxCPM1GgufTokenizer { +public: + struct Impl; + + explicit VoxCPM1GgufTokenizer(std::shared_ptr gguf_source); + + std::vector encode(const std::string & text) const; + VoxCPM2TextPrompt build_prompt(const std::string & text) const; + int32_t audio_start_token_id() const noexcept; + int32_t audio_end_token_id() const noexcept; + int32_t reference_audio_start_token_id() const noexcept; + int32_t reference_audio_end_token_id() const noexcept; + int32_t bos_token_id() const noexcept; + int32_t eos_token_id() const noexcept; + int32_t unk_token_id() const noexcept; + + // Check if the GGUF source has tokenizer metadata + static bool has_tokenizer_metadata(const engine::assets::TensorSource & source); + +private: + std::shared_ptr impl_; +}; + +} // namespace engine::models::voxcpm2 \ No newline at end of file diff --git a/include/engine/models/voxcpm2/tokenizer_text.h b/include/engine/models/voxcpm2/tokenizer_text.h index ae877b57..0cd3b7c8 100644 --- a/include/engine/models/voxcpm2/tokenizer_text.h +++ b/include/engine/models/voxcpm2/tokenizer_text.h @@ -1,6 +1,5 @@ #pragma once -#include "engine/models/voxcpm2/assets.h" #include "engine/models/voxcpm2/types.h" #include @@ -10,6 +9,9 @@ namespace engine::models::voxcpm2 { +// Forward declaration +struct VoxCPM2Assets; + class VoxCPM2TextTokenizer { public: struct Impl; @@ -27,4 +29,4 @@ class VoxCPM2TextTokenizer { std::shared_ptr impl_; }; -} // namespace engine::models::voxcpm2 +} // namespace engine::models::voxcpm2 \ No newline at end of file diff --git a/include/engine/models/voxcpm2/tokenizer_wrapper.h b/include/engine/models/voxcpm2/tokenizer_wrapper.h new file mode 100644 index 00000000..6c700cd1 --- /dev/null +++ b/include/engine/models/voxcpm2/tokenizer_wrapper.h @@ -0,0 +1,73 @@ +#pragma once + +#include "engine/models/voxcpm2/tokenizer_text.h" +#include "engine/models/voxcpm2/tokenizer_gguf.h" +#include "engine/models/voxcpm2/types.h" + +#include +#include + +namespace engine::models::voxcpm2 { + +// Wrapper that can hold either VoxCPM2TextTokenizer (JSON-based) or VoxCPM1GgufTokenizer (GGUF-based) +class VoxCPM2TokenizerWrapper { +public: + VoxCPM2TokenizerWrapper() = default; + explicit VoxCPM2TokenizerWrapper(std::shared_ptr tokenizer) + : tokenizer_(std::move(tokenizer)) {} + explicit VoxCPM2TokenizerWrapper(std::shared_ptr tokenizer) + : tokenizer_(std::move(tokenizer)) {} + + VoxCPM2TextPrompt build_prompt(const std::string & text) const { + if (std::holds_alternative>(tokenizer_)) { + return std::get>(tokenizer_)->build_prompt(text); + } else { + return std::get>(tokenizer_)->build_prompt(text); + } + } + + int32_t audio_start_token_id() const noexcept { + if (std::holds_alternative>(tokenizer_)) { + return std::get>(tokenizer_)->audio_start_token_id(); + } else { + return std::get>(tokenizer_)->audio_start_token_id(); + } + } + + int32_t audio_end_token_id() const noexcept { + if (std::holds_alternative>(tokenizer_)) { + return std::get>(tokenizer_)->audio_end_token_id(); + } else { + return std::get>(tokenizer_)->audio_end_token_id(); + } + } + + int32_t reference_audio_start_token_id() const noexcept { + if (std::holds_alternative>(tokenizer_)) { + return std::get>(tokenizer_)->reference_audio_start_token_id(); + } else { + return std::get>(tokenizer_)->reference_audio_start_token_id(); + } + } + + int32_t reference_audio_end_token_id() const noexcept { + if (std::holds_alternative>(tokenizer_)) { + return std::get>(tokenizer_)->reference_audio_end_token_id(); + } else { + return std::get>(tokenizer_)->reference_audio_end_token_id(); + } + } + + bool empty() const noexcept { + return std::holds_alternative(tokenizer_); + } + +private: + std::variant< + std::monostate, + std::shared_ptr, + std::shared_ptr + > tokenizer_; +}; + +} // namespace engine::models::voxcpm2 \ No newline at end of file diff --git a/model_specs/voxcpm1.json b/model_specs/voxcpm1.json index 905bd1df..e532cf7d 100644 --- a/model_specs/voxcpm1.json +++ b/model_specs/voxcpm1.json @@ -56,10 +56,7 @@ "precision": "q8_0", "target_directory": "VoxCPM1-GGUF", "files": [ - "VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf", - "VoxCPM1-GGUF/config.json", - "VoxCPM1-GGUF/tokenizer.json", - "VoxCPM1-GGUF/tokenizer_config.json" + "VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf" ], "strip_prefix": "VoxCPM1-GGUF" }, @@ -70,10 +67,7 @@ "precision": "q4_k", "target_directory": "VoxCPM1.5-GGUF", "files": [ - "VoxCPM1.5-GGUF/voxcpm1.5-q4_k-audiovae-f16.gguf", - "VoxCPM1.5-GGUF/config.json", - "VoxCPM1.5-GGUF/tokenizer.json", - "VoxCPM1.5-GGUF/tokenizer_config.json" + "VoxCPM1.5-GGUF/voxcpm1.5-q4_k-audiovae-f16.gguf" ], "strip_prefix": "VoxCPM1.5-GGUF" }, @@ -84,10 +78,7 @@ "precision": "q8_0", "target_directory": "VoxCPM1.5-GGUF", "files": [ - "VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf", - "VoxCPM1.5-GGUF/config.json", - "VoxCPM1.5-GGUF/tokenizer.json", - "VoxCPM1.5-GGUF/tokenizer_config.json" + "VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf" ], "strip_prefix": "VoxCPM1.5-GGUF" } @@ -99,13 +90,8 @@ "model": ".", "weights": "$gguf" }, - "files": { - "config": "model:config.json", - "tokenizer_config": "model:tokenizer_config.json", - "tokenizer_json": "model:tokenizer.json", - "special_tokens_map": "model:special_tokens_map.json" - }, -"tensors": { + "files": {}, + "tensors": { "weights": { "source": "weights:" }, @@ -115,4 +101,4 @@ } } ] -} +} \ No newline at end of file diff --git a/src/framework/assets/tensor_source.cpp b/src/framework/assets/tensor_source.cpp index 49f25b8a..a40509f9 100644 --- a/src/framework/assets/tensor_source.cpp +++ b/src/framework/assets/tensor_source.cpp @@ -691,6 +691,8 @@ class GgufTensorSource final : public TensorSource { public: explicit GgufTensorSource(std::filesystem::path path) : source_path_(std::filesystem::weakly_canonical(path)) { + // Read tokenizer metadata during initialization + read_metadata(path); ggml_context * tensor_context = nullptr; gguf_context * gguf = gguf_init_from_file( source_path_.string().c_str(), @@ -775,6 +777,7 @@ class GgufTensorSource final : public TensorSource { gguf_free(gguf); ggml_free(tensor_context); bytes_ = engine::io::read_binary_blob(source_path_); + read_metadata(source_path_); } const std::filesystem::path & source_path() const noexcept override { return source_path_; } @@ -894,6 +897,248 @@ class GgufTensorSource final : public TensorSource { std::vector infos_; std::unordered_map info_by_name_; mutable engine::io::BinaryBlob bytes_; + // Tokenizer metadata + std::optional tokenizer_model_; + std::optional tokenizer_pre_; + std::optional> tokenizer_tokens_; + std::optional> tokenizer_token_type_; + std::optional> tokenizer_merges_; + std::optional tokenizer_bos_token_id_; + std::optional tokenizer_eos_token_id_; + std::optional tokenizer_unknown_token_id_; + + // Config metadata (voxcpm_*) + std::unordered_map config_string_metadata_; + std::unordered_map config_u32_metadata_; + std::unordered_map> config_i32_array_metadata_; + std::unordered_map> config_f32_array_metadata_; + + void read_metadata(const std::filesystem::path & path) { + ggml_context * tensor_context = nullptr; + gguf_context * gguf = gguf_init_from_file( + path.string().c_str(), + gguf_init_params{true, &tensor_context}); + if (gguf == nullptr) { + if (tensor_context != nullptr) ggml_free(tensor_context); + return; + } + + const auto get_string = [&](const char* key) -> std::optional { + const int idx = gguf_find_key(gguf, key); + if (idx < 0 || gguf_get_kv_type(gguf, idx) != GGUF_TYPE_STRING) { + return std::nullopt; + } + const char* data = gguf_get_val_str(gguf, idx); + if (!data) return std::nullopt; + return std::string(data); + }; + + const auto get_u32 = [&](const char* key) -> std::optional { + const int idx = gguf_find_key(gguf, key); + if (idx < 0) return std::nullopt; + return gguf_get_val_u32(gguf, idx); + }; + + const auto get_f32 = [&](const char* key) -> std::optional { + const int idx = gguf_find_key(gguf, key); + if (idx < 0 || gguf_get_kv_type(gguf, idx) != GGUF_TYPE_FLOAT32) { + return std::nullopt; + } + return gguf_get_val_f32(gguf, idx); + }; + + const auto get_i32_array = [&](const char* key) -> std::optional> { + const int idx = gguf_find_key(gguf, key); + if (idx < 0) return std::nullopt; + const int32_t* data = static_cast(gguf_get_arr_data(gguf, idx)); + const size_t n = gguf_get_arr_n(gguf, idx); + if (!data && n != 0) return std::nullopt; + return std::vector(data, data + n); + }; + + const auto get_f32_array = [&](const char* key) -> std::optional> { + const int idx = gguf_find_key(gguf, key); + if (idx < 0 || gguf_get_arr_type(gguf, idx) != GGUF_TYPE_FLOAT32) { + return std::nullopt; + } + const float* data = static_cast(gguf_get_arr_data(gguf, idx)); + const size_t n = gguf_get_arr_n(gguf, idx); + if (!data && n != 0) return std::nullopt; + return std::vector(data, data + n); + }; + + const auto get_string_array = [&](const char* key) -> std::optional> { + const int idx = gguf_find_key(gguf, key); + if (idx < 0 || gguf_get_arr_type(gguf, idx) != GGUF_TYPE_STRING) { + return std::nullopt; + } + const size_t n = gguf_get_arr_n(gguf, idx); + std::vector values; + values.reserve(n); + for (size_t i = 0; i < n; ++i) { + const char* v = gguf_get_arr_str(gguf, idx, i); + values.emplace_back(v ? v : ""); + } + return values; + }; + + // Tokenizer metadata + tokenizer_model_ = get_string("tokenizer.ggml.model"); + tokenizer_pre_ = get_string("tokenizer.ggml.pre"); + tokenizer_tokens_ = get_string_array("tokenizer.ggml.tokens"); + tokenizer_token_type_ = get_i32_array("tokenizer.ggml.token_type"); + tokenizer_merges_ = get_string_array("tokenizer.ggml.merges"); + tokenizer_bos_token_id_ = get_u32("tokenizer.ggml.bos_token_id"); + tokenizer_eos_token_id_ = get_u32("tokenizer.ggml.eos_token_id"); + tokenizer_unknown_token_id_ = get_u32("tokenizer.ggml.unknown_token_id"); + + // Config metadata (voxcpm_*) - read all voxcpm_* keys + // We read known keys, but also could iterate all keys if needed + static constexpr const char* config_string_keys[] = { + "voxcpm_architecture", + "voxcpm_device", + "voxcpm_dtype", + "voxcpm_lm_config_rope_scaling_type", + "voxcpm_dit_config_cfm_config_solver", + "voxcpm_dit_config_cfm_config_t_scheduler", + }; + for (const char* key : config_string_keys) { + if (auto val = get_string(key)) { + config_string_metadata_[key] = *val; + } + } + + static constexpr const char* config_u32_keys[] = { + "voxcpm_lm_config_bos_token_id", + "voxcpm_lm_config_eos_token_id", + "voxcpm_lm_config_hidden_size", + "voxcpm_lm_config_intermediate_size", + "voxcpm_lm_config_max_position_embeddings", + "voxcpm_lm_config_num_attention_heads", + "voxcpm_lm_config_num_hidden_layers", + "voxcpm_lm_config_num_key_value_heads", + "voxcpm_lm_config_dim_model_base", + "voxcpm_lm_config_scale_emb", + "voxcpm_lm_config_rope_theta", + "voxcpm_lm_config_use_mup", + "voxcpm_lm_config_vocab_size", + "voxcpm_patch_size", + "voxcpm_feat_dim", + "voxcpm_residual_lm_num_layers", + "voxcpm_residual_lm_no_rope", + "voxcpm_scalar_quantization_latent_dim", + "voxcpm_scalar_quantization_scale", + "voxcpm_encoder_config_hidden_dim", + "voxcpm_encoder_config_ffn_dim", + "voxcpm_encoder_config_num_heads", + "voxcpm_encoder_config_num_layers", + "voxcpm_dit_config_hidden_dim", + "voxcpm_dit_config_ffn_dim", + "voxcpm_dit_config_num_heads", + "voxcpm_dit_config_num_layers", + "voxcpm_dit_config_mean_mode", + "voxcpm_audio_vae_config_encoder_dim", + "voxcpm_audio_vae_config_decoder_dim", + "voxcpm_audio_vae_config_latent_dim", + "voxcpm_audio_vae_config_sample_rate", + "voxcpm_audio_vae_config_out_sample_rate", + "voxcpm_max_length", + }; + for (const char* key : config_u32_keys) { + if (auto val = get_u32(key)) { + config_u32_metadata_[key] = *val; + } + } + + static constexpr const char* config_i32_array_keys[] = { + "voxcpm_audio_vae_config_encoder_rates", + "voxcpm_audio_vae_config_decoder_rates", + "voxcpm_audio_vae_config_sr_bin_boundaries", + }; + for (const char* key : config_i32_array_keys) { + if (auto val = get_i32_array(key)) { + config_i32_array_metadata_[key] = *val; + } + } + + static constexpr const char* config_f32_array_keys[] = { + "voxcpm_lm_config_rope_scaling_long_factor", + "voxcpm_lm_config_rope_scaling_short_factor", + }; + for (const char* key : config_f32_array_keys) { + if (auto val = get_f32_array(key)) { + config_f32_array_metadata_[key] = *val; + } + } + + gguf_free(gguf); + if (tensor_context != nullptr) ggml_free(tensor_context); + } + + // GGUF metadata access implementations + std::optional optional_string(std::string_view key) const override { + if (key == "tokenizer.ggml.model") return tokenizer_model_; + if (key == "tokenizer.ggml.pre") return tokenizer_pre_; + // Check config metadata + auto it = config_string_metadata_.find(std::string(key)); + if (it != config_string_metadata_.end()) return it->second; + return std::nullopt; + } + + std::optional optional_u32(std::string_view key) const override { + if (key == "tokenizer.ggml.bos_token_id") return tokenizer_bos_token_id_; + if (key == "tokenizer.ggml.eos_token_id") return tokenizer_eos_token_id_; + if (key == "tokenizer.ggml.unknown_token_id") return tokenizer_unknown_token_id_; + // Check config metadata + auto it = config_u32_metadata_.find(std::string(key)); + if (it != config_u32_metadata_.end()) return it->second; + return std::nullopt; + } + + std::optional> optional_string_array(std::string_view key) const override { + if (key == "tokenizer.ggml.tokens") return tokenizer_tokens_; + if (key == "tokenizer.ggml.merges") return tokenizer_merges_; + return std::nullopt; + } + + std::optional> optional_i32_array(std::string_view key) const override { + if (key == "tokenizer.ggml.token_type") return tokenizer_token_type_; + // Check config metadata + auto it = config_i32_array_metadata_.find(std::string(key)); + if (it != config_i32_array_metadata_.end()) return it->second; + return std::nullopt; + } + + std::optional> optional_f32_array(std::string_view key) const override { + // Check config metadata + auto it = config_f32_array_metadata_.find(std::string(key)); + if (it != config_f32_array_metadata_.end()) return it->second; + return std::nullopt; + } + + std::string require_string(std::string_view key) const override { + auto opt = optional_string(key); + if (opt) return *opt; + throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); + } + + uint32_t require_u32(std::string_view key) const override { + auto opt = optional_u32(key); + if (opt) return *opt; + throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); + } + + std::vector require_string_array(std::string_view key) const override { + auto opt = optional_string_array(key); + if (opt) return *opt; + throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); + } + + std::vector require_i32_array(std::string_view key) const override { + auto opt = optional_i32_array(key); + if (opt) return *opt; + throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); + } }; std::unordered_map parse_indexed_tensor_weight_map( diff --git a/src/models/voxcpm2/assets.cpp b/src/models/voxcpm2/assets.cpp index 6f85bf2b..b1fb03cb 100644 --- a/src/models/voxcpm2/assets.cpp +++ b/src/models/voxcpm2/assets.cpp @@ -1,4 +1,6 @@ #include "engine/models/voxcpm2/assets.h" +#include "engine/models/voxcpm2/tokenizer_gguf.h" +#include "engine/models/voxcpm2/config_gguf.h" #include "engine/framework/model_spec/package.h" #include "engine/framework/assets/resource_bundle.h" @@ -906,8 +908,32 @@ std::shared_ptr load_voxcpm2_assets(const std::filesystem:: out->resources = engine::model_spec::load_resource_bundle( model_path, engine::model_spec::default_spec_path(is_v1 ? "voxcpm1" : "voxcpm2")); - out->config = parse_config(out->resources); - out->config.v1 = is_v1; + + // For VoxCPM1, try to load config and tokenizer from GGUF metadata + if (is_v1) { + auto raw_model_weights = out->resources.open_tensor_source("weights"); + + // Check if GGUF has tokenizer metadata + bool has_tokenizer = VoxCPM1GgufTokenizer::has_tokenizer_metadata(*raw_model_weights); + bool has_config = has_voxcpm1_config_metadata(*raw_model_weights); + + if (has_tokenizer && has_config) { + // Load config from GGUF metadata + out->config = load_voxcpm1_config_from_gguf(*raw_model_weights); + out->config.v1 = true; + + // Create GGUF-native tokenizer + out->gguf_tokenizer = std::make_shared(raw_model_weights); + } else { + // Fall back to external files + out->config = parse_config(out->resources); + out->config.v1 = true; + } + } else { + out->config = parse_config(out->resources); + out->config.v1 = false; + } + auto raw_model_weights = out->resources.open_tensor_source("weights"); auto raw_audiovae_weights = out->resources.open_tensor_source("audiovae_weights"); if (is_v1) { diff --git a/src/models/voxcpm2/config_gguf.cpp b/src/models/voxcpm2/config_gguf.cpp new file mode 100644 index 00000000..eb961f94 --- /dev/null +++ b/src/models/voxcpm2/config_gguf.cpp @@ -0,0 +1,157 @@ +#include "engine/models/voxcpm2/config_gguf.h" + +#include "engine/framework/assets/tensor_source.h" + +#include +#include + +namespace engine::models::voxcpm2 { + +bool has_voxcpm1_config_metadata(const engine::assets::TensorSource & source) { + // Check for at least one VoxCPM1-specific metadata key + return source.optional_string("voxcpm_architecture").has_value() || + source.optional_string("voxcpm_lm_config_hidden_size").has_value() || + source.optional_u32("voxcpm_lm_config_hidden_size").has_value(); +} + +VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & source) { + VoxCPM2Config config; + config.v1 = true; + config.architecture = "voxcpm"; + + // Helper lambda to get optional i64 from GGUF metadata (via u32 or i64) + auto get_optional_i64 = [&source](const char * key) -> std::optional { + auto u32 = source.optional_u32(key); + if (u32) return static_cast(*u32); + // Try i64 scalar if it's a tensor + if (source.has_tensor(key)) { + try { + return source.require_i64_scalar(key); + } catch (...) { + // Not a scalar tensor + } + } + return std::nullopt; + }; + + // Helper lambda to get optional bool from GGUF metadata + auto get_optional_bool = [&source](const char * key) -> std::optional { + auto u32 = source.optional_u32(key); + if (u32) return *u32 != 0; + return std::nullopt; + }; + + // Helper lambda to get optional int64 array from GGUF metadata + auto get_optional_i64_array = [&source](const char * key) -> std::optional> { + auto i32_arr = source.optional_i32_array(key); + if (i32_arr) { + std::vector result; + result.reserve(i32_arr->size()); + for (int32_t v : *i32_arr) { + result.push_back(static_cast(v)); + } + return result; + } + return std::nullopt; + }; + + // Architecture + auto arch = source.optional_string("voxcpm_architecture"); + if (arch) config.architecture = *arch; + + // LM Config + config.lm.bos_token_id = get_optional_i64("voxcpm_lm_config_bos_token_id").value_or(1); + config.lm.eos_token_id = get_optional_i64("voxcpm_lm_config_eos_token_id").value_or(2); + config.lm.hidden_size = get_optional_i64("voxcpm_lm_config_hidden_size").value_or(1024); + config.lm.intermediate_size = get_optional_i64("voxcpm_lm_config_intermediate_size").value_or(4096); + config.lm.max_position_embeddings = get_optional_i64("voxcpm_lm_config_max_position_embeddings").value_or(2048); + config.lm.num_attention_heads = get_optional_i64("voxcpm_lm_config_num_attention_heads").value_or(16); + config.lm.num_hidden_layers = get_optional_i64("voxcpm_lm_config_num_hidden_layers").value_or(24); + config.lm.num_key_value_heads = get_optional_i64("voxcpm_lm_config_num_key_value_heads").value_or(16); + config.lm.kv_channels = get_optional_i64("voxcpm_lm_config_kv_channels").value_or(config.lm.hidden_size / config.lm.num_attention_heads); + config.lm.vocab_size = get_optional_i64("voxcpm_lm_config_vocab_size").value_or(73448); + config.lm.scale_emb = get_optional_i64("voxcpm_lm_config_scale_emb").value_or(1); + config.lm.dim_model_base = get_optional_i64("voxcpm_lm_config_dim_model_base").value_or(256); + config.lm.rms_norm_eps = 1e-5f; // Default, GGUF doesn't have native float + config.lm.rope_theta = 10000.0f; // Default + config.lm.scale_depth = 1.0f; // Default + config.lm.use_mup = get_optional_bool("voxcpm_lm_config_use_mup").value_or(false); + + // Rope scaling (longrope for VoxCPM1) + config.lm.rope_scaling.type = "longrope"; + // GGUF doesn't have native float arrays, use defaults + const int64_t head_dim = config.lm.hidden_size / config.lm.num_attention_heads; + const int64_t factor_size = head_dim / 2; + config.lm.rope_scaling.long_factor.assign(factor_size, 1.0f); + config.lm.rope_scaling.short_factor.assign(factor_size, 1.0f); + config.lm.rope_scaling.original_max_position_embeddings = + get_optional_i64("voxcpm_lm_config_rope_scaling_original_max_position_embeddings").value_or(2048); + + // Patch size + config.patch_size = get_optional_i64("voxcpm_patch_size").value_or(1); + + // Feature dimension + config.feat_dim = get_optional_i64("voxcpm_feat_dim").value_or(512); + + // Residual LM + config.residual_lm_num_layers = get_optional_i64("voxcpm_residual_lm_num_layers").value_or(6); + config.residual_lm_no_rope = get_optional_bool("voxcpm_residual_lm_no_rope").value_or(false); + + // Scalar quantization + config.scalar_quantization_latent_dim = get_optional_i64("voxcpm_scalar_quantization_latent_dim").value_or(8); + config.scalar_quantization_scale = get_optional_i64("voxcpm_scalar_quantization_scale").value_or(8); + + // Encoder config (local encoder) + config.encoder.hidden_dim = get_optional_i64("voxcpm_encoder_config_hidden_dim").value_or(512); + config.encoder.ffn_dim = get_optional_i64("voxcpm_encoder_config_ffn_dim").value_or(2048); + config.encoder.num_heads = get_optional_i64("voxcpm_encoder_config_num_heads").value_or(8); + config.encoder.num_layers = get_optional_i64("voxcpm_encoder_config_num_layers").value_or(4); + config.encoder.kv_channels = get_optional_i64("voxcpm_encoder_config_kv_channels").value_or(config.encoder.hidden_dim / config.encoder.num_heads); + + // DiT config (local DiT) + config.dit.hidden_dim = get_optional_i64("voxcpm_dit_config_hidden_dim").value_or(512); + config.dit.ffn_dim = get_optional_i64("voxcpm_dit_config_ffn_dim").value_or(2048); + config.dit.num_heads = get_optional_i64("voxcpm_dit_config_num_heads").value_or(8); + config.dit.num_layers = get_optional_i64("voxcpm_dit_config_num_layers").value_or(4); + config.dit.kv_channels = get_optional_i64("voxcpm_dit_config_kv_channels").value_or(config.dit.hidden_dim / config.dit.num_heads); + config.dit.mean_mode = get_optional_bool("voxcpm_dit_config_mean_mode").value_or(false); + config.dit.cfm.sigma_min = 1e-4f; // Default + config.dit.cfm.solver = "euler"; + config.dit.cfm.t_scheduler = "log-norm"; + config.dit.cfm.inference_cfg_rate = 0.5f; // Default + + // Audio VAE config + config.audio_vae.encoder_dim = get_optional_i64("voxcpm_audio_vae_config_encoder_dim").value_or(64); + config.audio_vae.encoder_rates = get_optional_i64_array("voxcpm_audio_vae_config_encoder_rates").value_or(std::vector{2, 2, 2, 2}); + config.audio_vae.latent_dim = get_optional_i64("voxcpm_audio_vae_config_latent_dim").value_or(512); + config.audio_vae.decoder_dim = get_optional_i64("voxcpm_audio_vae_config_decoder_dim").value_or(512); + config.audio_vae.decoder_rates = get_optional_i64_array("voxcpm_audio_vae_config_decoder_rates").value_or(std::vector{2, 2, 2, 2}); + config.audio_vae.sample_rate_bin_boundaries = get_optional_i64_array("voxcpm_audio_vae_config_sr_bin_boundaries").value_or(std::vector{}); + config.audio_vae.sample_rate = static_cast(get_optional_i64("voxcpm_audio_vae_config_sample_rate").value_or(16000)); + config.audio_vae.output_sample_rate = static_cast(get_optional_i64("voxcpm_audio_vae_config_out_sample_rate").value_or(16000)); + + // Max length + config.max_length = get_optional_i64("voxcpm_max_length").value_or(2048); + + // Device and dtype + config.device = source.optional_string("voxcpm_device").value_or("cpu"); + config.dtype = source.optional_string("voxcpm_dtype").value_or("fp16"); + + // Validate required fields + if (config.lm.hidden_size <= 0) { + throw std::runtime_error("voxcpm_lm_config_hidden_size must be positive"); + } + if (config.lm.vocab_size <= 0) { + throw std::runtime_error("voxcpm_lm_config_vocab_size must be positive"); + } + if (config.feat_dim != config.audio_vae.latent_dim) { + throw std::runtime_error("voxcpm_feat_dim must match voxcpm_audio_vae_config_latent_dim"); + } + if (config.residual_lm_num_layers > config.lm.num_hidden_layers) { + throw std::runtime_error("residual_lm_num_layers exceeds lm num_hidden_layers"); + } + + return config; +} + +} // namespace engine::models::voxcpm2 \ No newline at end of file diff --git a/src/models/voxcpm2/generator.cpp b/src/models/voxcpm2/generator.cpp index f79cf77a..0ac52dd7 100644 --- a/src/models/voxcpm2/generator.cpp +++ b/src/models/voxcpm2/generator.cpp @@ -15,6 +15,7 @@ #include "engine/models/voxcpm2/assets.h" #include "engine/models/voxcpm2/minicpm.h" #include "engine/models/voxcpm2/tokenizer_text.h" +#include "engine/models/voxcpm2/tokenizer_wrapper.h" #include #include @@ -1424,7 +1425,10 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { weights_(std::make_shared( assets_, execution_context, config.weight_context_bytes, config.weight_storage_type)), - tokenizer_(assets_), + tokenizer_(assets_->gguf_tokenizer + ? VoxCPM2TokenizerWrapper(assets_->gguf_tokenizer) + : VoxCPM2TokenizerWrapper( + std::make_shared(assets_))), text_embedding_(weights_, config.text_embedding_graph_context_bytes, config.mem_saver), prefill_(weights_, config.lm_step_graph_context_bytes, @@ -1882,7 +1886,7 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { std::shared_ptr assets_; std::shared_ptr weights_; - VoxCPM2TextTokenizer tokenizer_; + VoxCPM2TokenizerWrapper tokenizer_; VoxCPM2TextEmbeddingRuntime text_embedding_; VoxCPM2PromptPrefillRuntime prefill_; VoxCPM2MiniCPMStepRuntime base_lm_; diff --git a/src/models/voxcpm2/tokenizer_gguf.cpp b/src/models/voxcpm2/tokenizer_gguf.cpp new file mode 100644 index 00000000..e849fff9 --- /dev/null +++ b/src/models/voxcpm2/tokenizer_gguf.cpp @@ -0,0 +1,358 @@ +#include "engine/models/voxcpm2/tokenizer_gguf.h" + +#include "engine/framework/assets/tensor_source.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace engine::models::voxcpm2 { +namespace { + +// UTF-8 handling functions (copied from tokenizer_text.cpp) +uint32_t next_utf8_codepoint(std::string_view text, size_t & offset) { + if (offset >= text.size()) { + throw std::runtime_error("VoxCPM2 tokenizer UTF-8 offset is out of range"); + } + const unsigned char first = static_cast(text[offset]); + uint32_t codepoint = 0; + size_t len = 1; + if ((first & 0x80U) == 0) { + codepoint = first; + } else if ((first & 0xE0U) == 0xC0U) { + len = 2; + codepoint = first & 0x1FU; + } else if ((first & 0xF0U) == 0xE0U) { + len = 3; + codepoint = first & 0x0FU; + } else if ((first & 0xF8U) == 0xF0U) { + len = 4; + codepoint = first & 0x07U; + } else { + throw std::runtime_error("VoxCPM2 tokenizer encountered invalid UTF-8"); + } + if (offset + len > text.size()) { + throw std::runtime_error("VoxCPM2 tokenizer encountered truncated UTF-8"); + } + for (size_t i = 1; i < len; ++i) { + const unsigned char ch = static_cast(text[offset + i]); + if ((ch & 0xC0U) != 0x80U) { + throw std::runtime_error("VoxCPM2 tokenizer encountered invalid UTF-8 continuation"); + } + codepoint = (codepoint << 6U) | (ch & 0x3FU); + } + offset += len; + return codepoint; +} + +std::vector utf8_codepoints(std::string_view text) { + std::vector out; + for (size_t offset = 0; offset < text.size();) { + const size_t start = offset; + (void) next_utf8_codepoint(text, offset); + out.emplace_back(text.substr(start, offset - start)); + } + return out; +} + +std::string normalize_text(std::string_view text) { + const std::string space = "\xE2\x96\x81"; + std::string out = space; + for (char ch : text) { + if (ch == ' ') { + out += space; + } else { + out.push_back(ch); + } + } + return out; +} + +std::string byte_fallback_token(unsigned char byte) { + constexpr char kHex[] = "0123456789ABCDEF"; + std::string out = "<0x"; + out.push_back(kHex[(byte >> 4U) & 0x0FU]); + out.push_back(kHex[byte & 0x0FU]); + out.push_back('>'); + return out; +} + +std::vector bpe_initial_pieces( + std::string_view normalized_text, + const std::unordered_map & vocab) { + std::vector pieces; + for (size_t offset = 0; offset < normalized_text.size();) { + const size_t start = offset; + (void) next_utf8_codepoint(normalized_text, offset); + std::string piece(normalized_text.substr(start, offset - start)); + if (vocab.find(piece) != vocab.end()) { + pieces.push_back(std::move(piece)); + continue; + } + for (size_t i = start; i < offset; ++i) { + pieces.push_back(byte_fallback_token(static_cast(normalized_text[i]))); + } + } + return pieces; +} + +bool is_cjk_codepoint(uint32_t codepoint) { + return (codepoint >= 0x4E00 && codepoint <= 0x9FFF) || + (codepoint >= 0x3400 && codepoint <= 0x4DBF) || + (codepoint >= 0xF900 && codepoint <= 0xFAFF) || + (codepoint >= 0x20000 && codepoint <= 0x2A6DF); +} + +bool is_pure_multichar_cjk(std::string_view text) { + size_t count = 0; + for (size_t offset = 0; offset < text.size();) { + if (!is_cjk_codepoint(next_utf8_codepoint(text, offset))) { + return false; + } + ++count; + } + return count >= 2; +} + +std::string strip_sentencepiece_prefix(std::string token) { + const std::string prefix = "\xE2\x96\x81"; + size_t pos = 0; + while ((pos = token.find(prefix, pos)) != std::string::npos) { + token.erase(pos, prefix.size()); + } + return token; +} + +bool starts_with_at(std::string_view text, size_t pos, std::string_view prefix) { + return pos + prefix.size() <= text.size() && text.substr(pos, prefix.size()) == prefix; +} + +std::string pair_key(const std::string & left, const std::string & right) { + std::string key = left; + key.push_back('\0'); + key += right; + return key; +} + +} // namespace + +struct VoxCPM1GgufTokenizer::Impl { + std::unordered_map vocab; + std::unordered_map id_to_token; + std::unordered_map special_tokens; + std::unordered_map merge_ranks; + std::unordered_map> cjk_split_map; + int32_t audio_start_token_id = 101; + int32_t audio_end_token_id = 102; + int32_t reference_audio_start_token_id = 103; + int32_t reference_audio_end_token_id = 104; + int32_t bos_token_id_ = 1; + int32_t eos_token_id_ = 2; + int32_t unk_token_id_ = 3; + + std::vector bpe(std::string_view normalized_text) const { + std::vector word = bpe_initial_pieces(normalized_text, vocab); + if (word.size() <= 1) { + return word; + } + while (true) { + int32_t best_rank = std::numeric_limits::max(); + size_t best_index = word.size(); + for (size_t i = 0; i + 1 < word.size(); ++i) { + const auto it = merge_ranks.find(pair_key(word[i], word[i + 1])); + if (it != merge_ranks.end() && it->second < best_rank) { + best_rank = it->second; + best_index = i; + } + } + if (best_index == word.size()) { + break; + } + word[best_index] += word[best_index + 1]; + word.erase(word.begin() + static_cast(best_index + 1)); + if (word.size() <= 1) { + break; + } + } + return word; + } + + void append_expanded_id(std::vector & ids, int32_t id) const { + const auto split = cjk_split_map.find(id); + if (split == cjk_split_map.end()) { + ids.push_back(id); + return; + } + ids.insert(ids.end(), split->second.begin(), split->second.end()); + } +}; + +VoxCPM1GgufTokenizer::VoxCPM1GgufTokenizer(std::shared_ptr gguf_source) { + if (!gguf_source) { + throw std::runtime_error("VoxCPM1 GGUF tokenizer requires a valid GGUF tensor source"); + } + impl_ = std::make_shared(); + auto & impl = *impl_; + + // Read tokenizer metadata from GGUF directly in constructor + const std::string tokenizer_model = gguf_source->require_string("tokenizer.ggml.model"); + const std::string tokenizer_pre = gguf_source->require_string("tokenizer.ggml.pre"); + const std::vector tokens = gguf_source->require_string_array("tokenizer.ggml.tokens"); + const std::vector token_types = gguf_source->require_i32_array("tokenizer.ggml.token_type"); + const std::vector merges = gguf_source->require_string_array("tokenizer.ggml.merges"); + const uint32_t bos_id = gguf_source->require_u32("tokenizer.ggml.bos_token_id"); + const uint32_t eos_id = gguf_source->require_u32("tokenizer.ggml.eos_token_id"); + const uint32_t unk_id = gguf_source->require_u32("tokenizer.ggml.unknown_token_id"); + + if (tokenizer_model != "gpt2" || tokens.empty() || merges.empty() || token_types.size() != tokens.size()) { + throw std::runtime_error("Invalid VoxCPM1 GGUF tokenizer metadata"); + } + + constexpr int32_t kTokenTypeNormal = 1; + constexpr int32_t kTokenTypeByte = 6; + + for (size_t i = 0; i < tokens.size(); ++i) { + const int32_t id = static_cast(i); + impl.vocab.emplace(tokens[i], id); + impl.id_to_token.emplace(id, tokens[i]); + if (token_types[i] != kTokenTypeNormal && token_types[i] != kTokenTypeByte) { + impl.special_tokens.emplace(tokens[i], id); + } + } + + impl.bos_token_id_ = static_cast(bos_id); + impl.eos_token_id_ = static_cast(eos_id); + impl.unk_token_id_ = static_cast(unk_id); + + // Build merge ranks + int32_t rank = 0; + for (const std::string & merge_text : merges) { + const size_t split = merge_text.find(' '); + if (split == std::string::npos) { + ++rank; + continue; + } + const std::string left = merge_text.substr(0, split); + const std::string right = merge_text.substr(split + 1); + const auto left_it = impl.vocab.find(left); + const auto right_it = impl.vocab.find(right); + const auto merged_it = impl.vocab.find(left + right); + if (left_it != impl.vocab.end() && right_it != impl.vocab.end() && merged_it != impl.vocab.end()) { + impl.merge_ranks.emplace(pair_key(left, right), rank); + } + ++rank; + } + + if (impl.merge_ranks.empty()) { + throw std::runtime_error("VoxCPM1 GGUF tokenizer has no valid merge rules"); + } + + // Build CJK split map + for (const auto & [id, token] : impl.id_to_token) { + const std::string clean = strip_sentencepiece_prefix(token); + if (!is_pure_multichar_cjk(clean)) { + continue; + } + std::vector char_ids; + for (const auto & ch : utf8_codepoints(clean)) { + const auto it = impl.vocab.find(ch); + if (it == impl.vocab.end()) { + char_ids.clear(); + break; + } + char_ids.push_back(it->second); + } + if (!char_ids.empty()) { + impl.cjk_split_map.emplace(id, std::move(char_ids)); + } + } +} + +std::vector VoxCPM1GgufTokenizer::encode(const std::string & text) const { + std::vector ids; + for (size_t i = 0; i < text.size();) { + const auto special_it = std::find_if( + impl_->special_tokens.begin(), + impl_->special_tokens.end(), + [&](const auto & item) { return starts_with_at(text, i, item.first); }); + if (special_it != impl_->special_tokens.end()) { + impl_->append_expanded_id(ids, special_it->second); + i += special_it->first.size(); + continue; + } + + size_t next_special = text.size(); + for (const auto & [special, _] : impl_->special_tokens) { + const size_t pos = text.find(special, i); + if (pos != std::string::npos) { + next_special = std::min(next_special, pos); + } + } + const std::string normalized = normalize_text(std::string_view( + text.data() + static_cast(i), + next_special - i)); + for (const auto & bpe_token : impl_->bpe(normalized)) { + const auto vocab_it = impl_->vocab.find(bpe_token); + if (vocab_it == impl_->vocab.end()) { + throw std::runtime_error("VoxCPM1 tokenizer produced token not present in vocab: " + bpe_token); + } + impl_->append_expanded_id(ids, vocab_it->second); + } + i = next_special; + } + return ids; +} + +VoxCPM2TextPrompt VoxCPM1GgufTokenizer::build_prompt(const std::string & text) const { + if (text.empty()) { + throw std::runtime_error("VoxCPM1 requires non-empty text input"); + } + VoxCPM2TextPrompt prompt; + prompt.text = text; + prompt.input_ids = encode(text); + if (prompt.input_ids.empty()) { + throw std::runtime_error("VoxCPM1 tokenizer produced no tokens"); + } + return prompt; +} + +int32_t VoxCPM1GgufTokenizer::audio_start_token_id() const noexcept { + return impl_->audio_start_token_id; +} + +int32_t VoxCPM1GgufTokenizer::audio_end_token_id() const noexcept { + return impl_->audio_end_token_id; +} + +int32_t VoxCPM1GgufTokenizer::reference_audio_start_token_id() const noexcept { + return impl_->reference_audio_start_token_id; +} + +int32_t VoxCPM1GgufTokenizer::reference_audio_end_token_id() const noexcept { + return impl_->reference_audio_end_token_id; +} + +int32_t VoxCPM1GgufTokenizer::bos_token_id() const noexcept { + return impl_->bos_token_id_; +} + +int32_t VoxCPM1GgufTokenizer::eos_token_id() const noexcept { + return impl_->eos_token_id_; +} + +int32_t VoxCPM1GgufTokenizer::unk_token_id() const noexcept { + return impl_->unk_token_id_; +} + +bool VoxCPM1GgufTokenizer::has_tokenizer_metadata(const engine::assets::TensorSource & source) { + return source.optional_string("tokenizer.ggml.model").has_value() && + source.optional_string_array("tokenizer.ggml.tokens").has_value() && + source.optional_string_array("tokenizer.ggml.merges").has_value(); +} + +} // namespace engine::models::voxcpm2 \ No newline at end of file diff --git a/src/models/voxcpm2/tokenizer_text.cpp b/src/models/voxcpm2/tokenizer_text.cpp index 8f49c619..ecdc62e6 100644 --- a/src/models/voxcpm2/tokenizer_text.cpp +++ b/src/models/voxcpm2/tokenizer_text.cpp @@ -1,4 +1,5 @@ #include "engine/models/voxcpm2/tokenizer_text.h" +#include "engine/models/voxcpm2/assets.h" #include "engine/framework/io/json.h" From bc80b8b3cc3c13760f83db7473cac3665dfb0123 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Tue, 18 Aug 2026 01:52:27 +0200 Subject: [PATCH 05/23] Fix VoxCPM1 issues: sample rate (V1.5) and early stopping (V1 0.5B) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - config_gguf.cpp: output_sample_rate now falls back to sample_rate (not 16000) VoxCPM1.5 GGUF has sample_rate=44100 but no out_sample_rate → was defaulting to 16kHz - session.cpp: add V1-specific default min_tokens to prevent early stop token trigger VoxCPM1 (patch_size=2): min_tokens=20, VoxCPM1.5 (patch_size=4): min_tokens=12 Without this, stop token triggers at ~2 tokens causing 1.28s cutoff - Stop predictor weights correctly loaded via V1 relaxed rank (no transpose needed) GGUF stores [1024,2] (GGML), expected logical [2,1024] → to_ggml_dims → [1024,2] ✓ Results: VoxCPM1 (0.5B): durations scale 1.76s→4.32s with text length VoxCPM1.5 (1.5B): durations scale 2.56s→5.12s, correct 44.1kHz sample rate VoxCPM2: regression passes (48kHz, 1.28s) Files: config_gguf.cpp (+6), session.cpp (+14) --- src/models/voxcpm2/config_gguf.cpp | 8 ++++++-- src/models/voxcpm2/session.cpp | 14 ++++++++++++++ 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/src/models/voxcpm2/config_gguf.cpp b/src/models/voxcpm2/config_gguf.cpp index eb961f94..37255586 100644 --- a/src/models/voxcpm2/config_gguf.cpp +++ b/src/models/voxcpm2/config_gguf.cpp @@ -127,8 +127,12 @@ VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & config.audio_vae.decoder_dim = get_optional_i64("voxcpm_audio_vae_config_decoder_dim").value_or(512); config.audio_vae.decoder_rates = get_optional_i64_array("voxcpm_audio_vae_config_decoder_rates").value_or(std::vector{2, 2, 2, 2}); config.audio_vae.sample_rate_bin_boundaries = get_optional_i64_array("voxcpm_audio_vae_config_sr_bin_boundaries").value_or(std::vector{}); - config.audio_vae.sample_rate = static_cast(get_optional_i64("voxcpm_audio_vae_config_sample_rate").value_or(16000)); - config.audio_vae.output_sample_rate = static_cast(get_optional_i64("voxcpm_audio_vae_config_out_sample_rate").value_or(16000)); + auto sample_rate_opt = get_optional_i64("voxcpm_audio_vae_config_sample_rate"); + config.audio_vae.sample_rate = static_cast(sample_rate_opt.value_or(16000)); + config.audio_vae.output_sample_rate = static_cast( + get_optional_i64("voxcpm_audio_vae_config_out_sample_rate") + .value_or(sample_rate_opt.value_or(16000)) + ); // Max length config.max_length = get_optional_i64("voxcpm_max_length").value_or(2048); diff --git a/src/models/voxcpm2/session.cpp b/src/models/voxcpm2/session.cpp index 780ae8e9..21375fc4 100644 --- a/src/models/voxcpm2/session.cpp +++ b/src/models/voxcpm2/session.cpp @@ -480,9 +480,23 @@ const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( VoxCPM2GenerationOptions VoxCPM2SessionBase::generation_options_from_request( const runtime::TaskRequest &request) const { VoxCPM2GenerationOptions options; + bool min_tokens_explicit = false; if (const auto value = runtime::parse_i64_option( request.options, {"voxcpm2.min_tokens", "min_tokens"})) { options.min_tokens = *value; + min_tokens_explicit = true; + } + // Set V1-specific default min_tokens if not explicitly provided + if (!min_tokens_explicit && assets_->config.v1) { + // VoxCPM1 (0.5B) has patch_size=2, VoxCPM1.5 (1.5B) has patch_size=4 + // Use model-specific defaults for optimal speech speed + if (assets_->config.patch_size == 2) { + options.min_tokens = 20; // 20, VoxCPM1 0.5B + } else if (assets_->config.patch_size == 4) { + options.min_tokens = 12; // VoxCPM1.5 1.5B + } else { + options.min_tokens = 15; // Other V1 models + } } if (const auto value = runtime::parse_i64_option( request.options, {"max_tokens", "voxcpm2.max_tokens"})) { From 85e5859da569488cfecc4392681a32cf1670ed29 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Tue, 18 Aug 2026 12:58:34 +0200 Subject: [PATCH 06/23] **feat(voxcpm1): enable voice clone & streaming support (parity with VoxCPM2)** MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit VoxCPM1 (0.5B/1.5B) models now support voice cloning (`--voice-ref`) and streaming output (`--mode streaming`), matching the VoxCPM2 feature surface. The inference math was already shared; this unblocks the capability/option/reporting layer. **Root causes fixed (5 gaps):** - Capability advertisement: now exposes `Tts + {Offline, Streaming}` for V1 (was TTS-only) - Family identity: `family_impl()` returns `"voxcpm1"` for V1 models (was hardcoded `"voxcpm2"`) - Session options: `normalize_v1_session_options()` rewrites `voxcpm1.*` → `voxcpm2.*` keys so aliases work - Request options: added `voxcpm1.*` aliases for all params (`prompt_text`, `min_tokens`, `guidance_scale`, `retry_badcase`, etc.) - Model spec: `voxcpm1.json` adds `streaming` mode, correct sample rates (16kHz/44.1kHz) **Changes:** 7 files, +167/−32 lines - `src/models/voxcpm2/session.cpp` — option normalization, family-aware errors, request-option aliases - `src/models/voxcpm2/loader.cpp` — capability advertisement, family-labeled errors - `model_specs/voxcpm1.json` — streaming mode, tags, corrected description - `docs/tts.md` — V1 streaming/voice-clone examples, `retry_badcase=false` requirement - `tools/audiocpp_cli/audiocpp_cli_path_cases.json` — 3 new V1 path tests - `webui/configs/models_catalog.json` + `model_params.json` — V1 WebUI entries **Verified (CPU):** | Test | Result | |------|--------| | V1 offline TTS | `family=voxcpm1` ✓ | | V1 voice clone | 16kHz, 5.12s, RMS 0.115 ✓ | | V1 streaming | 40×1280 chunks, 16kHz ✓ | | V1 `voxcpm1.*` session/request options | accepted & applied ✓ | | V1 capability inspection | `modes=offline,streaming` ✓ | | V2 regression (offline/streaming) | 48kHz, parity maintained ✓ | Streaming requires `retry_badcase=false` (same as V2, pre-existing design). No V2 behavior changes. **Issue**: The audio quality is still bad --- docs/tts.md | 20 +- model_specs/voxcpm1.json | 11 +- src/models/voxcpm2/loader.cpp | 11 +- src/models/voxcpm2/session.cpp | 81 +- src/models/voxcpm2/session.cpp~ | 708 ++++++++++++++++++ .../audiocpp_cli/audiocpp_cli_path_cases.json | 68 ++ webui/configs/model_params.json | 7 + webui/configs/models_catalog.json | 1 + 8 files changed, 875 insertions(+), 32 deletions(-) create mode 100644 src/models/voxcpm2/session.cpp~ diff --git a/docs/tts.md b/docs/tts.md index e77ccd8f..92767fe9 100644 --- a/docs/tts.md +++ b/docs/tts.md @@ -402,16 +402,16 @@ audiocpp_cli --task tts --family pocket_tts --model models/pocket-tts --backend ## VoxCPM1 -VoxCPM1 supports offline TTS. It reuses the VoxCPM2 runtime tree with a GGUF tensor-adaptation layer that understands the OpenBMB folded AudioVAE weights, so the same `--family voxcpm1` path serves both the 16 kHz 0.5B model and the 44.1 kHz 1.5B variants. +VoxCPM1 supports offline and streaming TTS plus short-reference voice cloning. It reuses the VoxCPM2 runtime tree with a GGUF tensor-adaptation layer that understands the OpenBMB folded AudioVAE weights, so the same `--family voxcpm1` path serves both the 16 kHz 0.5B model and the 44.1 kHz 1.5B variants. | Field | Value | |---|---| | Family | `voxcpm1` | | Model directory | `models/VoxCPM1-GGUF` (0.5B), `models/VoxCPM1.5-GGUF` (1.5B) | | Task | `tts` | -| Modes | `offline` | +| Modes | `offline`, `streaming` | | Languages | Model auto-handles supported languages | -| Voice input | Optional reference WAV | +| Voice input | Optional reference WAV; optional transcript through `--reference-text` | | Built-in voices | Not exposed | Text to speech: @@ -426,9 +426,23 @@ audiocpp_cli --task tts --family voxcpm1 --model models/VoxCPM1-GGUF/voxcpm-0.5b audiocpp_cli --task tts --family voxcpm1 --model models/VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf --backend cpu --text "Hello from VoxCPM1." --out out.wav ``` +Voice clone: + +```bash +audiocpp_cli --task tts --family voxcpm1 --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf --backend cpu --text "Hello from VoxCPM1." --voice-ref assets/resources/b.wav --out out.wav +``` + +Streaming output: + +```bash +audiocpp_cli --task tts --family voxcpm1 --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf --backend cpu --mode streaming --text "Hello from VoxCPM1." --request-option retry_badcase=false --out out.wav +``` + | Option | Values | Default | Meaning | |---|---:|---:|---| | `--voice-ref` | WAV path | not set | Reference speaker audio. | +| `--reference-text` | text | empty string | Transcript for the reference audio (clone prompting). | +| `--mode` | `offline`, `streaming` | `offline` | Full-output or streaming run mode; streaming requires `retry_badcase=false`. | | `--session-option voxcpm1.mem_saver=true\|false` | bool | `false` | Use tighter graph workspaces and release MiniCPM/AudioVAE request graphs after completion to reduce resident VRAM. | | `--session-option voxcpm1.prompt_cache_slots=` | integer | `1` | Prompt and prompt-audio embedding cache slots. Set to `0` to disable prompt caching. | | `--max-tokens` | integer | `4096` | Maximum generated AR tokens. | diff --git a/model_specs/voxcpm1.json b/model_specs/voxcpm1.json index e532cf7d..ee9d0ebe 100644 --- a/model_specs/voxcpm1.json +++ b/model_specs/voxcpm1.json @@ -1,7 +1,7 @@ { "family": "voxcpm1", "display_name": "VoxCPM1", - "description": "OpenBMB VoxCPM 0.5B and 1.5B tokenizer-free TTS models with 24kHz output.", + "description": "OpenBMB VoxCPM 0.5B and 1.5B tokenizer-free TTS models supporting short-reference voice cloning and streaming output (16kHz for 0.5B, 44.1kHz for 1.5B).", "category": "tts", "status": "supported", "tasks": [ @@ -9,7 +9,8 @@ "clone" ], "modes": [ - "offline" + "offline", + "streaming" ], "languages": [ "zh", @@ -24,7 +25,8 @@ }, "runtime": { "tags": [ - "gguf" + "gguf", + "stream" ] }, "ui": { @@ -32,7 +34,8 @@ "tags": [ "TTS", "Clone", - "GGUF" + "GGUF", + "Stream" ], "docs": [ "docs/tts.md", diff --git a/src/models/voxcpm2/loader.cpp b/src/models/voxcpm2/loader.cpp index 33df9175..3754b0a6 100644 --- a/src/models/voxcpm2/loader.cpp +++ b/src/models/voxcpm2/loader.cpp @@ -105,7 +105,7 @@ runtime::CapabilitySet capabilities_v1(const VoxCPM2Assets &) { runtime::CapabilitySet out; out.supported_tasks = { {runtime::VoiceTaskKind::Tts, - {runtime::RunMode::Offline}}, + {runtime::RunMode::Offline, runtime::RunMode::Streaming}}, }; out.languages = {"Auto"}; out.supports_speaker_reference = true; @@ -150,7 +150,7 @@ class VoxCPM1Loader final : public runtime::IVoiceModelLoader { runtime::CapabilitySet out; out.supported_tasks = { {runtime::VoiceTaskKind::Tts, - {runtime::RunMode::Offline}}, + {runtime::RunMode::Offline, runtime::RunMode::Streaming}}, }; out.supports_speaker_reference = true; return out; @@ -219,13 +219,14 @@ std::unique_ptr VoxCPM2LoadedModel::create_task_session( const runtime::TaskSpec &task, const runtime::SessionOptions &options) const { + const std::string family_label = metadata_.family == "voxcpm1" ? "VoxCPM1" : "VoxCPM2"; if (task.mode != runtime::RunMode::Offline && task.mode != runtime::RunMode::Streaming) { - throw std::runtime_error( - "VoxCPM2 only supports offline and streaming sessions"); + throw std::runtime_error(family_label + + " only supports offline and streaming sessions"); } if (task.task != runtime::VoiceTaskKind::Tts) { - throw std::runtime_error("VoxCPM2 only supports the Tts task"); + throw std::runtime_error(family_label + " only supports the Tts task"); } if (task.mode == runtime::RunMode::Streaming) { return std::make_unique(task, options, assets_); diff --git a/src/models/voxcpm2/session.cpp b/src/models/voxcpm2/session.cpp index 21375fc4..1f8f1680 100644 --- a/src/models/voxcpm2/session.cpp +++ b/src/models/voxcpm2/session.cpp @@ -50,6 +50,24 @@ void reject_denoiser_option( } } +std::unordered_map normalize_v1_session_options( + std::unordered_map options) { + // V1 sessions share this runtime but advertise "voxcpm1.*" options; alias + // them to "voxcpm2.*" so the shared parsing below accepts both spellings. + std::unordered_map out; + out.reserve(options.size()); + for (auto &[key, value] : options) { + constexpr std::string_view kV1Prefix = "voxcpm1."; + if (key.rfind(kV1Prefix, 0) == 0) { + out[std::string("voxcpm2.") + + key.substr(kV1Prefix.size())] = std::move(value); + } else { + out[std::move(key)] = std::move(value); + } + } + return out; +} + bool audio_buffer_equal(const runtime::AudioBuffer &lhs, const runtime::AudioBuffer &rhs) { return lhs.sample_rate == rhs.sample_rate && lhs.channels == rhs.channels && @@ -67,9 +85,9 @@ bool optional_audio_equal(const std::optional &lhs, size_t prompt_cache_slots_from_options( const std::unordered_map &options) { constexpr int64_t kDefaultPromptCacheSlots = 1; - const int64_t slots = - runtime::parse_i64_option(options, {"voxcpm2.prompt_cache_slots"}) - .value_or(kDefaultPromptCacheSlots); + const int64_t slots = runtime::parse_i64_option( + options, {"voxcpm2.prompt_cache_slots", "voxcpm1.prompt_cache_slots"}) + .value_or(kDefaultPromptCacheSlots); if (slots < 0) { throw std::runtime_error("voxcpm2.prompt_cache_slots must be non-negative"); } @@ -159,12 +177,17 @@ VoxCPM2SessionBase::VoxCPM2SessionBase(runtime::TaskSpec task, if (task_.mode != runtime::RunMode::Offline && task_.mode != runtime::RunMode::Streaming) { throw std::runtime_error( - "VoxCPM2 only supports offline and streaming sessions"); + std::string(assets_->config.v1 ? "VoxCPM1" : "VoxCPM2") + + " only supports offline and streaming sessions"); } if (task_.task != runtime::VoiceTaskKind::Tts) { - throw std::runtime_error("VoxCPM2 only supports the Tts task"); + throw std::runtime_error( + std::string(assets_->config.v1 ? "VoxCPM1" : "VoxCPM2") + + " only supports the Tts task"); } + options.options = normalize_v1_session_options(std::move(options.options)); + reject_enabled_denoise(options.options, {"voxcpm2.denoise"}); reject_enabled_denoise(options.options, {"voxcpm2.load_denoiser"}); reject_denoiser_option(options.options, {"voxcpm2.denoiser"}); @@ -225,7 +248,9 @@ VoxCPM2SessionBase::VoxCPM2SessionBase(runtime::TaskSpec task, VoxCPM2SessionBase::~VoxCPM2SessionBase() = default; -std::string VoxCPM2SessionBase::family_impl() const { return "voxcpm2"; } +std::string VoxCPM2SessionBase::family_impl() const { + return assets_->config.v1 ? "voxcpm1" : "voxcpm2"; +} runtime::VoiceTaskKind VoxCPM2SessionBase::task_kind_impl() const { return task_.task; } @@ -263,6 +288,7 @@ runtime::TaskResult VoxCPM2SessionBase::run_offline_request(const runtime::TaskR const auto generation_options = generation_options_from_request(request); const auto prompt_text = runtime::find_option(request.options, {"voxcpm2.prompt_text", + "voxcpm1.prompt_text", "prompt_text", "reference_text"}) .value_or(""); std::optional reference_audio; @@ -340,6 +366,7 @@ VoxCPM2SessionBase::run_streaming_request( auto generation_options = generation_options_from_request(request); const auto prompt_text = runtime::find_option(request.options, {"voxcpm2.prompt_text", + "voxcpm1.prompt_text", "prompt_text", "reference_text"}) .value_or(""); std::optional reference_audio; @@ -482,7 +509,8 @@ VoxCPM2GenerationOptions VoxCPM2SessionBase::generation_options_from_request( VoxCPM2GenerationOptions options; bool min_tokens_explicit = false; if (const auto value = runtime::parse_i64_option( - request.options, {"voxcpm2.min_tokens", "min_tokens"})) { + request.options, + {"voxcpm2.min_tokens", "voxcpm1.min_tokens", "min_tokens"})) { options.min_tokens = *value; min_tokens_explicit = true; } @@ -499,40 +527,50 @@ VoxCPM2GenerationOptions VoxCPM2SessionBase::generation_options_from_request( } } if (const auto value = runtime::parse_i64_option( - request.options, {"max_tokens", "voxcpm2.max_tokens"})) { + request.options, + {"max_tokens", "voxcpm2.max_tokens", "voxcpm1.max_tokens"})) { options.max_tokens = *value; } if (const auto value = runtime::parse_i64_option( request.options, - {"num_inference_steps", "voxcpm2.num_inference_steps"})) { + {"num_inference_steps", "voxcpm2.num_inference_steps", + "voxcpm1.num_inference_steps"})) { options.num_inference_steps = *value; } if (const auto value = runtime::parse_finite_float_option( - request.options, {"guidance_scale", "voxcpm2.guidance_scale"})) { + request.options, + {"guidance_scale", "voxcpm2.guidance_scale", + "voxcpm1.guidance_scale"})) { options.guidance_scale = *value; } if (const auto match = runtime::find_option_match( - request.options, {"voxcpm2.retry_badcase", "retry_badcase"})) { + request.options, + {"voxcpm2.retry_badcase", "voxcpm1.retry_badcase", + "retry_badcase"})) { options.retry_badcase = runtime::parse_bool_option(match->value, match->key); } if (const auto value = runtime::parse_i64_option( request.options, - {"voxcpm2.retry_badcase_max_times", "retry_badcase_max_times"})) { + {"voxcpm2.retry_badcase_max_times", + "voxcpm1.retry_badcase_max_times", "retry_badcase_max_times"})) { options.retry_badcase_max_times = *value; } if (const auto value = runtime::parse_finite_float_option( - request.options, {"voxcpm2.retry_badcase_ratio_threshold", - "retry_badcase_ratio_threshold"})) { + request.options, + {"voxcpm2.retry_badcase_ratio_threshold", + "voxcpm1.retry_badcase_ratio_threshold", + "retry_badcase_ratio_threshold"})) { options.retry_badcase_ratio_threshold = *value; } - if (const auto value = runtime::parse_u32_option(request.options, - {"voxcpm2.seed", "seed"})) { + if (const auto value = runtime::parse_u32_option( + request.options, {"voxcpm2.seed", "voxcpm1.seed", "seed"})) { options.seed = *value; } options.cfm_noise_file = runtime::find_option(request.options, - {"voxcpm2.cfm_noise_file", "cfm_noise_file"}) + {"voxcpm2.cfm_noise_file", "voxcpm1.cfm_noise_file", + "cfm_noise_file"}) .value_or(""); if (options.min_tokens < 0) { throw std::runtime_error("VoxCPM2 min_tokens must be non-negative"); @@ -565,10 +603,13 @@ VoxCPM2GenerationOptions VoxCPM2SessionBase::generation_options_from_request( throw std::runtime_error( "VoxCPM2 retry_badcase_ratio_threshold must be positive"); } - reject_enabled_denoise(request.options, {"voxcpm2.denoise", "denoise"}); reject_enabled_denoise(request.options, - {"voxcpm2.load_denoiser", "load_denoiser"}); - reject_denoiser_option(request.options, {"voxcpm2.denoiser", "denoiser"}); + {"voxcpm2.denoise", "voxcpm1.denoise", "denoise"}); + reject_enabled_denoise(request.options, + {"voxcpm2.load_denoiser", "voxcpm1.load_denoiser", + "load_denoiser"}); + reject_denoiser_option(request.options, + {"voxcpm2.denoiser", "voxcpm1.denoiser", "denoiser"}); return options; } diff --git a/src/models/voxcpm2/session.cpp~ b/src/models/voxcpm2/session.cpp~ new file mode 100644 index 00000000..c3cf78bf --- /dev/null +++ b/src/models/voxcpm2/session.cpp~ @@ -0,0 +1,708 @@ +#include "engine/models/voxcpm2/session.h" + +#include "engine/framework/debug/profiler.h" +#include "engine/framework/runtime/options.h" +#include "engine/framework/text/chunking.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace engine::models::voxcpm2 { +namespace { + +using Clock = std::chrono::steady_clock; +constexpr int64_t kDefaultTextChunkSize = 2048; + +std::shared_ptr +require_assets(std::shared_ptr assets) { + if (assets == nullptr) { + throw std::runtime_error("VoxCPM2 session requires assets"); + } + return assets; +} + +void reject_enabled_denoise( + const std::unordered_map &options, + std::initializer_list keys) { + const auto match = runtime::find_option_match(options, keys); + if (match.has_value() && + runtime::parse_bool_option(match->value, match->key)) { + throw std::runtime_error( + "VoxCPM2 denoise is disabled in this implementation"); + } +} + +void reject_denoiser_option( + const std::unordered_map &options, + std::initializer_list keys) { + if (runtime::find_option_match(options, keys).has_value()) { + throw std::runtime_error( + "VoxCPM2 denoise is disabled in this implementation"); + } +} + +bool audio_buffer_equal(const runtime::AudioBuffer &lhs, + const runtime::AudioBuffer &rhs) { + return lhs.sample_rate == rhs.sample_rate && lhs.channels == rhs.channels && + lhs.samples == rhs.samples; +} + +bool optional_audio_equal(const std::optional &lhs, + const std::optional &rhs) { + if (lhs.has_value() != rhs.has_value()) { + return false; + } + return !lhs.has_value() || audio_buffer_equal(*lhs, *rhs); +} + +size_t prompt_cache_slots_from_options( + const std::unordered_map &options) { + constexpr int64_t kDefaultPromptCacheSlots = 1; + const int64_t slots = + runtime::parse_i64_option(options, {"voxcpm2.prompt_cache_slots"}) + .value_or(kDefaultPromptCacheSlots); + if (slots < 0) { + throw std::runtime_error("voxcpm2.prompt_cache_slots must be non-negative"); + } + return static_cast(slots); +} + +void validate_weight_storage(engine::assets::TensorStorageType storage_type, + const char *option_name) { + if (storage_type == engine::assets::TensorStorageType::Native || + storage_type == engine::assets::TensorStorageType::F32 || + storage_type == engine::assets::TensorStorageType::F16 || + storage_type == engine::assets::TensorStorageType::BF16 || + storage_type == engine::assets::TensorStorageType::Q8_0) { + return; + } + throw std::runtime_error(std::string(option_name) + + " supports only native, f32, f16, bf16, and q8_0"); +} + +void parse_weight_type( + const std::unordered_map &options, + const char *key, engine::assets::TensorStorageType &storage_type) { + const auto it = options.find(key); + if (it == options.end()) { + return; + } + storage_type = engine::assets::parse_tensor_storage_type(it->second); + validate_weight_storage(storage_type, key); +} + +void validate_session_options( + const std::unordered_map &options) { + for (const auto &[key, value] : options) { + (void)value; + if (key.rfind("voxcpm2.", 0) != 0) { + continue; + } + if (key == "voxcpm2.weight_context_mb" || + key == "voxcpm2.text_embedding_graph_context_mb" || + key == "voxcpm2.lm_step_graph_context_mb" || + key == "voxcpm2.projection_graph_context_mb" || + key == "voxcpm2.local_encoder_graph_context_mb" || + key == "voxcpm2.dit_graph_context_mb" || + key == "voxcpm2.audiovae_weight_context_mb" || + key == "voxcpm2.audiovae_graph_context_mb" || + key == "voxcpm2.audiovae_encoder_graph_context_mb" || + key == "voxcpm2.audiovae_latent_capacity" || + key == "voxcpm2.audiovae_encoder_sample_capacity" || + key == "voxcpm2.weight_type" || + key == "voxcpm2.audiovae_weight_type" || + key == "voxcpm2.prompt_cache_slots" || + key == "voxcpm2.mem_saver" || + key == "voxcpm2.denoise" || key == "voxcpm2.load_denoiser") { + continue; + } + throw std::runtime_error("unknown VoxCPM2 session option: " + key); + } +} + +int64_t product(const std::vector &values) { + int64_t out = 1; + for (const int64_t value : values) { + if (value <= 0) { + throw std::runtime_error("VoxCPM2 AudioVAE decoder rate is invalid"); + } + out *= value; + } + return out; +} + +} // namespace + +bool VoxCPM2SessionBase::EncodedPromptCacheKeyEqual::operator()( + const EncodedPromptCacheKey &lhs, + const EncodedPromptCacheKey &rhs) const { + return lhs.prompt_text == rhs.prompt_text && + optional_audio_equal(lhs.prompt_audio, rhs.prompt_audio) && + optional_audio_equal(lhs.reference_audio, rhs.reference_audio); +} + +VoxCPM2SessionBase::VoxCPM2SessionBase(runtime::TaskSpec task, + runtime::SessionOptions options, + std::shared_ptr assets) + : RuntimeSessionBase(options), task_(task), + assets_(require_assets(std::move(assets))), + encoded_prompt_cache_(prompt_cache_slots_from_options(options.options)) { + if (task_.mode != runtime::RunMode::Offline && + task_.mode != runtime::RunMode::Streaming) { + throw std::runtime_error( + "VoxCPM2 only supports offline and streaming sessions"); + } + if (task_.task != runtime::VoiceTaskKind::Tts) { + throw std::runtime_error("VoxCPM2 only supports the Tts task"); + } + + reject_enabled_denoise(options.options, {"voxcpm2.denoise"}); + reject_enabled_denoise(options.options, {"voxcpm2.load_denoiser"}); + reject_denoiser_option(options.options, {"voxcpm2.denoiser"}); + validate_session_options(options.options); + + generator_config_.weight_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm2.weight_context_mb"}, + generator_config_.weight_context_bytes); + generator_config_.text_embedding_graph_context_bytes = + runtime::parse_size_mb_option( + options.options, {"voxcpm2.text_embedding_graph_context_mb"}, + generator_config_.text_embedding_graph_context_bytes); + generator_config_.lm_step_graph_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm2.lm_step_graph_context_mb"}, + generator_config_.lm_step_graph_context_bytes); + generator_config_.projection_graph_context_bytes = + runtime::parse_size_mb_option( + options.options, {"voxcpm2.projection_graph_context_mb"}, + generator_config_.projection_graph_context_bytes); + generator_config_.local_encoder_graph_context_bytes = + runtime::parse_size_mb_option( + options.options, {"voxcpm2.local_encoder_graph_context_mb"}, + generator_config_.local_encoder_graph_context_bytes); + generator_config_.dit_graph_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm2.dit_graph_context_mb"}, + generator_config_.dit_graph_context_bytes); + generator_config_.prompt_cache_slots = encoded_prompt_cache_.capacity(); + decoder_config_.weight_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm2.audiovae_weight_context_mb"}, + decoder_config_.weight_context_bytes); + decoder_config_.graph_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm2.audiovae_graph_context_mb"}, + decoder_config_.graph_context_bytes); + decoder_config_.encoder_graph_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm2.audiovae_encoder_graph_context_mb"}, + decoder_config_.encoder_graph_context_bytes); + decoder_config_.latent_frame_capacity = runtime::parse_positive_i64_option( + options.options, {"voxcpm2.audiovae_latent_capacity"}, + decoder_config_.latent_frame_capacity); + decoder_config_.encoder_sample_capacity = runtime::parse_positive_i64_option( + options.options, {"voxcpm2.audiovae_encoder_sample_capacity"}, + decoder_config_.encoder_sample_capacity); + parse_weight_type(options.options, "voxcpm2.weight_type", + generator_config_.weight_storage_type); + parse_weight_type(options.options, "voxcpm2.audiovae_weight_type", + decoder_config_.weight_storage_type); + if (const auto mem_saver = + runtime::find_option(options.options, {"voxcpm2.mem_saver"})) { + generator_config_.mem_saver = + runtime::parse_bool_option(*mem_saver, "voxcpm2.mem_saver"); + } + + generator_ = std::make_unique( + assets_, execution_context(), generator_config_); + decoder_ = std::make_unique( + assets_, execution_context(), decoder_config_); +} + +VoxCPM2SessionBase::~VoxCPM2SessionBase() = default; + +std::string VoxCPM2SessionBase::family_impl() const { return "voxcpm2"; } + +runtime::VoiceTaskKind VoxCPM2SessionBase::task_kind_impl() const { return task_.task; } + +runtime::RunMode VoxCPM2SessionBase::run_mode_impl() const { return task_.mode; } + +void VoxCPM2SessionBase::prepare_impl( + const runtime::SessionPreparationRequest &request) { + (void)request; + mark_prepared(); +} + +runtime::TaskResult VoxCPM2SessionBase::run_offline_request(const runtime::TaskRequest &request) { + require_prepared("VoxCPM2 run"); + if (task_.mode != runtime::RunMode::Offline) { + throw std::runtime_error("VoxCPM2 run requires an offline session"); + } + validate_request(request); + auto release_runtime_memory = [this](VoxCPM2SessionBase *self) { + if (self != nullptr) { + self->release_request_runtime_memory(); + } + }; + std::unique_ptr + release_guard(generator_config_.mem_saver ? this : nullptr, + release_runtime_memory); + + const auto wall_start = Clock::now(); + const int64_t text_chunk_size = + engine::text::parse_text_chunk_size_override(request.options).value_or(kDefaultTextChunkSize); + const auto text_chunk_mode = + engine::text::parse_text_chunk_mode_override(request.options) + .value_or(engine::text::TextChunkMode::TagAware); + const auto chunk_requests = + runtime::chunk_text_request(request, text_chunk_size, text_chunk_mode); + const auto generation_options = generation_options_from_request(request); + const auto prompt_text = + runtime::find_option(request.options, {"voxcpm2.prompt_text", + "prompt_text", "reference_text"}) + .value_or(""); + std::optional reference_audio; + if (request.voice.has_value() && request.voice->speaker.has_value() && + request.voice->speaker->audio.has_value()) { + reference_audio = *request.voice->speaker->audio; + } + const VoxCPM2EncodedPrompt *prompt = + encoded_prompt_for_request(request.audio_input, prompt_text, + reference_audio); + + runtime::TaskResult result; + double generator_ms = 0.0; + double decoder_ms = 0.0; + runtime::AudioBuffer merged_audio; + for (const auto & chunk_request : chunk_requests) { + const auto generator_start = Clock::now(); + const auto generated = generator_->generate( + chunk_request.text_input->text, prompt, generation_options); + generator_ms += engine::debug::elapsed_ms(generator_start, Clock::now()); + + const auto decoder_start = Clock::now(); + auto audio = decoder_->decode_features(generated.decode_features, + generated.decode_patches); + if (generated.decode_trim_patches > 0) { + const int64_t trim_samples = + generated.decode_trim_patches * assets_->config.patch_size * + product(assets_->config.audio_vae.decoder_rates); + if (trim_samples > static_cast(audio.samples.size())) { + throw std::runtime_error( + "VoxCPM2 decoded continuation trim exceeds audio length"); + } + audio.samples.erase( + audio.samples.begin(), + audio.samples.begin() + static_cast(trim_samples)); + } + decoder_ms += engine::debug::elapsed_ms(decoder_start, Clock::now()); + runtime::append_audio_buffer(merged_audio, audio); + } + result.audio_output = std::move(merged_audio); + + const auto wall_end = Clock::now(); + debug::trace_log_scalar("voxcpm2.text_chunk_size", text_chunk_size); + debug::trace_log_scalar("voxcpm2.text_chunk_mode", + engine::text::text_chunk_mode_name(text_chunk_mode)); + debug::trace_log_scalar("voxcpm2.text_chunk_count", + static_cast(chunk_requests.size())); + debug::timing_log_scalar("voxcpm2.generator_ms", generator_ms); + debug::timing_log_scalar("voxcpm2.audiovae_decoder_ms", decoder_ms); + debug::timing_log_scalar("session.wall_ms", + engine::debug::elapsed_ms(wall_start, wall_end)); + return result; +} + +runtime::TaskResult +VoxCPM2SessionBase::run_streaming_request( + const runtime::TaskRequest &request, + const runtime::StreamEventCallback &stream_event_sink) { + require_prepared("VoxCPM2 run_streaming"); + if (task_.mode != runtime::RunMode::Streaming) { + throw std::runtime_error( + "VoxCPM2 run_streaming requires a streaming session"); + } + validate_request(request); + auto release_runtime_memory = [this](VoxCPM2SessionBase *self) { + if (self != nullptr) { + self->release_request_runtime_memory(); + } + }; + std::unique_ptr + release_guard(generator_config_.mem_saver ? this : nullptr, + release_runtime_memory); + + const auto wall_start = Clock::now(); + auto generation_options = generation_options_from_request(request); + const auto prompt_text = + runtime::find_option(request.options, {"voxcpm2.prompt_text", + "prompt_text", "reference_text"}) + .value_or(""); + std::optional reference_audio; + if (request.voice.has_value() && request.voice->speaker.has_value() && + request.voice->speaker->audio.has_value()) { + reference_audio = *request.voice->speaker->audio; + } + const VoxCPM2EncodedPrompt *prompt = + encoded_prompt_for_request(request.audio_input, prompt_text, + reference_audio); + + runtime::TaskResult result; + runtime::AudioBuffer merged; + merged.sample_rate = assets_->config.audio_vae.output_sample_rate; + merged.channels = 1; + double decoder_ms = 0.0; + size_t emitted_chunks = 0; + auto emit_chunk = [&](const VoxCPM2StreamingChunk &chunk) { + const auto decoder_start = Clock::now(); + auto audio = decoder_->decode_features(chunk.decode_features, + chunk.decode_patches); + decoder_ms += engine::debug::elapsed_ms(decoder_start, Clock::now()); + if (emitted_chunks == 0) { + merged.sample_rate = audio.sample_rate; + merged.channels = audio.channels; + } else if (audio.sample_rate != merged.sample_rate || + audio.channels != merged.channels) { + throw std::runtime_error( + "VoxCPM2 streaming decoder chunk format changed"); + } + merged.samples.insert(merged.samples.end(), audio.samples.begin(), + audio.samples.end()); + runtime::NamedAudioBuffer named; + named.id = "chunk_" + std::to_string(emitted_chunks); + named.audio = std::move(audio); + named.meta.insert_or_assign( + "generated_patches", std::to_string(chunk.generated_patches)); + if (stream_event_sink) { + runtime::StreamEvent event; + event.named_audio_outputs.push_back(named); + stream_event_sink(event); + } + result.named_audio_outputs.push_back(std::move(named)); + ++emitted_chunks; + }; + + const auto generator_start = Clock::now(); + (void)generator_->generate_streaming(request.text_input->text, prompt, + generation_options, emit_chunk); + const auto generator_end = Clock::now(); + const double generator_with_callbacks_ms = + engine::debug::elapsed_ms(generator_start, generator_end); + + result.audio_output = std::move(merged); + + const auto wall_end = Clock::now(); + debug::timing_log_scalar( + "voxcpm2.generator_ms", + std::max(0.0, generator_with_callbacks_ms - decoder_ms)); + debug::timing_log_scalar("voxcpm2.generator_streaming_callbacks_ms", + generator_with_callbacks_ms); + debug::timing_log_scalar("voxcpm2.audiovae_decoder_ms", decoder_ms); + debug::timing_log_scalar("voxcpm2.streaming_chunks", + static_cast(emitted_chunks)); + debug::timing_log_scalar("session.wall_ms", + engine::debug::elapsed_ms(wall_start, wall_end)); + return result; +} + +void VoxCPM2SessionBase::release_request_runtime_memory() { + if (!generator_config_.mem_saver) { + return; + } + generator_->release_runtime_memory(); + decoder_->release_runtime_memory(); +} + +const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( + const std::optional &prompt_audio, + const std::string &prompt_text, + const std::optional &reference_audio) { + if (!prompt_audio.has_value() && !reference_audio.has_value()) { + return nullptr; + } + EncodedPromptCacheKey key; + key.prompt_text = prompt_text; + key.prompt_audio = prompt_audio; + key.reference_audio = reference_audio; + if (auto *cached = encoded_prompt_cache_.find(key)) { + debug::trace_log_scalar("voxcpm2.prompt_cache.hit", 1); + debug::trace_log_scalar("voxcpm2.prompt_cache.slots", + static_cast( + encoded_prompt_cache_.capacity())); + debug::trace_log_scalar("voxcpm2.prompt_cache.entries", + static_cast(encoded_prompt_cache_.size())); + debug::trace_log_scalar("voxcpm2.prompt_cache.evicted", 0); + debug::timing_log_scalar("voxcpm2.prompt_encode_ms", 0.0); + return &cached->encoded; + } + + const auto encode_start = Clock::now(); + EncodedPromptCacheEntry entry; + entry.encoded = + decoder_->encode_prompt_audio(prompt_audio, prompt_text, reference_audio); + const double encode_ms = engine::debug::elapsed_ms(encode_start); + if (encoded_prompt_cache_.capacity() == 0) { + uncached_encoded_prompt_ = std::move(entry); + debug::trace_log_scalar("voxcpm2.prompt_cache.hit", 0); + debug::trace_log_scalar("voxcpm2.prompt_cache.slots", 0); + debug::trace_log_scalar("voxcpm2.prompt_cache.entries", 0); + debug::trace_log_scalar("voxcpm2.prompt_cache.evicted", 0); + debug::timing_log_scalar("voxcpm2.prompt_encode_ms", encode_ms); + return &uncached_encoded_prompt_->encoded; + } + const bool will_evict = + encoded_prompt_cache_.size() >= encoded_prompt_cache_.capacity(); + encoded_prompt_cache_.put(std::move(key), std::move(entry)); + EncodedPromptCacheKey lookup; + lookup.prompt_text = prompt_text; + lookup.prompt_audio = prompt_audio; + lookup.reference_audio = reference_audio; + auto *cached = encoded_prompt_cache_.find(lookup); + if (cached == nullptr) { + throw std::runtime_error("VoxCPM2 prompt cache insert failed"); + } + debug::trace_log_scalar("voxcpm2.prompt_cache.hit", 0); + debug::trace_log_scalar("voxcpm2.prompt_cache.slots", + static_cast( + encoded_prompt_cache_.capacity())); + debug::trace_log_scalar("voxcpm2.prompt_cache.entries", + static_cast(encoded_prompt_cache_.size())); + debug::trace_log_scalar("voxcpm2.prompt_cache.evicted", will_evict ? 1 : 0); + debug::timing_log_scalar("voxcpm2.prompt_encode_ms", + encode_ms); + return &cached->encoded; +} + +VoxCPM2GenerationOptions VoxCPM2SessionBase::generation_options_from_request( + const runtime::TaskRequest &request) const { + VoxCPM2GenerationOptions options; + bool min_tokens_explicit = false; + if (const auto value = runtime::parse_i64_option( + request.options, {"voxcpm2.min_tokens", "min_tokens"})) { + options.min_tokens = *value; + min_tokens_explicit = true; + } + // Set V1-specific default min_tokens if not explicitly provided + if (!min_tokens_explicit && assets_->config.v1) { + // VoxCPM1 (0.5B) has patch_size=2, VoxCPM1.5 (1.5B) has patch_size=4 + // Use model-specific defaults for optimal speech speed + if (assets_->config.patch_size == 2) { + options.min_tokens = 15; // 20, VoxCPM1 0.5B + } else if (assets_->config.patch_size == 4) { + options.min_tokens = 12; // VoxCPM1.5 1.5B + } else { + options.min_tokens = 15; // Other V1 models + } + } + if (const auto value = runtime::parse_i64_option( + request.options, {"max_tokens", "voxcpm2.max_tokens"})) { + options.max_tokens = *value; + } + if (const auto value = runtime::parse_i64_option( + request.options, + {"num_inference_steps", "voxcpm2.num_inference_steps"})) { + options.num_inference_steps = *value; + } + if (const auto value = runtime::parse_finite_float_option( + request.options, {"guidance_scale", "voxcpm2.guidance_scale"})) { + options.guidance_scale = *value; + } + if (const auto match = runtime::find_option_match( + request.options, {"voxcpm2.retry_badcase", "retry_badcase"})) { + options.retry_badcase = + runtime::parse_bool_option(match->value, match->key); + } + if (const auto value = runtime::parse_i64_option( + request.options, + {"voxcpm2.retry_badcase_max_times", "retry_badcase_max_times"})) { + options.retry_badcase_max_times = *value; + } + if (const auto value = runtime::parse_finite_float_option( + request.options, {"voxcpm2.retry_badcase_ratio_threshold", + "retry_badcase_ratio_threshold"})) { + options.retry_badcase_ratio_threshold = *value; + } + if (const auto value = runtime::parse_u32_option(request.options, + {"voxcpm2.seed", "seed"})) { + options.seed = *value; + } + options.cfm_noise_file = + runtime::find_option(request.options, + {"voxcpm2.cfm_noise_file", "cfm_noise_file"}) + .value_or(""); + if (options.min_tokens < 0) { + throw std::runtime_error("VoxCPM2 min_tokens must be non-negative"); + } + if (options.max_tokens < 0) { + throw std::runtime_error("VoxCPM2 max_tokens must be non-negative"); + } + if (options.max_tokens == 0) { + options.max_tokens = assets_->config.max_length; + } + if (options.min_tokens > options.max_tokens) { + throw std::runtime_error("VoxCPM2 min_tokens must not exceed max_tokens"); + } + if (options.max_tokens > assets_->config.max_length) { + throw std::runtime_error( + "VoxCPM2 max_tokens exceeds model config max_length"); + } + if (options.num_inference_steps <= 0) { + throw std::runtime_error( + "VoxCPM2 num_inference_steps must be positive"); + } + if (options.guidance_scale < 0.0F) { + throw std::runtime_error("VoxCPM2 guidance_scale must be non-negative"); + } + if (options.retry_badcase_max_times <= 0) { + throw std::runtime_error( + "VoxCPM2 retry_badcase_max_times must be positive"); + } + if (options.retry_badcase_ratio_threshold <= 0.0F) { + throw std::runtime_error( + "VoxCPM2 retry_badcase_ratio_threshold must be positive"); + } + reject_enabled_denoise(request.options, {"voxcpm2.denoise", "denoise"}); + reject_enabled_denoise(request.options, + {"voxcpm2.load_denoiser", "load_denoiser"}); + reject_denoiser_option(request.options, {"voxcpm2.denoiser", "denoiser"}); + return options; +} + +void VoxCPM2SessionBase::validate_request( + const runtime::TaskRequest &request) const { + if (!request.text_input.has_value()) { + throw std::runtime_error("VoxCPM2 requires text input"); + } + if (request.text_input->text.empty()) { + throw std::runtime_error("VoxCPM2 text input must not be empty"); + } + if (request.voice.has_value()) { + if (request.voice->style.has_value()) { + throw std::runtime_error( + "VoxCPM2 C++ session does not consume style conditions"); + } + if (request.voice->speaker.has_value()) { + const auto &speaker = *request.voice->speaker; + if (speaker.cached_voice_id.has_value()) { + throw std::runtime_error("VoxCPM2 C++ session requires speaker " + "reference audio, not a cached voice id"); + } + if (!speaker.audio.has_value()) { + throw std::runtime_error( + "VoxCPM2 C++ session speaker condition requires audio"); + } + } + } + if (!request.input_artifacts.empty()) { + throw std::runtime_error( + "VoxCPM2 C++ session does not consume input artifacts"); + } +} + +VoxCPM2OfflineSession::VoxCPM2OfflineSession( + runtime::TaskSpec task, + runtime::SessionOptions options, + std::shared_ptr assets) + : VoxCPM2SessionBase(task, std::move(options), std::move(assets)) {} + +std::string VoxCPM2OfflineSession::family() const { return family_impl(); } + +runtime::VoiceTaskKind VoxCPM2OfflineSession::task_kind() const { + return task_kind_impl(); +} + +runtime::RunMode VoxCPM2OfflineSession::run_mode() const { + return run_mode_impl(); +} + +void VoxCPM2OfflineSession::prepare( + const runtime::SessionPreparationRequest &request) { + prepare_impl(request); +} + +runtime::TaskResult +VoxCPM2OfflineSession::run(const runtime::TaskRequest &request) { + return run_offline_request(request); +} + +VoxCPM2StreamingSession::VoxCPM2StreamingSession( + runtime::TaskSpec task, + runtime::SessionOptions options, + std::shared_ptr assets) + : VoxCPM2SessionBase(task, std::move(options), std::move(assets)) {} + +std::string VoxCPM2StreamingSession::family() const { return family_impl(); } + +runtime::VoiceTaskKind VoxCPM2StreamingSession::task_kind() const { + return task_kind_impl(); +} + +runtime::RunMode VoxCPM2StreamingSession::run_mode() const { + return run_mode_impl(); +} + +void VoxCPM2StreamingSession::prepare( + const runtime::SessionPreparationRequest &request) { + prepare_impl(request); +} + +runtime::StreamingPolicy VoxCPM2StreamingSession::streaming_policy() const { + runtime::StreamingPolicy policy; + policy.input = runtime::StreamingInputKind::None; + policy.output = runtime::StreamingOutputKind::FinalResult; + return policy; +} + +void VoxCPM2StreamingSession::start_stream(const runtime::TaskRequest &request) { + reset(); + result_ = run_streaming_request(request, stream_event_sink_); + started_ = true; +} + +void VoxCPM2StreamingSession::set_stream_event_sink(runtime::StreamEventCallback sink) { + stream_event_sink_ = std::move(sink); +} + +std::optional VoxCPM2StreamingSession::next_stream_event() { + if (!started_) { + throw std::runtime_error("VoxCPM2 streaming has not been started"); + } + if (next_chunk_index_ >= result_.named_audio_outputs.size()) { + return std::nullopt; + } + const auto & named = result_.named_audio_outputs[next_chunk_index_++]; + runtime::StreamEvent event; + event.named_audio_outputs.push_back(named); + return event; +} + +runtime::TaskResult VoxCPM2StreamingSession::finish_stream() { + if (!started_) { + throw std::runtime_error("VoxCPM2 streaming has not been started"); + } + started_ = false; + next_chunk_index_ = 0; + return std::move(result_); +} + +void VoxCPM2StreamingSession::reset() { + result_ = runtime::TaskResult{}; + next_chunk_index_ = 0; + started_ = false; +} + +runtime::StreamEvent VoxCPM2StreamingSession::process_audio_chunk( + const runtime::AudioChunk &chunk) { + (void)chunk; + throw std::runtime_error("VoxCPM2 streaming does not consume audio chunks"); +} + +runtime::TaskResult VoxCPM2StreamingSession::finalize() { + return finish_stream(); +} + +} // namespace engine::models::voxcpm2 diff --git a/tools/audiocpp_cli/audiocpp_cli_path_cases.json b/tools/audiocpp_cli/audiocpp_cli_path_cases.json index 2513515c..f522a666 100644 --- a/tools/audiocpp_cli/audiocpp_cli_path_cases.json +++ b/tools/audiocpp_cli/audiocpp_cli_path_cases.json @@ -737,6 +737,74 @@ } ] }, + { + "id": "voxcpm1_tts", + "coverage": "VoxCPM1 text-to-speech path with MiniCPM generation, diffusion feature generation, and AudioVAE decode", + "family": "voxcpm1", + "model": "models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf", + "task": "tts", + "mode": "offline", + "outputs": [ + "audio" + ], + "requests": [ + { + "id": "tts", + "text": "This VoxCPM1 path test checks the text-to-speech interface through AudioCPP CLI.", + "seed": 1234, + "max_tokens": 160, + "guidance_scale": 2.0, + "num_inference_steps": 10 + } + ] + }, + { + "id": "voxcpm1_voice_clone", + "coverage": "VoxCPM1 voice clone path with reference audio encoding, MiniCPM generation, diffusion feature generation, and AudioVAE decode", + "family": "voxcpm1", + "model": "models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf", + "task": "tts", + "mode": "offline", + "outputs": [ + "audio" + ], + "requests": [ + { + "id": "clone", + "text": "This VoxCPM1 path test clones the reference speaker for a short review sentence.", + "voice_ref": "resources/sample.wav", + "seed": 1234, + "max_tokens": 160, + "guidance_scale": 2.0, + "num_inference_steps": 10 + } + ] + }, + { + "id": "voxcpm1_streaming_tts", + "coverage": "VoxCPM1 streaming text-to-speech path with MiniCPM streaming generation, diffusion feature generation, and AudioVAE chunk decode", + "family": "voxcpm1", + "model": "models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf", + "task": "tts", + "mode": "streaming", + "chunk_size": 512, + "outputs": [ + "audio" + ], + "requests": [ + { + "id": "streaming_tts", + "text": "This VoxCPM1 streaming path test checks that the CLI can emit audio chunks for a longer request while preserving a steady speaking style.", + "seed": 1234, + "max_tokens": 160, + "guidance_scale": 2.0, + "num_inference_steps": 10, + "options": { + "retry_badcase": false + } + } + ] + }, { "id": "higgs_audio_tts_voice_clone_chunked", "coverage": "Higgs Audio v3 voice clone path with framework text chunking, AR generation, and codec decode", diff --git a/webui/configs/model_params.json b/webui/configs/model_params.json index 6214ce46..01d679e6 100644 --- a/webui/configs/model_params.json +++ b/webui/configs/model_params.json @@ -28,6 +28,13 @@ {"name": "retry_badcase", "type": "bool", "label": "retry_badcase（自动重试异常输出）", "default": true} ], + "voxcpm1": [ + {"name": "num_inference_steps", "type": "number", "label": "num_inference_steps", "default": 10, "minimum": 1, "step": 1, "precision": 0, "info": "CFM/DiT 步数"}, + {"name": "guidance_scale", "type": "slider", "label": "guidance_scale", "default": 2.0, "minimum": 0.0, "maximum": 5.0, "step": 0.1}, + {"name": "min_tokens", "type": "number", "label": "min_tokens", "default": 2, "minimum": 0, "step": 1, "precision": 0}, + {"name": "retry_badcase", "type": "bool", "label": "retry_badcase（自动重试异常输出）", "default": true} + ], + "miotts": [ {"name": "temperature", "type": "slider", "label": "temperature", "default": 0.8, "minimum": 0.0, "maximum": 2.0, "step": 0.05}, {"name": "top_k", "type": "number", "label": "top_k", "default": 50, "minimum": 0, "step": 1, "precision": 0}, diff --git a/webui/configs/models_catalog.json b/webui/configs/models_catalog.json index 05cefa2e..7cecec68 100644 --- a/webui/configs/models_catalog.json +++ b/webui/configs/models_catalog.json @@ -16,6 +16,7 @@ { "id": "qwen3-tts-1.7b-custom", "display_name": "Qwen3-TTS 1.7B CustomVoice (tts)", "family": "qwen3_tts", "path": "models/Qwen3-TTS-12Hz-1.7B-CustomVoice", "task": "tts", "mode": "offline", "download_id": "qwen3_tts_1_7b_custom_voice", "min_vram_gb": 8 }, { "id": "miotts", "display_name": "MioTTS 1.7B (tts; needs MioCodec)", "family": "miotts", "path": "models/MioTTS-1.7B", "task": "tts", "mode": "offline", "download_id": "miotts_1_7b", "min_vram_gb": 8 }, { "id": "voxcpm2", "display_name": "VoxCPM2 (tts)", "family": "voxcpm2", "path": "models/VoxCPM2", "task": "tts", "mode": "offline", "download_id": "voxcpm2", "session_options": { "voxcpm2.weight_type": "q8_0" }, "min_vram_gb": 6 }, + { "id": "voxcpm1", "display_name": "VoxCPM1 0.5B (tts + clone)", "family": "voxcpm1", "path": "models/VoxCPM1-GGUF", "task": "tts", "mode": "offline", "download_id": "voxcpm1_0.5b_q8_0", "min_vram_gb": 4 }, { "id": "vibevoice", "display_name": "VibeVoice 1.5B (tts, long-form/multi-speaker)", "family": "vibevoice", "path": "models/VibeVoice-1.5B", "task": "tts", "mode": "offline", "download_id": "vibevoice_1_5b", "min_vram_gb": 7 }, { "id": "index-tts2", "display_name": "IndexTTS2 (tts 中英克隆+情感)", "display_name_en": "IndexTTS2 (tts, zh/en clone + emotion)", "family": "index_tts2", "path": "models/IndexTTS-2", "task": "tts", "mode": "offline", "download_id": "index_tts2", "min_vram_gb": 8 }, { "id": "index-tts2.5", "display_name": "IndexTTS2.5 (tts 多语种克隆+情感, GGUF Q8)", "display_name_en": "IndexTTS2.5 (tts, zh/en/ja/es/ar clone + emotion, GGUF Q8)", "family": "index_tts2", "path": "models/IndexTTS2.5-GGUF", "task": "tts", "mode": "offline", "download_id": "index_tts2_5_q8_0", "min_vram_gb": 8, From a8fef3dc6281b0bdacafaf0f1491dab9c3761daf Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Tue, 18 Aug 2026 13:58:01 +0200 Subject: [PATCH 07/23] fix(voxcpm1): load real RoPE longrope factors and align min_tokens floor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit VoxCPM1 attention used identity longrope factors and a padded stop-token floor. The GGUF's real F32 factor arrays are now read and applied (prefill, stop behavior and duration match the VoxCPM.cpp reference), and the V1 default `min_tokens` is lowered to the reference floor so short utterances are no longer padded with trailing silence. **Root causes fixed (2):** - RoPE longrope factors were hardcoded to `1.0f` in the GGUF config path ("GGUF doesn't have native float arrays" was wrong — `GgufTensorSource` already parses them); every attention computation across all four transformers (base LM, residual LM, local encoder, local DiT) used identity positional encodings - V1 default `min_tokens=20` (per patch_size) vs reference `kMinLen=2` — forced ~1.6s+ of audio and padded short utterances with trailing silence after the stop predictor fired **Changes:** 2 files, +29/−12 lines - `src/models/voxcpm2/config_gguf.cpp` — read `voxcpm_lm_config_rope_scaling_{short,long}_factor` f32 arrays via `optional_f32_array()` with size validation (`head_dim/2`), identity fallback only when the keys are absent - `src/models/voxcpm2/session.cpp` — V1 default `min_tokens = 2` (≡ reference `step > kMinLen`), keeping the `--request-option min_tokens` override **Verified (CPU, against reference `/workspace/pi/VoxCPM.cpp`):** | Test | Result | |------|--------| | Prefill lm_hidden | l2 within ~2% of reference (was diverged) | | Stop predictor ("This is a test run for the fix") | fires at pos=19 (was: never fired) | | Duration | 1.60s (ref 1.68s), trailing silence 0.13s (ref 0.44s) | | V2 regression | 48kHz output maintained ✓ | | Embedding + fusion | `[73448,1024]` transpose intact, `has_fusion_proj=false` ✓ | **Issue**: Voice clone is still not supported — `--task clon` is rejected and passing reference audio + text (`--task tts --voice-ref `) generates noise rather than cloned speech. Needs a port-audit of the VoxCPM1 reference-audio conditioning path. Full evidence in `docs/reports/2026-08-18_1128_VoxCPM1_RoPE_Longrope_Factors_Stop_Floor_Fix.md`. --- src/models/voxcpm2/config_gguf.cpp | 27 ++++++++++++++++++++++++--- src/models/voxcpm2/session.cpp | 14 +++++--------- 2 files changed, 29 insertions(+), 12 deletions(-) diff --git a/src/models/voxcpm2/config_gguf.cpp b/src/models/voxcpm2/config_gguf.cpp index 37255586..218ca747 100644 --- a/src/models/voxcpm2/config_gguf.cpp +++ b/src/models/voxcpm2/config_gguf.cpp @@ -79,11 +79,32 @@ VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & // Rope scaling (longrope for VoxCPM1) config.lm.rope_scaling.type = "longrope"; - // GGUF doesn't have native float arrays, use defaults const int64_t head_dim = config.lm.hidden_size / config.lm.num_attention_heads; const int64_t factor_size = head_dim / 2; - config.lm.rope_scaling.long_factor.assign(factor_size, 1.0f); - config.lm.rope_scaling.short_factor.assign(factor_size, 1.0f); + // The GGUF stores the real longrope factor arrays as F32 metadata arrays + // (32 values for a 64-dim head, ~1.0004 to ~49.85 for VoxCPM1). Read them + // instead of the old identity fallback: identity factors silently degrade + // every RoPE computation across all four transformers. + auto short_factor = + source.optional_f32_array("voxcpm_lm_config_rope_scaling_short_factor"); + auto long_factor = + source.optional_f32_array("voxcpm_lm_config_rope_scaling_long_factor"); + if (short_factor && + static_cast(short_factor->size()) != factor_size) { + throw std::runtime_error( + "voxcpm_lm_config_rope_scaling_short_factor must have head_dim / 2 " + "elements"); + } + if (long_factor && + static_cast(long_factor->size()) != factor_size) { + throw std::runtime_error( + "voxcpm_lm_config_rope_scaling_long_factor must have head_dim / 2 " + "elements"); + } + config.lm.rope_scaling.short_factor = + short_factor.value_or(std::vector(factor_size, 1.0f)); + config.lm.rope_scaling.long_factor = + long_factor.value_or(std::vector(factor_size, 1.0f)); config.lm.rope_scaling.original_max_position_embeddings = get_optional_i64("voxcpm_lm_config_rope_scaling_original_max_position_embeddings").value_or(2048); diff --git a/src/models/voxcpm2/session.cpp b/src/models/voxcpm2/session.cpp index 1f8f1680..8b9ea0ec 100644 --- a/src/models/voxcpm2/session.cpp +++ b/src/models/voxcpm2/session.cpp @@ -516,15 +516,11 @@ VoxCPM2GenerationOptions VoxCPM2SessionBase::generation_options_from_request( } // Set V1-specific default min_tokens if not explicitly provided if (!min_tokens_explicit && assets_->config.v1) { - // VoxCPM1 (0.5B) has patch_size=2, VoxCPM1.5 (1.5B) has patch_size=4 - // Use model-specific defaults for optimal speech speed - if (assets_->config.patch_size == 2) { - options.min_tokens = 20; // 20, VoxCPM1 0.5B - } else if (assets_->config.patch_size == 4) { - options.min_tokens = 12; // VoxCPM1.5 1.5B - } else { - options.min_tokens = 15; // Other V1 models - } + // Reference VoxCPM.cpp uses kMinLen=2 (stop may fire from the 4th patch); + // the decode loop gates on `index > min_tokens`, which is the same check. + // A higher floor (e.g. 20) forces ~1.6 s of audio and pads short + // utterances with trailing silence after the stop predictor fires. + options.min_tokens = 2; } if (const auto value = runtime::parse_i64_option( request.options, From 65ea6a13f53d5511c30f1bde9664e3c2129873b9 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Wed, 19 Aug 2026 11:38:39 +0000 Subject: [PATCH 08/23] fix(voxcpm1): align voice-clone conditioning and VAE encoder with golden impl Restore working voice cloning by fixing the reference-audio conditioning and AudioVAE encoder alignment against the golden VoxCPM.cpp port: - generator: only set the CFM `prefix_cond` from prefill rows carrying audio (audio_mask). Previously the trailing text row's zero feature overwrote the reference patch, feeding the DiT a zero acoustic anchor for voice cloning (matches torch feat[:, -1] semantics) - audiovae: re-enable VAD silence trimming for prompt/reference audio (matches golden server_common.cpp:842/878), then pad to patch alignment before VAE encoding (left for prompt, right for reference) - audiovae: drop the `stride % 2` output_padding on the encoder downsample conv so causal padding matches the reference encoder - assets: declare base_lm.embed_tokens.weight as [vocab, hidden] so V1 GGUFs storing the embedding transposed ([hidden, vocab]) load correctly - audiovae: add VOXCPM_DUMP_REF_MONO / REF_FEAT / ENC_STAGE debug dumps Validation (sensevoice-small STT, continuation-mode clone with the Anna reference): 6/6 target sentences transcribe exactly; text-only TTS unchanged. Reference-only cloning (ref_start/ref_end tokens) still fails identically in the golden VoxCPM.cpp - a model-level limitation. --- src/models/voxcpm2/assets.cpp | 2 + src/models/voxcpm2/audiovae.cpp | 132 +++++++++++++++++++++++++++++-- src/models/voxcpm2/generator.cpp | 4 +- 3 files changed, 132 insertions(+), 6 deletions(-) diff --git a/src/models/voxcpm2/assets.cpp b/src/models/voxcpm2/assets.cpp index b1fb03cb..b419c58b 100644 --- a/src/models/voxcpm2/assets.cpp +++ b/src/models/voxcpm2/assets.cpp @@ -600,6 +600,8 @@ class TransformingTensorSource final : public assets::TensorSource { // feat_quant: {N, F} -> {N, F, 1} // merge: {N, D} -> {N, D, 1} // downsample/upsample: {out, in} -> {out, in, k, k} (k=3 for 3x3) + // V1 embedding: token_embd.weight [hidden, vocab] -> base_lm.embed_tokens.weight [vocab, hidden] + {"base_lm.embed_tokens.weight", {config_.lm.vocab_size, config_.lm.hidden_size}}, }; // Folded AudioVAE conv weights: v1 GGUF stores weight-norm weights diff --git a/src/models/voxcpm2/audiovae.cpp b/src/models/voxcpm2/audiovae.cpp index 2d890f5e..40893122 100644 --- a/src/models/voxcpm2/audiovae.cpp +++ b/src/models/voxcpm2/audiovae.cpp @@ -40,6 +40,78 @@ using Clock = std::chrono::steady_clock; constexpr int64_t kResidualKernel = 7; +enum class PaddingMode { Left, Right }; + +std::vector trim_audio_silence_vad(const std::vector& input, + int sample_rate, + float max_silence_ms = 100.0f, + float top_db = 30.0f) { + if (input.empty() || sample_rate <= 0) { + return input; + } + + constexpr int kFrameLength = 2048; + constexpr int kHopLength = 512; + const float ref = *std::max_element(input.begin(), input.end(), [](float a, float b) { + return std::fabs(a) < std::fabs(b); + }); + if (std::fabs(ref) <= 0.0f) { + return input; + } + + const float threshold = std::fabs(ref) * std::pow(10.0f, -top_db / 20.0f); + const size_t n = input.size(); + int first_voice_frame = -1; + int last_voice_frame = -1; + + for (size_t idx = 0, frame = 0; idx < n; idx += kHopLength, ++frame) { + const size_t frame_end = std::min(idx + static_cast(kFrameLength), n); + const size_t frame_size = frame_end - idx; + if (frame_size == 0) { + break; + } + double energy = 0.0; + for (size_t i = idx; i < frame_end; ++i) { + energy += static_cast(input[i]) * static_cast(input[i]); + } + const float rms = static_cast(std::sqrt(energy / static_cast(frame_size))); + if (rms >= threshold) { + if (first_voice_frame < 0) { + first_voice_frame = static_cast(frame); + } + last_voice_frame = static_cast(frame); + } + if (frame_end == n) { + break; + } + } + + if (first_voice_frame < 0 || last_voice_frame < 0) { + return input; + } + + const int max_silence_samples = std::max(0, static_cast(std::lround(max_silence_ms * sample_rate / 1000.0f))); + const int start = std::max(0, first_voice_frame * kHopLength - max_silence_samples); + const int end = std::min(static_cast(n), + (last_voice_frame + 1) * kHopLength + (kFrameLength - kHopLength) + max_silence_samples); + if (start >= end) { + return input; + } + return std::vector(input.begin() + start, input.begin() + end); +} + +void pad_audio_for_patch_alignment(std::vector& audio, size_t patch_len, PaddingMode mode) { + if (patch_len == 0 || audio.empty() || (audio.size() % patch_len) == 0) { + return; + } + const size_t padding = patch_len - (audio.size() % patch_len); + if (mode == PaddingMode::Left) { + audio.insert(audio.begin(), padding, 0.0f); + } else { + audio.insert(audio.end(), padding, 0.0f); + } +} + struct GgmlContextDeleter { void operator()(ggml_context *ctx) const noexcept { if (ctx != nullptr) { @@ -525,9 +597,8 @@ core::TensorValue encoder_block(core::ModuleBuildContext &ctx, hidden = residual_unit(ctx, hidden, weights.residual_units[2], 9); hidden = snake_exact(ctx, hidden, weights.snake, weights.input_channels); const int padding = static_cast((weights.stride + 1) / 2); - const int output_padding = weights.stride % 2; return causal_conv1d(ctx, hidden, weights.downsample, weights.stride, padding, - 1, output_padding); + 1); } core::TensorValue decoder_block(core::ModuleBuildContext &ctx, @@ -668,13 +739,26 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { mono = engine::audio::resample_mono_soxr_or_linear( mono, audio.sample_rate, vae.sample_rate, options); } - const int64_t patch_samples = assets_->config.patch_size * encoder_stride_; - if (patch_samples <= 0) { - throw std::runtime_error("VoxCPM2 AudioVAE patch sample size is invalid"); + // VAD trim silence (match VoxCPM.cpp server_common.cpp:842/878) + mono = trim_audio_silence_vad(mono, vae.sample_rate); + if (const char *dump_path = std::getenv("VOXCPM_DUMP_REF_MONO")) { + FILE *f = std::fopen(dump_path, "wb"); + if (f != nullptr) { + std::fwrite(mono.data(), sizeof(float), mono.size(), f); + std::fclose(f); + } } + // Patch-aligned padding (Left for prompt, Right for reference) + const int64_t patch_samples = assets_->config.patch_size * encoder_stride_; + pad_audio_for_patch_alignment(mono, static_cast(patch_samples), + left_pad ? PaddingMode::Left : PaddingMode::Right); + // Final padding to encoder_sample_capacity const int64_t sample_count = static_cast(mono.size()); const int64_t padded_samples = ((sample_count + patch_samples - 1) / patch_samples) * patch_samples; + if (patch_samples <= 0) { + throw std::runtime_error("VoxCPM2 AudioVAE patch sample size is invalid"); + } if (padded_samples > config_.encoder_sample_capacity) { throw std::runtime_error( "VoxCPM2 AudioVAE encoder sample capacity exceeded"); @@ -694,6 +778,19 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { if (status != GGML_STATUS_SUCCESS) { throw std::runtime_error("VoxCPM2 AudioVAE encoder graph compute failed"); } + if (const char *stage_path = std::getenv("VOXCPM_DUMP_ENC_STAGE")) { + const std::string dir(stage_path); + for (size_t i = 0; i < encoder_stages_.size(); ++i) { + ggml_tensor *stage = encoder_stages_[i]; + std::vector buf(static_cast(ggml_nelements(stage)), 0.0F); + ggml_backend_tensor_get(stage, buf.data(), 0, buf.size() * sizeof(float)); + FILE *f = std::fopen((dir + "/stage_" + std::to_string(i) + ".bin").c_str(), "wb"); + if (f != nullptr) { + std::fwrite(buf.data(), sizeof(float), buf.size(), f); + std::fclose(f); + } + } + } const int64_t latent_frames = padded_samples / encoder_stride_; const int64_t expected_capacity_frames = @@ -716,6 +813,14 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { full[static_cast(c * expected_capacity_frames + t)]; } } + if (const char *dump_path = std::getenv("VOXCPM_DUMP_REF_FEAT")) { + FILE *f = std::fopen(dump_path, "wb"); + if (f != nullptr) { + std::fwrite(encoded.features.data(), sizeof(float), + encoded.features.size(), f); + std::fclose(f); + } + } return encoded; } @@ -872,15 +977,31 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { core::TensorShape::from_dims({1, 1, config_.encoder_sample_capacity})); encoder_input_ = hidden.tensor; ggml_set_input(encoder_input_); + encoder_stages_.clear(); + if (std::getenv("VOXCPM_DUMP_ENC_STAGE") != nullptr) { + encoder_stages_.push_back(encoder_input_); + } hidden = causal_conv1d(ctx, hidden, weights_.encoder_first, 1, 3, 1); + if (std::getenv("VOXCPM_DUMP_ENC_STAGE") != nullptr) { + encoder_stages_.push_back(hidden.tensor); + } for (const auto &block : weights_.encoder_blocks) { hidden = encoder_block(ctx, hidden, block); + if (std::getenv("VOXCPM_DUMP_ENC_STAGE") != nullptr) { + encoder_stages_.push_back(hidden.tensor); + } } hidden = causal_conv1d(ctx, hidden, weights_.encoder_fc_mu, 1, 1, 1); encoder_output_ = hidden.tensor; ggml_set_output(encoder_output_); + for (ggml_tensor *stage : encoder_stages_) { + ggml_set_output(stage); + } encoder_graph_ = ggml_new_graph_custom(encoder_ctx_.get(), 65536, false); ggml_build_forward_expand(encoder_graph_, encoder_output_); + for (ggml_tensor *stage : encoder_stages_) { + ggml_build_forward_expand(encoder_graph_, stage); + } encoder_gallocr_ = ggml_gallocr_new( ggml_backend_get_default_buffer_type(execution_context_.backend())); if (encoder_gallocr_ == nullptr || @@ -902,6 +1023,7 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { ggml_tensor *output_ = nullptr; ggml_tensor *encoder_input_ = nullptr; ggml_tensor *encoder_output_ = nullptr; + std::vector encoder_stages_; ggml_cgraph *graph_ = nullptr; ggml_cgraph *encoder_graph_ = nullptr; ggml_gallocr_t gallocr_ = nullptr; diff --git a/src/models/voxcpm2/generator.cpp b/src/models/voxcpm2/generator.cpp index b3a353b0..f62e1b05 100644 --- a/src/models/voxcpm2/generator.cpp +++ b/src/models/voxcpm2/generator.cpp @@ -1760,7 +1760,9 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { } input_embedding = current_embed; } - prefix_cond = row.feature; + if (row.audio_mask) { + prefix_cond = row.feature; + } prefill_input.input_embeddings.insert(prefill_input.input_embeddings.end(), input_embedding.begin(), input_embedding.end()); From 00ea405ffe536f96ade3273dea7ff63feb6556f1 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Wed, 19 Aug 2026 14:48:14 +0000 Subject: [PATCH 09/23] fix(voxcpm1): route --voice-ref through prompt path so V1 cloning works MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit VoxCPM1 voice cloning via `--voice-ref ` produced non-cloned speech, while `--audio ` (plus `--reference-text`) cloned correctly. Both flags carried the same user intent, but the CLI mapped them to different request fields that the session treated as two distinct audio roles. **Root cause:** `--voice-ref` set `request.voice->speaker->audio`, which the session consumed as *reference audio*. For VoxCPM1 the reference path is wrong in two ways: - `encode_prompt_audio()` only copies `prompt_text` inside the `prompt_audio` branch, so a reference-only request dropped the reference transcript entirely (the LM never saw it). - The reference role right-pads the audio and prepends it wrapped in the `` tokens 103/104. Those belong to VoxCPM2's "reference-mode plumbing"; the V1 LM was only trained for prompt-continuation cloning (golden VoxCPM.cpp uses `--prompt-audio` + `--prompt-text`, and its V1 server never calls `encode_reference_audio`). **Fix:** in `VoxCPM2SessionBase::encoded_prompt_for_request()`, when the model is V1 and only a reference audio is supplied (no `--audio`), route it through the prompt path — the audio becomes `prompt_audio` (left-padded, after ``) and `--reference-text` becomes `prompt_text` (concatenated with the target text). V2 keeps the reference-mode path untouched. Applies to both offline and streaming runs (single shared function). Without `--reference-text` the request now fails with the golden's exact rule ("prompt audio requires prompt_text or reference_text"). **Changes:** 1 file, +24/−12 lines - `src/models/voxcpm2/session.cpp` — V1 reference→prompt routing with cache key/lookup/encode all using the effective audio roles **Verified (CPU, 0.5B Q8_0):** | Test | Result | |------|--------| | V1 `--voice-ref` + ref-text | byte-identical WAV to `--audio` + ref-text (same clone) | | V1 `--voice-ref` without ref-text | clean error (matches golden iff rule) | | V1 `--audio` regression | byte-identical output | | V2 `--voice-ref` regression | 48kHz, byte-identical to pre-fix (reference mode preserved) | | 5-voice clone batch (ana/eric/andrew/jenny/nicole) | 16kHz speech, RMS 0.06–0.08 ✓ | **Note:** V1.5 (44.1kHz) fails at load with "encoder sample capacity must be divisible by encoder stride" — pre-existing config gap (stride 1764 ∤ default capacity 240000), identical on `--audio` before this fix. --- src/models/voxcpm2/session.cpp | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/src/models/voxcpm2/session.cpp b/src/models/voxcpm2/session.cpp index 8b9ea0ec..238ef3d8 100644 --- a/src/models/voxcpm2/session.cpp +++ b/src/models/voxcpm2/session.cpp @@ -451,10 +451,22 @@ const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( if (!prompt_audio.has_value() && !reference_audio.has_value()) { return nullptr; } + // VoxCPM1 clones only via prompt-continuation mode (golden VoxCPM.cpp + // uses --prompt-audio + --prompt-text); the V2 reference-mode path wraps + // audio in tokens 103/104, which the V1 LM was never trained on. Route a + // V1 reference audio through the prompt path so --voice-ref clones like + // --audio. + std::optional effective_prompt_audio = prompt_audio; + std::optional effective_reference_audio = reference_audio; + if (assets_->config.v1 && !effective_prompt_audio.has_value() && + effective_reference_audio.has_value()) { + effective_prompt_audio = effective_reference_audio; + effective_reference_audio.reset(); + } EncodedPromptCacheKey key; key.prompt_text = prompt_text; - key.prompt_audio = prompt_audio; - key.reference_audio = reference_audio; + key.prompt_audio = effective_prompt_audio; + key.reference_audio = effective_reference_audio; if (auto *cached = encoded_prompt_cache_.find(key)) { debug::trace_log_scalar("voxcpm2.prompt_cache.hit", 1); debug::trace_log_scalar("voxcpm2.prompt_cache.slots", @@ -469,8 +481,8 @@ const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( const auto encode_start = Clock::now(); EncodedPromptCacheEntry entry; - entry.encoded = - decoder_->encode_prompt_audio(prompt_audio, prompt_text, reference_audio); + entry.encoded = decoder_->encode_prompt_audio( + effective_prompt_audio, prompt_text, effective_reference_audio); const double encode_ms = engine::debug::elapsed_ms(encode_start); if (encoded_prompt_cache_.capacity() == 0) { uncached_encoded_prompt_ = std::move(entry); @@ -486,8 +498,8 @@ const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( encoded_prompt_cache_.put(std::move(key), std::move(entry)); EncodedPromptCacheKey lookup; lookup.prompt_text = prompt_text; - lookup.prompt_audio = prompt_audio; - lookup.reference_audio = reference_audio; + lookup.prompt_audio = effective_prompt_audio; + lookup.reference_audio = effective_reference_audio; auto *cached = encoded_prompt_cache_.find(lookup); if (cached == nullptr) { throw std::runtime_error("VoxCPM2 prompt cache insert failed"); From 57b649834bd1235c726aa36ee2520a81bef6aeca Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Wed, 19 Aug 2026 16:14:27 +0000 Subject: [PATCH 10/23] refactor: release tensor_source framework changes back to upstream Move VoxCPM GGUF tokenizer/config metadata reading out of the framework TensorSource interface into a new voxcpm2 GgufMetadataReader. Revert the validate_expected_shape relaxed_rank parameter and redundant include; tensor_source.h/.cpp now differ from upstream/main by a single line (is_synthesized). --- CMakeLists.txt | 2 + .../engine/framework/assets/tensor_source.h | 31 +- include/engine/models/voxcpm2/gguf_metadata.h | 47 +++ src/framework/assets/tensor_source.cpp | 277 +----------------- src/models/voxcpm2/assets.cpp | 2 +- src/models/voxcpm2/config_gguf.cpp | 31 +- src/models/voxcpm2/gguf_metadata.cpp | 136 +++++++++ src/models/voxcpm2/tokenizer_gguf.cpp | 28 +- 8 files changed, 230 insertions(+), 324 deletions(-) create mode 100644 include/engine/models/voxcpm2/gguf_metadata.h create mode 100644 src/models/voxcpm2/gguf_metadata.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index fa888f40..5b21254e 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -707,6 +707,7 @@ audiocpp_add_model(voxcpm2 src/models/voxcpm2/audiovae.cpp src/models/voxcpm2/config_gguf.cpp src/models/voxcpm2/generator.cpp + src/models/voxcpm2/gguf_metadata.cpp src/models/voxcpm2/loader.cpp src/models/voxcpm2/minicpm.cpp src/models/voxcpm2/session.cpp @@ -724,6 +725,7 @@ audiocpp_add_model(voxcpm1 src/models/voxcpm2/audiovae.cpp src/models/voxcpm2/config_gguf.cpp src/models/voxcpm2/generator.cpp + src/models/voxcpm2/gguf_metadata.cpp src/models/voxcpm2/loader.cpp src/models/voxcpm2/minicpm.cpp src/models/voxcpm2/session.cpp diff --git a/include/engine/framework/assets/tensor_source.h b/include/engine/framework/assets/tensor_source.h index a1648a01..b8088582 100644 --- a/include/engine/framework/assets/tensor_source.h +++ b/include/engine/framework/assets/tensor_source.h @@ -118,36 +118,7 @@ class TensorSource { [[nodiscard]] std::string require_tensor_name( std::initializer_list candidates) const; [[nodiscard]] virtual int64_t require_i64_scalar(std::string_view name) const = 0; - [[nodiscard]] virtual bool is_synthesized(std::string_view name) const noexcept { return false; } - - // GGUF metadata access (optional, only implemented by GgufTensorSource) - [[nodiscard]] virtual std::optional optional_string(std::string_view key) const { - return std::nullopt; - } - [[nodiscard]] virtual std::optional optional_u32(std::string_view key) const { - return std::nullopt; - } - [[nodiscard]] virtual std::optional> optional_string_array(std::string_view key) const { - return std::nullopt; - } - [[nodiscard]] virtual std::optional> optional_i32_array(std::string_view key) const { - return std::nullopt; - } - [[nodiscard]] virtual std::optional> optional_f32_array(std::string_view key) const { - return std::nullopt; - } - [[nodiscard]] virtual std::string require_string(std::string_view key) const { - throw std::runtime_error("require_string not supported by this TensorSource"); - } - [[nodiscard]] virtual uint32_t require_u32(std::string_view key) const { - throw std::runtime_error("require_u32 not supported by this TensorSource"); - } - [[nodiscard]] virtual std::vector require_string_array(std::string_view key) const { - throw std::runtime_error("require_string_array not supported by this TensorSource"); - } - [[nodiscard]] virtual std::vector require_i32_array(std::string_view key) const { - throw std::runtime_error("require_i32_array not supported by this TensorSource"); - } + [[nodiscard]] virtual bool is_synthesized(std::string_view) const noexcept { return false; } }; [[nodiscard]] TensorStorageType parse_tensor_storage_type(std::string_view value); diff --git a/include/engine/models/voxcpm2/gguf_metadata.h b/include/engine/models/voxcpm2/gguf_metadata.h new file mode 100644 index 00000000..0f538f36 --- /dev/null +++ b/include/engine/models/voxcpm2/gguf_metadata.h @@ -0,0 +1,47 @@ +#pragma once + +#include +#include +#include +#include +#include + +struct gguf_context; + +namespace engine::assets { +class TensorSource; +} + +namespace engine::models::voxcpm2 { + +// Reads GGUF KV metadata (tokenizer.ggml.*, voxcpm_*) directly from the file +// backing a TensorSource. Only meaningful for GGUF sources: for any other +// source type valid() is false and all accessors return nullopt (optional_*) +// or throw (require_*). This keeps VoxCPM schema knowledge out of the +// framework TensorSource interface. +class GgufMetadataReader { +public: + explicit GgufMetadataReader(const engine::assets::TensorSource & source); + ~GgufMetadataReader(); + + GgufMetadataReader(const GgufMetadataReader &) = delete; + GgufMetadataReader & operator=(const GgufMetadataReader &) = delete; + + bool valid() const noexcept { return gguf_ != nullptr; } + + [[nodiscard]] std::optional optional_string(std::string_view key) const; + [[nodiscard]] std::optional optional_u32(std::string_view key) const; + [[nodiscard]] std::optional> optional_string_array(std::string_view key) const; + [[nodiscard]] std::optional> optional_i32_array(std::string_view key) const; + [[nodiscard]] std::optional> optional_f32_array(std::string_view key) const; + + [[nodiscard]] std::string require_string(std::string_view key) const; + [[nodiscard]] uint32_t require_u32(std::string_view key) const; + [[nodiscard]] std::vector require_string_array(std::string_view key) const; + [[nodiscard]] std::vector require_i32_array(std::string_view key) const; + +private: + struct gguf_context * gguf_ = nullptr; +}; + +} // namespace engine::models::voxcpm2 \ No newline at end of file diff --git a/src/framework/assets/tensor_source.cpp b/src/framework/assets/tensor_source.cpp index a40509f9..189ea251 100644 --- a/src/framework/assets/tensor_source.cpp +++ b/src/framework/assets/tensor_source.cpp @@ -15,7 +15,6 @@ #include #include #include -#include #include #include #include @@ -72,25 +71,12 @@ core::TensorShape shape_from_dims(const std::vector & dims) { void validate_expected_shape( std::string_view name, const std::vector & actual_shape, - const std::optional> & expected_shape, - bool relaxed_rank) { + const std::optional> & expected_shape) { if (expected_shape.has_value() && actual_shape != *expected_shape) { - if (!relaxed_rank) { - throw std::runtime_error("tensor shape mismatch for " + std::string(name)); - } - int64_t expected_elems = 1; - for (const int64_t dim : *expected_shape) { - expected_elems *= dim; - } - int64_t actual_elems = 1; - for (const int64_t dim : actual_shape) { - actual_elems *= dim; - } - if (actual_elems != expected_elems) { - throw std::runtime_error("tensor element count mismatch for " + std::string(name)); - } + throw std::runtime_error("tensor shape mismatch for " + std::string(name)); } } + std::string lower_ascii(std::string_view value) { std::string out(value); for (char & ch : out) { @@ -579,7 +565,7 @@ class SafeTensorSource final : public TensorSource { if (info == nullptr) { throw std::runtime_error("missing tensor: " + std::string(name)); } - validate_expected_shape(name, info->shape, expected_shape, false); + validate_expected_shape(name, info->shape, expected_shape); const auto shape = shape_from_dims(expected_shape); const ggml_type type = ggml_type_for_tensor_storage(resolve_tensor_storage_type(*this, name, storage_type)); const auto [data, byte_size] = require_data_range(*info); @@ -608,7 +594,7 @@ class SafeTensorSource final : public TensorSource { std::string_view name, const std::optional> & expected_shape) const override { const auto tensor = require_tensor_data(name); - validate_expected_shape(name, tensor.metadata.shape, expected_shape, false); + validate_expected_shape(name, tensor.metadata.shape, expected_shape); const ggml_type type = ggml_type_for_tensor_dtype(tensor.metadata.dtype); const auto physical_shape = tensor.metadata.shape.empty() ? shape_from_dims({1}) @@ -691,8 +677,6 @@ class GgufTensorSource final : public TensorSource { public: explicit GgufTensorSource(std::filesystem::path path) : source_path_(std::filesystem::weakly_canonical(path)) { - // Read tokenizer metadata during initialization - read_metadata(path); ggml_context * tensor_context = nullptr; gguf_context * gguf = gguf_init_from_file( source_path_.string().c_str(), @@ -777,7 +761,6 @@ class GgufTensorSource final : public TensorSource { gguf_free(gguf); ggml_free(tensor_context); bytes_ = engine::io::read_binary_blob(source_path_); - read_metadata(source_path_); } const std::filesystem::path & source_path() const noexcept override { return source_path_; } @@ -820,7 +803,7 @@ class GgufTensorSource final : public TensorSource { TensorStorageType storage_type, const std::vector & expected_shape) const override { const auto & info = require_info(name); - validate_expected_shape(name, info.shape, expected_shape, false); + validate_expected_shape(name, info.shape, expected_shape); const auto shape = shape_from_dims(expected_shape); const ggml_type type = ggml_type_for_tensor_storage(resolve_tensor_storage_type(*this, name, storage_type)); const auto [data, byte_size] = require_data_range(info); @@ -848,7 +831,7 @@ class GgufTensorSource final : public TensorSource { std::string_view name, const std::optional> & expected_shape) const override { const auto tensor = require_tensor_data(name); - validate_expected_shape(name, tensor.metadata.shape, expected_shape, false); + validate_expected_shape(name, tensor.metadata.shape, expected_shape); const auto physical_shape = tensor.metadata.shape.empty() ? shape_from_dims({1}) : shape_from_dims(tensor.metadata.shape); @@ -897,248 +880,6 @@ class GgufTensorSource final : public TensorSource { std::vector infos_; std::unordered_map info_by_name_; mutable engine::io::BinaryBlob bytes_; - // Tokenizer metadata - std::optional tokenizer_model_; - std::optional tokenizer_pre_; - std::optional> tokenizer_tokens_; - std::optional> tokenizer_token_type_; - std::optional> tokenizer_merges_; - std::optional tokenizer_bos_token_id_; - std::optional tokenizer_eos_token_id_; - std::optional tokenizer_unknown_token_id_; - - // Config metadata (voxcpm_*) - std::unordered_map config_string_metadata_; - std::unordered_map config_u32_metadata_; - std::unordered_map> config_i32_array_metadata_; - std::unordered_map> config_f32_array_metadata_; - - void read_metadata(const std::filesystem::path & path) { - ggml_context * tensor_context = nullptr; - gguf_context * gguf = gguf_init_from_file( - path.string().c_str(), - gguf_init_params{true, &tensor_context}); - if (gguf == nullptr) { - if (tensor_context != nullptr) ggml_free(tensor_context); - return; - } - - const auto get_string = [&](const char* key) -> std::optional { - const int idx = gguf_find_key(gguf, key); - if (idx < 0 || gguf_get_kv_type(gguf, idx) != GGUF_TYPE_STRING) { - return std::nullopt; - } - const char* data = gguf_get_val_str(gguf, idx); - if (!data) return std::nullopt; - return std::string(data); - }; - - const auto get_u32 = [&](const char* key) -> std::optional { - const int idx = gguf_find_key(gguf, key); - if (idx < 0) return std::nullopt; - return gguf_get_val_u32(gguf, idx); - }; - - const auto get_f32 = [&](const char* key) -> std::optional { - const int idx = gguf_find_key(gguf, key); - if (idx < 0 || gguf_get_kv_type(gguf, idx) != GGUF_TYPE_FLOAT32) { - return std::nullopt; - } - return gguf_get_val_f32(gguf, idx); - }; - - const auto get_i32_array = [&](const char* key) -> std::optional> { - const int idx = gguf_find_key(gguf, key); - if (idx < 0) return std::nullopt; - const int32_t* data = static_cast(gguf_get_arr_data(gguf, idx)); - const size_t n = gguf_get_arr_n(gguf, idx); - if (!data && n != 0) return std::nullopt; - return std::vector(data, data + n); - }; - - const auto get_f32_array = [&](const char* key) -> std::optional> { - const int idx = gguf_find_key(gguf, key); - if (idx < 0 || gguf_get_arr_type(gguf, idx) != GGUF_TYPE_FLOAT32) { - return std::nullopt; - } - const float* data = static_cast(gguf_get_arr_data(gguf, idx)); - const size_t n = gguf_get_arr_n(gguf, idx); - if (!data && n != 0) return std::nullopt; - return std::vector(data, data + n); - }; - - const auto get_string_array = [&](const char* key) -> std::optional> { - const int idx = gguf_find_key(gguf, key); - if (idx < 0 || gguf_get_arr_type(gguf, idx) != GGUF_TYPE_STRING) { - return std::nullopt; - } - const size_t n = gguf_get_arr_n(gguf, idx); - std::vector values; - values.reserve(n); - for (size_t i = 0; i < n; ++i) { - const char* v = gguf_get_arr_str(gguf, idx, i); - values.emplace_back(v ? v : ""); - } - return values; - }; - - // Tokenizer metadata - tokenizer_model_ = get_string("tokenizer.ggml.model"); - tokenizer_pre_ = get_string("tokenizer.ggml.pre"); - tokenizer_tokens_ = get_string_array("tokenizer.ggml.tokens"); - tokenizer_token_type_ = get_i32_array("tokenizer.ggml.token_type"); - tokenizer_merges_ = get_string_array("tokenizer.ggml.merges"); - tokenizer_bos_token_id_ = get_u32("tokenizer.ggml.bos_token_id"); - tokenizer_eos_token_id_ = get_u32("tokenizer.ggml.eos_token_id"); - tokenizer_unknown_token_id_ = get_u32("tokenizer.ggml.unknown_token_id"); - - // Config metadata (voxcpm_*) - read all voxcpm_* keys - // We read known keys, but also could iterate all keys if needed - static constexpr const char* config_string_keys[] = { - "voxcpm_architecture", - "voxcpm_device", - "voxcpm_dtype", - "voxcpm_lm_config_rope_scaling_type", - "voxcpm_dit_config_cfm_config_solver", - "voxcpm_dit_config_cfm_config_t_scheduler", - }; - for (const char* key : config_string_keys) { - if (auto val = get_string(key)) { - config_string_metadata_[key] = *val; - } - } - - static constexpr const char* config_u32_keys[] = { - "voxcpm_lm_config_bos_token_id", - "voxcpm_lm_config_eos_token_id", - "voxcpm_lm_config_hidden_size", - "voxcpm_lm_config_intermediate_size", - "voxcpm_lm_config_max_position_embeddings", - "voxcpm_lm_config_num_attention_heads", - "voxcpm_lm_config_num_hidden_layers", - "voxcpm_lm_config_num_key_value_heads", - "voxcpm_lm_config_dim_model_base", - "voxcpm_lm_config_scale_emb", - "voxcpm_lm_config_rope_theta", - "voxcpm_lm_config_use_mup", - "voxcpm_lm_config_vocab_size", - "voxcpm_patch_size", - "voxcpm_feat_dim", - "voxcpm_residual_lm_num_layers", - "voxcpm_residual_lm_no_rope", - "voxcpm_scalar_quantization_latent_dim", - "voxcpm_scalar_quantization_scale", - "voxcpm_encoder_config_hidden_dim", - "voxcpm_encoder_config_ffn_dim", - "voxcpm_encoder_config_num_heads", - "voxcpm_encoder_config_num_layers", - "voxcpm_dit_config_hidden_dim", - "voxcpm_dit_config_ffn_dim", - "voxcpm_dit_config_num_heads", - "voxcpm_dit_config_num_layers", - "voxcpm_dit_config_mean_mode", - "voxcpm_audio_vae_config_encoder_dim", - "voxcpm_audio_vae_config_decoder_dim", - "voxcpm_audio_vae_config_latent_dim", - "voxcpm_audio_vae_config_sample_rate", - "voxcpm_audio_vae_config_out_sample_rate", - "voxcpm_max_length", - }; - for (const char* key : config_u32_keys) { - if (auto val = get_u32(key)) { - config_u32_metadata_[key] = *val; - } - } - - static constexpr const char* config_i32_array_keys[] = { - "voxcpm_audio_vae_config_encoder_rates", - "voxcpm_audio_vae_config_decoder_rates", - "voxcpm_audio_vae_config_sr_bin_boundaries", - }; - for (const char* key : config_i32_array_keys) { - if (auto val = get_i32_array(key)) { - config_i32_array_metadata_[key] = *val; - } - } - - static constexpr const char* config_f32_array_keys[] = { - "voxcpm_lm_config_rope_scaling_long_factor", - "voxcpm_lm_config_rope_scaling_short_factor", - }; - for (const char* key : config_f32_array_keys) { - if (auto val = get_f32_array(key)) { - config_f32_array_metadata_[key] = *val; - } - } - - gguf_free(gguf); - if (tensor_context != nullptr) ggml_free(tensor_context); - } - - // GGUF metadata access implementations - std::optional optional_string(std::string_view key) const override { - if (key == "tokenizer.ggml.model") return tokenizer_model_; - if (key == "tokenizer.ggml.pre") return tokenizer_pre_; - // Check config metadata - auto it = config_string_metadata_.find(std::string(key)); - if (it != config_string_metadata_.end()) return it->second; - return std::nullopt; - } - - std::optional optional_u32(std::string_view key) const override { - if (key == "tokenizer.ggml.bos_token_id") return tokenizer_bos_token_id_; - if (key == "tokenizer.ggml.eos_token_id") return tokenizer_eos_token_id_; - if (key == "tokenizer.ggml.unknown_token_id") return tokenizer_unknown_token_id_; - // Check config metadata - auto it = config_u32_metadata_.find(std::string(key)); - if (it != config_u32_metadata_.end()) return it->second; - return std::nullopt; - } - - std::optional> optional_string_array(std::string_view key) const override { - if (key == "tokenizer.ggml.tokens") return tokenizer_tokens_; - if (key == "tokenizer.ggml.merges") return tokenizer_merges_; - return std::nullopt; - } - - std::optional> optional_i32_array(std::string_view key) const override { - if (key == "tokenizer.ggml.token_type") return tokenizer_token_type_; - // Check config metadata - auto it = config_i32_array_metadata_.find(std::string(key)); - if (it != config_i32_array_metadata_.end()) return it->second; - return std::nullopt; - } - - std::optional> optional_f32_array(std::string_view key) const override { - // Check config metadata - auto it = config_f32_array_metadata_.find(std::string(key)); - if (it != config_f32_array_metadata_.end()) return it->second; - return std::nullopt; - } - - std::string require_string(std::string_view key) const override { - auto opt = optional_string(key); - if (opt) return *opt; - throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); - } - - uint32_t require_u32(std::string_view key) const override { - auto opt = optional_u32(key); - if (opt) return *opt; - throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); - } - - std::vector require_string_array(std::string_view key) const override { - auto opt = optional_string_array(key); - if (opt) return *opt; - throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); - } - - std::vector require_i32_array(std::string_view key) const override { - auto opt = optional_i32_array(key); - if (opt) return *opt; - throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); - } }; std::unordered_map parse_indexed_tensor_weight_map( @@ -1493,7 +1234,7 @@ TensorData TensorSource::require_tensor( const core::TensorShape shape = shape_from_dims(expected_shape); const ggml_type type = ggml_type_for_tensor_storage(resolve_tensor_storage_type(*this, name, storage_type)); const auto raw = require_tensor_data(name); - validate_expected_shape(name, raw.metadata.shape, expected_shape, false); + validate_expected_shape(name, raw.metadata.shape, expected_shape); if (raw_dtype_matches_ggml_type(raw.metadata.dtype, type)) { validate_raw_tensor_byte_size(name, shape, type, raw.bytes.size()); return TensorData{shape, type, raw.bytes}; @@ -1516,7 +1257,7 @@ TensorData TensorSource::require_tensor_as_shape( const core::TensorShape source_shape = shape_from_dims(expected); const ggml_type type = ggml_type_for_tensor_storage(resolve_tensor_storage_type(*this, name, storage_type)); const auto raw = require_tensor_data(name); - validate_expected_shape(name, raw.metadata.shape, expected, false); + validate_expected_shape(name, raw.metadata.shape, expected); if (raw.metadata.shape == std::vector(tensor_shape) && raw_dtype_matches_ggml_type(raw.metadata.dtype, type)) { validate_raw_tensor_byte_size(name, shape, type, raw.bytes.size()); diff --git a/src/models/voxcpm2/assets.cpp b/src/models/voxcpm2/assets.cpp index b419c58b..a68833ae 100644 --- a/src/models/voxcpm2/assets.cpp +++ b/src/models/voxcpm2/assets.cpp @@ -720,7 +720,7 @@ class TransformingTensorSource final : public assets::TensorSource { } assets::RawTensorData reshape_tensor_data(const assets::RawTensorData & data, - const std::vector & target_shape) const { + const std::vector &) const { // For now, just return the data as-is (validation happens elsewhere) // The actual reshape happens in require_f32 return data; diff --git a/src/models/voxcpm2/config_gguf.cpp b/src/models/voxcpm2/config_gguf.cpp index 218ca747..794313e2 100644 --- a/src/models/voxcpm2/config_gguf.cpp +++ b/src/models/voxcpm2/config_gguf.cpp @@ -1,6 +1,7 @@ #include "engine/models/voxcpm2/config_gguf.h" #include "engine/framework/assets/tensor_source.h" +#include "engine/models/voxcpm2/gguf_metadata.h" #include #include @@ -8,20 +9,22 @@ namespace engine::models::voxcpm2 { bool has_voxcpm1_config_metadata(const engine::assets::TensorSource & source) { + const GgufMetadataReader metadata(source); // Check for at least one VoxCPM1-specific metadata key - return source.optional_string("voxcpm_architecture").has_value() || - source.optional_string("voxcpm_lm_config_hidden_size").has_value() || - source.optional_u32("voxcpm_lm_config_hidden_size").has_value(); + return metadata.optional_string("voxcpm_architecture").has_value() || + metadata.optional_string("voxcpm_lm_config_hidden_size").has_value() || + metadata.optional_u32("voxcpm_lm_config_hidden_size").has_value(); } VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & source) { VoxCPM2Config config; + const GgufMetadataReader metadata(source); config.v1 = true; config.architecture = "voxcpm"; // Helper lambda to get optional i64 from GGUF metadata (via u32 or i64) - auto get_optional_i64 = [&source](const char * key) -> std::optional { - auto u32 = source.optional_u32(key); + auto get_optional_i64 = [&source, &metadata](const char * key) -> std::optional { + auto u32 = metadata.optional_u32(key); if (u32) return static_cast(*u32); // Try i64 scalar if it's a tensor if (source.has_tensor(key)) { @@ -35,15 +38,15 @@ VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & }; // Helper lambda to get optional bool from GGUF metadata - auto get_optional_bool = [&source](const char * key) -> std::optional { - auto u32 = source.optional_u32(key); + auto get_optional_bool = [&metadata](const char * key) -> std::optional { + auto u32 = metadata.optional_u32(key); if (u32) return *u32 != 0; return std::nullopt; }; // Helper lambda to get optional int64 array from GGUF metadata - auto get_optional_i64_array = [&source](const char * key) -> std::optional> { - auto i32_arr = source.optional_i32_array(key); + auto get_optional_i64_array = [&metadata](const char * key) -> std::optional> { + auto i32_arr = metadata.optional_i32_array(key); if (i32_arr) { std::vector result; result.reserve(i32_arr->size()); @@ -56,7 +59,7 @@ VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & }; // Architecture - auto arch = source.optional_string("voxcpm_architecture"); + auto arch = metadata.optional_string("voxcpm_architecture"); if (arch) config.architecture = *arch; // LM Config @@ -86,9 +89,9 @@ VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & // instead of the old identity fallback: identity factors silently degrade // every RoPE computation across all four transformers. auto short_factor = - source.optional_f32_array("voxcpm_lm_config_rope_scaling_short_factor"); + metadata.optional_f32_array("voxcpm_lm_config_rope_scaling_short_factor"); auto long_factor = - source.optional_f32_array("voxcpm_lm_config_rope_scaling_long_factor"); + metadata.optional_f32_array("voxcpm_lm_config_rope_scaling_long_factor"); if (short_factor && static_cast(short_factor->size()) != factor_size) { throw std::runtime_error( @@ -159,8 +162,8 @@ VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & config.max_length = get_optional_i64("voxcpm_max_length").value_or(2048); // Device and dtype - config.device = source.optional_string("voxcpm_device").value_or("cpu"); - config.dtype = source.optional_string("voxcpm_dtype").value_or("fp16"); + config.device = metadata.optional_string("voxcpm_device").value_or("cpu"); + config.dtype = metadata.optional_string("voxcpm_dtype").value_or("fp16"); // Validate required fields if (config.lm.hidden_size <= 0) { diff --git a/src/models/voxcpm2/gguf_metadata.cpp b/src/models/voxcpm2/gguf_metadata.cpp new file mode 100644 index 00000000..13981512 --- /dev/null +++ b/src/models/voxcpm2/gguf_metadata.cpp @@ -0,0 +1,136 @@ +#include "engine/models/voxcpm2/gguf_metadata.h" + +#include "engine/framework/assets/tensor_source.h" + +#include + +#include + +namespace engine::models::voxcpm2 { + +GgufMetadataReader::GgufMetadataReader(const engine::assets::TensorSource & source) { + // Metadata-only open: no_alloc=true with no ggml context parses the GGUF + // header and KV section without ever touching tensor data. + gguf_ = gguf_init_from_file( + source.source_path().string().c_str(), + gguf_init_params{true, nullptr}); +} + +GgufMetadataReader::~GgufMetadataReader() { + if (gguf_ != nullptr) { + gguf_free(gguf_); + } +} + +std::optional GgufMetadataReader::optional_string(std::string_view key) const { + if (gguf_ == nullptr) { + return std::nullopt; + } + const int64_t idx = gguf_find_key(gguf_, std::string(key).c_str()); + if (idx < 0 || gguf_get_kv_type(gguf_, idx) != GGUF_TYPE_STRING) { + return std::nullopt; + } + const char * data = gguf_get_val_str(gguf_, idx); + if (data == nullptr) { + return std::nullopt; + } + return std::string(data); +} + +std::optional GgufMetadataReader::optional_u32(std::string_view key) const { + if (gguf_ == nullptr) { + return std::nullopt; + } + // No KV type check: VoxCPM stores boolean flags (use_mup, no_rope, + // mean_mode) as scalar values that gguf_get_val_u32 reads regardless of + // their declared scalar type. + const int64_t idx = gguf_find_key(gguf_, std::string(key).c_str()); + if (idx < 0) { + return std::nullopt; + } + return gguf_get_val_u32(gguf_, idx); +} + +std::optional> GgufMetadataReader::optional_string_array(std::string_view key) const { + if (gguf_ == nullptr) { + return std::nullopt; + } + const int64_t idx = gguf_find_key(gguf_, std::string(key).c_str()); + if (idx < 0 || gguf_get_arr_type(gguf_, idx) != GGUF_TYPE_STRING) { + return std::nullopt; + } + const size_t n = gguf_get_arr_n(gguf_, idx); + std::vector values; + values.reserve(n); + for (size_t i = 0; i < n; ++i) { + const char * v = gguf_get_arr_str(gguf_, idx, i); + values.emplace_back(v != nullptr ? v : ""); + } + return values; +} + +std::optional> GgufMetadataReader::optional_i32_array(std::string_view key) const { + if (gguf_ == nullptr) { + return std::nullopt; + } + const int64_t idx = gguf_find_key(gguf_, std::string(key).c_str()); + if (idx < 0) { + return std::nullopt; + } + const auto * data = static_cast(gguf_get_arr_data(gguf_, idx)); + const size_t n = gguf_get_arr_n(gguf_, idx); + if (data == nullptr) { + return n == 0 ? std::optional>(std::vector{}) : std::nullopt; + } + return std::vector(data, data + n); +} + +std::optional> GgufMetadataReader::optional_f32_array(std::string_view key) const { + if (gguf_ == nullptr) { + return std::nullopt; + } + const int64_t idx = gguf_find_key(gguf_, std::string(key).c_str()); + if (idx < 0 || gguf_get_arr_type(gguf_, idx) != GGUF_TYPE_FLOAT32) { + return std::nullopt; + } + const auto * data = static_cast(gguf_get_arr_data(gguf_, idx)); + const size_t n = gguf_get_arr_n(gguf_, idx); + if (data == nullptr) { + return n == 0 ? std::optional>(std::vector{}) : std::nullopt; + } + return std::vector(data, data + n); +} + +std::string GgufMetadataReader::require_string(std::string_view key) const { + auto opt = optional_string(key); + if (opt) { + return *opt; + } + throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); +} + +uint32_t GgufMetadataReader::require_u32(std::string_view key) const { + auto opt = optional_u32(key); + if (opt) { + return *opt; + } + throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); +} + +std::vector GgufMetadataReader::require_string_array(std::string_view key) const { + auto opt = optional_string_array(key); + if (opt) { + return *opt; + } + throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); +} + +std::vector GgufMetadataReader::require_i32_array(std::string_view key) const { + auto opt = optional_i32_array(key); + if (opt) { + return *opt; + } + throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); +} + +} // namespace engine::models::voxcpm2 \ No newline at end of file diff --git a/src/models/voxcpm2/tokenizer_gguf.cpp b/src/models/voxcpm2/tokenizer_gguf.cpp index e849fff9..f3230653 100644 --- a/src/models/voxcpm2/tokenizer_gguf.cpp +++ b/src/models/voxcpm2/tokenizer_gguf.cpp @@ -1,6 +1,7 @@ #include "engine/models/voxcpm2/tokenizer_gguf.h" #include "engine/framework/assets/tensor_source.h" +#include "engine/models/voxcpm2/gguf_metadata.h" #include #include @@ -196,18 +197,22 @@ VoxCPM1GgufTokenizer::VoxCPM1GgufTokenizer(std::shared_ptr(); auto & impl = *impl_; // Read tokenizer metadata from GGUF directly in constructor - const std::string tokenizer_model = gguf_source->require_string("tokenizer.ggml.model"); - const std::string tokenizer_pre = gguf_source->require_string("tokenizer.ggml.pre"); - const std::vector tokens = gguf_source->require_string_array("tokenizer.ggml.tokens"); - const std::vector token_types = gguf_source->require_i32_array("tokenizer.ggml.token_type"); - const std::vector merges = gguf_source->require_string_array("tokenizer.ggml.merges"); - const uint32_t bos_id = gguf_source->require_u32("tokenizer.ggml.bos_token_id"); - const uint32_t eos_id = gguf_source->require_u32("tokenizer.ggml.eos_token_id"); - const uint32_t unk_id = gguf_source->require_u32("tokenizer.ggml.unknown_token_id"); + const std::string tokenizer_model = metadata.require_string("tokenizer.ggml.model"); + const std::string tokenizer_pre = metadata.require_string("tokenizer.ggml.pre"); + const std::vector tokens = metadata.require_string_array("tokenizer.ggml.tokens"); + const std::vector token_types = metadata.require_i32_array("tokenizer.ggml.token_type"); + const std::vector merges = metadata.require_string_array("tokenizer.ggml.merges"); + const uint32_t bos_id = metadata.require_u32("tokenizer.ggml.bos_token_id"); + const uint32_t eos_id = metadata.require_u32("tokenizer.ggml.eos_token_id"); + const uint32_t unk_id = metadata.require_u32("tokenizer.ggml.unknown_token_id"); if (tokenizer_model != "gpt2" || tokens.empty() || merges.empty() || token_types.size() != tokens.size()) { throw std::runtime_error("Invalid VoxCPM1 GGUF tokenizer metadata"); @@ -350,9 +355,10 @@ int32_t VoxCPM1GgufTokenizer::unk_token_id() const noexcept { } bool VoxCPM1GgufTokenizer::has_tokenizer_metadata(const engine::assets::TensorSource & source) { - return source.optional_string("tokenizer.ggml.model").has_value() && - source.optional_string_array("tokenizer.ggml.tokens").has_value() && - source.optional_string_array("tokenizer.ggml.merges").has_value(); + const GgufMetadataReader metadata(source); + return metadata.optional_string("tokenizer.ggml.model").has_value() && + metadata.optional_string_array("tokenizer.ggml.tokens").has_value() && + metadata.optional_string_array("tokenizer.ggml.merges").has_value(); } } // namespace engine::models::voxcpm2 \ No newline at end of file From 2b65a0d59a295699c3e388be1253b9215873625d Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Wed, 19 Aug 2026 18:16:26 +0200 Subject: [PATCH 11/23] fix(voxcpm1): Add Webui support for VoxCPM v1 (0.5B) --- webui/native/dist/index.html | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/webui/native/dist/index.html b/webui/native/dist/index.html index a9f22547..6a456372 100644 --- a/webui/native/dist/index.html +++ b/webui/native/dist/index.html @@ -13,20 +13,20 @@
From 1d9c59b5e021796abe61e3fa26adc3633b360861 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Wed, 19 Aug 2026 19:26:57 +0000 Subject: [PATCH 12/23] perf(voxcpm1): release text-length-scaled VRAM after each request Only the cloned voice is cached across requests; prompt-prefill and AudioVAE encoder/decoder graphs are freed at request end and rebuilt fresh on the next request. Idle VRAM drops to ~1.4GB after generation; very long text may require up to ~3.5GB VRAM during generation. --- include/engine/models/voxcpm2/audiovae.h | 1 + include/engine/models/voxcpm2/generator.h | 1 + include/engine/models/voxcpm2/minicpm.h | 2 + src/models/voxcpm2/audiovae.cpp | 10 +- src/models/voxcpm2/generator.cpp | 151 ++++++++++++++++++---- src/models/voxcpm2/minicpm.cpp | 38 +++++- src/models/voxcpm2/session.cpp | 42 ++++-- 7 files changed, 200 insertions(+), 45 deletions(-) diff --git a/include/engine/models/voxcpm2/audiovae.h b/include/engine/models/voxcpm2/audiovae.h index 5e1b038c..d6a250aa 100644 --- a/include/engine/models/voxcpm2/audiovae.h +++ b/include/engine/models/voxcpm2/audiovae.h @@ -43,6 +43,7 @@ class VoxCPM2AudioVAEDecoderRuntime final { const std::string &prompt_text, const std::optional &reference_audio); void release_runtime_memory(); + void release_encoder_graph(); private: class Impl; diff --git a/include/engine/models/voxcpm2/generator.h b/include/engine/models/voxcpm2/generator.h index abbcf8be..cf46d955 100644 --- a/include/engine/models/voxcpm2/generator.h +++ b/include/engine/models/voxcpm2/generator.h @@ -49,6 +49,7 @@ class VoxCPM2FeatureGeneratorRuntime final { const std::function &chunk_callback = nullptr); void release_runtime_memory(); + void release_text_length_memory(); private: class Impl; diff --git a/include/engine/models/voxcpm2/minicpm.h b/include/engine/models/voxcpm2/minicpm.h index 1aa57fc3..5b40a03b 100644 --- a/include/engine/models/voxcpm2/minicpm.h +++ b/include/engine/models/voxcpm2/minicpm.h @@ -135,6 +135,7 @@ class VoxCPM2TextEmbeddingRuntime final { ~VoxCPM2TextEmbeddingRuntime(); std::vector embed_token(int32_t token_id); + void release_runtime_memory(); private: class Impl; @@ -150,6 +151,7 @@ class VoxCPM2PromptPrefillRuntime final { ~VoxCPM2PromptPrefillRuntime(); VoxCPM2PromptPrefillOutput run(const VoxCPM2PromptPrefillInput &input); + void release_runtime_memory(); private: class Impl; diff --git a/src/models/voxcpm2/audiovae.cpp b/src/models/voxcpm2/audiovae.cpp index 40893122..f59fe280 100644 --- a/src/models/voxcpm2/audiovae.cpp +++ b/src/models/voxcpm2/audiovae.cpp @@ -710,9 +710,11 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { void release_runtime_memory() { release_decoder_graph(); - release_encoder_graph(); + release_encoder_graph_impl(); } + void release_encoder_graph() { release_encoder_graph_impl(); } + private: struct EncodedFeatures { std::vector features; @@ -937,7 +939,7 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { decoder_latent_frame_capacity_ = latent_frame_capacity; } - void release_encoder_graph() { + void release_encoder_graph_impl() { if (encoder_graph_ != nullptr) { core::release_backend_graph_resources(execution_context_.backend(), encoder_graph_); @@ -1059,4 +1061,8 @@ void VoxCPM2AudioVAEDecoderRuntime::release_runtime_memory() { impl_->release_runtime_memory(); } +void VoxCPM2AudioVAEDecoderRuntime::release_encoder_graph() { + impl_->release_encoder_graph(); +} + } // namespace engine::models::voxcpm2 diff --git a/src/models/voxcpm2/generator.cpp b/src/models/voxcpm2/generator.cpp index f62e1b05..4f1e7af2 100644 --- a/src/models/voxcpm2/generator.cpp +++ b/src/models/voxcpm2/generator.cpp @@ -206,6 +206,7 @@ class VoxCPM2StepProjectionRuntime final { VoxCPM2StepProjectionOutput run(const std::vector &lm_hidden, const std::vector &residual_hidden, const std::vector ¤t_embed); + void release_runtime_memory(); private: class Impl; @@ -221,6 +222,7 @@ class VoxCPM2LocalEncoderRuntime final { std::vector encode_patch(const std::vector &patch_features) const; + void release_runtime_memory(); private: class Impl; @@ -239,6 +241,7 @@ class VoxCPM2DiTEstimatorRuntime final { const std::vector &cond, const std::vector &time_embedding, const std::vector &delta_time_embedding); + void release_runtime_memory(); private: class Impl; @@ -259,6 +262,7 @@ class VoxCPM2CFMRuntime final { uint64_t noise_start_index = 0, const std::string &noise_file = {}, float temperature = 1.0F); + void release_runtime_memory(); private: class Impl; @@ -277,15 +281,9 @@ class VoxCPM2StepProjectionRuntime::Impl { build(graph_context_bytes); } - ~Impl() { - engine::core::release_backend_graph_resources(weights_->backend(), graph_); - if (buffer_ != nullptr) { - ggml_backend_buffer_free(buffer_); - } - if (gallocr_ != nullptr) { - ggml_gallocr_free(gallocr_); - } - } + ~Impl() { release_graph(); } + + void release_runtime_memory() { release_graph(); } VoxCPM2StepProjectionOutput run(const std::vector &lm_hidden, const std::vector &residual_hidden, @@ -356,6 +354,33 @@ class VoxCPM2StepProjectionRuntime::Impl { } private: + void release_graph() { + if (graph_ != nullptr) { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + } + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + buffer_ = nullptr; + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + graph_ = nullptr; + lm_hidden_ = nullptr; + residual_hidden_ = nullptr; + current_embed_ = nullptr; + fsq_hidden_output_ = nullptr; + current_residual_input_output_ = nullptr; + residual_input_output_ = nullptr; + current_lm_dit_output_ = nullptr; + fsq_lm_dit_output_ = nullptr; + residual_dit_output_ = nullptr; + current_stop_logits_output_ = nullptr; + fsq_stop_logits_output_ = nullptr; + ctx_.reset(); + } + void build(size_t graph_context_bytes) { const auto &config = weights_->assets().config; if (graph_context_bytes == 0) { @@ -579,6 +604,10 @@ VoxCPM2StepProjectionRuntime::VoxCPM2StepProjectionRuntime( VoxCPM2StepProjectionRuntime::~VoxCPM2StepProjectionRuntime() = default; +void VoxCPM2StepProjectionRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + VoxCPM2StepProjectionOutput VoxCPM2StepProjectionRuntime::run(const std::vector &lm_hidden, const std::vector &residual_hidden, @@ -598,15 +627,9 @@ class VoxCPM2LocalEncoderRuntime::Impl { build(graph_context_bytes); } - ~Impl() { - engine::core::release_backend_graph_resources(weights_->backend(), graph_); - if (buffer_ != nullptr) { - ggml_backend_buffer_free(buffer_); - } - if (gallocr_ != nullptr) { - ggml_gallocr_free(gallocr_); - } - } + ~Impl() { release_graph(); } + + void release_runtime_memory() { release_graph(); } std::vector encode_patch(const std::vector &patch_features) const { @@ -632,6 +655,25 @@ class VoxCPM2LocalEncoderRuntime::Impl { } private: + void release_graph() { + if (graph_ != nullptr) { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + } + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + buffer_ = nullptr; + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + graph_ = nullptr; + input_ = nullptr; + positions_ = nullptr; + output_ = nullptr; + ctx_.reset(); + } + void build(size_t graph_context_bytes) { const auto &root_config = weights_->assets().config; if (graph_context_bytes == 0) { @@ -736,6 +778,10 @@ VoxCPM2LocalEncoderRuntime::VoxCPM2LocalEncoderRuntime( VoxCPM2LocalEncoderRuntime::~VoxCPM2LocalEncoderRuntime() = default; +void VoxCPM2LocalEncoderRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + std::vector VoxCPM2LocalEncoderRuntime::encode_patch( const std::vector &patch_features) const { return impl_->encode_patch(patch_features); @@ -753,15 +799,9 @@ class VoxCPM2DiTEstimatorRuntime::Impl { build(graph_context_bytes); } - ~Impl() { - engine::core::release_backend_graph_resources(weights_->backend(), graph_); - if (buffer_ != nullptr) { - ggml_backend_buffer_free(buffer_); - } - if (gallocr_ != nullptr) { - ggml_gallocr_free(gallocr_); - } - } + ~Impl() { release_graph(); } + + void release_runtime_memory() { release_graph(); } std::vector run(const std::vector &x, const std::vector &mu, @@ -937,6 +977,29 @@ class VoxCPM2DiTEstimatorRuntime::Impl { } private: + void release_graph() { + if (graph_ != nullptr) { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + } + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + buffer_ = nullptr; + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + graph_ = nullptr; + x_ = nullptr; + mu_ = nullptr; + cond_ = nullptr; + time_embedding_ = nullptr; + delta_time_embedding_ = nullptr; + positions_ = nullptr; + output_ = nullptr; + ctx_.reset(); + } + void build(size_t graph_context_bytes) { const auto &root_config = weights_->assets().config; const auto &config = root_config.dit; @@ -1164,6 +1227,10 @@ VoxCPM2DiTEstimatorRuntime::VoxCPM2DiTEstimatorRuntime( VoxCPM2DiTEstimatorRuntime::~VoxCPM2DiTEstimatorRuntime() = default; +void VoxCPM2DiTEstimatorRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + std::vector VoxCPM2DiTEstimatorRuntime::run( const std::vector &x, const std::vector &mu, const std::vector &cond, const std::vector &time_embedding, @@ -1201,6 +1268,8 @@ class VoxCPM2CFMRuntime::Impl { } } + void release_runtime_memory() { estimator_.release_runtime_memory(); } + std::vector generate_patch(const std::vector &mu, const std::vector &cond_patch, int64_t timesteps, float cfg_value, @@ -1408,6 +1477,10 @@ VoxCPM2CFMRuntime::VoxCPM2CFMRuntime( VoxCPM2CFMRuntime::~VoxCPM2CFMRuntime() = default; +void VoxCPM2CFMRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + std::vector VoxCPM2CFMRuntime::generate_patch( const std::vector &mu, const std::vector &cond_patch, int64_t timesteps, float cfg_value, uint64_t seed, @@ -1503,8 +1576,23 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { } void release_runtime_memory() { + // Release every staged graph so a session can idle at weight-only + // VRAM. Each runtime lazily rebuilds its graph on the next use. + text_embedding_.release_runtime_memory(); + prefill_.release_runtime_memory(); base_lm_.release_runtime_memory(); residual_lm_.release_runtime_memory(); + projection_.release_runtime_memory(); + cfm_.release_runtime_memory(); + local_encoder_.release_runtime_memory(); + } + + void release_text_length_memory() { + // Only the prompt-prefill graph is sized by the request text/prompt + // length; the other generator graphs have fixed-size workspaces. Drop it + // after every request so a long-lived session does not retain buffers + // that scale with text length; the next request rebuilds it fresh. + prefill_.release_runtime_memory(); } private: @@ -1795,6 +1883,11 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { residual_lm_.import_state(prefill_output.residual_state); std::vector lm_hidden = prefill_output.lm_hidden; std::vector residual_hidden = prefill_output.residual_hidden; + // The prefill graph holds the largest sequence-shaped workspace; its + // outputs have been copied to host hiddens and its KV state imported + // into the step runtimes, so nothing below references it. Drop it now + // so the token loop runs against the much smaller step graphs. + prefill_.release_runtime_memory(); VoxCPM2Result result; std::vector context_rows; @@ -1947,4 +2040,8 @@ void VoxCPM2FeatureGeneratorRuntime::release_runtime_memory() { impl_->release_runtime_memory(); } +void VoxCPM2FeatureGeneratorRuntime::release_text_length_memory() { + impl_->release_text_length_memory(); +} + } // namespace engine::models::voxcpm2 diff --git a/src/models/voxcpm2/minicpm.cpp b/src/models/voxcpm2/minicpm.cpp index e03a02e7..409aec91 100644 --- a/src/models/voxcpm2/minicpm.cpp +++ b/src/models/voxcpm2/minicpm.cpp @@ -338,15 +338,11 @@ class VoxCPM2TextEmbeddingRuntime::Impl { } ~Impl() { - engine::core::release_backend_graph_resources(weights_->backend(), graph_); - if (buffer_ != nullptr) { - ggml_backend_buffer_free(buffer_); - } - if (gallocr_ != nullptr) { - ggml_gallocr_free(gallocr_); - } + release_graph(); } + void release_runtime_memory() { release_graph(); } + std::vector embed_token(int32_t token_id) { const auto &config = weights_->assets().config.lm; if (token_id < 0 || token_id >= config.vocab_size) { @@ -368,6 +364,24 @@ class VoxCPM2TextEmbeddingRuntime::Impl { } private: + void release_graph() { + if (graph_ != nullptr) { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + } + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + buffer_ = nullptr; + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + graph_ = nullptr; + token_id_ = nullptr; + output_ = nullptr; + ctx_.reset(); + } + void build(size_t graph_context_bytes) { const auto &config = weights_->assets().config.lm; if (graph_context_bytes == 0) { @@ -450,6 +464,10 @@ std::vector VoxCPM2TextEmbeddingRuntime::embed_token(int32_t token_id) { return impl_->embed_token(token_id); } +void VoxCPM2TextEmbeddingRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + struct MiniCPMLayerWithCacheOutput { engine::core::TensorValue output; engine::core::TensorValue key; @@ -578,6 +596,8 @@ class VoxCPM2PromptPrefillRuntime::Impl { ~Impl() { release_graph(); } + void release_runtime_memory() { release_graph(); } + VoxCPM2PromptPrefillOutput run(const VoxCPM2PromptPrefillInput &input) { const auto &config = weights_->assets().config; const int64_t hidden_size = config.lm.hidden_size; @@ -909,6 +929,10 @@ VoxCPM2PromptPrefillRuntime::run(const VoxCPM2PromptPrefillInput &input) { return impl_->run(input); } +void VoxCPM2PromptPrefillRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + engine::core::TensorValue minicpm_layer_with_static_cache(engine::core::ModuleBuildContext &ctx, const engine::core::TensorValue &input, diff --git a/src/models/voxcpm2/session.cpp b/src/models/voxcpm2/session.cpp index 238ef3d8..45072ba2 100644 --- a/src/models/voxcpm2/session.cpp +++ b/src/models/voxcpm2/session.cpp @@ -274,8 +274,7 @@ runtime::TaskResult VoxCPM2SessionBase::run_offline_request(const runtime::TaskR } }; std::unique_ptr - release_guard(generator_config_.mem_saver ? this : nullptr, - release_runtime_memory); + release_guard(this, release_runtime_memory); const auto wall_start = Clock::now(); const int64_t text_chunk_size = @@ -299,17 +298,32 @@ runtime::TaskResult VoxCPM2SessionBase::run_offline_request(const runtime::TaskR const VoxCPM2EncodedPrompt *prompt = encoded_prompt_for_request(request.audio_input, prompt_text, reference_audio); + // The encoded prompt (voice-clone conditioning) is cached as host-side + // vectors; the VAE encoder graph that produced it is not needed again until + // a different voice is encoded. Drop it before the generator runs so the + // generator and decoder phases never coexist with the encoder graph. + decoder_->release_encoder_graph(); runtime::TaskResult result; double generator_ms = 0.0; double decoder_ms = 0.0; runtime::AudioBuffer merged_audio; - for (const auto & chunk_request : chunk_requests) { + const bool mem_saver = generator_config_.mem_saver; + for (size_t chunk_index = 0; chunk_index < chunk_requests.size(); + ++chunk_index) { + const auto &chunk_request = chunk_requests[chunk_index]; const auto generator_start = Clock::now(); const auto generated = generator_->generate( chunk_request.text_input->text, prompt, generation_options); generator_ms += engine::debug::elapsed_ms(generator_start, Clock::now()); + if (mem_saver && chunk_index + 1 == chunk_requests.size()) { + // Last chunk: free the generator graphs before the AudioVAE decode so + // the final decode peaks at weight + decoder graph instead of weight + + // generator + decoder. Graphs rebuild lazily on the next request. + generator_->release_runtime_memory(); + } + const auto decoder_start = Clock::now(); auto audio = decoder_->decode_features(generated.decode_features, generated.decode_patches); @@ -359,8 +373,7 @@ VoxCPM2SessionBase::run_streaming_request( } }; std::unique_ptr - release_guard(generator_config_.mem_saver ? this : nullptr, - release_runtime_memory); + release_guard(this, release_runtime_memory); const auto wall_start = Clock::now(); auto generation_options = generation_options_from_request(request); @@ -377,6 +390,10 @@ VoxCPM2SessionBase::run_streaming_request( const VoxCPM2EncodedPrompt *prompt = encoded_prompt_for_request(request.audio_input, prompt_text, reference_audio); + // Same host-side clone-conditioning cache invariant as the offline path: + // the encoder graph is only needed to produce the cached vectors, so free + // it before the streaming generation starts. + decoder_->release_encoder_graph(); runtime::TaskResult result; runtime::AudioBuffer merged; @@ -437,11 +454,18 @@ VoxCPM2SessionBase::run_streaming_request( } void VoxCPM2SessionBase::release_request_runtime_memory() { - if (!generator_config_.mem_saver) { - return; - } - generator_->release_runtime_memory(); + // Only the cloned voice is cached across requests (host-side encoded + // vectors in encoded_prompt_cache_). Every graph whose size follows the + // request text/audio length (prompt prefill, VAE encoder/decoder) is + // dropped so a long-lived server session returns to baseline VRAM and + // reallocates fresh buffers sized to the next request. + generator_->release_text_length_memory(); decoder_->release_runtime_memory(); + if (generator_config_.mem_saver) { + // mem_saver additionally drops the fixed-size generator graphs so the + // session idles at weight-only VRAM. + generator_->release_runtime_memory(); + } } const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( From 1d6576c942bf761b9deb9ac2d8126528c64c678d Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Wed, 19 Aug 2026 23:14:49 +0200 Subject: [PATCH 13/23] remove local --- docs/reports/voxcpm1_pr.md | 243 ------------------------------------- 1 file changed, 243 deletions(-) delete mode 100644 docs/reports/voxcpm1_pr.md diff --git a/docs/reports/voxcpm1_pr.md b/docs/reports/voxcpm1_pr.md deleted file mode 100644 index a5d3b70e..00000000 --- a/docs/reports/voxcpm1_pr.md +++ /dev/null @@ -1,243 +0,0 @@ -# PR: VoxCPM1 — lightweight VoxCPM TTS support (0.5B / 1.5B) - -> **Status: first porting attempt — runtime works end-to-end, output quality NOT yet acceptable** -> -> The port successfully loads and runs all three VoxCPM v1 GGUF variants (anchors pass, graphs -> execute, WAV files are produced at the correct sample rates/durations with active signal). -> **Known issue:** the generated audio is almost pure noise with only a faint trace of human -> voice. The pipeline is correct mechanically, but output quality requires further debugging -> (hypotheses and investigation plan in [Known issue](#known-issue-noisy-output)). - ---- - -## 1. Overview - -This PR adds support for the **OpenBMB VoxCPM v1** family of lightweight TTS models to -audio.cpp, reusing the existing and already-released `voxcpm2` model tree: - -| Model | Params | Output sample rate | GGUF file | -|---|---|---|---| -| VoxCPM-0.5B | 0.5B | **16 kHz** | `voxcpm-0.5b-q8_0-audiovae-f16.gguf` | -| VoxCPM-1.5B | 1.5B | **44.1 kHz** | `voxcpm1.5-q8_0.gguf` | -| VoxCPM-1.5B | 1.5B | **44.1 kHz** | `voxcpm1.5-q4_k-audiovae-f16.gguf` | - -The three models are architecturally **different variants** (they cannot share one config): - -- **0.5B:** VAE encoder 128 / decoder 1536, encoder_rates `[2,5,8,8]`, decoder_rates - `[8,8,5,2]`, patch_size 2, residual_lm 6 layers, encoder/dit 4 layers, 16 kHz, max_len 4096. -- **1.5B:** VAE encoder 64 / decoder 2048, encoder_rates `[2,3,6,7,7]`, decoder_rates - `[7,7,6,3,2]`, patch_size 4, residual_lm 8 layers, encoder/dit 8 layers, 44.1 kHz, max_len 8192. - -Since the v1 GGUFs store a different tensor convention than v2 (folded AudioVAE weights, no -`weight_v`/`weight_g` split, no `sr_cond_model` tensors, `voxcpm` architecture name), the port -wraps the v2 loader with a GGUF tensor-adaptation layer and adds `config.v1`-guarded branches -in the generator, mirroring the reference implementation (`VoxCPM.cpp`). - ---- - -## 2. Porting activities - -1. **Regenerated the 0.5B `config.json` from the GGUF metadata** — the previously shipped - sidecar was wrong on ~8 axes (patch, residual_lm/encoder/dit layer counts, VAE dims and - rates, sample rate 44.1 kHz vs the actual 16 kHz, max_len). -2. **Diagnosed the v1 GGUF conventions** (tensor dump + reference converter analysis): - - AudioVAE conv weights are stored **already folded** (weight-norm folded), with no - `weight_v`/`weight_g` decomposition and no `sr_cond_model.*` tensors. - - GGUF file dims == ggml `ne` order; the v1 GGUFs carry **no** `audiocpp.tensor_shapes` - override metadata (v2 does), so the adapter must present shapes itself. - - The 1.5B **Q8_0** file stores VAE conv weights **2D-flattened** (`{out, in·k}`, kernel - folded into dim1) while Q4_K and 0.5B store 3D `{out, in, k}` — both must load. -3. **Designed the identity-fold adapter** (see §4) so the existing `load_vae_weights` loader - works unchanged against folded v1 weights byte-for-byte. -4. **Mirrored the reference generator math** for the no-fusion (no `fusion_concat_proj`) - case: elementwise-add fusion inputs, elementwise-add dit-mu, and a real residual_lm - autoregressive step. -5. **Set up per-variant model directories** (`VoxCPM1-GGUF/` for 0.5B, `VoxCPM1.5-GGUF/` - for 1.5B) each with a config regenerated from its own GGUF metadata + tokenizer sidecars, - and updated `model_specs/voxcpm1.json` package targets accordingly. -6. **Verified end-to-end runs** for all three GGUFs on the CPU backend (see - [Validation](#6-validation-performed)). - ---- - -## 3. Changes per file - -| File | Change | -|---|---| -| `CMakeLists.txt` | Added `audiocpp_add_model(voxcpm1 ...)` reusing the 7 voxcpm2 sources; registers `engine::models::voxcpm2::make_voxcpm1_loader`. | -| `include/engine/models/voxcpm2/loader.h` | Declared `make_voxcpm1_loader()`. | -| `include/engine/models/voxcpm2/assets.h` | Added `VoxCPM2Config::v1 = false`; `load_voxcpm2_assets()` now takes `bool is_v1`. | -| `src/models/voxcpm2/loader.cpp` | Added `VoxCPM1Loader` (family `"voxcpm1"`), `load_voxcpm1_model()`, `make_voxcpm1_loader()`, `metadata_v1` / `capabilities_v1` / `cli_v1`. Offline-only TTS + speaker-reference clone, `text_prefix` policy, GGUF via `load_voxcpm2_assets(path, is_v1=true)`. | -| `src/models/voxcpm2/assets.cpp` | Added `TransformingTensorSource` v1 adapter (biggest chunk):
• v1→v2 tensor-name rename map (`token_embd.weight`→`base_lm.embed_tokens.weight`, gguf `blk.N.*`→`base_lm.layers.N.*` / `feat_encoder.encoder.layers.*` / `feat_decoder.estimator.decoder.layers.*` / `residual_lm.layers.*`, `attn_norm`→`input_layernorm`, `ffn_norm`→`post_attention_layernorm`, `attn_*`→`self_attn.*_proj`, `ffn_*`→`mlp.*_proj`, `time_mlp.*` (preserving `.linear_N`), `output_norm.weight`→`base_lm.norm.weight`, projection/fsq/stop mappings)
• **Folded weight-norm synthesis**: for every `audio_vae.*.weight` conv, `X.weight_v` → folded tensor data as-is, `X.weight_g` → per-row L2 norms (identity fold, see §4)
• Identity `decoder.sr_cond_model.{2..5}.scale_embed.weight` (ones) / `.bias_embed.weight` (zeros) since v1 GGUFs carry no SR-conditioning tensors
• Synthesized missing v1 tensors (`feat_encoder.scale_embed/bias_embed`, `feat_encoder.fc_logvar`, `feat_encoder.diag`, `feat_encoder.merge`, `token_embd.extra_bias`, `fusion_concat_proj.weight/bias`, `stop_proj.weight`, `stop_head.weight`)
• Rank-tolerant `require_f32` (accept element-count-equal, shape-different fetches — handles 2D-flattened convs and `{C,1}` alphas) + relaxed-rank VAE weight_v anchors for v1
• `has_tensor` / `require_metadata` / `require_tensor_data` folded + synthesized lookups
• **Anchor fix:** `encoder.fc_mu.weight_v` now uses computed encoder-in (`encoder_dim << #rates` = 2048), not `decoder_dim` (1536) | -| `src/models/voxcpm2/generator.cpp` | • v1 fusion guard: residual input = `AddModule(lm_hidden, current_embed)` / `AddModule(fsq, current_embed)` instead of concat+linear (matches reference `build_residual_fusion_input`)
• Added `add_dit_mu()` helper; v1 `mu` = elementwise add of `current_lm_dit_hidden + residual_dit_hidden` (matches reference `build_dit_mu`, `mu_dim = hidden·(fusion?2:1)`, v1 → hidden)
• CFM `mu` size check is now v1-aware (`hidden_dim * (v1 ? 1 : 2)`)
• v1 decode loop runs `residual_lm_.run_step(next_projected.residual_input).hidden` (the earlier `fsq_lm_dit_hidden` shortcut removed — v1 GGUFs have 6/8 residual_lm layers) | -| `src/models/voxcpm2/minicpm.cpp` | Prompt-prefill graph: v1 `residual_input` = `AddModule(lm_hidden, masked_current)` instead of concat+linear; residual_lm always runs (previously the concat path would have produced a wrong-dimension residual input for v1). | -| `model_specs/voxcpm1.json` | Package targets: `voxcpm1_0.5b_q8_0` → `VoxCPM1-GGUF`; `voxcpm1_1.5b_q4_k` and `voxcpm1_1.5b_q8_0` → `VoxCPM1.5-GGUF` (per-variant config/tokenizer). | -| `docs/tts.md` | Added VoxCPM1 section + TOC entry (usage, options, sample-rate notes). | -| `README.md` | Added `voxcpm1` row to the supported-model table. | -| `docs/reports/voxcpm1_port_status.md` | Port status log (analysis, decisions, timestamps, remaining tasks). | -| `models/VoxCPM1-GGUF/config.json` | **Regenerated** from 0.5B GGUF metadata. | -| `models/VoxCPM1.5-GGUF/config.json` | **New**, regenerated from 1.5B GGUF metadata. | -| `models/VoxCPM1.5-GGUF/tokenizer.json` (+config/special tokens) | Copied from 0.5B dir (same 73,448-vocab BPE tokenizer). | - ---- - -## 4. Key design: the identity-fold adapter - -The v1 GGUF (OpenBMB reference converter) stores AudioVAE conv weights **already folded** -(`weight = weight_g · weight_v / ‖weight_v‖`), with no `weight_v`/`weight_g` split, while -`audiovae.cpp` requests the decomposed names directly via `require_f32`. The adapter solves -this without touching the VAE loader: - -``` -X.weight_v := folded GGUF tensor data (as-is) -X.weight_g := per-row L2 norms of the folded tensor, - computed with the loader's own row grouping - (groups = expected_shape.front(), inner = elements/groups) -``` - -Because `fold_weight_norm` multiplies row `d0` by `weight_g[d0] / ‖row d0‖ = 1`, the loader -output equals the GGUF data **byte-for-byte** — an exact identity, with no layout drift -relative to the reference runtime's consumption of the same bytes. The same mechanism works -for 3D `{out, in, k}` and 2D-flattened `{out, in·k}` conversions (element counts must match; -ranks may differ, covered by rank-tolerant `require_f32` + relaxed-rank anchors). - ---- - -## 5. Usage - -### Build - -```bash -scripts/build_linux.sh --backend cpu --target audiocpp_cli -# or, with the standard full model set: -cmake -S . -B build/linux-cpu-release -DCMAKE_BUILD_TYPE=Release -cmake --build build/linux-cpu-release --target audiocpp_cli -j 8 -``` - -### Run — 0.5B (16 kHz output) - -```bash -build/linux-cpu-release/bin/audiocpp_cli \ - --task tts --family voxcpm1 \ - --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf \ - --backend cpu --text "Hello from VoxCPM1." --out out.wav -``` - -### Run — 1.5B (44.1 kHz output) - -```bash -build/linux-cpu-release/bin/audiocpp_cli \ - --task tts --family voxcpm1 \ - --model models/VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf \ - --backend cpu --text "Hello from VoxCPM1." --out out.wav -``` - -### Options - -| Option | Values | Default | Meaning | -|---|---:|---:|---| -| `--task` | `tts` | required | Task kind. | -| `--family` | `voxcpm1` | auto-detect | Selects the v1 loader. | -| `--backend` | `cpu`, `cuda`, `vulkan`, `metal`, `hip`, `best` | `best` | Backend. | -| `--voice-ref` | WAV path | not set | Reference speaker audio (clone). | -| `--max-tokens` | integer | `4096` | Maximum generated AR tokens. | -| `--num-inference-steps` | integer | `10` | Flow-matching steps. | -| `--guidance-scale` | float | `2.0` | CFG strength. | -| `--session-option voxcpm1.mem_saver=true\|false` | bool | `false` | Tighter graph workspaces + release request graphs after completion. | -| `--session-option voxcpm1.prompt_cache_slots=` | integer | `1` | Prompt/prompt-audio embedding cache slots. | -| `--text-chunk-mode` | `default`, `tag_aware`, `japanese`, `endline` | `tag_aware` | Long-form chunking mode. | - ---- - -## 6. Validation performed - -- **Load + anchors:** all three GGUFs pass `validate_weight_anchors` and - `load_vae_weights`/`load_model_weights` on CPU. This includes the 0.5B 3D convs, the 1.5B - Q4_K 3D convs, and the 1.5B Q8_0 2D-flattened convs. -- **End-to-end:** `--task tts` completes for all three models; outputs are written as WAV at - the correct sample rate (16 kHz for 0.5B, 44.1 kHz for 1.5B) with active signal and - speech-plausible duration/envelope. -- **Regression:** the released voxcpm2 path is untouched (guard style `config.v1`, v2 default - `false`); voxcpm2 was not re-benchmarked but the changed code paths are v1-gated or - v1/v2-neutral. - -> ⚠️ **Quality caveat:** "end-to-end completes" does **not** mean the output is usable yet. -> See the known issue below — the audio is predominantly noise. - ---- - -## 7. Supported modes - -| Mode | Supported | Notes | -|---|---|---| -| **Offline TTS** | ✅ implemented | Default and only advertised mode. | -| **Streaming** | ❌ not implemented for v1 | `voxcpm1` advertises offline-only. Streaming is a v2 capability; it has not been validated (or enabled) for v1. | -| **Voice clone** | ⚠️ surface present | Speaker-reference options are advertised (`--voice-ref`), but quality is gated on the same known issue as plain TTS. | - ---- - -## Known issue: noisy output - -**Symptom.** Generated v1 voices are almost pure noise with a little human voice mixed in — -the signal is dominated by broadband/noise content. This affects all three GGUFs. - -**What is confirmed working.** Model loading, tensor adaptation, anchor validation, graph -construction, graph execution, and WAV output plumbing are all correct (no crashes, no -shape/size errors, correct sample rates and durations). The failure is therefore in the -**numerics of synthesis**, i.e. the audio content itself. - -**Most likely causes (in rough priority order).** - -1. **Weight data interpretation** — the identity fold preserves bytes, but if some AudioVAE - layer's storage layout (depthwise vs pointwise handling, 2D-flattened Q8_0, transposed - decoder `{in,out,k}` conventions, per-group row ordering of `weight_g`) differs from what - `ggml_conv_1d` / `conv_transpose` expects, the VAE decoder outputs garbage while loading - still "succeeds" (element counts match). -2. **Synthesized tensor semantics** — `feat_encoder.scale_embed/bias_embed`, `merge`, - `diag`, `extra_bias`, `fusion_concat_proj`, `stop_*`, and the identity `sr_cond_model` - tensors were synthesized with plausible but unverified semantics; if any is required to be - learned/zero-`scale` (or absent entirely in the reference runtime), the feature stream - feeding the LM/CFM is wrong. -3. **Graph parity vs the reference** — fusion = add and dit-mu = add were taken from - reference `build_residual_fusion_input`/`build_dit_mu`, but adjacent details (masking, - slice indices, position ids, prompt handling, FSQ rounding, CFM conditioning inputs, - ordering of `nn.Module` sub-blocks in the residual_lm stack) may differ. -4. **Sample-rate/codec mismatch** — 0.5B output asserted 16 kHz but the reference may expect - a specific internal feature rate; patch_size/feat_dim interplay (2·64 vs 4·64) feeding the - CFM estimator could be off by a constant factor, producing frozen-then-noisy patches. -5. **Quantization path** — the 1.5B Q8_0 GGUF quantizes the VAE itself (2D-flattened); - dequantized values feed `require_f32`, but a transpose or block-order mismatch would - corrupt every activation. - -**Debugging plan (next iteration).** - -- [ ] Port a small deterministic parity harness: run the same prompt through the reference - `VoxCPM.cpp` and audio.cpp, dump intermediate tensors (lm hidden, residual hidden, - CFM mu, VAE latent, decoder output) at each major stage, and diff numerically. -- [ ] Verify `encoder.fc_mu` / `decoder.model.{0,1,N}` folded data against the Python - reference weights with a strict per-element comparison on non-quantized tensors - (f16 VAE files), including row-grouping of `weight_g`. -- [ ] Check whether the reference runtime actually instantiates `sr_cond_model` and - `feat_encoder` synthesizable blocks for v1; remove or zero-scale any block the - reference does not run. -- [ ] Experimentally force one suspected block to a no-op (e.g. sr_cond identity, merge - zeros, scale_embed 0/1) and measure whether noise level drops. -- [ ] Validate CFM mu dimension/conditioning against the reference expectation for - `patch=2` (0.5B) and `patch=4` (1.5B). -- [ ] After the numerics match, run a human listening + loudness/spectral sanity check - (the current output has a spectral envelope consistent with noise + faint voice). - ---- - -## 8. Remaining tasks - -- [x] Loader registration, tensor adaptation, generator v1 branches, configs, model spec -- [x] End-to-end execution for 0.5B Q8_0, 1.5B Q4_K, 1.5B Q8_0 -- [ ] **Fix noisy output (known issue above) — top priority** -- [ ] Numerical parity harness vs `VoxCPM.cpp` reference (stage-by-stage tensor diff) -- [ ] `tests/voxcpm1/` automated path tests mirroring `tests/voxcpm2/` -- [ ] WebUI catalog entry (`webui/configs/models_catalog.json`) -- [ ] `docs/gguf.md` support-table entry -- [ ] CUDA-backend verification + RTF measurement (expect voxcpm2-like speedups) -- [ ] Streaming support for v1 (only meaningful after numerics are fixed) -- [ ] Commit + release packaging for `audio.cpp-gguf` (0.5B and 1.5B packages) \ No newline at end of file From 5b949e78b1408528036472c76c4bd4ba78966dfc Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Thu, 20 Aug 2026 00:25:56 +0200 Subject: [PATCH 14/23] generate new index.html based on merged code --- webui/native/dist/index.html | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/webui/native/dist/index.html b/webui/native/dist/index.html index 9ce05069..b875ab6b 100644 --- a/webui/native/dist/index.html +++ b/webui/native/dist/index.html @@ -5,7 +5,7 @@ - + @@ -13,20 +13,20 @@
From 47a7e7e5cd898810148555a74948ebd1ee33e91d Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Thu, 20 Aug 2026 11:40:58 +0200 Subject: [PATCH 15/23] fix(vpxcpm1): release the last framework change. udpate index.html for merge --- include/engine/framework/assets/tensor_source.h | 1 - src/models/voxcpm2/generator.cpp | 10 +++++----- webui/native/dist/index.html | 16 ++++++++-------- 3 files changed, 13 insertions(+), 14 deletions(-) diff --git a/include/engine/framework/assets/tensor_source.h b/include/engine/framework/assets/tensor_source.h index b8088582..6735f208 100644 --- a/include/engine/framework/assets/tensor_source.h +++ b/include/engine/framework/assets/tensor_source.h @@ -118,7 +118,6 @@ class TensorSource { [[nodiscard]] std::string require_tensor_name( std::initializer_list candidates) const; [[nodiscard]] virtual int64_t require_i64_scalar(std::string_view name) const = 0; - [[nodiscard]] virtual bool is_synthesized(std::string_view) const noexcept { return false; } }; [[nodiscard]] TensorStorageType parse_tensor_storage_type(std::string_view value); diff --git a/src/models/voxcpm2/generator.cpp b/src/models/voxcpm2/generator.cpp index 4f1e7af2..607cb2d6 100644 --- a/src/models/voxcpm2/generator.cpp +++ b/src/models/voxcpm2/generator.cpp @@ -445,7 +445,7 @@ class VoxCPM2StepProjectionRuntime::Impl { // For V1 models, synthesized weights (Xavier init) should not count as present const bool has_fusion_proj = proj.fusion_concat_proj.weight.tensor != nullptr && - !weights_->assets().model_weights->is_synthesized("fusion_concat_proj.weight"); + config.architecture == "voxcpm2"; if (has_fusion_proj) { // Concat + Linear (used by V2 and some V1 models trained with fusion) @@ -814,7 +814,7 @@ class VoxCPM2DiTEstimatorRuntime::Impl { // For V1 models, synthesized weights (Xavier init) should not count as present const bool has_fusion_proj = weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && - !weights_->assets().model_weights->is_synthesized("fusion_concat_proj.weight"); + config.architecture == "voxcpm2"; const int64_t patch_elems = 2 * config.feat_dim * config.patch_size; if (static_cast(x.size()) != patch_elems) { throw std::runtime_error("VoxCPM2 DiT estimator x size mismatch"); @@ -1036,7 +1036,7 @@ class VoxCPM2DiTEstimatorRuntime::Impl { // For V1 models, synthesized weights (Xavier init) should not count as present const bool has_fusion_proj = weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && - !weights_->assets().model_weights->is_synthesized("fusion_concat_proj.weight"); + root_config.architecture == "voxcpm2"; mu_ = engine::core::make_tensor( ctx, GGML_TYPE_F32, has_fusion_proj @@ -1282,7 +1282,7 @@ class VoxCPM2CFMRuntime::Impl { // For V1 models, synthesized weights (Xavier init) should not count as present const bool has_fusion_proj = weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && - !weights_->assets().model_weights->is_synthesized("fusion_concat_proj.weight"); + config.architecture == "voxcpm2"; if (timesteps <= 0) { throw std::runtime_error("VoxCPM2 CFM requires positive timesteps"); } @@ -1933,7 +1933,7 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { // For V1 models, synthesized weights (Xavier init) should not count as present const bool has_fusion_proj = weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && - !weights_->assets().model_weights->is_synthesized("fusion_concat_proj.weight"); + config.architecture == "voxcpm2"; const auto mu = has_fusion_proj ? concat_dit_mu(projected.current_lm_dit_hidden, projected.residual_dit_hidden) diff --git a/webui/native/dist/index.html b/webui/native/dist/index.html index 73385fee..a1c903ef 100644 --- a/webui/native/dist/index.html +++ b/webui/native/dist/index.html @@ -13,20 +13,20 @@
From 730fb37c21577f70e23d1748b24bcce660be6964 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Thu, 20 Aug 2026 11:44:37 +0200 Subject: [PATCH 16/23] remove loacal file --- src/models/voxcpm2/session.cpp~ | 708 -------------------------------- 1 file changed, 708 deletions(-) delete mode 100644 src/models/voxcpm2/session.cpp~ diff --git a/src/models/voxcpm2/session.cpp~ b/src/models/voxcpm2/session.cpp~ deleted file mode 100644 index c3cf78bf..00000000 --- a/src/models/voxcpm2/session.cpp~ +++ /dev/null @@ -1,708 +0,0 @@ -#include "engine/models/voxcpm2/session.h" - -#include "engine/framework/debug/profiler.h" -#include "engine/framework/runtime/options.h" -#include "engine/framework/text/chunking.h" - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -namespace engine::models::voxcpm2 { -namespace { - -using Clock = std::chrono::steady_clock; -constexpr int64_t kDefaultTextChunkSize = 2048; - -std::shared_ptr -require_assets(std::shared_ptr assets) { - if (assets == nullptr) { - throw std::runtime_error("VoxCPM2 session requires assets"); - } - return assets; -} - -void reject_enabled_denoise( - const std::unordered_map &options, - std::initializer_list keys) { - const auto match = runtime::find_option_match(options, keys); - if (match.has_value() && - runtime::parse_bool_option(match->value, match->key)) { - throw std::runtime_error( - "VoxCPM2 denoise is disabled in this implementation"); - } -} - -void reject_denoiser_option( - const std::unordered_map &options, - std::initializer_list keys) { - if (runtime::find_option_match(options, keys).has_value()) { - throw std::runtime_error( - "VoxCPM2 denoise is disabled in this implementation"); - } -} - -bool audio_buffer_equal(const runtime::AudioBuffer &lhs, - const runtime::AudioBuffer &rhs) { - return lhs.sample_rate == rhs.sample_rate && lhs.channels == rhs.channels && - lhs.samples == rhs.samples; -} - -bool optional_audio_equal(const std::optional &lhs, - const std::optional &rhs) { - if (lhs.has_value() != rhs.has_value()) { - return false; - } - return !lhs.has_value() || audio_buffer_equal(*lhs, *rhs); -} - -size_t prompt_cache_slots_from_options( - const std::unordered_map &options) { - constexpr int64_t kDefaultPromptCacheSlots = 1; - const int64_t slots = - runtime::parse_i64_option(options, {"voxcpm2.prompt_cache_slots"}) - .value_or(kDefaultPromptCacheSlots); - if (slots < 0) { - throw std::runtime_error("voxcpm2.prompt_cache_slots must be non-negative"); - } - return static_cast(slots); -} - -void validate_weight_storage(engine::assets::TensorStorageType storage_type, - const char *option_name) { - if (storage_type == engine::assets::TensorStorageType::Native || - storage_type == engine::assets::TensorStorageType::F32 || - storage_type == engine::assets::TensorStorageType::F16 || - storage_type == engine::assets::TensorStorageType::BF16 || - storage_type == engine::assets::TensorStorageType::Q8_0) { - return; - } - throw std::runtime_error(std::string(option_name) + - " supports only native, f32, f16, bf16, and q8_0"); -} - -void parse_weight_type( - const std::unordered_map &options, - const char *key, engine::assets::TensorStorageType &storage_type) { - const auto it = options.find(key); - if (it == options.end()) { - return; - } - storage_type = engine::assets::parse_tensor_storage_type(it->second); - validate_weight_storage(storage_type, key); -} - -void validate_session_options( - const std::unordered_map &options) { - for (const auto &[key, value] : options) { - (void)value; - if (key.rfind("voxcpm2.", 0) != 0) { - continue; - } - if (key == "voxcpm2.weight_context_mb" || - key == "voxcpm2.text_embedding_graph_context_mb" || - key == "voxcpm2.lm_step_graph_context_mb" || - key == "voxcpm2.projection_graph_context_mb" || - key == "voxcpm2.local_encoder_graph_context_mb" || - key == "voxcpm2.dit_graph_context_mb" || - key == "voxcpm2.audiovae_weight_context_mb" || - key == "voxcpm2.audiovae_graph_context_mb" || - key == "voxcpm2.audiovae_encoder_graph_context_mb" || - key == "voxcpm2.audiovae_latent_capacity" || - key == "voxcpm2.audiovae_encoder_sample_capacity" || - key == "voxcpm2.weight_type" || - key == "voxcpm2.audiovae_weight_type" || - key == "voxcpm2.prompt_cache_slots" || - key == "voxcpm2.mem_saver" || - key == "voxcpm2.denoise" || key == "voxcpm2.load_denoiser") { - continue; - } - throw std::runtime_error("unknown VoxCPM2 session option: " + key); - } -} - -int64_t product(const std::vector &values) { - int64_t out = 1; - for (const int64_t value : values) { - if (value <= 0) { - throw std::runtime_error("VoxCPM2 AudioVAE decoder rate is invalid"); - } - out *= value; - } - return out; -} - -} // namespace - -bool VoxCPM2SessionBase::EncodedPromptCacheKeyEqual::operator()( - const EncodedPromptCacheKey &lhs, - const EncodedPromptCacheKey &rhs) const { - return lhs.prompt_text == rhs.prompt_text && - optional_audio_equal(lhs.prompt_audio, rhs.prompt_audio) && - optional_audio_equal(lhs.reference_audio, rhs.reference_audio); -} - -VoxCPM2SessionBase::VoxCPM2SessionBase(runtime::TaskSpec task, - runtime::SessionOptions options, - std::shared_ptr assets) - : RuntimeSessionBase(options), task_(task), - assets_(require_assets(std::move(assets))), - encoded_prompt_cache_(prompt_cache_slots_from_options(options.options)) { - if (task_.mode != runtime::RunMode::Offline && - task_.mode != runtime::RunMode::Streaming) { - throw std::runtime_error( - "VoxCPM2 only supports offline and streaming sessions"); - } - if (task_.task != runtime::VoiceTaskKind::Tts) { - throw std::runtime_error("VoxCPM2 only supports the Tts task"); - } - - reject_enabled_denoise(options.options, {"voxcpm2.denoise"}); - reject_enabled_denoise(options.options, {"voxcpm2.load_denoiser"}); - reject_denoiser_option(options.options, {"voxcpm2.denoiser"}); - validate_session_options(options.options); - - generator_config_.weight_context_bytes = runtime::parse_size_mb_option( - options.options, {"voxcpm2.weight_context_mb"}, - generator_config_.weight_context_bytes); - generator_config_.text_embedding_graph_context_bytes = - runtime::parse_size_mb_option( - options.options, {"voxcpm2.text_embedding_graph_context_mb"}, - generator_config_.text_embedding_graph_context_bytes); - generator_config_.lm_step_graph_context_bytes = runtime::parse_size_mb_option( - options.options, {"voxcpm2.lm_step_graph_context_mb"}, - generator_config_.lm_step_graph_context_bytes); - generator_config_.projection_graph_context_bytes = - runtime::parse_size_mb_option( - options.options, {"voxcpm2.projection_graph_context_mb"}, - generator_config_.projection_graph_context_bytes); - generator_config_.local_encoder_graph_context_bytes = - runtime::parse_size_mb_option( - options.options, {"voxcpm2.local_encoder_graph_context_mb"}, - generator_config_.local_encoder_graph_context_bytes); - generator_config_.dit_graph_context_bytes = runtime::parse_size_mb_option( - options.options, {"voxcpm2.dit_graph_context_mb"}, - generator_config_.dit_graph_context_bytes); - generator_config_.prompt_cache_slots = encoded_prompt_cache_.capacity(); - decoder_config_.weight_context_bytes = runtime::parse_size_mb_option( - options.options, {"voxcpm2.audiovae_weight_context_mb"}, - decoder_config_.weight_context_bytes); - decoder_config_.graph_context_bytes = runtime::parse_size_mb_option( - options.options, {"voxcpm2.audiovae_graph_context_mb"}, - decoder_config_.graph_context_bytes); - decoder_config_.encoder_graph_context_bytes = runtime::parse_size_mb_option( - options.options, {"voxcpm2.audiovae_encoder_graph_context_mb"}, - decoder_config_.encoder_graph_context_bytes); - decoder_config_.latent_frame_capacity = runtime::parse_positive_i64_option( - options.options, {"voxcpm2.audiovae_latent_capacity"}, - decoder_config_.latent_frame_capacity); - decoder_config_.encoder_sample_capacity = runtime::parse_positive_i64_option( - options.options, {"voxcpm2.audiovae_encoder_sample_capacity"}, - decoder_config_.encoder_sample_capacity); - parse_weight_type(options.options, "voxcpm2.weight_type", - generator_config_.weight_storage_type); - parse_weight_type(options.options, "voxcpm2.audiovae_weight_type", - decoder_config_.weight_storage_type); - if (const auto mem_saver = - runtime::find_option(options.options, {"voxcpm2.mem_saver"})) { - generator_config_.mem_saver = - runtime::parse_bool_option(*mem_saver, "voxcpm2.mem_saver"); - } - - generator_ = std::make_unique( - assets_, execution_context(), generator_config_); - decoder_ = std::make_unique( - assets_, execution_context(), decoder_config_); -} - -VoxCPM2SessionBase::~VoxCPM2SessionBase() = default; - -std::string VoxCPM2SessionBase::family_impl() const { return "voxcpm2"; } - -runtime::VoiceTaskKind VoxCPM2SessionBase::task_kind_impl() const { return task_.task; } - -runtime::RunMode VoxCPM2SessionBase::run_mode_impl() const { return task_.mode; } - -void VoxCPM2SessionBase::prepare_impl( - const runtime::SessionPreparationRequest &request) { - (void)request; - mark_prepared(); -} - -runtime::TaskResult VoxCPM2SessionBase::run_offline_request(const runtime::TaskRequest &request) { - require_prepared("VoxCPM2 run"); - if (task_.mode != runtime::RunMode::Offline) { - throw std::runtime_error("VoxCPM2 run requires an offline session"); - } - validate_request(request); - auto release_runtime_memory = [this](VoxCPM2SessionBase *self) { - if (self != nullptr) { - self->release_request_runtime_memory(); - } - }; - std::unique_ptr - release_guard(generator_config_.mem_saver ? this : nullptr, - release_runtime_memory); - - const auto wall_start = Clock::now(); - const int64_t text_chunk_size = - engine::text::parse_text_chunk_size_override(request.options).value_or(kDefaultTextChunkSize); - const auto text_chunk_mode = - engine::text::parse_text_chunk_mode_override(request.options) - .value_or(engine::text::TextChunkMode::TagAware); - const auto chunk_requests = - runtime::chunk_text_request(request, text_chunk_size, text_chunk_mode); - const auto generation_options = generation_options_from_request(request); - const auto prompt_text = - runtime::find_option(request.options, {"voxcpm2.prompt_text", - "prompt_text", "reference_text"}) - .value_or(""); - std::optional reference_audio; - if (request.voice.has_value() && request.voice->speaker.has_value() && - request.voice->speaker->audio.has_value()) { - reference_audio = *request.voice->speaker->audio; - } - const VoxCPM2EncodedPrompt *prompt = - encoded_prompt_for_request(request.audio_input, prompt_text, - reference_audio); - - runtime::TaskResult result; - double generator_ms = 0.0; - double decoder_ms = 0.0; - runtime::AudioBuffer merged_audio; - for (const auto & chunk_request : chunk_requests) { - const auto generator_start = Clock::now(); - const auto generated = generator_->generate( - chunk_request.text_input->text, prompt, generation_options); - generator_ms += engine::debug::elapsed_ms(generator_start, Clock::now()); - - const auto decoder_start = Clock::now(); - auto audio = decoder_->decode_features(generated.decode_features, - generated.decode_patches); - if (generated.decode_trim_patches > 0) { - const int64_t trim_samples = - generated.decode_trim_patches * assets_->config.patch_size * - product(assets_->config.audio_vae.decoder_rates); - if (trim_samples > static_cast(audio.samples.size())) { - throw std::runtime_error( - "VoxCPM2 decoded continuation trim exceeds audio length"); - } - audio.samples.erase( - audio.samples.begin(), - audio.samples.begin() + static_cast(trim_samples)); - } - decoder_ms += engine::debug::elapsed_ms(decoder_start, Clock::now()); - runtime::append_audio_buffer(merged_audio, audio); - } - result.audio_output = std::move(merged_audio); - - const auto wall_end = Clock::now(); - debug::trace_log_scalar("voxcpm2.text_chunk_size", text_chunk_size); - debug::trace_log_scalar("voxcpm2.text_chunk_mode", - engine::text::text_chunk_mode_name(text_chunk_mode)); - debug::trace_log_scalar("voxcpm2.text_chunk_count", - static_cast(chunk_requests.size())); - debug::timing_log_scalar("voxcpm2.generator_ms", generator_ms); - debug::timing_log_scalar("voxcpm2.audiovae_decoder_ms", decoder_ms); - debug::timing_log_scalar("session.wall_ms", - engine::debug::elapsed_ms(wall_start, wall_end)); - return result; -} - -runtime::TaskResult -VoxCPM2SessionBase::run_streaming_request( - const runtime::TaskRequest &request, - const runtime::StreamEventCallback &stream_event_sink) { - require_prepared("VoxCPM2 run_streaming"); - if (task_.mode != runtime::RunMode::Streaming) { - throw std::runtime_error( - "VoxCPM2 run_streaming requires a streaming session"); - } - validate_request(request); - auto release_runtime_memory = [this](VoxCPM2SessionBase *self) { - if (self != nullptr) { - self->release_request_runtime_memory(); - } - }; - std::unique_ptr - release_guard(generator_config_.mem_saver ? this : nullptr, - release_runtime_memory); - - const auto wall_start = Clock::now(); - auto generation_options = generation_options_from_request(request); - const auto prompt_text = - runtime::find_option(request.options, {"voxcpm2.prompt_text", - "prompt_text", "reference_text"}) - .value_or(""); - std::optional reference_audio; - if (request.voice.has_value() && request.voice->speaker.has_value() && - request.voice->speaker->audio.has_value()) { - reference_audio = *request.voice->speaker->audio; - } - const VoxCPM2EncodedPrompt *prompt = - encoded_prompt_for_request(request.audio_input, prompt_text, - reference_audio); - - runtime::TaskResult result; - runtime::AudioBuffer merged; - merged.sample_rate = assets_->config.audio_vae.output_sample_rate; - merged.channels = 1; - double decoder_ms = 0.0; - size_t emitted_chunks = 0; - auto emit_chunk = [&](const VoxCPM2StreamingChunk &chunk) { - const auto decoder_start = Clock::now(); - auto audio = decoder_->decode_features(chunk.decode_features, - chunk.decode_patches); - decoder_ms += engine::debug::elapsed_ms(decoder_start, Clock::now()); - if (emitted_chunks == 0) { - merged.sample_rate = audio.sample_rate; - merged.channels = audio.channels; - } else if (audio.sample_rate != merged.sample_rate || - audio.channels != merged.channels) { - throw std::runtime_error( - "VoxCPM2 streaming decoder chunk format changed"); - } - merged.samples.insert(merged.samples.end(), audio.samples.begin(), - audio.samples.end()); - runtime::NamedAudioBuffer named; - named.id = "chunk_" + std::to_string(emitted_chunks); - named.audio = std::move(audio); - named.meta.insert_or_assign( - "generated_patches", std::to_string(chunk.generated_patches)); - if (stream_event_sink) { - runtime::StreamEvent event; - event.named_audio_outputs.push_back(named); - stream_event_sink(event); - } - result.named_audio_outputs.push_back(std::move(named)); - ++emitted_chunks; - }; - - const auto generator_start = Clock::now(); - (void)generator_->generate_streaming(request.text_input->text, prompt, - generation_options, emit_chunk); - const auto generator_end = Clock::now(); - const double generator_with_callbacks_ms = - engine::debug::elapsed_ms(generator_start, generator_end); - - result.audio_output = std::move(merged); - - const auto wall_end = Clock::now(); - debug::timing_log_scalar( - "voxcpm2.generator_ms", - std::max(0.0, generator_with_callbacks_ms - decoder_ms)); - debug::timing_log_scalar("voxcpm2.generator_streaming_callbacks_ms", - generator_with_callbacks_ms); - debug::timing_log_scalar("voxcpm2.audiovae_decoder_ms", decoder_ms); - debug::timing_log_scalar("voxcpm2.streaming_chunks", - static_cast(emitted_chunks)); - debug::timing_log_scalar("session.wall_ms", - engine::debug::elapsed_ms(wall_start, wall_end)); - return result; -} - -void VoxCPM2SessionBase::release_request_runtime_memory() { - if (!generator_config_.mem_saver) { - return; - } - generator_->release_runtime_memory(); - decoder_->release_runtime_memory(); -} - -const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( - const std::optional &prompt_audio, - const std::string &prompt_text, - const std::optional &reference_audio) { - if (!prompt_audio.has_value() && !reference_audio.has_value()) { - return nullptr; - } - EncodedPromptCacheKey key; - key.prompt_text = prompt_text; - key.prompt_audio = prompt_audio; - key.reference_audio = reference_audio; - if (auto *cached = encoded_prompt_cache_.find(key)) { - debug::trace_log_scalar("voxcpm2.prompt_cache.hit", 1); - debug::trace_log_scalar("voxcpm2.prompt_cache.slots", - static_cast( - encoded_prompt_cache_.capacity())); - debug::trace_log_scalar("voxcpm2.prompt_cache.entries", - static_cast(encoded_prompt_cache_.size())); - debug::trace_log_scalar("voxcpm2.prompt_cache.evicted", 0); - debug::timing_log_scalar("voxcpm2.prompt_encode_ms", 0.0); - return &cached->encoded; - } - - const auto encode_start = Clock::now(); - EncodedPromptCacheEntry entry; - entry.encoded = - decoder_->encode_prompt_audio(prompt_audio, prompt_text, reference_audio); - const double encode_ms = engine::debug::elapsed_ms(encode_start); - if (encoded_prompt_cache_.capacity() == 0) { - uncached_encoded_prompt_ = std::move(entry); - debug::trace_log_scalar("voxcpm2.prompt_cache.hit", 0); - debug::trace_log_scalar("voxcpm2.prompt_cache.slots", 0); - debug::trace_log_scalar("voxcpm2.prompt_cache.entries", 0); - debug::trace_log_scalar("voxcpm2.prompt_cache.evicted", 0); - debug::timing_log_scalar("voxcpm2.prompt_encode_ms", encode_ms); - return &uncached_encoded_prompt_->encoded; - } - const bool will_evict = - encoded_prompt_cache_.size() >= encoded_prompt_cache_.capacity(); - encoded_prompt_cache_.put(std::move(key), std::move(entry)); - EncodedPromptCacheKey lookup; - lookup.prompt_text = prompt_text; - lookup.prompt_audio = prompt_audio; - lookup.reference_audio = reference_audio; - auto *cached = encoded_prompt_cache_.find(lookup); - if (cached == nullptr) { - throw std::runtime_error("VoxCPM2 prompt cache insert failed"); - } - debug::trace_log_scalar("voxcpm2.prompt_cache.hit", 0); - debug::trace_log_scalar("voxcpm2.prompt_cache.slots", - static_cast( - encoded_prompt_cache_.capacity())); - debug::trace_log_scalar("voxcpm2.prompt_cache.entries", - static_cast(encoded_prompt_cache_.size())); - debug::trace_log_scalar("voxcpm2.prompt_cache.evicted", will_evict ? 1 : 0); - debug::timing_log_scalar("voxcpm2.prompt_encode_ms", - encode_ms); - return &cached->encoded; -} - -VoxCPM2GenerationOptions VoxCPM2SessionBase::generation_options_from_request( - const runtime::TaskRequest &request) const { - VoxCPM2GenerationOptions options; - bool min_tokens_explicit = false; - if (const auto value = runtime::parse_i64_option( - request.options, {"voxcpm2.min_tokens", "min_tokens"})) { - options.min_tokens = *value; - min_tokens_explicit = true; - } - // Set V1-specific default min_tokens if not explicitly provided - if (!min_tokens_explicit && assets_->config.v1) { - // VoxCPM1 (0.5B) has patch_size=2, VoxCPM1.5 (1.5B) has patch_size=4 - // Use model-specific defaults for optimal speech speed - if (assets_->config.patch_size == 2) { - options.min_tokens = 15; // 20, VoxCPM1 0.5B - } else if (assets_->config.patch_size == 4) { - options.min_tokens = 12; // VoxCPM1.5 1.5B - } else { - options.min_tokens = 15; // Other V1 models - } - } - if (const auto value = runtime::parse_i64_option( - request.options, {"max_tokens", "voxcpm2.max_tokens"})) { - options.max_tokens = *value; - } - if (const auto value = runtime::parse_i64_option( - request.options, - {"num_inference_steps", "voxcpm2.num_inference_steps"})) { - options.num_inference_steps = *value; - } - if (const auto value = runtime::parse_finite_float_option( - request.options, {"guidance_scale", "voxcpm2.guidance_scale"})) { - options.guidance_scale = *value; - } - if (const auto match = runtime::find_option_match( - request.options, {"voxcpm2.retry_badcase", "retry_badcase"})) { - options.retry_badcase = - runtime::parse_bool_option(match->value, match->key); - } - if (const auto value = runtime::parse_i64_option( - request.options, - {"voxcpm2.retry_badcase_max_times", "retry_badcase_max_times"})) { - options.retry_badcase_max_times = *value; - } - if (const auto value = runtime::parse_finite_float_option( - request.options, {"voxcpm2.retry_badcase_ratio_threshold", - "retry_badcase_ratio_threshold"})) { - options.retry_badcase_ratio_threshold = *value; - } - if (const auto value = runtime::parse_u32_option(request.options, - {"voxcpm2.seed", "seed"})) { - options.seed = *value; - } - options.cfm_noise_file = - runtime::find_option(request.options, - {"voxcpm2.cfm_noise_file", "cfm_noise_file"}) - .value_or(""); - if (options.min_tokens < 0) { - throw std::runtime_error("VoxCPM2 min_tokens must be non-negative"); - } - if (options.max_tokens < 0) { - throw std::runtime_error("VoxCPM2 max_tokens must be non-negative"); - } - if (options.max_tokens == 0) { - options.max_tokens = assets_->config.max_length; - } - if (options.min_tokens > options.max_tokens) { - throw std::runtime_error("VoxCPM2 min_tokens must not exceed max_tokens"); - } - if (options.max_tokens > assets_->config.max_length) { - throw std::runtime_error( - "VoxCPM2 max_tokens exceeds model config max_length"); - } - if (options.num_inference_steps <= 0) { - throw std::runtime_error( - "VoxCPM2 num_inference_steps must be positive"); - } - if (options.guidance_scale < 0.0F) { - throw std::runtime_error("VoxCPM2 guidance_scale must be non-negative"); - } - if (options.retry_badcase_max_times <= 0) { - throw std::runtime_error( - "VoxCPM2 retry_badcase_max_times must be positive"); - } - if (options.retry_badcase_ratio_threshold <= 0.0F) { - throw std::runtime_error( - "VoxCPM2 retry_badcase_ratio_threshold must be positive"); - } - reject_enabled_denoise(request.options, {"voxcpm2.denoise", "denoise"}); - reject_enabled_denoise(request.options, - {"voxcpm2.load_denoiser", "load_denoiser"}); - reject_denoiser_option(request.options, {"voxcpm2.denoiser", "denoiser"}); - return options; -} - -void VoxCPM2SessionBase::validate_request( - const runtime::TaskRequest &request) const { - if (!request.text_input.has_value()) { - throw std::runtime_error("VoxCPM2 requires text input"); - } - if (request.text_input->text.empty()) { - throw std::runtime_error("VoxCPM2 text input must not be empty"); - } - if (request.voice.has_value()) { - if (request.voice->style.has_value()) { - throw std::runtime_error( - "VoxCPM2 C++ session does not consume style conditions"); - } - if (request.voice->speaker.has_value()) { - const auto &speaker = *request.voice->speaker; - if (speaker.cached_voice_id.has_value()) { - throw std::runtime_error("VoxCPM2 C++ session requires speaker " - "reference audio, not a cached voice id"); - } - if (!speaker.audio.has_value()) { - throw std::runtime_error( - "VoxCPM2 C++ session speaker condition requires audio"); - } - } - } - if (!request.input_artifacts.empty()) { - throw std::runtime_error( - "VoxCPM2 C++ session does not consume input artifacts"); - } -} - -VoxCPM2OfflineSession::VoxCPM2OfflineSession( - runtime::TaskSpec task, - runtime::SessionOptions options, - std::shared_ptr assets) - : VoxCPM2SessionBase(task, std::move(options), std::move(assets)) {} - -std::string VoxCPM2OfflineSession::family() const { return family_impl(); } - -runtime::VoiceTaskKind VoxCPM2OfflineSession::task_kind() const { - return task_kind_impl(); -} - -runtime::RunMode VoxCPM2OfflineSession::run_mode() const { - return run_mode_impl(); -} - -void VoxCPM2OfflineSession::prepare( - const runtime::SessionPreparationRequest &request) { - prepare_impl(request); -} - -runtime::TaskResult -VoxCPM2OfflineSession::run(const runtime::TaskRequest &request) { - return run_offline_request(request); -} - -VoxCPM2StreamingSession::VoxCPM2StreamingSession( - runtime::TaskSpec task, - runtime::SessionOptions options, - std::shared_ptr assets) - : VoxCPM2SessionBase(task, std::move(options), std::move(assets)) {} - -std::string VoxCPM2StreamingSession::family() const { return family_impl(); } - -runtime::VoiceTaskKind VoxCPM2StreamingSession::task_kind() const { - return task_kind_impl(); -} - -runtime::RunMode VoxCPM2StreamingSession::run_mode() const { - return run_mode_impl(); -} - -void VoxCPM2StreamingSession::prepare( - const runtime::SessionPreparationRequest &request) { - prepare_impl(request); -} - -runtime::StreamingPolicy VoxCPM2StreamingSession::streaming_policy() const { - runtime::StreamingPolicy policy; - policy.input = runtime::StreamingInputKind::None; - policy.output = runtime::StreamingOutputKind::FinalResult; - return policy; -} - -void VoxCPM2StreamingSession::start_stream(const runtime::TaskRequest &request) { - reset(); - result_ = run_streaming_request(request, stream_event_sink_); - started_ = true; -} - -void VoxCPM2StreamingSession::set_stream_event_sink(runtime::StreamEventCallback sink) { - stream_event_sink_ = std::move(sink); -} - -std::optional VoxCPM2StreamingSession::next_stream_event() { - if (!started_) { - throw std::runtime_error("VoxCPM2 streaming has not been started"); - } - if (next_chunk_index_ >= result_.named_audio_outputs.size()) { - return std::nullopt; - } - const auto & named = result_.named_audio_outputs[next_chunk_index_++]; - runtime::StreamEvent event; - event.named_audio_outputs.push_back(named); - return event; -} - -runtime::TaskResult VoxCPM2StreamingSession::finish_stream() { - if (!started_) { - throw std::runtime_error("VoxCPM2 streaming has not been started"); - } - started_ = false; - next_chunk_index_ = 0; - return std::move(result_); -} - -void VoxCPM2StreamingSession::reset() { - result_ = runtime::TaskResult{}; - next_chunk_index_ = 0; - started_ = false; -} - -runtime::StreamEvent VoxCPM2StreamingSession::process_audio_chunk( - const runtime::AudioChunk &chunk) { - (void)chunk; - throw std::runtime_error("VoxCPM2 streaming does not consume audio chunks"); -} - -runtime::TaskResult VoxCPM2StreamingSession::finalize() { - return finish_stream(); -} - -} // namespace engine::models::voxcpm2 From 03e18ca39c5e378985fd5c5c37f0b1b238a3c4d4 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Thu, 20 Aug 2026 12:47:32 +0200 Subject: [PATCH 17/23] add missing file --- src/models/voxcpm2/assets.cpp | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/models/voxcpm2/assets.cpp b/src/models/voxcpm2/assets.cpp index a68833ae..08b98c07 100644 --- a/src/models/voxcpm2/assets.cpp +++ b/src/models/voxcpm2/assets.cpp @@ -235,11 +235,6 @@ class TransformingTensorSource final : public assets::TensorSource { return false; } - [[nodiscard]] bool is_synthesized(std::string_view name) const noexcept override { - const std::string key{std::string(name)}; - return synthesized_tensors_.find(key) != synthesized_tensors_.end(); - } - assets::TensorMetadata require_metadata(std::string_view name) const override { const auto it = synthesized_tensors_.find(std::string(name)); if (it != synthesized_tensors_.end()) { From a252b6de6b6746b465ba3612c0f10e70c88d4a89 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Thu, 20 Aug 2026 14:57:40 +0200 Subject: [PATCH 18/23] fix(voxcpm1/2): remove strange stream blocking setting. --- src/models/voxcpm2/generator.cpp | 5 +++-- src/models/voxcpm2/session.cpp | 11 +++++++++++ 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/src/models/voxcpm2/generator.cpp b/src/models/voxcpm2/generator.cpp index 607cb2d6..7d74924b 100644 --- a/src/models/voxcpm2/generator.cpp +++ b/src/models/voxcpm2/generator.cpp @@ -1560,8 +1560,9 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { &chunk_callback) { validate_generation_options(options); if (options.retry_badcase) { - throw std::runtime_error( - "VoxCPM2 streaming generation requires retry_badcase=false"); + fprintf(stderr, + "[VoxCPM2] warning: retry_badcase ignored in streaming " + "generation\n"); } const auto prefill = build_prefill_sequence(text, prompt); const int64_t max_tokens = diff --git a/src/models/voxcpm2/session.cpp b/src/models/voxcpm2/session.cpp index 45072ba2..021ae765 100644 --- a/src/models/voxcpm2/session.cpp +++ b/src/models/voxcpm2/session.cpp @@ -575,12 +575,23 @@ VoxCPM2GenerationOptions VoxCPM2SessionBase::generation_options_from_request( "voxcpm1.guidance_scale"})) { options.guidance_scale = *value; } + bool retry_badcase_explicit = false; if (const auto match = runtime::find_option_match( request.options, {"voxcpm2.retry_badcase", "voxcpm1.retry_badcase", "retry_badcase"})) { options.retry_badcase = runtime::parse_bool_option(match->value, match->key); + retry_badcase_explicit = true; + } + // Streaming emits decoded chunks to the client in real time, so a bad-case + // retry (regenerate from scratch, discard earlier output) is impossible by + // construction. The struct default retry_badcase=true exists for the + // offline path; it must not leak into the streaming path and block every + // streaming request. Relax it to false unless the caller explicitly asked + // for retry, which the generator accepts and warns about. + if (!retry_badcase_explicit && task_.mode == runtime::RunMode::Streaming) { + options.retry_badcase = false; } if (const auto value = runtime::parse_i64_option( request.options, From 44d38be6cf13604e871c5dcffe3a8cc753ff48f9 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Thu, 20 Aug 2026 15:14:51 +0000 Subject: [PATCH 19/23] feat(webui): expose VoxCPM1 1.5B variants in catalog --- webui/configs/models_catalog.json | 2 ++ 1 file changed, 2 insertions(+) diff --git a/webui/configs/models_catalog.json b/webui/configs/models_catalog.json index 3a0d939f..9d8784ae 100644 --- a/webui/configs/models_catalog.json +++ b/webui/configs/models_catalog.json @@ -17,6 +17,8 @@ { "id": "miotts", "display_name": "MioTTS 1.7B (tts; needs MioCodec)", "family": "miotts", "path": "models/MioTTS-1.7B", "task": "tts", "mode": "offline", "download_id": "miotts_1_7b", "min_vram_gb": 8 }, { "id": "voxcpm2", "display_name": "VoxCPM2 (tts)", "family": "voxcpm2", "path": "models/VoxCPM2", "task": "tts", "mode": "offline", "download_id": "voxcpm2", "session_options": { "voxcpm2.weight_type": "q8_0" }, "min_vram_gb": 6 }, { "id": "voxcpm1", "display_name": "VoxCPM1 0.5B (tts + clone)", "family": "voxcpm1", "path": "models/VoxCPM1-GGUF", "task": "tts", "mode": "offline", "download_id": "voxcpm1_0.5b_q8_0", "min_vram_gb": 4 }, + { "id": "voxcpm1_1_5b_q8_0", "display_name": "VoxCPM1 1.5B Q8_0 (tts + clone)", "family": "voxcpm1", "path": "models/VoxCPM1.5-GGUF", "task": "tts", "mode": "offline", "download_id": "voxcpm1_1.5b_q8_0", "min_vram_gb": 8 }, + { "id": "voxcpm1_1_5b_q4_k", "display_name": "VoxCPM1 1.5B Q4_K (tts + clone)", "family": "voxcpm1", "path": "models/VoxCPM1.5-GGUF", "task": "tts", "mode": "offline", "download_id": "voxcpm1_1.5b_q4_k", "min_vram_gb": 6 }, { "id": "vibevoice", "display_name": "VibeVoice 1.5B (tts, long-form/multi-speaker)", "family": "vibevoice", "path": "models/VibeVoice-1.5B", "task": "tts", "mode": "offline", "download_id": "vibevoice_1_5b", "min_vram_gb": 7 }, { "id": "index-tts2", "display_name": "IndexTTS2 (tts 中英克隆+情感)", "display_name_en": "IndexTTS2 (tts, zh/en clone + emotion)", "family": "index_tts2", "path": "models/IndexTTS-2", "task": "tts", "mode": "offline", "download_id": "index_tts2", "min_vram_gb": 8 }, { "id": "index-tts2.5", "display_name": "IndexTTS2.5 (tts 多语种克隆+情感, GGUF Q8)", "display_name_en": "IndexTTS2.5 (tts, zh/en/ja/es/ar clone + emotion, GGUF Q8)", "family": "index_tts2", "path": "models/IndexTTS2.5-GGUF", "task": "tts", "mode": "offline", "download_id": "index_tts2_5_q8_0", "min_vram_gb": 8, From 0d23e6cf958333d37f538837bb76b5374c40b978 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Thu, 20 Aug 2026 15:28:19 +0000 Subject: [PATCH 20/23] fix(voxcpm1): read CFM config keys from GGUF with golden defaults --- include/engine/models/voxcpm2/gguf_metadata.h | 1 + src/models/voxcpm2/config_gguf.cpp | 12 ++++++++---- src/models/voxcpm2/gguf_metadata.cpp | 11 +++++++++++ 3 files changed, 20 insertions(+), 4 deletions(-) diff --git a/include/engine/models/voxcpm2/gguf_metadata.h b/include/engine/models/voxcpm2/gguf_metadata.h index 0f538f36..d7ff10c5 100644 --- a/include/engine/models/voxcpm2/gguf_metadata.h +++ b/include/engine/models/voxcpm2/gguf_metadata.h @@ -31,6 +31,7 @@ class GgufMetadataReader { [[nodiscard]] std::optional optional_string(std::string_view key) const; [[nodiscard]] std::optional optional_u32(std::string_view key) const; + [[nodiscard]] std::optional optional_f32(std::string_view key) const; [[nodiscard]] std::optional> optional_string_array(std::string_view key) const; [[nodiscard]] std::optional> optional_i32_array(std::string_view key) const; [[nodiscard]] std::optional> optional_f32_array(std::string_view key) const; diff --git a/src/models/voxcpm2/config_gguf.cpp b/src/models/voxcpm2/config_gguf.cpp index 794313e2..830b86ee 100644 --- a/src/models/voxcpm2/config_gguf.cpp +++ b/src/models/voxcpm2/config_gguf.cpp @@ -139,10 +139,14 @@ VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & config.dit.num_layers = get_optional_i64("voxcpm_dit_config_num_layers").value_or(4); config.dit.kv_channels = get_optional_i64("voxcpm_dit_config_kv_channels").value_or(config.dit.hidden_dim / config.dit.num_heads); config.dit.mean_mode = get_optional_bool("voxcpm_dit_config_mean_mode").value_or(false); - config.dit.cfm.sigma_min = 1e-4f; // Default - config.dit.cfm.solver = "euler"; - config.dit.cfm.t_scheduler = "log-norm"; - config.dit.cfm.inference_cfg_rate = 0.5f; // Default + config.dit.cfm.sigma_min = + metadata.optional_f32("voxcpm_dit_config_cfm_config_sigma_min").value_or(1.0e-6F); + config.dit.cfm.solver = + metadata.optional_string("voxcpm_dit_config_cfm_config_solver").value_or("euler"); + config.dit.cfm.t_scheduler = + metadata.optional_string("voxcpm_dit_config_cfm_config_t_scheduler").value_or("log-norm"); + config.dit.cfm.inference_cfg_rate = + metadata.optional_f32("voxcpm_dit_config_cfm_config_inference_cfg_rate").value_or(2.0F); // Audio VAE config config.audio_vae.encoder_dim = get_optional_i64("voxcpm_audio_vae_config_encoder_dim").value_or(64); diff --git a/src/models/voxcpm2/gguf_metadata.cpp b/src/models/voxcpm2/gguf_metadata.cpp index 13981512..da9ddcac 100644 --- a/src/models/voxcpm2/gguf_metadata.cpp +++ b/src/models/voxcpm2/gguf_metadata.cpp @@ -51,6 +51,17 @@ std::optional GgufMetadataReader::optional_u32(std::string_view key) c return gguf_get_val_u32(gguf_, idx); } +std::optional GgufMetadataReader::optional_f32(std::string_view key) const { + if (gguf_ == nullptr) { + return std::nullopt; + } + const int64_t idx = gguf_find_key(gguf_, std::string(key).c_str()); + if (idx < 0 || gguf_get_kv_type(gguf_, idx) == GGUF_TYPE_ARRAY) { + return std::nullopt; + } + return gguf_get_val_f32(gguf_, idx); +} + std::optional> GgufMetadataReader::optional_string_array(std::string_view key) const { if (gguf_ == nullptr) { return std::nullopt; From 28915eb4409855ae9110901bbcb21b53f5809047 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Thu, 20 Aug 2026 15:34:44 +0000 Subject: [PATCH 21/23] feat(tools): add VoxCPM1 1.5B q8_0+q4_k cli path-test cases --- .../audiocpp_cli/audiocpp_cli_path_cases.json | 110 ++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/tools/audiocpp_cli/audiocpp_cli_path_cases.json b/tools/audiocpp_cli/audiocpp_cli_path_cases.json index f522a666..c1bc3293 100644 --- a/tools/audiocpp_cli/audiocpp_cli_path_cases.json +++ b/tools/audiocpp_cli/audiocpp_cli_path_cases.json @@ -805,6 +805,116 @@ } ] }, + { + "id": "voxcpm1_1.5b_q4k_load", + "coverage": "VoxCPM1 1.5B q4_k GGUF load and minimal offline tts sanity", + "family": "voxcpm1", + "model": "models/VoxCPM1.5-GGUF/voxcpm1.5-q4_k-audiovae-f16.gguf", + "task": "tts", + "mode": "offline", + "outputs": [ + "audio" + ], + "requests": [ + { + "id": "q4k_load", + "text": "This minimal q4k test loads the VoxCPM1 1.5B quantized variant.", + "seed": 1234, + "max_tokens": 160, + "guidance_scale": 2.0, + "num_inference_steps": 10 + } + ] + }, + { + "id": "voxcpm1_1.5b_q4k_tts", + "coverage": "VoxCPM1 1.5B q4_k text-to-speech path with MiniCPM generation, diffusion feature generation, and AudioVAE decode", + "family": "voxcpm1", + "model": "models/VoxCPM1.5-GGUF/voxcpm1.5-q4_k-audiovae-f16.gguf", + "task": "tts", + "mode": "offline", + "outputs": [ + "audio" + ], + "requests": [ + { + "id": "q4k_tts", + "text": "This VoxCPM1 path test checks the text-to-speech interface through AudioCPP CLI.", + "seed": 1234, + "max_tokens": 160, + "guidance_scale": 2.0, + "num_inference_steps": 10 + } + ] + }, + { + "id": "voxcpm1_1.5b_tts", + "coverage": "VoxCPM1 1.5B text-to-speech path with MiniCPM generation, diffusion feature generation, and AudioVAE decode", + "family": "voxcpm1", + "model": "models/VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf", + "task": "tts", + "mode": "offline", + "outputs": [ + "audio" + ], + "requests": [ + { + "id": "tts", + "text": "This VoxCPM1 path test checks the text-to-speech interface through AudioCPP CLI.", + "seed": 1234, + "max_tokens": 160, + "guidance_scale": 2.0, + "num_inference_steps": 10 + } + ] + }, + { + "id": "voxcpm1_1.5b_voice_clone", + "coverage": "VoxCPM1 1.5B voice clone path with reference audio encoding, MiniCPM generation, diffusion feature generation, and AudioVAE decode", + "family": "voxcpm1", + "model": "models/VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf", + "task": "tts", + "mode": "offline", + "outputs": [ + "audio" + ], + "requests": [ + { + "id": "clone", + "text": "This VoxCPM1 path test clones the reference speaker for a short review sentence.", + "voice_ref": "resources/sample.wav", + "seed": 1234, + "max_tokens": 160, + "guidance_scale": 2.0, + "num_inference_steps": 10 + } + ] + }, + { + "id": "voxcpm1_1.5b_streaming_tts", + "coverage": "VoxCPM1 1.5B streaming text-to-speech path with MiniCPM streaming generation, diffusion feature generation, and AudioVAE chunk decode", + "family": "voxcpm1", + "model": "models/VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf", + "task": "tts", + "mode": "streaming", + "chunk_size": 512, + "outputs": [ + "audio" + ], + "requests": [ + { + "id": "streaming_tts", + "text": "This VoxCPM1 streaming path test checks that the CLI can emit audio chunks for a longer request while preserving a steady speaking style.", + "seed": 1234, + "max_tokens": 160, + "guidance_scale": 2.0, + "num_inference_steps": 10, + "options": { + "retry_badcase": false + } + } + ] + }, { "id": "higgs_audio_tts_voice_clone_chunked", "coverage": "Higgs Audio v3 voice clone path with framework text chunking, AR generation, and codec decode", From 75152ffdf244b28d92b5b3aca6b51d1907dd5c33 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Fri, 21 Aug 2026 09:49:41 +0000 Subject: [PATCH 22/23] feat(voxcpm1): add VoxCPM1 community model family and fix V1 weight loading Port VoxCPM1 (tokenizer-free 0.5B TTS) as a community model under community_models/voxcpm1, reusing the VoxCPM2 / Local-DiT / CFM stack. Fix V1 GGUF loading: - embed_tokens weight transpose to [vocab, hidden] for ggml get_rows - correct fusion-projection handling so true V1 models do not use the synthesized Xavier weight as a real fusion weight - distinguish synthesized vs loaded tensors (is_synthesized) Refactor shared voxcpm2 components accordingly. Updates CMake registration, model_specs/voxcpm1.json, webui catalog entries, and cli path-test cases. Verified via STT: VoxCPM1/VoxCPM2 TTS and voice-clone generate content-correct speech; VoxCPM2 retains 48kHz output. --- CMakeLists.txt | 26 +- docs/tts.md | 10 +- .../engine/community_models/voxcpm1/assets.h | 102 + .../community_models/voxcpm1/audiovae.h | 53 + .../voxcpm1}/config_gguf.h | 8 +- .../community_models/voxcpm1/generator.h | 59 + .../voxcpm1}/gguf_metadata.h | 4 +- .../engine/community_models/voxcpm1/minicpm.h | 180 ++ .../engine/community_models/voxcpm1/session.h | 111 + .../voxcpm1}/tokenizer_gguf.h | 10 +- .../community_models/voxcpm1/tokenizer_text.h | 32 + .../voxcpm1}/tokenizer_wrapper.h | 44 +- .../engine/community_models/voxcpm1/types.h | 69 + include/engine/models/voxcpm2/assets.h | 5 +- include/engine/models/voxcpm2/audiovae.h | 1 - include/engine/models/voxcpm2/generator.h | 1 - include/engine/models/voxcpm2/loader.h | 1 - include/engine/models/voxcpm2/minicpm.h | 2 - .../engine/models/voxcpm2/tokenizer_text.h | 6 +- model_specs/voxcpm1.json | 214 +- src/community_models/voxcpm1/assets.cpp | 933 ++++++++ src/community_models/voxcpm1/audiovae.cpp | 1068 +++++++++ .../voxcpm1}/config_gguf.cpp | 12 +- src/community_models/voxcpm1/generator.cpp | 2048 +++++++++++++++++ .../voxcpm1}/gguf_metadata.cpp | 6 +- src/community_models/voxcpm1/minicpm.cpp | 1268 ++++++++++ src/community_models/voxcpm1/minicpm_blocks.h | 272 +++ src/community_models/voxcpm1/session.cpp | 938 ++++++++ .../voxcpm1}/tokenizer_gguf.cpp | 20 +- .../voxcpm1/tokenizer_text.cpp | 353 +++ src/models/voxcpm2/assets.cpp | 747 +----- src/models/voxcpm2/audiovae.cpp | 142 +- src/models/voxcpm2/generator.cpp | 524 +---- src/models/voxcpm2/loader.cpp | 122 +- src/models/voxcpm2/minicpm.cpp | 54 +- src/models/voxcpm2/minicpm_blocks.h | 16 +- src/models/voxcpm2/session.cpp | 168 +- src/models/voxcpm2/tokenizer_text.cpp | 1 - .../audiocpp_cli/audiocpp_cli_path_cases.json | 110 - webui/configs/models_catalog.json | 2 - 40 files changed, 7887 insertions(+), 1855 deletions(-) create mode 100644 include/engine/community_models/voxcpm1/assets.h create mode 100644 include/engine/community_models/voxcpm1/audiovae.h rename include/engine/{models/voxcpm2 => community_models/voxcpm1}/config_gguf.h (60%) create mode 100644 include/engine/community_models/voxcpm1/generator.h rename include/engine/{models/voxcpm2 => community_models/voxcpm1}/gguf_metadata.h (95%) create mode 100644 include/engine/community_models/voxcpm1/minicpm.h create mode 100644 include/engine/community_models/voxcpm1/session.h rename include/engine/{models/voxcpm2 => community_models/voxcpm1}/tokenizer_gguf.h (80%) create mode 100644 include/engine/community_models/voxcpm1/tokenizer_text.h rename include/engine/{models/voxcpm2 => community_models/voxcpm1}/tokenizer_wrapper.h (57%) create mode 100644 include/engine/community_models/voxcpm1/types.h create mode 100644 src/community_models/voxcpm1/assets.cpp create mode 100644 src/community_models/voxcpm1/audiovae.cpp rename src/{models/voxcpm2 => community_models/voxcpm1}/config_gguf.cpp (96%) create mode 100644 src/community_models/voxcpm1/generator.cpp rename src/{models/voxcpm2 => community_models/voxcpm1}/gguf_metadata.cpp (96%) create mode 100644 src/community_models/voxcpm1/minicpm.cpp create mode 100644 src/community_models/voxcpm1/minicpm_blocks.h create mode 100644 src/community_models/voxcpm1/session.cpp rename src/{models/voxcpm2 => community_models/voxcpm1}/tokenizer_gguf.cpp (95%) create mode 100644 src/community_models/voxcpm1/tokenizer_text.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index fcbb3df6..2ab9c686 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -734,13 +734,10 @@ audiocpp_add_model(voxcpm2 SOURCES src/models/voxcpm2/assets.cpp src/models/voxcpm2/audiovae.cpp - src/models/voxcpm2/config_gguf.cpp src/models/voxcpm2/generator.cpp - src/models/voxcpm2/gguf_metadata.cpp src/models/voxcpm2/loader.cpp src/models/voxcpm2/minicpm.cpp src/models/voxcpm2/session.cpp - src/models/voxcpm2/tokenizer_gguf.cpp src/models/voxcpm2/tokenizer_text.cpp INCLUDES engine/models/voxcpm2/loader.h @@ -750,20 +747,19 @@ audiocpp_add_model(voxcpm2 audiocpp_add_model(voxcpm1 SOURCES - src/models/voxcpm2/assets.cpp - src/models/voxcpm2/audiovae.cpp - src/models/voxcpm2/config_gguf.cpp - src/models/voxcpm2/generator.cpp - src/models/voxcpm2/gguf_metadata.cpp - src/models/voxcpm2/loader.cpp - src/models/voxcpm2/minicpm.cpp - src/models/voxcpm2/session.cpp - src/models/voxcpm2/tokenizer_gguf.cpp - src/models/voxcpm2/tokenizer_text.cpp + src/community_models/voxcpm1/assets.cpp + src/community_models/voxcpm1/audiovae.cpp + src/community_models/voxcpm1/config_gguf.cpp + src/community_models/voxcpm1/generator.cpp + src/community_models/voxcpm1/gguf_metadata.cpp + src/community_models/voxcpm1/minicpm.cpp + src/community_models/voxcpm1/session.cpp + src/community_models/voxcpm1/tokenizer_gguf.cpp + src/community_models/voxcpm1/tokenizer_text.cpp INCLUDES - engine/models/voxcpm2/loader.h + engine/community_models/voxcpm1/session.h LOADERS - engine::models::voxcpm2::make_voxcpm1_loader + engine::community_models::voxcpm1::make_voxcpm1_loader ) audiocpp_add_model(vibevoice diff --git a/docs/tts.md b/docs/tts.md index 0a2bb296..4a7c9ff6 100644 --- a/docs/tts.md +++ b/docs/tts.md @@ -425,12 +425,12 @@ audiocpp_cli --task tts --family pocket_tts --model models/pocket-tts --backend ## VoxCPM1 -VoxCPM1 supports offline and streaming TTS plus short-reference voice cloning. It reuses the VoxCPM2 runtime tree with a GGUF tensor-adaptation layer that understands the OpenBMB folded AudioVAE weights, so the same `--family voxcpm1` path serves both the 16 kHz 0.5B model and the 44.1 kHz 1.5B variants. +VoxCPM1 supports offline and streaming TTS plus short-reference voice cloning. It reuses the VoxCPM2 runtime tree with a GGUF tensor-adaptation layer that understands the OpenBMB folded AudioVAE weights. The registered package is the 16 kHz 0.5B model; the runtime is size-agnostic, so a different VoxCPM1 GGUF can still be loaded via an explicit `--model `. | Field | Value | |---|---| | Family | `voxcpm1` | -| Model directory | `models/VoxCPM1-GGUF` (0.5B), `models/VoxCPM1.5-GGUF` (1.5B) | +| Model directory | `models/VoxCPM1-GGUF` (0.5B) | | Task | `tts` | | Modes | `offline`, `streaming` | | Languages | Model auto-handles supported languages | @@ -443,12 +443,6 @@ Text to speech: audiocpp_cli --task tts --family voxcpm1 --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf --backend cpu --text "Hello from VoxCPM1." --out out.wav ``` -1.5B variant (44.1 kHz output): - -```bash -audiocpp_cli --task tts --family voxcpm1 --model models/VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf --backend cpu --text "Hello from VoxCPM1." --out out.wav -``` - Voice clone: ```bash diff --git a/include/engine/community_models/voxcpm1/assets.h b/include/engine/community_models/voxcpm1/assets.h new file mode 100644 index 00000000..cf837a8d --- /dev/null +++ b/include/engine/community_models/voxcpm1/assets.h @@ -0,0 +1,102 @@ +#pragma once + +#include "engine/framework/assets/resource_bundle.h" +#include "engine/framework/assets/tensor_source.h" +#include "engine/community_models/voxcpm1/tokenizer_gguf.h" + +#include +#include +#include +#include +#include + +namespace engine::community_models::voxcpm1 { + +struct VoxCPM1RopeScalingConfig { + std::string type; + std::vector long_factor; + std::vector short_factor; + int64_t original_max_position_embeddings = 0; +}; + +struct VoxCPM1MiniCPMConfig { + int64_t bos_token_id = 1; + int64_t eos_token_id = 2; + int64_t hidden_size = 0; + int64_t intermediate_size = 0; + int64_t max_position_embeddings = 0; + int64_t num_attention_heads = 0; + int64_t num_hidden_layers = 0; + int64_t num_key_value_heads = 0; + int64_t kv_channels = 0; + int64_t vocab_size = 0; + int64_t scale_emb = 1; + int64_t dim_model_base = 0; + float rms_norm_eps = 1.0e-5F; + float rope_theta = 10000.0F; + float scale_depth = 1.0F; + bool use_mup = false; + bool no_rope = false; + VoxCPM1RopeScalingConfig rope_scaling; +}; + +struct VoxCPM1LocalTransformerConfig { + int64_t hidden_dim = 0; + int64_t ffn_dim = 0; + int64_t num_heads = 0; + int64_t num_layers = 0; + int64_t kv_channels = 0; +}; + +struct VoxCPM1CFMConfig { + float sigma_min = 1.0e-6F; + std::string solver = "euler"; + std::string t_scheduler = "log-norm"; + float inference_cfg_rate = 2.0F; +}; + +struct VoxCPM1DiTConfig : VoxCPM1LocalTransformerConfig { + bool mean_mode = false; + VoxCPM1CFMConfig cfm; +}; + +struct VoxCPM1AudioVAEConfig { + int64_t encoder_dim = 0; + std::vector encoder_rates; + int64_t latent_dim = 0; + int64_t decoder_dim = 0; + std::vector decoder_rates; + std::vector sample_rate_bin_boundaries; + int sample_rate = 0; + int output_sample_rate = 0; +}; + +struct VoxCPM1Config { + std::string architecture; + VoxCPM1MiniCPMConfig lm; + int64_t patch_size = 4; + int64_t feat_dim = 64; + int64_t residual_lm_num_layers = 8; + bool residual_lm_no_rope = false; + int64_t scalar_quantization_latent_dim = 512; + int64_t scalar_quantization_scale = 9; + VoxCPM1LocalTransformerConfig encoder; + VoxCPM1DiTConfig dit; + VoxCPM1AudioVAEConfig audio_vae; + int64_t max_length = 8192; + std::string device = "cuda"; + std::string dtype = "bfloat16"; + bool v1 = false; +}; + +struct VoxCPM1Assets { + assets::ResourceBundle resources; + VoxCPM1Config config; + std::shared_ptr model_weights; + std::shared_ptr audiovae_weights; + std::shared_ptr gguf_tokenizer; +}; + +std::shared_ptr load_voxcpm1_assets(const std::filesystem::path & model_path); + +} // namespace engine::community_models::voxcpm1 diff --git a/include/engine/community_models/voxcpm1/audiovae.h b/include/engine/community_models/voxcpm1/audiovae.h new file mode 100644 index 00000000..2bf1c321 --- /dev/null +++ b/include/engine/community_models/voxcpm1/audiovae.h @@ -0,0 +1,53 @@ +#pragma once + +#include "engine/framework/core/backend.h" +#include "engine/framework/runtime/session.h" +#include "engine/community_models/voxcpm1/assets.h" +#include "engine/community_models/voxcpm1/types.h" + +#include +#include +#include +#include +#include +#include + +namespace engine::core { +class ExecutionContext; +} + +namespace engine::community_models::voxcpm1 { + +struct VoxCPM1AudioVAEDecoderConfig { + size_t weight_context_bytes = 768ull * 1024ull * 1024ull; + size_t graph_context_bytes = 1024ull * 1024ull * 1024ull; + size_t encoder_graph_context_bytes = 1024ull * 1024ull * 1024ull; + int64_t latent_frame_capacity = 0; + int64_t encoder_sample_capacity = 240000; + engine::assets::TensorStorageType weight_storage_type = + engine::assets::TensorStorageType::F32; +}; + +class VoxCPM1AudioVAEDecoderRuntime final { +public: + VoxCPM1AudioVAEDecoderRuntime( + std::shared_ptr assets, + engine::core::ExecutionContext &execution_context, + VoxCPM1AudioVAEDecoderConfig config = {}); + ~VoxCPM1AudioVAEDecoderRuntime(); + + runtime::AudioBuffer decode_features(const std::vector &features, + int64_t patches); + VoxCPM1EncodedPrompt encode_prompt_audio( + const std::optional &prompt_audio, + const std::string &prompt_text, + const std::optional &reference_audio); + void release_runtime_memory(); + void release_encoder_graph(); + +private: + class Impl; + std::unique_ptr impl_; +}; + +} // namespace engine::community_models::voxcpm1 diff --git a/include/engine/models/voxcpm2/config_gguf.h b/include/engine/community_models/voxcpm1/config_gguf.h similarity index 60% rename from include/engine/models/voxcpm2/config_gguf.h rename to include/engine/community_models/voxcpm1/config_gguf.h index c0867a6c..1657da16 100644 --- a/include/engine/models/voxcpm2/config_gguf.h +++ b/include/engine/community_models/voxcpm1/config_gguf.h @@ -1,18 +1,18 @@ #pragma once -#include "engine/models/voxcpm2/assets.h" +#include "engine/community_models/voxcpm1/assets.h" #include "engine/framework/assets/tensor_source.h" #include #include #include -namespace engine::models::voxcpm2 { +namespace engine::community_models::voxcpm1 { // Load VoxCPM1 config from GGUF metadata -VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & source); +VoxCPM1Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & source); // Check if GGUF has VoxCPM1 config metadata bool has_voxcpm1_config_metadata(const engine::assets::TensorSource & source); -} // namespace engine::models::voxcpm2 \ No newline at end of file +} // namespace engine::community_models::voxcpm1 \ No newline at end of file diff --git a/include/engine/community_models/voxcpm1/generator.h b/include/engine/community_models/voxcpm1/generator.h new file mode 100644 index 00000000..d7bb0431 --- /dev/null +++ b/include/engine/community_models/voxcpm1/generator.h @@ -0,0 +1,59 @@ +#pragma once + +#include "engine/framework/assets/tensor_source.h" +#include "engine/community_models/voxcpm1/types.h" + +#include +#include +#include +#include + +namespace engine::core { +class ExecutionContext; +} + +namespace engine::community_models::voxcpm1 { + +struct VoxCPM1Assets; + +struct VoxCPM1FeatureGeneratorConfig { + size_t weight_context_bytes = 3ull * 1024ull * 1024ull * 1024ull; + size_t text_embedding_graph_context_bytes = 64ull * 1024ull * 1024ull; + size_t lm_step_graph_context_bytes = 1024ull * 1024ull * 1024ull; + size_t projection_graph_context_bytes = 256ull * 1024ull * 1024ull; + size_t local_encoder_graph_context_bytes = 512ull * 1024ull * 1024ull; + size_t dit_graph_context_bytes = 1024ull * 1024ull * 1024ull; + size_t prompt_cache_slots = 1; + bool mem_saver = false; + engine::assets::TensorStorageType weight_storage_type = + engine::assets::TensorStorageType::Native; +}; + +class VoxCPM1FeatureGeneratorRuntime final { +public: + VoxCPM1FeatureGeneratorRuntime( + std::shared_ptr assets, + engine::core::ExecutionContext &execution_context, + VoxCPM1FeatureGeneratorConfig config = {}); + ~VoxCPM1FeatureGeneratorRuntime(); + + VoxCPM1Result generate_zero_shot(const std::string &text, + const VoxCPM1GenerationOptions &options); + VoxCPM1Result generate(const std::string &text, + const VoxCPM1EncodedPrompt *prompt, + const VoxCPM1GenerationOptions &options); + VoxCPM1StreamingResult + generate_streaming(const std::string &text, + const VoxCPM1EncodedPrompt *prompt, + const VoxCPM1GenerationOptions &options, + const std::function + &chunk_callback = nullptr); + void release_runtime_memory(); + void release_text_length_memory(); + +private: + class Impl; + std::unique_ptr impl_; +}; + +} // namespace engine::community_models::voxcpm1 diff --git a/include/engine/models/voxcpm2/gguf_metadata.h b/include/engine/community_models/voxcpm1/gguf_metadata.h similarity index 95% rename from include/engine/models/voxcpm2/gguf_metadata.h rename to include/engine/community_models/voxcpm1/gguf_metadata.h index d7ff10c5..174a6860 100644 --- a/include/engine/models/voxcpm2/gguf_metadata.h +++ b/include/engine/community_models/voxcpm1/gguf_metadata.h @@ -12,7 +12,7 @@ namespace engine::assets { class TensorSource; } -namespace engine::models::voxcpm2 { +namespace engine::community_models::voxcpm1 { // Reads GGUF KV metadata (tokenizer.ggml.*, voxcpm_*) directly from the file // backing a TensorSource. Only meaningful for GGUF sources: for any other @@ -45,4 +45,4 @@ class GgufMetadataReader { struct gguf_context * gguf_ = nullptr; }; -} // namespace engine::models::voxcpm2 \ No newline at end of file +} // namespace engine::community_models::voxcpm1 \ No newline at end of file diff --git a/include/engine/community_models/voxcpm1/minicpm.h b/include/engine/community_models/voxcpm1/minicpm.h new file mode 100644 index 00000000..07bbd801 --- /dev/null +++ b/include/engine/community_models/voxcpm1/minicpm.h @@ -0,0 +1,180 @@ +#pragma once + +#include "engine/framework/assets/tensor_source.h" +#include "engine/framework/core/backend_weight_store.h" +#include "engine/framework/core/module.h" +#include "engine/framework/modules/linear_module.h" +#include "engine/framework/modules/norm_modules.h" +#include "engine/framework/core/backend.h" +#include "engine/framework/runtime/kv_cache.h" +#include "engine/community_models/voxcpm1/assets.h" + +#include +#include +#include +#include +#include +#include + +namespace engine::core { +class ExecutionContext; +} + +namespace engine::community_models::voxcpm1 { + +struct VoxCPM1MiniCPMLayerWeights { + engine::modules::NormWeights input_norm; + engine::modules::LinearWeights q_proj; + engine::modules::LinearWeights k_proj; + engine::modules::LinearWeights v_proj; + engine::modules::LinearWeights o_proj; + engine::modules::NormWeights post_norm; + engine::modules::LinearWeights gate_proj; + engine::modules::LinearWeights up_proj; + engine::modules::LinearWeights down_proj; +}; + +struct VoxCPM1MiniCPMWeights { + VoxCPM1MiniCPMConfig config; + std::vector layers; + engine::modules::NormWeights norm; + std::optional token_embedding; + std::optional rope_factors; + float rope_attn_factor = 1.0F; +}; + +struct VoxCPM1FeatEncoderWeights { + engine::core::TensorValue special_token; + engine::modules::LinearWeights in_proj; + VoxCPM1MiniCPMWeights encoder; +}; + +struct VoxCPM1DiTWeights { + engine::modules::LinearWeights in_proj; + engine::modules::LinearWeights cond_proj; + engine::modules::LinearWeights out_proj; + engine::modules::LinearWeights time_mlp_1; + engine::modules::LinearWeights time_mlp_2; + engine::modules::LinearWeights delta_time_mlp_1; + engine::modules::LinearWeights delta_time_mlp_2; + VoxCPM1MiniCPMWeights decoder; +}; + +struct VoxCPM1ProjectionWeights { + engine::modules::LinearWeights fsq_in_proj; + engine::modules::LinearWeights fsq_out_proj; + engine::modules::LinearWeights enc_to_lm_proj; + engine::modules::LinearWeights lm_to_dit_proj; + engine::modules::LinearWeights res_to_dit_proj; + engine::modules::LinearWeights fusion_concat_proj; + engine::modules::LinearWeights stop_proj; + engine::modules::LinearWeights stop_head; +}; + +struct VoxCPM1ModelWeights { + std::shared_ptr store; + VoxCPM1MiniCPMWeights base_lm; + VoxCPM1MiniCPMWeights residual_lm; + VoxCPM1FeatEncoderWeights feat_encoder; + VoxCPM1DiTWeights dit; + VoxCPM1ProjectionWeights projections; +}; + +int64_t head_dim(const VoxCPM1MiniCPMConfig &config); + +enum class VoxCPM1MiniCPMKind { + BaseLM, + ResidualLM, +}; + +struct VoxCPM1MiniCPMStepOutput { + std::vector hidden; + int64_t position = 0; +}; + +struct VoxCPM1PromptPrefillInput { + std::vector input_embeddings; + std::vector current_embeddings; + std::vector text_mask; + std::vector audio_mask; + int64_t steps = 0; +}; + +struct VoxCPM1PromptPrefillOutput { + std::vector lm_hidden; + std::vector residual_hidden; + engine::runtime::TransformerKVState base_state; + engine::runtime::TransformerKVState residual_state; +}; + +class VoxCPM1WeightsRuntime final { +public: + VoxCPM1WeightsRuntime(std::shared_ptr assets, + engine::core::ExecutionContext &execution_context, + size_t weight_context_bytes, + engine::assets::TensorStorageType weight_storage_type); + ~VoxCPM1WeightsRuntime(); + + const VoxCPM1Assets &assets() const noexcept; + const VoxCPM1ModelWeights &weights() const noexcept; + ggml_backend_t backend() const noexcept; + int threads() const noexcept; + bool weights_uploaded() const noexcept; + +private: + class Impl; + std::unique_ptr impl_; +}; + +class VoxCPM1TextEmbeddingRuntime final { +public: + VoxCPM1TextEmbeddingRuntime( + std::shared_ptr weights, + size_t graph_context_bytes, + bool mem_saver = false); + ~VoxCPM1TextEmbeddingRuntime(); + + std::vector embed_token(int32_t token_id); + void release_runtime_memory(); + +private: + class Impl; + std::unique_ptr impl_; +}; + +class VoxCPM1PromptPrefillRuntime final { +public: + VoxCPM1PromptPrefillRuntime( + std::shared_ptr weights, + size_t graph_context_bytes, + bool mem_saver = false); + ~VoxCPM1PromptPrefillRuntime(); + + VoxCPM1PromptPrefillOutput run(const VoxCPM1PromptPrefillInput &input); + void release_runtime_memory(); + +private: + class Impl; + std::unique_ptr impl_; +}; + +class VoxCPM1MiniCPMStepRuntime final { +public: + VoxCPM1MiniCPMStepRuntime( + std::shared_ptr weights, + VoxCPM1MiniCPMKind kind, int64_t cache_steps, + size_t graph_context_bytes); + ~VoxCPM1MiniCPMStepRuntime(); + + void reset(); + void import_state(const engine::runtime::TransformerKVState &state); + engine::runtime::TransformerKVState export_state() const; + VoxCPM1MiniCPMStepOutput run_step(const std::vector &embedding); + void release_runtime_memory(); + +private: + class Impl; + std::unique_ptr impl_; +}; + +} // namespace engine::community_models::voxcpm1 diff --git a/include/engine/community_models/voxcpm1/session.h b/include/engine/community_models/voxcpm1/session.h new file mode 100644 index 00000000..ab48df5a --- /dev/null +++ b/include/engine/community_models/voxcpm1/session.h @@ -0,0 +1,111 @@ +#pragma once + +#include "engine/framework/runtime/cache_slots.h" +#include "engine/framework/runtime/model.h" +#include "engine/framework/runtime/session_base.h" +#include "engine/community_models/voxcpm1/assets.h" +#include "engine/community_models/voxcpm1/audiovae.h" +#include "engine/community_models/voxcpm1/generator.h" + +#include +#include +#include +#include + +namespace engine::community_models::voxcpm1 { + +class VoxCPM1SessionBase : public runtime::RuntimeSessionBase { +public: + VoxCPM1SessionBase(runtime::TaskSpec task, runtime::SessionOptions options, + std::shared_ptr assets); + ~VoxCPM1SessionBase() override; + +protected: + std::string family_impl() const; + runtime::VoiceTaskKind task_kind_impl() const; + runtime::RunMode run_mode_impl() const; + void prepare_impl(const runtime::SessionPreparationRequest &request); + + struct EncodedPromptCacheKey { + std::string prompt_text; + std::optional prompt_audio; + std::optional reference_audio; + }; + + struct EncodedPromptCacheKeyEqual { + bool operator()(const EncodedPromptCacheKey &lhs, + const EncodedPromptCacheKey &rhs) const; + }; + + struct EncodedPromptCacheEntry { + VoxCPM1EncodedPrompt encoded; + }; + + VoxCPM1GenerationOptions + generation_options_from_request(const runtime::TaskRequest &request) const; + void validate_request(const runtime::TaskRequest &request) const; + const VoxCPM1EncodedPrompt *encoded_prompt_for_request( + const std::optional &prompt_audio, + const std::string &prompt_text, + const std::optional &reference_audio); + + runtime::TaskResult run_offline_request(const runtime::TaskRequest &request); + runtime::TaskResult run_streaming_request( + const runtime::TaskRequest &request, + const runtime::StreamEventCallback &stream_event_sink = nullptr); + void release_request_runtime_memory(); + + runtime::TaskSpec task_; + std::shared_ptr assets_; + VoxCPM1FeatureGeneratorConfig generator_config_; + VoxCPM1AudioVAEDecoderConfig decoder_config_; + std::unique_ptr generator_; + std::unique_ptr decoder_; + runtime::CacheSlots + encoded_prompt_cache_; + std::optional uncached_encoded_prompt_; +}; + +class VoxCPM1OfflineSession final : public VoxCPM1SessionBase, + public runtime::IOfflineVoiceTaskSession { +public: + VoxCPM1OfflineSession(runtime::TaskSpec task, runtime::SessionOptions options, + std::shared_ptr assets); + + std::string family() const override; + runtime::VoiceTaskKind task_kind() const override; + runtime::RunMode run_mode() const override; + void prepare(const runtime::SessionPreparationRequest &request) override; + runtime::TaskResult run(const runtime::TaskRequest &request) override; +}; + +class VoxCPM1StreamingSession final : public VoxCPM1SessionBase, + public runtime::IStreamingVoiceTaskSession { +public: + VoxCPM1StreamingSession(runtime::TaskSpec task, runtime::SessionOptions options, + std::shared_ptr assets); + + std::string family() const override; + runtime::VoiceTaskKind task_kind() const override; + runtime::RunMode run_mode() const override; + void prepare(const runtime::SessionPreparationRequest &request) override; + runtime::StreamingPolicy streaming_policy() const override; + void start_stream(const runtime::TaskRequest &request) override; + std::optional next_stream_event() override; + void set_stream_event_sink(runtime::StreamEventCallback sink) override; + runtime::TaskResult finish_stream() override; + void reset() override; + runtime::StreamEvent process_audio_chunk(const runtime::AudioChunk &chunk) override; + runtime::TaskResult finalize() override; + +private: + runtime::TaskResult result_; + size_t next_chunk_index_ = 0; + bool started_ = false; + runtime::StreamEventCallback stream_event_sink_; +}; + +std::shared_ptr make_voxcpm1_loader(); + +} // namespace engine::community_models::voxcpm1 diff --git a/include/engine/models/voxcpm2/tokenizer_gguf.h b/include/engine/community_models/voxcpm1/tokenizer_gguf.h similarity index 80% rename from include/engine/models/voxcpm2/tokenizer_gguf.h rename to include/engine/community_models/voxcpm1/tokenizer_gguf.h index d076e87a..0a406ea4 100644 --- a/include/engine/models/voxcpm2/tokenizer_gguf.h +++ b/include/engine/community_models/voxcpm1/tokenizer_gguf.h @@ -1,16 +1,16 @@ #pragma once -#include "engine/models/voxcpm2/types.h" +#include "engine/community_models/voxcpm1/types.h" #include "engine/framework/assets/tensor_source.h" #include #include #include -namespace engine::models::voxcpm2 { +namespace engine::community_models::voxcpm1 { // Forward declaration -struct VoxCPM2TextPrompt; +struct VoxCPM1TextPrompt; // GGUF-native tokenizer that reads tokenizer metadata directly from GGUF class VoxCPM1GgufTokenizer { @@ -20,7 +20,7 @@ class VoxCPM1GgufTokenizer { explicit VoxCPM1GgufTokenizer(std::shared_ptr gguf_source); std::vector encode(const std::string & text) const; - VoxCPM2TextPrompt build_prompt(const std::string & text) const; + VoxCPM1TextPrompt build_prompt(const std::string & text) const; int32_t audio_start_token_id() const noexcept; int32_t audio_end_token_id() const noexcept; int32_t reference_audio_start_token_id() const noexcept; @@ -36,4 +36,4 @@ class VoxCPM1GgufTokenizer { std::shared_ptr impl_; }; -} // namespace engine::models::voxcpm2 \ No newline at end of file +} // namespace engine::community_models::voxcpm1 \ No newline at end of file diff --git a/include/engine/community_models/voxcpm1/tokenizer_text.h b/include/engine/community_models/voxcpm1/tokenizer_text.h new file mode 100644 index 00000000..a7ae18ce --- /dev/null +++ b/include/engine/community_models/voxcpm1/tokenizer_text.h @@ -0,0 +1,32 @@ +#pragma once + +#include "engine/community_models/voxcpm1/types.h" + +#include +#include +#include +#include + +namespace engine::community_models::voxcpm1 { + +// Forward declaration +struct VoxCPM1Assets; + +class VoxCPM1TextTokenizer { +public: + struct Impl; + + explicit VoxCPM1TextTokenizer(std::shared_ptr assets); + + std::vector encode(const std::string & text) const; + VoxCPM1TextPrompt build_prompt(const std::string & text) const; + int32_t audio_start_token_id() const noexcept; + int32_t audio_end_token_id() const noexcept; + int32_t reference_audio_start_token_id() const noexcept; + int32_t reference_audio_end_token_id() const noexcept; + +private: + std::shared_ptr impl_; +}; + +} // namespace engine::community_models::voxcpm1 \ No newline at end of file diff --git a/include/engine/models/voxcpm2/tokenizer_wrapper.h b/include/engine/community_models/voxcpm1/tokenizer_wrapper.h similarity index 57% rename from include/engine/models/voxcpm2/tokenizer_wrapper.h rename to include/engine/community_models/voxcpm1/tokenizer_wrapper.h index 6c700cd1..183c4478 100644 --- a/include/engine/models/voxcpm2/tokenizer_wrapper.h +++ b/include/engine/community_models/voxcpm1/tokenizer_wrapper.h @@ -1,58 +1,58 @@ #pragma once -#include "engine/models/voxcpm2/tokenizer_text.h" -#include "engine/models/voxcpm2/tokenizer_gguf.h" -#include "engine/models/voxcpm2/types.h" +#include "engine/community_models/voxcpm1/tokenizer_text.h" +#include "engine/community_models/voxcpm1/tokenizer_gguf.h" +#include "engine/community_models/voxcpm1/types.h" #include #include -namespace engine::models::voxcpm2 { +namespace engine::community_models::voxcpm1 { -// Wrapper that can hold either VoxCPM2TextTokenizer (JSON-based) or VoxCPM1GgufTokenizer (GGUF-based) -class VoxCPM2TokenizerWrapper { +// Wrapper that can hold either VoxCPM1TextTokenizer (JSON-based) or VoxCPM1GgufTokenizer (GGUF-based) +class VoxCPM1TokenizerWrapper { public: - VoxCPM2TokenizerWrapper() = default; - explicit VoxCPM2TokenizerWrapper(std::shared_ptr tokenizer) + VoxCPM1TokenizerWrapper() = default; + explicit VoxCPM1TokenizerWrapper(std::shared_ptr tokenizer) : tokenizer_(std::move(tokenizer)) {} - explicit VoxCPM2TokenizerWrapper(std::shared_ptr tokenizer) + explicit VoxCPM1TokenizerWrapper(std::shared_ptr tokenizer) : tokenizer_(std::move(tokenizer)) {} - VoxCPM2TextPrompt build_prompt(const std::string & text) const { - if (std::holds_alternative>(tokenizer_)) { - return std::get>(tokenizer_)->build_prompt(text); + VoxCPM1TextPrompt build_prompt(const std::string & text) const { + if (std::holds_alternative>(tokenizer_)) { + return std::get>(tokenizer_)->build_prompt(text); } else { return std::get>(tokenizer_)->build_prompt(text); } } int32_t audio_start_token_id() const noexcept { - if (std::holds_alternative>(tokenizer_)) { - return std::get>(tokenizer_)->audio_start_token_id(); + if (std::holds_alternative>(tokenizer_)) { + return std::get>(tokenizer_)->audio_start_token_id(); } else { return std::get>(tokenizer_)->audio_start_token_id(); } } int32_t audio_end_token_id() const noexcept { - if (std::holds_alternative>(tokenizer_)) { - return std::get>(tokenizer_)->audio_end_token_id(); + if (std::holds_alternative>(tokenizer_)) { + return std::get>(tokenizer_)->audio_end_token_id(); } else { return std::get>(tokenizer_)->audio_end_token_id(); } } int32_t reference_audio_start_token_id() const noexcept { - if (std::holds_alternative>(tokenizer_)) { - return std::get>(tokenizer_)->reference_audio_start_token_id(); + if (std::holds_alternative>(tokenizer_)) { + return std::get>(tokenizer_)->reference_audio_start_token_id(); } else { return std::get>(tokenizer_)->reference_audio_start_token_id(); } } int32_t reference_audio_end_token_id() const noexcept { - if (std::holds_alternative>(tokenizer_)) { - return std::get>(tokenizer_)->reference_audio_end_token_id(); + if (std::holds_alternative>(tokenizer_)) { + return std::get>(tokenizer_)->reference_audio_end_token_id(); } else { return std::get>(tokenizer_)->reference_audio_end_token_id(); } @@ -65,9 +65,9 @@ class VoxCPM2TokenizerWrapper { private: std::variant< std::monostate, - std::shared_ptr, + std::shared_ptr, std::shared_ptr > tokenizer_; }; -} // namespace engine::models::voxcpm2 \ No newline at end of file +} // namespace engine::community_models::voxcpm1 \ No newline at end of file diff --git a/include/engine/community_models/voxcpm1/types.h b/include/engine/community_models/voxcpm1/types.h new file mode 100644 index 00000000..73c0fb52 --- /dev/null +++ b/include/engine/community_models/voxcpm1/types.h @@ -0,0 +1,69 @@ +#pragma once + +#include "engine/framework/runtime/session.h" + +#include +#include +#include +#include + +namespace engine::community_models::voxcpm1 { + +struct VoxCPM1GenerationOptions { + int64_t min_tokens = 2; + int64_t max_tokens = 4096; + int64_t num_inference_steps = 10; + float guidance_scale = 2.0F; + bool retry_badcase = true; + int64_t retry_badcase_max_times = 3; + float retry_badcase_ratio_threshold = 6.0F; + uint32_t seed = 1234; + std::string cfm_noise_file; +}; + +struct VoxCPM1PromptAudio { + runtime::AudioBuffer audio; + std::string text; +}; + +struct VoxCPM1EncodedPrompt { + std::string prompt_text; + std::vector prompt_features; + int64_t prompt_patches = 0; + std::vector reference_features; + int64_t reference_patches = 0; +}; + +struct VoxCPM1Request { + std::string text; + std::optional prompt = std::nullopt; + std::optional reference_audio = std::nullopt; + VoxCPM1GenerationOptions generation; +}; + +struct VoxCPM1TextPrompt { + std::string text; + std::vector input_ids; +}; + +struct VoxCPM1Result { + runtime::AudioBuffer audio; + std::vector generated_features; + int64_t generated_patches = 0; + std::vector decode_features; + int64_t decode_patches = 0; + int64_t decode_trim_patches = 0; +}; + +struct VoxCPM1StreamingChunk { + std::vector decode_features; + int64_t decode_patches = 0; + int64_t generated_patches = 0; +}; + +struct VoxCPM1StreamingResult { + std::vector chunks; + int64_t generated_patches = 0; +}; + +} // namespace engine::community_models::voxcpm1 diff --git a/include/engine/models/voxcpm2/assets.h b/include/engine/models/voxcpm2/assets.h index 4e29d06a..89fe156b 100644 --- a/include/engine/models/voxcpm2/assets.h +++ b/include/engine/models/voxcpm2/assets.h @@ -2,7 +2,6 @@ #include "engine/framework/assets/resource_bundle.h" #include "engine/framework/assets/tensor_source.h" -#include "engine/models/voxcpm2/tokenizer_gguf.h" #include #include @@ -86,7 +85,6 @@ struct VoxCPM2Config { int64_t max_length = 8192; std::string device = "cuda"; std::string dtype = "bfloat16"; - bool v1 = false; }; struct VoxCPM2Assets { @@ -94,9 +92,8 @@ struct VoxCPM2Assets { VoxCPM2Config config; std::shared_ptr model_weights; std::shared_ptr audiovae_weights; - std::shared_ptr gguf_tokenizer; }; -std::shared_ptr load_voxcpm2_assets(const std::filesystem::path & model_path, bool is_v1); +std::shared_ptr load_voxcpm2_assets(const std::filesystem::path & model_path); } // namespace engine::models::voxcpm2 diff --git a/include/engine/models/voxcpm2/audiovae.h b/include/engine/models/voxcpm2/audiovae.h index d6a250aa..5e1b038c 100644 --- a/include/engine/models/voxcpm2/audiovae.h +++ b/include/engine/models/voxcpm2/audiovae.h @@ -43,7 +43,6 @@ class VoxCPM2AudioVAEDecoderRuntime final { const std::string &prompt_text, const std::optional &reference_audio); void release_runtime_memory(); - void release_encoder_graph(); private: class Impl; diff --git a/include/engine/models/voxcpm2/generator.h b/include/engine/models/voxcpm2/generator.h index cf46d955..abbcf8be 100644 --- a/include/engine/models/voxcpm2/generator.h +++ b/include/engine/models/voxcpm2/generator.h @@ -49,7 +49,6 @@ class VoxCPM2FeatureGeneratorRuntime final { const std::function &chunk_callback = nullptr); void release_runtime_memory(); - void release_text_length_memory(); private: class Impl; diff --git a/include/engine/models/voxcpm2/loader.h b/include/engine/models/voxcpm2/loader.h index 72f7f3c5..4c588482 100644 --- a/include/engine/models/voxcpm2/loader.h +++ b/include/engine/models/voxcpm2/loader.h @@ -29,6 +29,5 @@ class VoxCPM2LoadedModel final : public runtime::ILoadedVoiceModel { std::unique_ptr load_voxcpm2_model(const std::filesystem::path &model_path); std::shared_ptr make_voxcpm2_loader(); -std::shared_ptr make_voxcpm1_loader(); } // namespace engine::models::voxcpm2 diff --git a/include/engine/models/voxcpm2/minicpm.h b/include/engine/models/voxcpm2/minicpm.h index 5b40a03b..1aa57fc3 100644 --- a/include/engine/models/voxcpm2/minicpm.h +++ b/include/engine/models/voxcpm2/minicpm.h @@ -135,7 +135,6 @@ class VoxCPM2TextEmbeddingRuntime final { ~VoxCPM2TextEmbeddingRuntime(); std::vector embed_token(int32_t token_id); - void release_runtime_memory(); private: class Impl; @@ -151,7 +150,6 @@ class VoxCPM2PromptPrefillRuntime final { ~VoxCPM2PromptPrefillRuntime(); VoxCPM2PromptPrefillOutput run(const VoxCPM2PromptPrefillInput &input); - void release_runtime_memory(); private: class Impl; diff --git a/include/engine/models/voxcpm2/tokenizer_text.h b/include/engine/models/voxcpm2/tokenizer_text.h index 0cd3b7c8..ae877b57 100644 --- a/include/engine/models/voxcpm2/tokenizer_text.h +++ b/include/engine/models/voxcpm2/tokenizer_text.h @@ -1,5 +1,6 @@ #pragma once +#include "engine/models/voxcpm2/assets.h" #include "engine/models/voxcpm2/types.h" #include @@ -9,9 +10,6 @@ namespace engine::models::voxcpm2 { -// Forward declaration -struct VoxCPM2Assets; - class VoxCPM2TextTokenizer { public: struct Impl; @@ -29,4 +27,4 @@ class VoxCPM2TextTokenizer { std::shared_ptr impl_; }; -} // namespace engine::models::voxcpm2 \ No newline at end of file +} // namespace engine::models::voxcpm2 diff --git a/model_specs/voxcpm1.json b/model_specs/voxcpm1.json index ee9d0ebe..5e26a5c2 100644 --- a/model_specs/voxcpm1.json +++ b/model_specs/voxcpm1.json @@ -1,7 +1,8 @@ { + "schema_version": 1, "family": "voxcpm1", "display_name": "VoxCPM1", - "description": "OpenBMB VoxCPM 0.5B and 1.5B tokenizer-free TTS models supporting short-reference voice cloning and streaming output (16kHz for 0.5B, 44.1kHz for 1.5B).", + "description": "OpenBMB VoxCPM 0.5B tokenizer-free TTS model supporting short-reference voice cloning and streaming output (16kHz).", "category": "tts", "status": "supported", "tasks": [ @@ -23,6 +24,195 @@ "speaker_reference" ] }, + "dependencies": [], + "options": { + "request": [ + { + "name": "text_chunk_mode", + "type": "enum", + "description": "Text chunking mode; default tag_aware.", + "preset": "text_chunk_mode_full", + "required": false, + "default": "tag_aware" + }, + { + "name": "seed", + "type": "int", + "description": "Random seed for MiniCPM and diffusion sampling.", + "required": false + }, + { + "name": "max_tokens", + "type": "int", + "description": "Maximum MiniCPM output tokens.", + "required": false, + "default": 1024 + }, + { + "name": "min_tokens", + "type": "int", + "description": "Minimum MiniCPM output tokens before an EOS stop is honored.", + "required": false, + "default": 0 + }, + { + "name": "num_inference_steps", + "type": "int", + "description": "CFM diffusion sampling steps.", + "required": false, + "default": 50 + }, + { + "name": "guidance_scale", + "type": "float", + "description": "CFM classifier-free guidance rate.", + "required": false, + "default": 2.0 + }, + { + "name": "retry_badcase", + "type": "bool", + "description": "Retry the request when generation is detected as a bad case.", + "required": false, + "default": true + }, + { + "name": "retry_badcase_max_times", + "type": "int", + "description": "Maximum bad-case retry count.", + "required": false, + "default": 2 + }, + { + "name": "retry_badcase_ratio_threshold", + "type": "float", + "description": "Bad-case ratio threshold for retry decisions.", + "required": false + }, + { + "name": "prompt_text", + "type": "string", + "description": "Text prompt for prompt-continuation voice cloning.", + "required": false + } + ], + "session": [ + { + "name": "mem_saver", + "type": "bool", + "description": "Use tighter graph workspaces and release request runtime graphs; default false.", + "required": false, + "default": false + }, + { + "name": "prompt_cache_slots", + "type": "int", + "description": "Prompt and prompt-audio embedding cache slots; default 1.", + "required": false, + "default": 1 + }, + { + "name": "weight_type", + "type": "enum", + "description": "Model weight storage type.", + "preset": "weight_type_full", + "required": false, + "default": "native" + }, + { + "name": "audiovae_weight_type", + "type": "enum", + "description": "AudioVAE weight storage type.", + "preset": "weight_type_full", + "required": false, + "default": "native" + }, + { + "name": "weight_context_mb", + "type": "int", + "description": "Model weight graph context size in MB.", + "required": false + }, + { + "name": "text_embedding_graph_context_mb", + "type": "int", + "description": "Text embedding graph context size in MB.", + "required": false + }, + { + "name": "lm_step_graph_context_mb", + "type": "int", + "description": "LM step graph context size in MB.", + "required": false + }, + { + "name": "projection_graph_context_mb", + "type": "int", + "description": "Projection graph context size in MB.", + "required": false + }, + { + "name": "local_encoder_graph_context_mb", + "type": "int", + "description": "Local encoder graph context size in MB.", + "required": false + }, + { + "name": "dit_graph_context_mb", + "type": "int", + "description": "DiT estimator graph context size in MB.", + "required": false + }, + { + "name": "audiovae_weight_context_mb", + "type": "int", + "description": "AudioVAE weight graph context size in MB.", + "required": false + }, + { + "name": "audiovae_graph_context_mb", + "type": "int", + "description": "AudioVAE decoder graph context size in MB.", + "required": false + }, + { + "name": "audiovae_encoder_graph_context_mb", + "type": "int", + "description": "AudioVAE encoder graph context size in MB.", + "required": false + }, + { + "name": "audiovae_latent_capacity", + "type": "int", + "description": "AudioVAE decoder latent frame capacity.", + "required": false + }, + { + "name": "audiovae_encoder_sample_capacity", + "type": "int", + "description": "AudioVAE encoder sample capacity.", + "required": false + } + ], + "load": [ + { + "name": "weight_type", + "type": "enum", + "description": "Model weight storage type selected at load time.", + "preset": "weight_type_full", + "required": false, + "default": "native" + }, + { + "name": "audiovae_weight_type", + "type": "enum", + "description": "AudioVAE weight storage type selected at load time.", + "preset": "weight_type_full", + "required": false, + "default": "native" + } + ] + }, "runtime": { "tags": [ "gguf", @@ -62,28 +252,6 @@ "VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf" ], "strip_prefix": "VoxCPM1-GGUF" - }, - { - "id": "voxcpm1_1.5b_q4_k", - "display_name": "VoxCPM 1.5B Q4_K GGUF", - "format": "gguf", - "precision": "q4_k", - "target_directory": "VoxCPM1.5-GGUF", - "files": [ - "VoxCPM1.5-GGUF/voxcpm1.5-q4_k-audiovae-f16.gguf" - ], - "strip_prefix": "VoxCPM1.5-GGUF" - }, - { - "id": "voxcpm1_1.5b_q8_0", - "display_name": "VoxCPM 1.5B Q8_0 GGUF", - "format": "gguf", - "precision": "q8_0", - "target_directory": "VoxCPM1.5-GGUF", - "files": [ - "VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf" - ], - "strip_prefix": "VoxCPM1.5-GGUF" } ], "sources": [ diff --git a/src/community_models/voxcpm1/assets.cpp b/src/community_models/voxcpm1/assets.cpp new file mode 100644 index 00000000..c0d08f52 --- /dev/null +++ b/src/community_models/voxcpm1/assets.cpp @@ -0,0 +1,933 @@ +#include "engine/community_models/voxcpm1/assets.h" +#include "engine/community_models/voxcpm1/tokenizer_gguf.h" +#include "engine/community_models/voxcpm1/config_gguf.h" + +#include "engine/framework/model_spec/package.h" +#include "engine/framework/assets/resource_bundle.h" +#include "engine/framework/assets/tensor_source.h" +#include "engine/framework/io/config.h" +#include "engine/framework/io/json.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace engine::community_models::voxcpm1 { +namespace json = engine::io::json; +namespace { + +VoxCPM1RopeScalingConfig parse_rope_scaling(const json::Value & value) { + VoxCPM1RopeScalingConfig config; + config.type = json::optional_string(value, "type", ""); + config.long_factor = json::optional_f32_array(value, "long_factor"); + config.short_factor = json::optional_f32_array(value, "short_factor"); + config.original_max_position_embeddings = + json::optional_i64(value, "original_max_position_embeddings", 0); + return config; +} + +VoxCPM1MiniCPMConfig parse_lm_config(const json::Value & value) { + VoxCPM1MiniCPMConfig config; + config.bos_token_id = json::optional_i64(value, "bos_token_id", config.bos_token_id); + config.eos_token_id = json::optional_i64(value, "eos_token_id", config.eos_token_id); + config.hidden_size = json::require_i64(value, "hidden_size"); + config.intermediate_size = json::require_i64(value, "intermediate_size"); + config.max_position_embeddings = json::require_i64(value, "max_position_embeddings"); + config.num_attention_heads = json::require_i64(value, "num_attention_heads"); + config.num_hidden_layers = json::require_i64(value, "num_hidden_layers"); + config.num_key_value_heads = json::require_i64(value, "num_key_value_heads"); + config.kv_channels = json::optional_i64(value, "kv_channels", config.hidden_size / config.num_attention_heads); + config.vocab_size = json::require_i64(value, "vocab_size"); + config.scale_emb = json::optional_i64(value, "scale_emb", config.scale_emb); + config.dim_model_base = json::optional_i64(value, "dim_model_base", config.dim_model_base); + config.rms_norm_eps = json::optional_f32(value, "rms_norm_eps", config.rms_norm_eps); + config.rope_theta = json::optional_f32(value, "rope_theta", config.rope_theta); + config.scale_depth = json::optional_f32(value, "scale_depth", config.scale_depth); + config.use_mup = json::optional_bool(value, "use_mup", config.use_mup); + if (const auto * rope_scaling = value.find("rope_scaling"); rope_scaling != nullptr) { + config.rope_scaling = parse_rope_scaling(*rope_scaling); + } + engine::io::require_positive(config.hidden_size, "lm hidden_size"); + engine::io::require_positive(config.intermediate_size, "lm intermediate_size"); + engine::io::require_positive(config.max_position_embeddings, "lm max_position_embeddings"); + engine::io::require_positive(config.num_attention_heads, "lm num_attention_heads"); + engine::io::require_positive(config.num_hidden_layers, "lm num_hidden_layers"); + engine::io::require_positive(config.num_key_value_heads, "lm num_key_value_heads"); + engine::io::require_positive(config.kv_channels, "lm kv_channels"); + engine::io::require_positive(config.vocab_size, "lm vocab_size"); + engine::io::require_divisible(config.hidden_size, config.num_attention_heads, "lm hidden_size / num_attention_heads"); + engine::io::require_divisible(config.num_attention_heads, config.num_key_value_heads, "lm attention heads"); + if (!config.rope_scaling.type.empty()) { + if (config.rope_scaling.type != "longrope") { + throw std::runtime_error("VoxCPM1 currently expects longrope rope_scaling"); + } + const int64_t expected = config.hidden_size / config.num_attention_heads / 2; + if (static_cast(config.rope_scaling.long_factor.size()) != expected || + static_cast(config.rope_scaling.short_factor.size()) != expected) { + throw std::runtime_error("VoxCPM1 rope_scaling factor length does not match head_dim / 2"); + } + } + return config; +} + +VoxCPM1LocalTransformerConfig parse_local_transformer_config( + const json::Value & value, + const char * label) { + VoxCPM1LocalTransformerConfig config; + config.hidden_dim = json::require_i64(value, "hidden_dim"); + config.ffn_dim = json::require_i64(value, "ffn_dim"); + config.num_heads = json::require_i64(value, "num_heads"); + config.num_layers = json::require_i64(value, "num_layers"); + config.kv_channels = json::optional_i64(value, "kv_channels", config.hidden_dim / config.num_heads); + engine::io::require_positive(config.hidden_dim, label); + engine::io::require_positive(config.ffn_dim, label); + engine::io::require_positive(config.num_heads, label); + engine::io::require_positive(config.num_layers, label); + engine::io::require_positive(config.kv_channels, label); + engine::io::require_divisible(config.hidden_dim, config.num_heads, label); + return config; +} + +VoxCPM1DiTConfig parse_dit_config(const json::Value & value) { + const auto base = parse_local_transformer_config(value, "dit transformer"); + VoxCPM1DiTConfig config; + config.hidden_dim = base.hidden_dim; + config.ffn_dim = base.ffn_dim; + config.num_heads = base.num_heads; + config.num_layers = base.num_layers; + config.kv_channels = base.kv_channels; + config.mean_mode = json::optional_bool(value, "dit_mean_mode", json::optional_bool(value, "mean_mode", false)); + const auto & cfm = value.require("cfm_config"); + config.cfm.sigma_min = json::optional_f32(cfm, "sigma_min", config.cfm.sigma_min); + config.cfm.solver = json::optional_string(cfm, "solver", config.cfm.solver); + config.cfm.t_scheduler = json::optional_string(cfm, "t_scheduler", config.cfm.t_scheduler); + config.cfm.inference_cfg_rate = json::optional_f32(cfm, "inference_cfg_rate", config.cfm.inference_cfg_rate); + if (config.cfm.solver != "euler") { + throw std::runtime_error("VoxCPM1 CFM currently expects euler solver"); + } + if (config.cfm.t_scheduler != "log-norm") { + throw std::runtime_error("VoxCPM1 CFM currently expects log-norm scheduler"); + } + return config; +} + +VoxCPM1AudioVAEConfig parse_audio_vae_config(const json::Value & value) { + VoxCPM1AudioVAEConfig config; + config.encoder_dim = json::require_i64(value, "encoder_dim"); + config.encoder_rates = json::require_i64_array(value, "encoder_rates"); + config.latent_dim = json::require_i64(value, "latent_dim"); + config.decoder_dim = json::require_i64(value, "decoder_dim"); + config.decoder_rates = json::require_i64_array(value, "decoder_rates"); + config.sample_rate_bin_boundaries = json::optional_i64_array(value, "sr_bin_boundaries"); + config.sample_rate = static_cast(json::require_i64(value, "sample_rate")); + config.output_sample_rate = static_cast(json::require_i64(value, "out_sample_rate")); + engine::io::require_positive(config.encoder_dim, "AudioVAE encoder_dim"); + engine::io::require_positive(config.latent_dim, "AudioVAE latent_dim"); + engine::io::require_positive(config.decoder_dim, "AudioVAE decoder_dim"); + engine::io::require_positive(config.sample_rate, "AudioVAE sample_rate"); + engine::io::require_positive(config.output_sample_rate, "AudioVAE out_sample_rate"); + if (config.encoder_rates.empty() || config.decoder_rates.empty()) { + throw std::runtime_error("VoxCPM1 AudioVAE rates must be non-empty"); + } + for (const auto rate : config.encoder_rates) { + engine::io::require_positive(rate, "AudioVAE encoder rate"); + } + for (const auto rate : config.decoder_rates) { + engine::io::require_positive(rate, "AudioVAE decoder rate"); + } + return config; +} + +VoxCPM1Config parse_config(const assets::ResourceBundle & resources) { + const auto root = resources.parse_json("config"); + VoxCPM1Config config; + config.architecture = json::require_string(root, "architecture"); + if (config.architecture != "voxcpm2" && config.architecture != "voxcpm") { + throw std::runtime_error("VoxCPM config architecture mismatch: " + config.architecture); + } + config.lm = parse_lm_config(root.require("lm_config")); + config.patch_size = json::optional_i64(root, "patch_size", config.patch_size); + config.feat_dim = json::optional_i64(root, "feat_dim", config.feat_dim); + config.residual_lm_num_layers = + json::optional_i64(root, "residual_lm_num_layers", config.residual_lm_num_layers); + config.residual_lm_no_rope = json::optional_bool(root, "residual_lm_no_rope", config.residual_lm_no_rope); + config.scalar_quantization_latent_dim = + json::optional_i64(root, "scalar_quantization_latent_dim", config.scalar_quantization_latent_dim); + config.scalar_quantization_scale = + json::optional_i64(root, "scalar_quantization_scale", config.scalar_quantization_scale); + config.encoder = parse_local_transformer_config(root.require("encoder_config"), "local encoder transformer"); + config.dit = parse_dit_config(root.require("dit_config")); + config.audio_vae = parse_audio_vae_config(root.require("audio_vae_config")); + config.max_length = json::optional_i64(root, "max_length", config.max_length); + config.device = json::optional_string(root, "device", config.device); + config.dtype = json::optional_string(root, "dtype", config.dtype); + engine::io::require_positive(config.patch_size, "patch_size"); + engine::io::require_positive(config.feat_dim, "feat_dim"); + engine::io::require_positive(config.residual_lm_num_layers, "residual_lm_num_layers"); + engine::io::require_positive(config.scalar_quantization_latent_dim, "scalar_quantization_latent_dim"); + engine::io::require_positive(config.scalar_quantization_scale, "scalar_quantization_scale"); + engine::io::require_positive(config.max_length, "max_length"); + if (config.feat_dim != config.audio_vae.latent_dim) { + throw std::runtime_error("VoxCPM1 feat_dim must match AudioVAE latent_dim"); + } + if (config.residual_lm_num_layers > config.lm.num_hidden_layers) { + throw std::runtime_error("VoxCPM1 residual_lm_num_layers exceeds lm num_hidden_layers"); + } + return config; +} + +namespace assets = engine::assets; + +namespace { +core::TensorShape make_tensor_shape(const std::vector & dims) { + if (dims.empty() || dims.size() > core::kMaxTensorRank) { + throw std::runtime_error("tensor rank must be between 1 and 4"); + } + switch (dims.size()) { + case 1: + return core::TensorShape::from_dims({dims[0]}); + case 2: + return core::TensorShape::from_dims({dims[0], dims[1]}); + case 3: + return core::TensorShape::from_dims({dims[0], dims[1], dims[2]}); + case 4: + return core::TensorShape::from_dims({dims[0], dims[1], dims[2], dims[3]}); + default: + throw std::runtime_error("unsupported tensor rank"); + } +} +} // namespace + +class TransformingTensorSource final : public assets::TensorSource { +public: + TransformingTensorSource( + std::shared_ptr source, + const VoxCPM1Config & config, + bool is_v1) + : source_(std::move(source)), config_(config), is_v1_(is_v1) { + build_routes(); + } + + const std::filesystem::path & source_path() const noexcept override { + return source_->source_path(); + } + + bool has_tensor(std::string_view name) const noexcept override { + const std::string key{std::string(name)}; + if (routes_.find(key) != routes_.end() || + synthesized_tensors_.find(key) != synthesized_tensors_.end()) { + return true; + } + if (is_v1_) { + // v1 GGUF stores folded AudioVAE conv weights; the loader asks for + // decomposed weight_v/weight_g names which we synthesize from the + // folded tensors on demand. + const auto base = folded_base_name(key); + if (!base.empty() && folded_convs_.find(base) != folded_convs_.end()) { + return true; + } + } + return false; + } + + assets::TensorMetadata require_metadata(std::string_view name) const override { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it != synthesized_tensors_.end()) { + return it->second; + } + const std::string key{std::string(name)}; + if (is_v1_) { + const auto base = folded_base_name(key); + if (!base.empty()) { + const auto folded_it = folded_convs_.find(base); + if (folded_it != folded_convs_.end()) { + auto metadata = source_->require_metadata(folded_it->second); + metadata.name = key; + if (has_suffix(key, ".weight_g") && !metadata.shape.empty()) { + metadata.shape = {metadata.shape.front(), 1, 1}; + } + return metadata; + } + } + } + const auto route_it = routes_.find(key); + if (route_it == routes_.end()) { + throw std::runtime_error("missing tensor: " + std::string(name)); + } + auto metadata = source_->require_metadata(route_it->second); + metadata.name = key; + // Apply shape transformations if needed + if (reshape_map_.find(key) != reshape_map_.end()) { + metadata.shape = reshape_map_.at(key); + } + return metadata; + } + + std::vector tensors() const override { + std::vector out; + out.reserve(routes_.size() + synthesized_tensors_.size()); + for (const auto & [name, route] : routes_) { + out.push_back(require_metadata(name)); + } + for (const auto & [name, metadata] : synthesized_tensors_) { + out.push_back(metadata); + } + std::sort(out.begin(), out.end(), + [](const assets::TensorMetadata & lhs, const assets::TensorMetadata & rhs) { + return lhs.name < rhs.name; + }); + return out; + } + + void release_storage() const override { source_->release_storage(); } + + assets::RawTensorData require_tensor_data(std::string_view name) const override { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it != synthesized_tensors_.end()) { + return generate_synthesized_tensor(name); + } + const std::string key{std::string(name)}; + if (is_v1_) { + const auto base = folded_base_name(key); + if (!base.empty() && folded_convs_.find(base) != folded_convs_.end()) { + auto data = source_->require_tensor_data(folded_convs_.at(base)); + data.metadata.name = key; + return data; + } + } + const auto route_it = routes_.find(key); + if (route_it == routes_.end()) { + throw std::runtime_error("missing tensor: " + std::string(name)); + } + auto data = source_->require_tensor_data(route_it->second); + data.metadata.name = key; + // Apply transformations + if (reshape_map_.find(key) != reshape_map_.end()) { + const auto & target_shape = reshape_map_.at(key); + if (data.metadata.shape != target_shape) { + // Reshape the data + data = reshape_tensor_data(data, target_shape); + } + } + return data; + } + + std::vector require_f32( + std::string_view name, + const std::optional> & expected_shape) const override { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it != synthesized_tensors_.end()) { + return generate_synthesized_f32(name); + } + const std::string key{std::string(name)}; + if (is_v1_) { + const auto base = folded_base_name(key); + if (!base.empty()) { + const auto folded_it = folded_convs_.find(base); + if (folded_it != folded_convs_.end()) { + const auto folded = source_->require_f32(folded_it->second, std::nullopt); + if (has_suffix(key, ".weight_g")) { + return folded_weight_g(folded, folded_it->second, expected_shape); + } + return folded; + } + } + } + const auto route_it = routes_.find(key); + if (route_it == routes_.end()) { + throw std::runtime_error("missing tensor: " + std::string(name)); + } + if (is_v1_ && expected_shape.has_value()) { + const auto meta = source_->require_metadata(route_it->second); + const int64_t expected_elems = checked_element_count("expected", *expected_shape); + const int64_t actual_elems = checked_element_count(route_it->second, meta.shape); + if (expected_elems == actual_elems && meta.shape != *expected_shape) { + return source_->require_f32(route_it->second, std::nullopt); + } + } + // Check if we need to reshape + if (reshape_map_.find(key) != reshape_map_.end()) { + const auto & target_shape = reshape_map_.at(key); + if (expected_shape.has_value() && *expected_shape != target_shape) { + // We'll fetch with target shape and then it will be validated + } + return source_->require_f32(route_it->second, target_shape); + } + return source_->require_f32(route_it->second, expected_shape); + } + + std::optional> optional_f32( + std::string_view name, + const std::optional> & expected_shape) const override { + if (!has_tensor(name)) return std::nullopt; + return require_f32(name, expected_shape); + } + + void set_backend_tensor( + ggml_tensor * tensor, + std::string_view name, + assets::TensorStorageType storage_type, + const std::vector & expected_shape) const override { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it != synthesized_tensors_.end()) { + const auto values = generate_synthesized_f32(name); + engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, values, + make_tensor_shape(expected_shape), + engine::assets::ggml_type_for_tensor_storage(storage_type)); + return; + } + const auto route_it = routes_.find(std::string(name)); + if (route_it == routes_.end()) { + throw std::runtime_error("missing tensor: " + std::string(name)); + } + // Check for weight norm decomposition (weight_v + weight_g) + const std::string logical_name = std::string(name); + if (weight_norm_map_.find(logical_name) != weight_norm_map_.end()) { + const auto & wn = weight_norm_map_.at(logical_name); + const auto weight_v = source_->require_f32(wn.weight_v_name, wn.weight_v_shape); + const auto weight_g = source_->require_f32(wn.weight_g_name, wn.weight_g_shape); + const auto folded = fold_weight_norm(weight_v, weight_g, wn.out_channels, wn.in_channels, wn.kernel_size); + const auto shape = make_tensor_shape(expected_shape); + const ggml_type type = engine::assets::ggml_type_for_tensor_storage( + engine::assets::resolve_tensor_storage_type(*this, name, storage_type)); + engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, folded, shape, type); + return; + } + // Check for reshape + if (reshape_map_.find(logical_name) != reshape_map_.end()) { + const auto & target_shape = reshape_map_.at(logical_name); + const auto values = source_->require_f32(route_it->second, target_shape); + const auto shape = make_tensor_shape(expected_shape); + const ggml_type type = engine::assets::ggml_type_for_tensor_storage( + engine::assets::resolve_tensor_storage_type(*this, name, storage_type)); + engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, values, shape, type); + return; + } + // Special handling for V1 embedding weight: token_embd.weight is transposed in GGUF + // V1 GGUF stores [hidden_size, vocab_size] but we need [vocab_size, hidden_size] + if (is_v1_ && logical_name == "base_lm.embed_tokens.weight") { + const auto source_values = source_->require_f32(route_it->second, std::nullopt); + const auto source_meta = source_->require_metadata(route_it->second); + if (source_meta.shape.size() == 2) { + const int64_t src_rows = source_meta.shape[0]; + const int64_t src_cols = source_meta.shape[1]; + const int64_t dst_rows = expected_shape.size() > 0 ? expected_shape[0] : src_cols; + const int64_t dst_cols = expected_shape.size() > 1 ? expected_shape[1] : src_rows; + if (src_rows == dst_cols && src_cols == dst_rows) { + // Transpose the weight matrix + std::vector transposed(static_cast(dst_rows * dst_cols)); + for (int64_t i = 0; i < src_rows; ++i) { + for (int64_t j = 0; j < src_cols; ++j) { + transposed[static_cast(j * dst_rows + i)] = source_values[static_cast(i * src_cols + j)]; + } + } + const auto shape = make_tensor_shape(expected_shape); + const ggml_type type = engine::assets::ggml_type_for_tensor_storage( + engine::assets::resolve_tensor_storage_type(*this, name, storage_type)); + engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, transposed, shape, type); + return; + } + } + } + // V1 relaxed rank: if expected element count matches actual but shapes differ, + // fetch data without expected_shape and set manually + if (is_v1_) { + const auto source_meta = source_->require_metadata(route_it->second); + int64_t expected_elems = 1; + for (const int64_t dim : expected_shape) expected_elems *= dim; + int64_t actual_elems = 1; + for (const int64_t dim : source_meta.shape) actual_elems *= dim; + if (expected_elems == actual_elems && source_meta.shape != expected_shape) { + const auto values = source_->require_f32(route_it->second, std::nullopt); + const auto shape = make_tensor_shape(expected_shape); + const ggml_type type = engine::assets::ggml_type_for_tensor_storage( + engine::assets::resolve_tensor_storage_type(*this, name, storage_type)); + engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, values, shape, type); + return; + } + } + source_->set_backend_tensor(tensor, route_it->second, storage_type, expected_shape); + } + + void set_backend_f32_tensor( + ggml_tensor * tensor, + std::string_view name, + const std::vector & expected_shape) const override { + set_backend_tensor(tensor, name, assets::TensorStorageType::F32, expected_shape); + } + + int64_t require_i64_scalar(std::string_view name) const override { + return source_->require_i64_scalar(name); + } + +private: + struct WeightNormInfo { + std::string weight_v_name; + std::string weight_g_name; + std::vector weight_v_shape; + std::vector weight_g_shape; + int64_t out_channels = 0; + int64_t in_channels = 0; + int64_t kernel_size = 0; + }; + + void build_routes() { + // V1 -> V2 tensor name mapping + std::unordered_map rename_map = { + // LM embeddings + {"token_embd.weight", "base_lm.embed_tokens.weight"}, + // LM blocks + {"blk.", "base_lm.layers."}, + {"attn_q.weight", "self_attn.q_proj.weight"}, + {"attn_k.weight", "self_attn.k_proj.weight"}, + {"attn_v.weight", "self_attn.v_proj.weight"}, + {"attn_norm.weight", "input_layernorm.weight"}, + {"attn_output.weight", "self_attn.o_proj.weight"}, + {"ffn_norm.weight", "post_attention_layernorm.weight"}, + {"ffn_gate.weight", "mlp.gate_proj.weight"}, + {"ffn_up.weight", "mlp.up_proj.weight"}, + {"ffn_down.weight", "mlp.down_proj.weight"}, + // Output norm + {"output_norm.weight", "base_lm.norm.weight"}, + // Residual LM + {"residual_lm.blk.", "residual_lm.layers."}, + {"residual_lm.output_norm.weight", "residual_lm.norm.weight"}, + // Local encoder (feat_encoder) + {"locenc.in_proj.weight", "feat_encoder.in_proj.weight"}, + {"locenc.in_proj.bias", "feat_encoder.in_proj.bias"}, + {"locenc.special_token", "feat_encoder.special_token"}, + {"locenc.blk.", "feat_encoder.encoder.layers."}, + {"locenc.output_norm.weight", "feat_encoder.encoder.norm.weight"}, + // Local DiT (feat_decoder) + {"locdit.in_proj.weight", "feat_decoder.estimator.in_proj.weight"}, + {"locdit.in_proj.bias", "feat_decoder.estimator.in_proj.bias"}, + {"locdit.cond_proj.weight", "feat_decoder.estimator.cond_proj.weight"}, + {"locdit.cond_proj.bias", "feat_decoder.estimator.cond_proj.bias"}, + {"locdit.out_proj.weight", "feat_decoder.estimator.out_proj.weight"}, + {"locdit.out_proj.bias", "feat_decoder.estimator.out_proj.bias"}, + {"locdit.time_mlp.linear_1.weight", "feat_decoder.estimator.time_mlp.linear_1.weight"}, + {"locdit.time_mlp.linear_1.bias", "feat_decoder.estimator.time_mlp.linear_1.bias"}, + {"locdit.time_mlp.linear_2.weight", "feat_decoder.estimator.time_mlp.linear_2.weight"}, + {"locdit.time_mlp.linear_2.bias", "feat_decoder.estimator.time_mlp.linear_2.bias"}, + {"locdit.delta_time_mlp.linear_1.weight", "feat_decoder.estimator.delta_time_mlp.linear_1.weight"}, + {"locdit.delta_time_mlp.linear_1.bias", "feat_decoder.estimator.delta_time_mlp.linear_1.bias"}, + {"locdit.delta_time_mlp.linear_2.weight", "feat_decoder.estimator.delta_time_mlp.linear_2.weight"}, + {"locdit.delta_time_mlp.linear_2.bias", "feat_decoder.estimator.delta_time_mlp.linear_2.bias"}, + {"locdit.output_norm.weight", "feat_decoder.estimator.decoder.norm.weight"}, + {"locdit.blk.", "feat_decoder.estimator.decoder.layers."}, + // Projections + {"proj.enc_to_lm.weight", "enc_to_lm_proj.weight"}, + {"proj.enc_to_lm.bias", "enc_to_lm_proj.bias"}, + {"proj.lm_to_dit.weight", "lm_to_dit_proj.weight"}, + {"proj.lm_to_dit.bias", "lm_to_dit_proj.bias"}, + {"proj.res_to_dit.weight", "res_to_dit_proj.weight"}, + {"proj.res_to_dit.bias", "res_to_dit_proj.bias"}, + // V1→V2 mapping for fusion_concat_proj (critical for V1 models with fusion) + {"proj.fusion_concat.weight", "fusion_concat_proj.weight"}, + {"proj.fusion_concat.bias", "fusion_concat_proj.bias"}, + {"fusion_concat_proj.weight", "fusion_concat_proj.weight"}, + {"stop.stop_proj.weight", "stop_proj.weight"}, + {"stop.stop_proj.bias", "stop_proj.bias"}, + {"stop.stop_head.weight", "stop_head.weight"}, + // FSQ + {"fsq.in_proj.weight", "fsq_layer.in_proj.weight"}, + {"fsq.in_proj.bias", "fsq_layer.in_proj.bias"}, + {"fsq.out_proj.weight", "fsq_layer.out_proj.weight"}, + {"fsq.out_proj.bias", "fsq_layer.out_proj.bias"}, + // Audio VAE (prefixed with audio_vae.) + {"audio_vae.encoder.block.", "encoder.block."}, + {"audio_vae.encoder.fc_mu", "encoder.fc_mu"}, + {"audio_vae.decoder.model.", "decoder.model."}, + {"audio_vae.decoder.sr_cond_model.", "decoder.sr_cond_model."}, + }; + + // Build routes by scanning source tensors + for (const auto & tensor : source_->tensors()) { + std::string v1_name = tensor.name; + std::string v2_name = v1_name; + + // Apply prefix replacements + for (const auto & [from, to] : rename_map) { + if (v2_name.rfind(from, 0) == 0) { + v2_name = to + v2_name.substr(from.size()); + break; + } + } + + // Handle blk.N.* -> layers.N.* (base LM, residual LM, locenc, locdit) + constexpr std::string_view kBlk = "blk."; + const size_t blk_pos = v1_name.find(kBlk); + if (blk_pos != std::string::npos) { + const size_t layer_start = blk_pos + kBlk.size(); + const size_t dot = v1_name.find('.', layer_start); + if (dot != std::string::npos) { + const std::string layer_idx = v1_name.substr(layer_start, dot - layer_start); + const std::string rest = v1_name.substr(dot + 1); + if (v1_name.rfind("residual_lm.", 0) == 0) { + v2_name = "residual_lm.layers." + layer_idx + "." + rest; + } else if (v1_name.rfind("locenc.", 0) == 0) { + v2_name = "feat_encoder.encoder.layers." + layer_idx + "." + rest; + } else if (v1_name.rfind("locdit.", 0) == 0) { + v2_name = "feat_decoder.estimator.decoder.layers." + layer_idx + "." + rest; + } else { + v2_name = "base_lm.layers." + layer_idx + "." + rest; + } + // Further sub-replacements + for (const auto & [from, to] : rename_map) { + size_t pos = v2_name.find(from); + if (pos != std::string::npos) { + v2_name.replace(pos, from.size(), to); + } + } + } + } + + routes_[v2_name] = v1_name; + } + + // Reshape map + reshape_map_ = { + // feat_quant: {N, F} -> {N, F, 1} + // merge: {N, D} -> {N, D, 1} + // downsample/upsample: {out, in} -> {out, in, k, k} (k=3 for 3x3) + // V1 embedding: token_embd.weight [hidden, vocab] -> base_lm.embed_tokens.weight [vocab, hidden] + {"base_lm.embed_tokens.weight", {config_.lm.vocab_size, config_.lm.hidden_size}}, + }; + + // Folded AudioVAE conv weights: v1 GGUF stores weight-norm weights + // already folded into a single `.weight` tensor, while the v2 loader + // requests decomposed `.weight_v`/`.weight_g` names. Register every + // audio_vae conv weight so those logical names resolve to the folded + // data (weight_v) and its per-channel row norms (weight_g), which makes + // the loader's fold_weight_norm an exact identity. + if (is_v1_) { + std::vector> folded; + for (const auto & [logical, source] : routes_) { + if (source.rfind("audio_vae.", 0) == 0 && has_suffix(logical, ".weight")) { + folded.emplace_back( + logical.substr(0, logical.size() - 7), source); + } + } + for (const auto & [base, source] : folded) { + folded_convs_[base] = source; + } + } + + // Synthesized tensors for V1 + const int64_t encoder_hidden = config_.encoder.hidden_dim; + const int64_t feat_dim = config_.feat_dim; + const int64_t lm_hidden = config_.lm.hidden_size; + + // feat_encoder.scale_embed (identity buckets) + synthesized_tensors_["feat_encoder.scale_embed.weight"] = + assets::TensorMetadata{"feat_encoder.scale_embed.weight", "F32", {32, encoder_hidden}}; + synthesized_tensors_["feat_encoder.bias_embed.weight"] = + assets::TensorMetadata{"feat_encoder.bias_embed.weight", "F32", {32, encoder_hidden}}; + + // feat_encoder.fc_logvar (zeros) + synthesized_tensors_["feat_encoder.fc_logvar.weight"] = + assets::TensorMetadata{"feat_encoder.fc_logvar.weight", "F32", {feat_dim, encoder_hidden}}; + + // feat_encoder.diag (identity) + synthesized_tensors_["feat_encoder.diag"] = + assets::TensorMetadata{"feat_encoder.diag", "F32", {feat_dim}}; + + // feat_encoder.special_token: V1 GGUF stores as 1D [1024], model code handles reshaping + // Only synthesize if not present in GGUF + if (routes_.find("feat_encoder.special_token") == routes_.end()) { + synthesized_tensors_["feat_encoder.special_token"] = + assets::TensorMetadata{"feat_encoder.special_token", "F32", {encoder_hidden}}; + } + + // token_embd.extra_bias (from logit_scale or zeros) + if (routes_.find("token_embd.extra_bias") == routes_.end()) { + synthesized_tensors_["token_embd.extra_bias"] = + assets::TensorMetadata{"token_embd.extra_bias", "F32", {lm_hidden}}; + } + + // feat_encoder.merge (zeros) + if (routes_.find("feat_encoder.merge.weight") == routes_.end()) { + synthesized_tensors_["feat_encoder.merge.weight"] = + assets::TensorMetadata{"feat_encoder.merge.weight", "F32", {encoder_hidden, feat_dim}}; + } + + // Identity SR-condition embeddings for V1 decoder blocks. VoxCPM1 + // GGUFs contain no sr_cond_model tensors (no SR conditioning), but the + // shared decoder loader requires scale_embed/bias_embed. + { + const auto & vae = config_.audio_vae; + const size_t num_blocks = vae.decoder_rates.size(); + for (size_t i = 0; i < num_blocks; ++i) { + const int64_t input_channels = + vae.decoder_dim / (int64_t{1} << static_cast(i)); + const std::string prefix = + "decoder.sr_cond_model." + std::to_string(i + 2) + "."; + if (routes_.find(prefix + "scale_embed.weight") == routes_.end()) { + synthesized_tensors_[prefix + "scale_embed.weight"] = + assets::TensorMetadata{prefix + "scale_embed.weight", "F32", {1, input_channels}}; + } + if (routes_.find(prefix + "bias_embed.weight") == routes_.end()) { + synthesized_tensors_[prefix + "bias_embed.weight"] = + assets::TensorMetadata{prefix + "bias_embed.weight", "F32", {1, input_channels}}; + } + } + } + + // Missing projection weights for V1 (not in VoxCPM1 GGUF) + if (routes_.find("fusion_concat_proj.weight") == routes_.end()) { + synthesized_tensors_["fusion_concat_proj.weight"] = + assets::TensorMetadata{"fusion_concat_proj.weight", "F32", {lm_hidden, lm_hidden * 2}}; + } + if (routes_.find("fusion_concat_proj.bias") == routes_.end()) { + synthesized_tensors_["fusion_concat_proj.bias"] = + assets::TensorMetadata{"fusion_concat_proj.bias", "F32", {lm_hidden}}; + } + } + + std::vector fold_weight_norm( + const std::vector & weight_v, + const std::vector & weight_g, + int64_t out_channels, int64_t in_channels, int64_t kernel_size) const { + if (static_cast(weight_v.size()) != out_channels * in_channels * kernel_size || + static_cast(weight_g.size()) != out_channels) { + throw std::runtime_error("VoxCPM1 weight-norm shape mismatch"); + } + std::vector out(weight_v.size(), 0.0F); + for (int64_t d0 = 0; d0 < out_channels; ++d0) { + const size_t base = static_cast(d0 * in_channels * kernel_size); + double norm_sq = 0.0; + for (int64_t i = 0; i < in_channels * kernel_size; ++i) { + const double value = weight_v[base + static_cast(i)]; + norm_sq += value * value; + } + const float scale = weight_g[static_cast(d0)] / + static_cast(std::sqrt(norm_sq + 1e-8)); + for (int64_t i = 0; i < in_channels * kernel_size; ++i) { + out[base + static_cast(i)] = weight_v[base + static_cast(i)] * scale; + } + } + return out; + } + + assets::RawTensorData reshape_tensor_data(const assets::RawTensorData & data, + const std::vector &) const { + // For now, just return the data as-is (validation happens elsewhere) + // The actual reshape happens in require_f32 + return data; + } + + assets::RawTensorData generate_synthesized_tensor(std::string_view name) const { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it == synthesized_tensors_.end()) { + throw std::runtime_error("no synthesized tensor: " + std::string(name)); + } + const auto & metadata = it->second; + const int64_t num_elements = std::accumulate(metadata.shape.begin(), metadata.shape.end(), 1, std::multiplies()); + std::vector bytes(num_elements * sizeof(float)); + std::memset(bytes.data(), 0, bytes.size()); + return {metadata, std::move(bytes)}; + } + + std::vector generate_synthesized_f32(std::string_view name) const { + const auto it = synthesized_tensors_.find(std::string(name)); + if (it == synthesized_tensors_.end()) { + throw std::runtime_error("no synthesized tensor: " + std::string(name)); + } + const auto & metadata = it->second; + const int64_t num_elements = std::accumulate(metadata.shape.begin(), metadata.shape.end(), 1, std::multiplies()); + if (name == "feat_encoder.diag") { + std::vector out(num_elements, 1.0F); + return out; + } + if (std::string_view prefix = "decoder.sr_cond_model."; + name.rfind(prefix, 0) == 0 && has_suffix(name, ".scale_embed.weight")) { + return std::vector(num_elements, 1.0F); + } + if (name == "feat_encoder.scale_embed.weight" || name == "feat_encoder.bias_embed.weight") { + // Identity-like initialization + std::vector out(num_elements, 0.0F); + // Fill with small values + for (size_t i = 0; i < out.size(); ++i) { + out[i] = 0.01F; + } + return out; + } + if (name == "fusion_concat_proj.weight") { + // Xavier/Glorot initialization for fusion_concat_proj weight + // shape is [lm_hidden, lm_hidden * 2] + std::vector out(num_elements); + const float scale = std::sqrt(2.0f / (config_.lm.hidden_size + config_.lm.hidden_size * 2)); + for (size_t i = 0; i < out.size(); ++i) { + // Simple uniform distribution in [-scale, scale] + out[i] = (static_cast(std::rand()) / RAND_MAX * 2.0f - 1.0f) * scale; + } + return out; + } + return std::vector(num_elements, 0.0F); + } + + static bool has_suffix(std::string_view value, std::string_view suffix) { + return value.size() >= suffix.size() && + value.substr(value.size() - suffix.size()) == suffix; + } + + std::string folded_base_name(const std::string & key) const { + constexpr std::string_view kWeightV = ".weight_v"; + constexpr std::string_view kWeightG = ".weight_g"; + if (has_suffix(key, kWeightV)) { + return key.substr(0, key.size() - kWeightV.size()); + } + if (has_suffix(key, kWeightG)) { + return key.substr(0, key.size() - kWeightG.size()); + } + return ""; + } + + std::vector folded_weight_g( + const std::vector & folded, + const std::string & folded_source_name, + const std::optional> & expected_shape) const { + const auto meta = source_->require_metadata(folded_source_name); + const int64_t groups = expected_shape.has_value() && !expected_shape->empty() + ? expected_shape->front() + : (meta.shape.empty() ? 0 : meta.shape.front()); + const int64_t rows = checked_element_count(folded_source_name, meta.shape); + if (groups <= 0 || rows == 0 || rows % groups != 0) { + throw std::runtime_error("folded weight_g shape mismatch: " + folded_source_name); + } + const int64_t inner = rows / groups; + std::vector out(static_cast(groups), 0.0F); + for (int64_t g = 0; g < groups; ++g) { + double norm_sq = 0.0; + for (int64_t i = 0; i < inner; ++i) { + const float value = folded[static_cast(g * inner + i)]; + norm_sq += static_cast(value) * static_cast(value); + } + out[static_cast(g)] = static_cast(std::sqrt(norm_sq)); + } + return out; + } + + static int64_t checked_element_count(std::string_view name, const std::vector & shape) { + int64_t count = 1; + for (const int64_t dim : shape) { + if (dim <= 0) { + throw std::runtime_error("tensor shape contains a non-positive dimension: " + std::string(name)); + } + if (count > std::numeric_limits::max() / dim) { + throw std::runtime_error("tensor element count overflow: " + std::string(name)); + } + count *= dim; + } + return count; + } + + std::shared_ptr source_; + VoxCPM1Config config_; + bool is_v1_; + std::unordered_map routes_; + std::unordered_map> reshape_map_; + std::unordered_map weight_norm_map_; + std::unordered_map synthesized_tensors_; + std::unordered_map folded_convs_; +}; + +void require_vae_weight_v_shape(const assets::TensorSource & source, + std::string_view name, + const std::vector & expected_shape, + bool relaxed_rank) { + const auto metadata = source.require_metadata(name); + if (metadata.shape == expected_shape) { + return; + } + if (!relaxed_rank) { + throw std::runtime_error("tensor shape mismatch for " + std::string(name)); + } + int64_t expected_elems = 1; + for (const int64_t dim : expected_shape) { + expected_elems *= dim; + } + int64_t actual_elems = 1; + for (const int64_t dim : metadata.shape) { + actual_elems *= dim; + } + if (actual_elems != expected_elems) { + throw std::runtime_error("tensor element count mismatch for " + std::string(name)); + } +} + +void validate_weight_anchors(const VoxCPM1Assets & assets) { + const auto & config = assets.config; + const auto & weights = *assets.model_weights; + assets::require_tensor_shape(weights, "base_lm.embed_tokens.weight", {config.lm.vocab_size, config.lm.hidden_size}); + assets::require_tensor_shape(weights, "base_lm.norm.weight", {config.lm.hidden_size}); + assets::require_tensor_shape(weights, "base_lm.layers.0.self_attn.q_proj.weight", {config.lm.hidden_size, config.lm.hidden_size}); + assets::require_tensor_shape(weights, "base_lm.layers.0.self_attn.k_proj.weight", + {config.lm.num_key_value_heads * config.lm.kv_channels, config.lm.hidden_size}); + assets::require_tensor_shape(weights, "base_lm.layers.0.mlp.gate_proj.weight", {config.lm.intermediate_size, config.lm.hidden_size}); + assets::require_tensor_shape(weights, "residual_lm.norm.weight", {config.lm.hidden_size}); + require_vae_weight_v_shape(weights, "feat_encoder.special_token", {1, 1, 1, config.encoder.hidden_dim}, config.v1); + assets::require_tensor_shape(weights, "feat_encoder.in_proj.weight", {config.encoder.hidden_dim, config.feat_dim}); + assets::require_tensor_shape(weights, "feat_encoder.encoder.norm.weight", {config.encoder.hidden_dim}); + assets::require_tensor_shape(weights, "feat_decoder.estimator.in_proj.weight", {config.dit.hidden_dim, config.feat_dim}); + assets::require_tensor_shape(weights, "feat_decoder.estimator.cond_proj.weight", {config.dit.hidden_dim, config.feat_dim}); + assets::require_tensor_shape(weights, "feat_decoder.estimator.out_proj.weight", {config.feat_dim, config.dit.hidden_dim}); + assets::require_tensor_shape(weights, "feat_decoder.estimator.decoder.norm.weight", {config.dit.hidden_dim}); + assets::require_tensor_shape(weights, "fsq_layer.in_proj.weight", {config.scalar_quantization_latent_dim, config.lm.hidden_size}); + assets::require_tensor_shape(weights, "fsq_layer.out_proj.weight", {config.lm.hidden_size, config.scalar_quantization_latent_dim}); + assets::require_tensor_shape(weights, "enc_to_lm_proj.weight", {config.lm.hidden_size, config.encoder.hidden_dim}); + assets::require_tensor_shape(weights, "lm_to_dit_proj.weight", {config.dit.hidden_dim, config.lm.hidden_size}); + assets::require_tensor_shape(weights, "res_to_dit_proj.weight", {config.dit.hidden_dim, config.lm.hidden_size}); + assets::require_tensor_shape(weights, "fusion_concat_proj.weight", {config.lm.hidden_size, config.lm.hidden_size * 2}); + assets::require_tensor_shape(weights, "stop_proj.weight", {config.lm.hidden_size, config.lm.hidden_size}); + assets::require_tensor_shape(weights, "stop_head.weight", {2, config.lm.hidden_size}); + + const auto & vae = *assets.audiovae_weights; + int64_t encoder_in_channels = config.audio_vae.encoder_dim; + for (size_t i = 0; i < config.audio_vae.encoder_rates.size(); ++i) { + encoder_in_channels *= 2; + } + require_vae_weight_v_shape(vae, "encoder.fc_mu.weight_v", {config.audio_vae.latent_dim, encoder_in_channels, 3}, config.v1); + assets::require_tensor_shape(vae, "encoder.fc_mu.bias", {config.audio_vae.latent_dim}); + require_vae_weight_v_shape(vae, "decoder.model.0.weight_v", {config.audio_vae.latent_dim, 1, 7}, config.v1); + require_vae_weight_v_shape(vae, "decoder.model.1.weight_v", {config.audio_vae.decoder_dim, config.audio_vae.latent_dim, 1}, config.v1); +} + +} + +std::shared_ptr load_voxcpm1_assets(const std::filesystem::path & model_path) { + auto out = std::make_shared(); + out->resources = engine::model_spec::load_resource_bundle( + model_path, + engine::model_spec::default_spec_path("voxcpm1")); + + { + auto raw_model_weights = out->resources.open_tensor_source("weights"); + + bool has_tokenizer = VoxCPM1GgufTokenizer::has_tokenizer_metadata(*raw_model_weights); + bool has_config = has_voxcpm1_config_metadata(*raw_model_weights); + + if (has_tokenizer && has_config) { + out->config = load_voxcpm1_config_from_gguf(*raw_model_weights); + out->config.v1 = true; + out->gguf_tokenizer = std::make_shared(raw_model_weights); + } else { + out->config = parse_config(out->resources); + out->config.v1 = true; + } + } + + auto raw_model_weights = out->resources.open_tensor_source("weights"); + auto raw_audiovae_weights = out->resources.open_tensor_source("audiovae_weights"); + out->model_weights = std::make_shared(raw_model_weights, out->config, true); + out->audiovae_weights = std::make_shared(raw_audiovae_weights, out->config, true); + validate_weight_anchors(*out); + return out; +} + +} // namespace engine::community_models::voxcpm1 diff --git a/src/community_models/voxcpm1/audiovae.cpp b/src/community_models/voxcpm1/audiovae.cpp new file mode 100644 index 00000000..c1c9b458 --- /dev/null +++ b/src/community_models/voxcpm1/audiovae.cpp @@ -0,0 +1,1068 @@ +#include "engine/community_models/voxcpm1/audiovae.h" + +#include "engine/framework/audio/conversion.h" +#include "engine/framework/audio/resampling.h" +#include "engine/framework/audio/waveform_ops.h" +#include "engine/framework/core/execution_context.h" +#include "engine/framework/core/backend_weight_store.h" +#include "engine/framework/core/module.h" +#include "engine/framework/debug/profiler.h" +#include "engine/framework/modules/activation_modules.h" +#include "engine/framework/modules/conv_modules.h" +#include "engine/framework/modules/primitive_modules.h" +#include "engine/framework/modules/streaming_conv_modules.h" +#include "engine/framework/modules/structural_modules.h" + +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace engine::community_models::voxcpm1 { +namespace { + +namespace core = engine::core; +namespace modules = engine::modules; +namespace assets_ns = engine::assets; + +using Clock = std::chrono::steady_clock; + +constexpr int64_t kResidualKernel = 7; + +enum class PaddingMode { Left, Right }; + +std::vector trim_audio_silence_vad(const std::vector& input, + int sample_rate, + float max_silence_ms = 100.0f, + float top_db = 30.0f) { + if (input.empty() || sample_rate <= 0) { + return input; + } + + constexpr int kFrameLength = 2048; + constexpr int kHopLength = 512; + const float ref = *std::max_element(input.begin(), input.end(), [](float a, float b) { + return std::fabs(a) < std::fabs(b); + }); + if (std::fabs(ref) <= 0.0f) { + return input; + } + + const float threshold = std::fabs(ref) * std::pow(10.0f, -top_db / 20.0f); + const size_t n = input.size(); + int first_voice_frame = -1; + int last_voice_frame = -1; + + for (size_t idx = 0, frame = 0; idx < n; idx += kHopLength, ++frame) { + const size_t frame_end = std::min(idx + static_cast(kFrameLength), n); + const size_t frame_size = frame_end - idx; + if (frame_size == 0) { + break; + } + double energy = 0.0; + for (size_t i = idx; i < frame_end; ++i) { + energy += static_cast(input[i]) * static_cast(input[i]); + } + const float rms = static_cast(std::sqrt(energy / static_cast(frame_size))); + if (rms >= threshold) { + if (first_voice_frame < 0) { + first_voice_frame = static_cast(frame); + } + last_voice_frame = static_cast(frame); + } + if (frame_end == n) { + break; + } + } + + if (first_voice_frame < 0 || last_voice_frame < 0) { + return input; + } + + const int max_silence_samples = std::max(0, static_cast(std::lround(max_silence_ms * sample_rate / 1000.0f))); + const int start = std::max(0, first_voice_frame * kHopLength - max_silence_samples); + const int end = std::min(static_cast(n), + (last_voice_frame + 1) * kHopLength + (kFrameLength - kHopLength) + max_silence_samples); + if (start >= end) { + return input; + } + return std::vector(input.begin() + start, input.begin() + end); +} + +void pad_audio_for_patch_alignment(std::vector& audio, size_t patch_len, PaddingMode mode) { + if (patch_len == 0 || audio.empty() || (audio.size() % patch_len) == 0) { + return; + } + const size_t padding = patch_len - (audio.size() % patch_len); + if (mode == PaddingMode::Left) { + audio.insert(audio.begin(), padding, 0.0f); + } else { + audio.insert(audio.end(), padding, 0.0f); + } +} + +struct GgmlContextDeleter { + void operator()(ggml_context *ctx) const noexcept { + if (ctx != nullptr) { + ggml_free(ctx); + } + } +}; + +struct VAEConv1dWeights { + modules::Conv1dWeights regular; + modules::DepthwiseConv1dWeights depthwise; + int64_t in_channels = 0; + int64_t out_channels = 0; + int64_t kernel_size = 0; + bool depthwise_layout = false; +}; + +struct VAEConvTranspose1dWeights { + modules::ConvTranspose1dWeights conv; + int64_t in_channels = 0; + int64_t out_channels = 0; + int64_t kernel_size = 0; +}; + +struct VAESnakeWeights { + core::TensorValue alpha; +}; + +struct VAESampleRateConditionWeights { + core::TensorValue scale; + core::TensorValue bias; +}; + +struct VAEResidualUnitWeights { + VAESnakeWeights snake1; + VAEConv1dWeights conv1; + VAESnakeWeights snake2; + VAEConv1dWeights conv2; +}; + +struct VAEDecoderBlockWeights { + VAESampleRateConditionWeights sr_cond; + VAESnakeWeights snake; + VAEConvTranspose1dWeights upsample; + std::vector residual_units; + int64_t input_channels = 0; + int64_t output_channels = 0; + int stride = 1; +}; + +struct VAEEncoderBlockWeights { + std::vector residual_units; + VAESnakeWeights snake; + VAEConv1dWeights downsample; + int64_t input_channels = 0; + int64_t output_channels = 0; + int stride = 1; +}; + +struct VAEWeights { + std::shared_ptr store; + VAEConv1dWeights encoder_first; + std::vector encoder_blocks; + VAEConv1dWeights encoder_fc_mu; + VAEConv1dWeights decoder_first_depthwise; + VAEConv1dWeights decoder_first_pointwise; + std::vector decoder_blocks; + VAESnakeWeights decoder_final_snake; + VAEConv1dWeights decoder_final_conv; +}; + +int sample_rate_bucket(const VoxCPM1AudioVAEConfig &config) { + int bucket = 0; + while (bucket < static_cast(config.sample_rate_bin_boundaries.size()) && + config.output_sample_rate > + config.sample_rate_bin_boundaries[static_cast(bucket)]) { + ++bucket; + } + return bucket; +} + +std::vector fold_weight_norm(const std::vector &weight_v, + const std::vector &weight_g, + int64_t dim0, int64_t dim1, + int64_t kernel) { + if (static_cast(weight_v.size()) != dim0 * dim1 * kernel || + static_cast(weight_g.size()) != dim0) { + throw std::runtime_error("VoxCPM1 AudioVAE weight-norm shape mismatch"); + } + std::vector out(weight_v.size(), 0.0F); + for (int64_t d0 = 0; d0 < dim0; ++d0) { + const size_t base = static_cast(d0 * dim1 * kernel); + double norm_sq = 0.0; + for (int64_t i = 0; i < dim1 * kernel; ++i) { + const double value = weight_v[base + static_cast(i)]; + norm_sq += value * value; + } + const float scale = weight_g[static_cast(d0)] / + static_cast(std::sqrt(norm_sq)); + for (int64_t i = 0; i < dim1 * kernel; ++i) { + out[base + static_cast(i)] = + weight_v[base + static_cast(i)] * scale; + } + } + return out; +} + +std::vector squeeze_weight_g(const std::vector &values, + int64_t channels) { + if (static_cast(values.size()) != channels) { + throw std::runtime_error("VoxCPM1 AudioVAE weight_g shape mismatch"); + } + return values; +} + +VAEConv1dWeights load_wn_conv1d(core::BackendWeightStore &store, + const assets_ns::TensorSource &source, + const std::string &prefix, int64_t out_channels, + int64_t in_channels, int64_t kernel_size, + bool depthwise, + assets_ns::TensorStorageType storage_type) { + const int64_t stored_in = depthwise ? 1 : in_channels; + const auto weight_v = source.require_f32( + prefix + ".weight_v", {out_channels, stored_in, kernel_size}); + const auto weight_g = squeeze_weight_g( + source.require_f32(prefix + ".weight_g", {out_channels, 1, 1}), + out_channels); + const auto folded = fold_weight_norm(weight_v, weight_g, out_channels, + stored_in, kernel_size); + VAEConv1dWeights out; + out.in_channels = in_channels; + out.out_channels = out_channels; + out.kernel_size = kernel_size; + out.depthwise_layout = depthwise; + if (depthwise) { + out.depthwise.weight = store.make_from_f32( + core::TensorShape::from_dims({out_channels, 1, kernel_size}), + storage_type, folded); + out.depthwise.bias = + store.load_f32_tensor(source, prefix + ".bias", {out_channels}); + } else { + out.regular.weight = store.make_from_f32( + core::TensorShape::from_dims({out_channels, in_channels, kernel_size}), + storage_type, folded); + out.regular.bias = + store.load_f32_tensor(source, prefix + ".bias", {out_channels}); + } + return out; +} + +VAEConvTranspose1dWeights +load_wn_conv_transpose1d(core::BackendWeightStore &store, + const assets_ns::TensorSource &source, + const std::string &prefix, int64_t in_channels, + int64_t out_channels, int64_t kernel_size, + assets_ns::TensorStorageType storage_type) { + const auto weight_v = source.require_f32( + prefix + ".weight_v", {in_channels, out_channels, kernel_size}); + const auto weight_g = squeeze_weight_g( + source.require_f32(prefix + ".weight_g", {in_channels, 1, 1}), + in_channels); + VAEConvTranspose1dWeights out; + out.in_channels = in_channels; + out.out_channels = out_channels; + out.kernel_size = kernel_size; + out.conv.weight = store.make_from_f32( + core::TensorShape::from_dims({in_channels, out_channels, kernel_size}), + storage_type, + fold_weight_norm(weight_v, weight_g, in_channels, out_channels, + kernel_size)); + out.conv.bias = + store.load_f32_tensor(source, prefix + ".bias", {out_channels}); + return out; +} + +VAESnakeWeights load_snake(core::BackendWeightStore &store, + const assets_ns::TensorSource &source, + const std::string &name, int64_t channels) { + VAESnakeWeights out; + out.alpha = store.make_from_f32(core::TensorShape::from_dims({channels}), + assets_ns::TensorStorageType::F32, + source.require_f32(name, {1, channels, 1})); + return out; +} + +VAESampleRateConditionWeights load_sr_condition( + core::BackendWeightStore &store, const assets_ns::TensorSource &source, + const std::string &prefix, int64_t channels, int bucket, int buckets) { + const auto scale = + source.require_f32(prefix + ".scale_embed.weight", {buckets, channels}); + const auto bias = + source.require_f32(prefix + ".bias_embed.weight", {buckets, channels}); + const auto offset = static_cast(bucket * channels); + VAESampleRateConditionWeights out; + out.scale = store.make_from_f32( + core::TensorShape::from_dims({channels}), + assets_ns::TensorStorageType::F32, + std::vector(scale.begin() + offset, + scale.begin() + offset + channels)); + out.bias = + store.make_from_f32(core::TensorShape::from_dims({channels}), + assets_ns::TensorStorageType::F32, + std::vector(bias.begin() + offset, + bias.begin() + offset + channels)); + return out; +} + +VAEResidualUnitWeights load_residual_unit(core::BackendWeightStore &store, + const assets_ns::TensorSource &source, + const std::string &prefix, + int64_t channels, + assets_ns::TensorStorageType storage_type) { + VAEResidualUnitWeights out; + out.snake1 = load_snake(store, source, prefix + ".block.0.alpha", channels); + out.conv1 = load_wn_conv1d(store, source, prefix + ".block.1", channels, + channels, kResidualKernel, true, storage_type); + out.snake2 = load_snake(store, source, prefix + ".block.2.alpha", channels); + out.conv2 = load_wn_conv1d(store, source, prefix + ".block.3", channels, + channels, 1, false, storage_type); + return out; +} + +VAEEncoderBlockWeights load_encoder_block(core::BackendWeightStore &store, + const assets_ns::TensorSource &source, + const std::string &prefix, + int64_t input_channels, + int64_t output_channels, int stride, + assets_ns::TensorStorageType storage_type) { + VAEEncoderBlockWeights block; + block.input_channels = input_channels; + block.output_channels = output_channels; + block.stride = stride; + block.residual_units.push_back( + load_residual_unit(store, source, prefix + ".block.0", input_channels, + storage_type)); + block.residual_units.push_back( + load_residual_unit(store, source, prefix + ".block.1", input_channels, + storage_type)); + block.residual_units.push_back( + load_residual_unit(store, source, prefix + ".block.2", input_channels, + storage_type)); + block.snake = + load_snake(store, source, prefix + ".block.3.alpha", input_channels); + block.downsample = + load_wn_conv1d(store, source, prefix + ".block.4", output_channels, + input_channels, 2 * stride, false, storage_type); + return block; +} + +int64_t product(const std::vector &values) { + int64_t out = 1; + for (const int64_t value : values) { + if (value <= 0) { + throw std::runtime_error("VoxCPM1 AudioVAE rate must be positive"); + } + out *= value; + } + return out; +} + +VAEWeights load_vae_weights(const VoxCPM1Assets &assets, + core::ExecutionContext &execution_context, + size_t weight_context_bytes, + assets_ns::TensorStorageType storage_type) { + const auto &config = assets.config.audio_vae; + const auto &source = *assets.audiovae_weights; + VAEWeights weights; + weights.store = std::make_shared( + execution_context.backend(), execution_context.backend_type(), + "voxcpm1.audiovae.weights", weight_context_bytes); + auto &store = *weights.store; + weights.encoder_first = load_wn_conv1d(store, source, "encoder.block.0", + config.encoder_dim, 1, 7, false, + storage_type); + int64_t encoder_in_channels = config.encoder_dim; + weights.encoder_blocks.reserve(config.encoder_rates.size()); + for (size_t i = 0; i < config.encoder_rates.size(); ++i) { + const int64_t encoder_out_channels = encoder_in_channels * 2; + weights.encoder_blocks.push_back(load_encoder_block( + store, source, "encoder.block." + std::to_string(i + 1), + encoder_in_channels, encoder_out_channels, + static_cast(config.encoder_rates[i]), storage_type)); + encoder_in_channels = encoder_out_channels; + } + weights.encoder_fc_mu = + load_wn_conv1d(store, source, "encoder.fc_mu", config.latent_dim, + encoder_in_channels, 3, false, storage_type); + + weights.decoder_first_depthwise = + load_wn_conv1d(store, source, "decoder.model.0", config.latent_dim, + config.latent_dim, 7, true, storage_type); + weights.decoder_first_pointwise = + load_wn_conv1d(store, source, "decoder.model.1", config.decoder_dim, + config.latent_dim, 1, false, storage_type); + + const int bucket = sample_rate_bucket(config); + const int buckets = + static_cast(config.sample_rate_bin_boundaries.size()) + 1; + weights.decoder_blocks.reserve(config.decoder_rates.size()); + for (size_t i = 0; i < config.decoder_rates.size(); ++i) { + const int64_t input_channels = + config.decoder_dim / (int64_t{1} << static_cast(i)); + const int64_t output_channels = + config.decoder_dim / (int64_t{1} << static_cast(i + 1)); + const int model_index = static_cast(i) + 2; + const std::string prefix = "decoder.model." + std::to_string(model_index); + VAEDecoderBlockWeights block; + block.input_channels = input_channels; + block.output_channels = output_channels; + block.stride = static_cast(config.decoder_rates[i]); + block.sr_cond = load_sr_condition( + store, source, "decoder.sr_cond_model." + std::to_string(model_index), + input_channels, bucket, buckets); + block.snake = + load_snake(store, source, prefix + ".block.0.alpha", input_channels); + block.upsample = load_wn_conv_transpose1d( + store, source, prefix + ".block.1", input_channels, output_channels, + 2 * block.stride, storage_type); + block.residual_units.push_back(load_residual_unit( + store, source, prefix + ".block.2", output_channels, storage_type)); + block.residual_units.push_back(load_residual_unit( + store, source, prefix + ".block.3", output_channels, storage_type)); + block.residual_units.push_back(load_residual_unit( + store, source, prefix + ".block.4", output_channels, storage_type)); + weights.decoder_blocks.push_back(std::move(block)); + } + + const int64_t decoder_final_channels = + config.decoder_dim / + (int64_t{1} << static_cast(config.decoder_rates.size())); + weights.decoder_final_snake = + load_snake(store, source, + "decoder.model." + + std::to_string(config.decoder_rates.size() + 2) + ".alpha", + decoder_final_channels); + weights.decoder_final_conv = load_wn_conv1d( + store, source, + "decoder.model." + std::to_string(config.decoder_rates.size() + 3), 1, + decoder_final_channels, 7, false, storage_type); + store.upload(); + return weights; +} + + +std::shared_ptr +require_assets(std::shared_ptr assets) { + if (assets == nullptr) { + throw std::runtime_error("VoxCPM1 AudioVAE decoder requires assets"); + } + return assets; +} + +core::TensorValue zeros_like_prefix(core::ModuleBuildContext &ctx, + const core::TensorValue &input, + int64_t frames) { + if (frames <= 0) { + return {}; + } + auto prefix = + modules::RepeatModule( + {core::TensorShape::from_dims( + {input.shape.dims[0], input.shape.dims[1], frames})}) + .build(ctx, modules::SliceModule({2, 0, 1}).build(ctx, input)); + auto contiguous = core::ensure_backend_addressable_layout(ctx, prefix); + return core::wrap_tensor(ggml_scale(ctx.ggml, contiguous.tensor, 0.0F), + prefix.shape, GGML_TYPE_F32); +} + +core::TensorValue causal_pad_left(core::ModuleBuildContext &ctx, + const core::TensorValue &input, + int64_t frames) { + if (frames <= 0) { + return input; + } + return modules::ConcatModule({2}).build( + ctx, zeros_like_prefix(ctx, input, frames), input); +} + +core::TensorValue snake_exact(core::ModuleBuildContext &ctx, + const core::TensorValue &input, + const VAESnakeWeights &weights, + int64_t channels) { + const auto input_f32 = core::ensure_backend_addressable_layout(ctx, input); + auto alpha = core::reshape_tensor( + ctx, weights.alpha, core::TensorShape::from_dims({1, channels, 1})); + alpha = + core::wrap_tensor(ggml_repeat(ctx.ggml, alpha.tensor, input_f32.tensor), + input.shape, GGML_TYPE_F32); + auto ax = + core::wrap_tensor(ggml_mul(ctx.ggml, input_f32.tensor, alpha.tensor), + input.shape, GGML_TYPE_F32); + auto s = core::wrap_tensor(ggml_sin(ctx.ggml, ax.tensor), input.shape, + GGML_TYPE_F32); + auto s2 = core::wrap_tensor(ggml_mul(ctx.ggml, s.tensor, s.tensor), + input.shape, GGML_TYPE_F32); + auto denom = + core::wrap_tensor(ggml_scale_bias(ctx.ggml, alpha.tensor, 1.0F, 1.0e-9F), + input.shape, GGML_TYPE_F32); + auto frac = core::wrap_tensor(ggml_div(ctx.ggml, s2.tensor, denom.tensor), + input.shape, GGML_TYPE_F32); + return core::wrap_tensor(ggml_add(ctx.ggml, input_f32.tensor, frac.tensor), + input.shape, GGML_TYPE_F32); +} + +core::TensorValue apply_sr_condition( + core::ModuleBuildContext &ctx, const core::TensorValue &input, + const VAESampleRateConditionWeights &weights, int64_t channels) { + const auto input_f32 = core::ensure_backend_addressable_layout(ctx, input); + auto scale = core::reshape_tensor( + ctx, weights.scale, core::TensorShape::from_dims({1, channels, 1})); + scale = + core::wrap_tensor(ggml_repeat(ctx.ggml, scale.tensor, input_f32.tensor), + input.shape, GGML_TYPE_F32); + auto bias = core::reshape_tensor( + ctx, weights.bias, core::TensorShape::from_dims({1, channels, 1})); + bias = core::wrap_tensor(ggml_repeat(ctx.ggml, bias.tensor, input_f32.tensor), + input.shape, GGML_TYPE_F32); + auto scaled = + core::wrap_tensor(ggml_mul(ctx.ggml, input_f32.tensor, scale.tensor), + input.shape, GGML_TYPE_F32); + return core::wrap_tensor(ggml_add(ctx.ggml, scaled.tensor, bias.tensor), + input.shape, GGML_TYPE_F32); +} + +core::TensorValue causal_conv1d(core::ModuleBuildContext &ctx, + const core::TensorValue &input, + const VAEConv1dWeights &weights, int stride, + int padding, int dilation, + int output_padding = 0) { + const int left_pad = 2 * padding - output_padding; + if (left_pad < 0) { + throw std::runtime_error( + "VoxCPM1 AudioVAE causal convolution padding is invalid"); + } + auto padded = causal_pad_left(ctx, input, left_pad); + if (weights.depthwise_layout) { + return modules::DepthwiseConv1dModule( + {weights.out_channels, weights.kernel_size, stride, 0, dilation, + weights.depthwise.bias.has_value()}) + .build(ctx, padded, weights.depthwise); + } + return modules::Conv1dModule({weights.in_channels, weights.out_channels, + weights.kernel_size, stride, 0, dilation, + weights.regular.bias.has_value()}) + .build(ctx, padded, weights.regular); +} + +core::TensorValue +causal_conv_transpose1d(core::ModuleBuildContext &ctx, + const core::TensorValue &input, + const VAEConvTranspose1dWeights &weights, int stride) { + auto full = + modules::ConvTranspose1dModule({weights.in_channels, weights.out_channels, + weights.kernel_size, stride, 0, 1, + weights.conv.bias.has_value()}) + .build(ctx, input, weights.conv); + const int64_t frames = input.shape.dims[2] * stride; + auto view = ggml_view_3d(ctx.ggml, full.tensor, frames, weights.out_channels, + 1, full.tensor->nb[1], full.tensor->nb[2], 0); + return core::wrap_tensor( + ggml_cont(ctx.ggml, view), + core::TensorShape::from_dims({1, weights.out_channels, frames}), + GGML_TYPE_F32); +} + +core::TensorValue residual_unit(core::ModuleBuildContext &ctx, + const core::TensorValue &input, + const VAEResidualUnitWeights &weights, + int dilation) { + const int padding = static_cast(((kResidualKernel - 1) * dilation) / 2); + auto hidden = snake_exact(ctx, input, weights.snake1, input.shape.dims[1]); + hidden = causal_conv1d(ctx, hidden, weights.conv1, 1, padding, dilation); + hidden = snake_exact(ctx, hidden, weights.snake2, input.shape.dims[1]); + hidden = causal_conv1d(ctx, hidden, weights.conv2, 1, 0, 1); + return modules::AddModule{}.build(ctx, input, hidden); +} + +core::TensorValue encoder_block(core::ModuleBuildContext &ctx, + const core::TensorValue &input, + const VAEEncoderBlockWeights &weights) { + auto hidden = residual_unit(ctx, input, weights.residual_units[0], 1); + hidden = residual_unit(ctx, hidden, weights.residual_units[1], 3); + hidden = residual_unit(ctx, hidden, weights.residual_units[2], 9); + hidden = snake_exact(ctx, hidden, weights.snake, weights.input_channels); + const int padding = static_cast((weights.stride + 1) / 2); + return causal_conv1d(ctx, hidden, weights.downsample, weights.stride, padding, + 1); +} + +core::TensorValue decoder_block(core::ModuleBuildContext &ctx, + const core::TensorValue &input, + const VAEDecoderBlockWeights &weights) { + auto hidden = + apply_sr_condition(ctx, input, weights.sr_cond, weights.input_channels); + hidden = snake_exact(ctx, hidden, weights.snake, weights.input_channels); + hidden = + causal_conv_transpose1d(ctx, hidden, weights.upsample, weights.stride); + hidden = residual_unit(ctx, hidden, weights.residual_units[0], 1); + hidden = residual_unit(ctx, hidden, weights.residual_units[1], 3); + hidden = residual_unit(ctx, hidden, weights.residual_units[2], 9); + return hidden; +} + +} // namespace + +class VoxCPM1AudioVAEDecoderRuntime::Impl { +public: + Impl(std::shared_ptr assets, + core::ExecutionContext &execution_context, + VoxCPM1AudioVAEDecoderConfig config) + : assets_(require_assets(std::move(assets))), + execution_context_(execution_context), config_(config), + weights_(load_vae_weights(*assets_, execution_context_, + config_.weight_context_bytes, + config_.weight_storage_type)) { + if (config_.latent_frame_capacity < 0) { + throw std::runtime_error( + "VoxCPM1 AudioVAE latent frame capacity must be non-negative"); + } + if (config_.encoder_sample_capacity <= 0) { + throw std::runtime_error( + "VoxCPM1 AudioVAE encoder sample capacity must be positive"); + } + } + + ~Impl() { + release_decoder_graph(); + release_encoder_graph(); + } + + runtime::AudioBuffer decode_features(const std::vector &features, + int64_t patches) { + const auto &vae = assets_->config.audio_vae; + if (patches < 0) { + throw std::runtime_error("VoxCPM1 AudioVAE patch count is negative"); + } + const int64_t latent_frames = patches * assets_->config.patch_size; + const int64_t expected = latent_frames * vae.latent_dim; + if (static_cast(features.size()) != expected) { + throw std::runtime_error("VoxCPM1 AudioVAE feature size mismatch"); + } + ensure_decoder_graph(latent_frames); + std::vector input( + static_cast(vae.latent_dim * decoder_latent_frame_capacity_), + 0.0F); + for (int64_t t = 0; t < latent_frames; ++t) { + for (int64_t c = 0; c < vae.latent_dim; ++c) { + input[static_cast(c * decoder_latent_frame_capacity_ + t)] = + features[static_cast(t * vae.latent_dim + c)]; + } + } + ggml_backend_tensor_set(input_, input.data(), 0, + input.size() * sizeof(float)); + core::set_backend_threads(execution_context_.backend(), + std::max(1, execution_context_.config().threads)); + const ggml_status status = + core::compute_backend_graph(execution_context_.backend(), graph_); + ggml_backend_synchronize(execution_context_.backend()); + if (status != GGML_STATUS_SUCCESS) { + throw std::runtime_error("VoxCPM1 AudioVAE decoder graph compute failed"); + } + const int64_t sample_count = latent_frames * decoder_stride_; + std::vector full(static_cast(output_frames_), 0.0F); + ggml_backend_tensor_get(output_, full.data(), 0, + full.size() * sizeof(float)); + runtime::AudioBuffer audio; + audio.sample_rate = vae.output_sample_rate; + audio.channels = 1; + audio.samples.assign( + full.begin(), full.begin() + static_cast(sample_count)); + return audio; + } + + VoxCPM1EncodedPrompt encode_prompt_audio( + const std::optional &prompt_audio, + const std::string &prompt_text, + const std::optional &reference_audio) { + VoxCPM1EncodedPrompt out; + if (prompt_audio.has_value()) { + if (prompt_text.empty()) { + throw std::runtime_error( + "VoxCPM1 prompt audio requires prompt_text or reference_text"); + } + out.prompt_text = prompt_text; + auto encoded = encode_audio(*prompt_audio, true); + out.prompt_features = std::move(encoded.features); + out.prompt_patches = encoded.patches; + } + if (reference_audio.has_value()) { + auto encoded = encode_audio(*reference_audio, false); + out.reference_features = std::move(encoded.features); + out.reference_patches = encoded.patches; + } + return out; + } + + void release_runtime_memory() { + release_decoder_graph(); + release_encoder_graph_impl(); + } + + void release_encoder_graph() { release_encoder_graph_impl(); } + +private: + struct EncodedFeatures { + std::vector features; + int64_t patches = 0; + }; + + EncodedFeatures encode_audio(const runtime::AudioBuffer &audio, + bool left_pad) { + ensure_encoder_graph(); + const auto &vae = assets_->config.audio_vae; + auto mono = engine::audio::mixdown_interleaved_to_mono_average( + audio.samples, audio.channels); + if (audio.sample_rate != vae.sample_rate) { + engine::audio::SoxrResampleOptions options; + options.profile = + engine::audio::SoxrResampleProfile::ExplicitFloat32Runtime; + options.output_length_policy = + engine::audio::SoxrOutputLengthPolicy::ExactExpected; + options.output_padding = 256; + options.require_full_input = true; + options.reject_empty_output = true; + options.warning_context = "VoxCPM1 AudioVAE encoder"; + options.fallback_description = "linear resampling"; + mono = engine::audio::resample_mono_soxr_or_linear( + mono, audio.sample_rate, vae.sample_rate, options); + } + // VAD trim silence (match VoxCPM.cpp server_common.cpp:842/878) + mono = trim_audio_silence_vad(mono, vae.sample_rate); + if (const char *dump_path = std::getenv("VOXCPM_DUMP_REF_MONO")) { + FILE *f = std::fopen(dump_path, "wb"); + if (f != nullptr) { + std::fwrite(mono.data(), sizeof(float), mono.size(), f); + std::fclose(f); + } + } + // Patch-aligned padding (Left for prompt, Right for reference) + const int64_t patch_samples = assets_->config.patch_size * encoder_stride_; + pad_audio_for_patch_alignment(mono, static_cast(patch_samples), + left_pad ? PaddingMode::Left : PaddingMode::Right); + // Final padding to encoder_sample_capacity + const int64_t sample_count = static_cast(mono.size()); + const int64_t padded_samples = + ((sample_count + patch_samples - 1) / patch_samples) * patch_samples; + if (patch_samples <= 0) { + throw std::runtime_error("VoxCPM1 AudioVAE patch sample size is invalid"); + } + if (padded_samples > config_.encoder_sample_capacity) { + throw std::runtime_error( + "VoxCPM1 AudioVAE encoder sample capacity exceeded"); + } + std::vector input( + static_cast(config_.encoder_sample_capacity), 0.0F); + const int64_t offset = left_pad ? padded_samples - sample_count : 0; + std::copy(mono.begin(), mono.end(), + input.begin() + static_cast(offset)); + ggml_backend_tensor_set(encoder_input_, input.data(), 0, + input.size() * sizeof(float)); + core::set_backend_threads(execution_context_.backend(), + std::max(1, execution_context_.config().threads)); + const ggml_status status = core::compute_backend_graph( + execution_context_.backend(), encoder_graph_); + ggml_backend_synchronize(execution_context_.backend()); + if (status != GGML_STATUS_SUCCESS) { + throw std::runtime_error("VoxCPM1 AudioVAE encoder graph compute failed"); + } + if (const char *stage_path = std::getenv("VOXCPM_DUMP_ENC_STAGE")) { + const std::string dir(stage_path); + for (size_t i = 0; i < encoder_stages_.size(); ++i) { + ggml_tensor *stage = encoder_stages_[i]; + std::vector buf(static_cast(ggml_nelements(stage)), 0.0F); + ggml_backend_tensor_get(stage, buf.data(), 0, buf.size() * sizeof(float)); + FILE *f = std::fopen((dir + "/stage_" + std::to_string(i) + ".bin").c_str(), "wb"); + if (f != nullptr) { + std::fwrite(buf.data(), sizeof(float), buf.size(), f); + std::fclose(f); + } + } + } + + const int64_t latent_frames = padded_samples / encoder_stride_; + const int64_t expected_capacity_frames = + config_.encoder_sample_capacity / encoder_stride_; + std::vector full( + static_cast(vae.latent_dim * expected_capacity_frames), 0.0F); + ggml_backend_tensor_get(encoder_output_, full.data(), 0, + full.size() * sizeof(float)); + if (latent_frames % assets_->config.patch_size != 0) { + throw std::runtime_error( + "VoxCPM1 AudioVAE encoded frames are not divisible by patch size"); + } + EncodedFeatures encoded; + encoded.patches = latent_frames / assets_->config.patch_size; + encoded.features.resize(static_cast(latent_frames * vae.latent_dim), + 0.0F); + for (int64_t t = 0; t < latent_frames; ++t) { + for (int64_t c = 0; c < vae.latent_dim; ++c) { + encoded.features[static_cast(t * vae.latent_dim + c)] = + full[static_cast(c * expected_capacity_frames + t)]; + } + } + if (const char *dump_path = std::getenv("VOXCPM_DUMP_REF_FEAT")) { + FILE *f = std::fopen(dump_path, "wb"); + if (f != nullptr) { + std::fwrite(encoded.features.data(), sizeof(float), + encoded.features.size(), f); + std::fclose(f); + } + } + return encoded; + } + + void ensure_encoder_graph() { + if (encoder_graph_ != nullptr) { + return; + } + build_encoder(); + } + + int64_t decoder_capacity_for(int64_t latent_frames) const { + const int64_t min_capacity = + config_.latent_frame_capacity > 0 ? config_.latent_frame_capacity + : assets_->config.patch_size; + int64_t capacity = std::max(min_capacity, assets_->config.patch_size); + while (capacity < latent_frames) { + capacity *= 2; + } + return capacity; + } + + void ensure_decoder_graph(int64_t latent_frames) { + if (graph_ != nullptr && latent_frames <= decoder_latent_frame_capacity_) { + engine::debug::timing_log_scalar( + "voxcpm1.audiovae.decoder.graph.rebuilt", false); + engine::debug::timing_log_scalar( + "voxcpm1.audiovae.decoder.graph.reused", true); + engine::debug::timing_log_scalar( + "voxcpm1.audiovae.decoder.graph.build_ms", 0.0); + engine::debug::timing_log_scalar( + "voxcpm1.audiovae.decoder.latent_capacity", + decoder_latent_frame_capacity_); + return; + } + const auto build_start = Clock::now(); + build_decoder(decoder_capacity_for(latent_frames)); + engine::debug::timing_log_scalar( + "voxcpm1.audiovae.decoder.graph.rebuilt", true); + engine::debug::timing_log_scalar( + "voxcpm1.audiovae.decoder.graph.reused", false); + engine::debug::timing_log_scalar( + "voxcpm1.audiovae.decoder.graph.build_ms", + engine::debug::elapsed_ms(build_start)); + engine::debug::timing_log_scalar( + "voxcpm1.audiovae.decoder.latent_capacity", + decoder_latent_frame_capacity_); + } + + void release_decoder_graph() { + if (graph_ != nullptr) { + core::release_backend_graph_resources(execution_context_.backend(), graph_); + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + graph_ = nullptr; + input_ = nullptr; + output_ = nullptr; + ctx_.reset(); + output_frames_ = 0; + decoder_latent_frame_capacity_ = 0; + } + + void build_decoder(int64_t latent_frame_capacity) { + const auto &vae = assets_->config.audio_vae; + if (latent_frame_capacity <= 0) { + throw std::runtime_error( + "VoxCPM1 AudioVAE decoder graph capacity must be positive"); + } + release_decoder_graph(); + if (config_.graph_context_bytes == 0) { + throw std::runtime_error( + "VoxCPM1 AudioVAE graph context bytes must be non-zero"); + } + decoder_stride_ = product(vae.decoder_rates); + output_frames_ = latent_frame_capacity * decoder_stride_; + ggml_init_params params{config_.graph_context_bytes, nullptr, true}; + ctx_.reset(ggml_init(params)); + if (ctx_ == nullptr) { + throw std::runtime_error( + "failed to initialize VoxCPM1 AudioVAE decoder graph context"); + } + core::ModuleBuildContext ctx{ctx_.get(), "voxcpm1.audiovae.decoder", + execution_context_.backend_type()}; + auto hidden = core::make_tensor( + ctx, GGML_TYPE_F32, + core::TensorShape::from_dims( + {1, vae.latent_dim, latent_frame_capacity})); + input_ = hidden.tensor; + ggml_set_input(input_); + hidden = + causal_conv1d(ctx, hidden, weights_.decoder_first_depthwise, 1, 3, 1); + hidden = + causal_conv1d(ctx, hidden, weights_.decoder_first_pointwise, 1, 0, 1); + for (const auto &block : weights_.decoder_blocks) { + hidden = decoder_block(ctx, hidden, block); + } + hidden = snake_exact(ctx, hidden, weights_.decoder_final_snake, + hidden.shape.dims[1]); + hidden = causal_conv1d(ctx, hidden, weights_.decoder_final_conv, 1, 3, 1); + hidden = modules::TanhModule{}.build(ctx, hidden); + output_ = hidden.tensor; + ggml_set_output(output_); + graph_ = ggml_new_graph_custom(ctx_.get(), 65536, false); + ggml_build_forward_expand(graph_, output_); + gallocr_ = ggml_gallocr_new( + ggml_backend_get_default_buffer_type(execution_context_.backend())); + if (gallocr_ == nullptr || !ggml_gallocr_reserve(gallocr_, graph_) || + !ggml_gallocr_alloc_graph(gallocr_, graph_)) { + release_decoder_graph(); + throw std::runtime_error("failed to allocate VoxCPM1 AudioVAE graph"); + } + decoder_latent_frame_capacity_ = latent_frame_capacity; + } + + void release_encoder_graph_impl() { + if (encoder_graph_ != nullptr) { + core::release_backend_graph_resources(execution_context_.backend(), + encoder_graph_); + } + if (encoder_gallocr_ != nullptr) { + ggml_gallocr_free(encoder_gallocr_); + encoder_gallocr_ = nullptr; + } + encoder_graph_ = nullptr; + encoder_input_ = nullptr; + encoder_output_ = nullptr; + encoder_ctx_.reset(); + } + + void build_encoder() { + const auto &vae = assets_->config.audio_vae; + release_encoder_graph(); + if (config_.encoder_graph_context_bytes == 0) { + throw std::runtime_error( + "VoxCPM1 AudioVAE encoder graph context bytes must be non-zero"); + } + encoder_stride_ = product(vae.encoder_rates); + if (config_.encoder_sample_capacity % encoder_stride_ != 0) { + throw std::runtime_error("VoxCPM1 AudioVAE encoder sample capacity must " + "be divisible by encoder stride"); + } + ggml_init_params params{config_.encoder_graph_context_bytes, nullptr, true}; + encoder_ctx_.reset(ggml_init(params)); + if (encoder_ctx_ == nullptr) { + throw std::runtime_error( + "failed to initialize VoxCPM1 AudioVAE encoder graph context"); + } + core::ModuleBuildContext ctx{encoder_ctx_.get(), "voxcpm1.audiovae.encoder", + execution_context_.backend_type()}; + auto hidden = core::make_tensor( + ctx, GGML_TYPE_F32, + core::TensorShape::from_dims({1, 1, config_.encoder_sample_capacity})); + encoder_input_ = hidden.tensor; + ggml_set_input(encoder_input_); + encoder_stages_.clear(); + if (std::getenv("VOXCPM_DUMP_ENC_STAGE") != nullptr) { + encoder_stages_.push_back(encoder_input_); + } + hidden = causal_conv1d(ctx, hidden, weights_.encoder_first, 1, 3, 1); + if (std::getenv("VOXCPM_DUMP_ENC_STAGE") != nullptr) { + encoder_stages_.push_back(hidden.tensor); + } + for (const auto &block : weights_.encoder_blocks) { + hidden = encoder_block(ctx, hidden, block); + if (std::getenv("VOXCPM_DUMP_ENC_STAGE") != nullptr) { + encoder_stages_.push_back(hidden.tensor); + } + } + hidden = causal_conv1d(ctx, hidden, weights_.encoder_fc_mu, 1, 1, 1); + encoder_output_ = hidden.tensor; + ggml_set_output(encoder_output_); + for (ggml_tensor *stage : encoder_stages_) { + ggml_set_output(stage); + } + encoder_graph_ = ggml_new_graph_custom(encoder_ctx_.get(), 65536, false); + ggml_build_forward_expand(encoder_graph_, encoder_output_); + for (ggml_tensor *stage : encoder_stages_) { + ggml_build_forward_expand(encoder_graph_, stage); + } + encoder_gallocr_ = ggml_gallocr_new( + ggml_backend_get_default_buffer_type(execution_context_.backend())); + if (encoder_gallocr_ == nullptr || + !ggml_gallocr_reserve(encoder_gallocr_, encoder_graph_) || + !ggml_gallocr_alloc_graph(encoder_gallocr_, encoder_graph_)) { + release_encoder_graph(); + throw std::runtime_error( + "failed to allocate VoxCPM1 AudioVAE encoder graph"); + } + } + + std::shared_ptr assets_; + core::ExecutionContext &execution_context_; + VoxCPM1AudioVAEDecoderConfig config_; + VAEWeights weights_; + std::unique_ptr ctx_; + std::unique_ptr encoder_ctx_; + ggml_tensor *input_ = nullptr; + ggml_tensor *output_ = nullptr; + ggml_tensor *encoder_input_ = nullptr; + ggml_tensor *encoder_output_ = nullptr; + std::vector encoder_stages_; + ggml_cgraph *graph_ = nullptr; + ggml_cgraph *encoder_graph_ = nullptr; + ggml_gallocr_t gallocr_ = nullptr; + ggml_gallocr_t encoder_gallocr_ = nullptr; + int64_t decoder_stride_ = 0; + int64_t encoder_stride_ = 0; + int64_t output_frames_ = 0; + int64_t decoder_latent_frame_capacity_ = 0; +}; + +VoxCPM1AudioVAEDecoderRuntime::VoxCPM1AudioVAEDecoderRuntime( + std::shared_ptr assets, + core::ExecutionContext &execution_context, + VoxCPM1AudioVAEDecoderConfig config) + : impl_(std::make_unique(std::move(assets), execution_context, + std::move(config))) {} + +VoxCPM1AudioVAEDecoderRuntime::~VoxCPM1AudioVAEDecoderRuntime() = default; + +runtime::AudioBuffer VoxCPM1AudioVAEDecoderRuntime::decode_features( + const std::vector &features, int64_t patches) { + return impl_->decode_features(features, patches); +} + +VoxCPM1EncodedPrompt VoxCPM1AudioVAEDecoderRuntime::encode_prompt_audio( + const std::optional &prompt_audio, + const std::string &prompt_text, + const std::optional &reference_audio) { + return impl_->encode_prompt_audio(prompt_audio, prompt_text, reference_audio); +} + +void VoxCPM1AudioVAEDecoderRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + +void VoxCPM1AudioVAEDecoderRuntime::release_encoder_graph() { + impl_->release_encoder_graph(); +} + +} // namespace engine::community_models::voxcpm1 diff --git a/src/models/voxcpm2/config_gguf.cpp b/src/community_models/voxcpm1/config_gguf.cpp similarity index 96% rename from src/models/voxcpm2/config_gguf.cpp rename to src/community_models/voxcpm1/config_gguf.cpp index 830b86ee..70800d76 100644 --- a/src/models/voxcpm2/config_gguf.cpp +++ b/src/community_models/voxcpm1/config_gguf.cpp @@ -1,12 +1,12 @@ -#include "engine/models/voxcpm2/config_gguf.h" +#include "engine/community_models/voxcpm1/config_gguf.h" #include "engine/framework/assets/tensor_source.h" -#include "engine/models/voxcpm2/gguf_metadata.h" +#include "engine/community_models/voxcpm1/gguf_metadata.h" #include #include -namespace engine::models::voxcpm2 { +namespace engine::community_models::voxcpm1 { bool has_voxcpm1_config_metadata(const engine::assets::TensorSource & source) { const GgufMetadataReader metadata(source); @@ -16,8 +16,8 @@ bool has_voxcpm1_config_metadata(const engine::assets::TensorSource & source) { metadata.optional_u32("voxcpm_lm_config_hidden_size").has_value(); } -VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & source) { - VoxCPM2Config config; +VoxCPM1Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & source) { + VoxCPM1Config config; const GgufMetadataReader metadata(source); config.v1 = true; config.architecture = "voxcpm"; @@ -186,4 +186,4 @@ VoxCPM2Config load_voxcpm1_config_from_gguf(const engine::assets::TensorSource & return config; } -} // namespace engine::models::voxcpm2 \ No newline at end of file +} // namespace engine::community_models::voxcpm1 \ No newline at end of file diff --git a/src/community_models/voxcpm1/generator.cpp b/src/community_models/voxcpm1/generator.cpp new file mode 100644 index 00000000..13b1e74b --- /dev/null +++ b/src/community_models/voxcpm1/generator.cpp @@ -0,0 +1,2048 @@ +#include "engine/community_models/voxcpm1/generator.h" + +#include "minicpm_blocks.h" + +#include "engine/framework/core/execution_context.h" +#include "engine/framework/debug/profiler.h" +#include "engine/framework/io/binary.h" +#include "engine/framework/modules/activation_modules.h" +#include "engine/framework/modules/linear_module.h" +#include "engine/framework/modules/primitive_modules.h" +#include "engine/framework/modules/structural_modules.h" +#include "engine/framework/modules/weight_binding.h" +#include "engine/framework/runtime/cache_slots.h" +#include "engine/framework/runtime/errors.h" +#include "engine/framework/sampling/torch_random.h" +#include "engine/community_models/voxcpm1/assets.h" +#include "engine/community_models/voxcpm1/minicpm.h" +#include "engine/community_models/voxcpm1/tokenizer_text.h" +#include "engine/community_models/voxcpm1/tokenizer_wrapper.h" + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace engine::community_models::voxcpm1 { +namespace { + +namespace binding = engine::modules::binding; + +using Clock = std::chrono::steady_clock; + +constexpr int32_t kRefAudioStartToken = 103; +constexpr int32_t kRefAudioEndToken = 104; +constexpr int64_t kStreamingPrefixLen = 4; + +struct PrefillRow { + int32_t token = 0; + std::vector feature; + std::vector embedding; + bool text_mask = false; + bool audio_mask = false; +}; + +struct PrefillSequence { + std::vector rows; + int64_t target_text_tokens = 0; +}; + +std::shared_ptr +require_assets(std::shared_ptr assets) { + if (assets == nullptr) { + throw std::runtime_error("VoxCPM1 feature generator requires assets"); + } + return assets; +} + +void validate_generation_options(const VoxCPM1GenerationOptions &options) { + if (options.min_tokens < 0) { + throw std::runtime_error("VoxCPM1 min_tokens must be non-negative"); + } + if (options.max_tokens < 0) { + throw std::runtime_error("VoxCPM1 max_tokens must be non-negative"); + } + if (options.num_inference_steps <= 0) { + throw std::runtime_error( + "VoxCPM1 num_inference_steps must be positive"); + } + if (!std::isfinite(options.guidance_scale)) { + throw std::runtime_error("VoxCPM1 guidance_scale must be finite"); + } + if (options.retry_badcase_max_times <= 0) { + throw std::runtime_error( + "VoxCPM1 retry_badcase_max_times must be positive"); + } + if (!std::isfinite(options.retry_badcase_ratio_threshold) || + options.retry_badcase_ratio_threshold <= 0.0F) { + throw std::runtime_error( + "VoxCPM1 retry_badcase_ratio_threshold must be positive and finite"); + } +} + +int64_t effective_max_tokens(const VoxCPM1GenerationOptions &options, + int64_t target_text_tokens) { + const auto ratio_bound = + static_cast(static_cast(target_text_tokens) * + options.retry_badcase_ratio_threshold + + 10.0F); + return std::min(ratio_bound, options.max_tokens); +} + +int stop_class(const std::vector &logits) { + if (logits.size() != 2) { + throw std::runtime_error("VoxCPM1 stop logits must have two classes"); + } + return logits[1] > logits[0] ? 1 : 0; +} + +std::vector concat_dit_mu(const std::vector &lm, + const std::vector &residual) { + std::vector out; + out.reserve(lm.size() + residual.size()); + out.insert(out.end(), lm.begin(), lm.end()); + out.insert(out.end(), residual.begin(), residual.end()); + return out; +} + +std::vector add_dit_mu(const std::vector &lm, + const std::vector &residual) { + if (lm.size() != residual.size()) { + throw std::runtime_error("VoxCPM1 dit mu inputs must have equal size"); + } + std::vector out(lm.size(), 0.0F); + for (size_t i = 0; i < lm.size(); ++i) { + out[i] = lm[i] + residual[i]; + } + return out; +} + +void append_patch(std::vector &features, const std::vector &patch, + int64_t expected_size) { + if (static_cast(patch.size()) != expected_size) { + throw std::runtime_error("VoxCPM1 generated patch size mismatch"); + } + features.insert(features.end(), patch.begin(), patch.end()); +} + +void validate_feature_block(const std::vector &features, int64_t patches, + int64_t patch_elems, const char *label) { + if (patches < 0) { + throw std::runtime_error(std::string("VoxCPM1 ") + label + + " patch count is negative"); + } + if (static_cast(features.size()) != patches * patch_elems) { + throw std::runtime_error(std::string("VoxCPM1 ") + label + + " feature size mismatch"); + } +} + +std::vector feature_patch(const std::vector &features, + int64_t index, int64_t patch_elems) { + const auto begin = + features.begin() + static_cast(index * patch_elems); + return std::vector(begin, + begin + static_cast(patch_elems)); +} + +bool has_prompt_audio(const VoxCPM1EncodedPrompt *prompt) { + return prompt != nullptr && prompt->prompt_patches > 0; +} + +bool has_reference_audio(const VoxCPM1EncodedPrompt *prompt) { + return prompt != nullptr && prompt->reference_patches > 0; +} + +std::string normalize_wrapper_text(const std::string &text) { + std::string out; + out.reserve(text.size()); + bool in_space = false; + for (const unsigned char ch : text) { + if (std::isspace(ch)) { + if (!in_space) { + out.push_back(' '); + in_space = true; + } + continue; + } + out.push_back(static_cast(ch)); + in_space = false; + } + return out; +} + +} // namespace + +struct VoxCPM1StepProjectionOutput { + std::vector fsq_hidden; + std::vector current_residual_input; + std::vector residual_input; + std::vector current_lm_dit_hidden; + std::vector fsq_lm_dit_hidden; + std::vector residual_dit_hidden; + std::vector current_stop_logits; + std::vector fsq_stop_logits; +}; + +class VoxCPM1StepProjectionRuntime final { +public: + VoxCPM1StepProjectionRuntime( + std::shared_ptr weights, + size_t graph_context_bytes, bool mem_saver = false); + ~VoxCPM1StepProjectionRuntime(); + + VoxCPM1StepProjectionOutput run(const std::vector &lm_hidden, + const std::vector &residual_hidden, + const std::vector ¤t_embed); + void release_runtime_memory(); + +private: + class Impl; + std::unique_ptr impl_; +}; + +class VoxCPM1LocalEncoderRuntime final { +public: + VoxCPM1LocalEncoderRuntime( + std::shared_ptr weights, + size_t graph_context_bytes, bool mem_saver = false); + ~VoxCPM1LocalEncoderRuntime(); + + std::vector + encode_patch(const std::vector &patch_features) const; + void release_runtime_memory(); + +private: + class Impl; + std::unique_ptr impl_; +}; + +class VoxCPM1DiTEstimatorRuntime final { +public: + VoxCPM1DiTEstimatorRuntime( + std::shared_ptr weights, + size_t graph_context_bytes, bool mem_saver = false); + ~VoxCPM1DiTEstimatorRuntime(); + + std::vector run(const std::vector &x, + const std::vector &mu, + const std::vector &cond, + const std::vector &time_embedding, + const std::vector &delta_time_embedding); + void release_runtime_memory(); + +private: + class Impl; + std::unique_ptr impl_; +}; + +class VoxCPM1CFMRuntime final { +public: + VoxCPM1CFMRuntime(std::shared_ptr weights, + size_t estimator_graph_context_bytes, + bool mem_saver = false); + ~VoxCPM1CFMRuntime(); + + std::vector generate_patch(const std::vector &mu, + const std::vector &cond_patch, + int64_t timesteps, float cfg_value, + uint64_t seed, + uint64_t noise_start_index = 0, + const std::string &noise_file = {}, + float temperature = 1.0F); + void release_runtime_memory(); + +private: + class Impl; + std::unique_ptr impl_; +}; + +class VoxCPM1StepProjectionRuntime::Impl { +public: + Impl(std::shared_ptr weights, + size_t graph_context_bytes, bool mem_saver) + : weights_(std::move(weights)), mem_saver_(mem_saver) { + if (weights_ == nullptr) { + throw std::runtime_error( + "VoxCPM1 step projection runtime requires weights"); + } + build(graph_context_bytes); + } + + ~Impl() { release_graph(); } + + void release_runtime_memory() { release_graph(); } + + VoxCPM1StepProjectionOutput run(const std::vector &lm_hidden, + const std::vector &residual_hidden, + const std::vector ¤t_embed) { + const auto &config = weights_->assets().config; + if (static_cast(lm_hidden.size()) != config.lm.hidden_size) { + throw std::runtime_error( + "VoxCPM1 step projection lm_hidden size mismatch"); + } + if (static_cast(residual_hidden.size()) != config.lm.hidden_size) { + throw std::runtime_error( + "VoxCPM1 step projection residual_hidden size mismatch"); + } + if (static_cast(current_embed.size()) != config.lm.hidden_size) { + throw std::runtime_error( + "VoxCPM1 step projection current_embed size mismatch"); + } + ggml_backend_tensor_set(lm_hidden_, lm_hidden.data(), 0, + lm_hidden.size() * sizeof(float)); + ggml_backend_tensor_set(residual_hidden_, residual_hidden.data(), 0, + residual_hidden.size() * sizeof(float)); + ggml_backend_tensor_set(current_embed_, current_embed.data(), 0, + current_embed.size() * sizeof(float)); + engine::core::set_backend_threads(weights_->backend(), weights_->threads()); + const ggml_status status = + engine::core::compute_backend_graph(weights_->backend(), graph_); + ggml_backend_synchronize(weights_->backend()); + if (status != GGML_STATUS_SUCCESS) { + throw std::runtime_error("VoxCPM1 step projection graph compute failed"); + } + VoxCPM1StepProjectionOutput output; + output.fsq_hidden.resize(static_cast(config.lm.hidden_size), 0.0F); + output.current_residual_input.resize( + static_cast(config.lm.hidden_size), 0.0F); + output.residual_input.resize(static_cast(config.lm.hidden_size), + 0.0F); + output.current_lm_dit_hidden.resize( + static_cast(config.dit.hidden_dim), 0.0F); + output.fsq_lm_dit_hidden.resize(static_cast(config.dit.hidden_dim), + 0.0F); + output.residual_dit_hidden.resize( + static_cast(config.dit.hidden_dim), 0.0F); + output.current_stop_logits.resize(2, 0.0F); + output.fsq_stop_logits.resize(2, 0.0F); + ggml_backend_tensor_get(fsq_hidden_output_, output.fsq_hidden.data(), 0, + output.fsq_hidden.size() * sizeof(float)); + ggml_backend_tensor_get( + current_residual_input_output_, output.current_residual_input.data(), 0, + output.current_residual_input.size() * sizeof(float)); + ggml_backend_tensor_get(residual_input_output_, + output.residual_input.data(), 0, + output.residual_input.size() * sizeof(float)); + ggml_backend_tensor_get( + current_lm_dit_output_, output.current_lm_dit_hidden.data(), 0, + output.current_lm_dit_hidden.size() * sizeof(float)); + ggml_backend_tensor_get(fsq_lm_dit_output_, output.fsq_lm_dit_hidden.data(), + 0, output.fsq_lm_dit_hidden.size() * sizeof(float)); + ggml_backend_tensor_get(residual_dit_output_, + output.residual_dit_hidden.data(), 0, + output.residual_dit_hidden.size() * sizeof(float)); + ggml_backend_tensor_get(current_stop_logits_output_, + output.current_stop_logits.data(), 0, + output.current_stop_logits.size() * sizeof(float)); + ggml_backend_tensor_get(fsq_stop_logits_output_, + output.fsq_stop_logits.data(), 0, + output.fsq_stop_logits.size() * sizeof(float)); + return output; + } + +private: + void release_graph() { + if (graph_ != nullptr) { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + } + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + buffer_ = nullptr; + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + graph_ = nullptr; + lm_hidden_ = nullptr; + residual_hidden_ = nullptr; + current_embed_ = nullptr; + fsq_hidden_output_ = nullptr; + current_residual_input_output_ = nullptr; + residual_input_output_ = nullptr; + current_lm_dit_output_ = nullptr; + fsq_lm_dit_output_ = nullptr; + residual_dit_output_ = nullptr; + current_stop_logits_output_ = nullptr; + fsq_stop_logits_output_ = nullptr; + ctx_.reset(); + } + + void build(size_t graph_context_bytes) { + const auto &config = weights_->assets().config; + if (graph_context_bytes == 0) { + throw std::runtime_error( + "VoxCPM1 step projection graph context bytes must be non-zero"); + } + ggml_init_params params{graph_context_bytes, nullptr, true}; + ctx_.reset(ggml_init(params)); + if (ctx_ == nullptr) { + throw std::runtime_error( + "failed to initialize VoxCPM1 step projection graph context"); + } + engine::core::ModuleBuildContext ctx{ctx_.get(), "voxcpm1.step_projection"}; + const auto &proj = weights_->weights().projections; + auto lm_hidden = engine::core::make_tensor( + ctx, GGML_TYPE_F32, + engine::core::TensorShape::from_dims({1, config.lm.hidden_size})); + lm_hidden_ = lm_hidden.tensor; + if (mem_saver_) { + ggml_set_input(lm_hidden_); + } + auto residual_hidden = engine::core::make_tensor( + ctx, GGML_TYPE_F32, + engine::core::TensorShape::from_dims({1, config.lm.hidden_size})); + residual_hidden_ = residual_hidden.tensor; + if (mem_saver_) { + ggml_set_input(residual_hidden_); + } + auto current_embed = engine::core::make_tensor( + ctx, GGML_TYPE_F32, + engine::core::TensorShape::from_dims({1, config.lm.hidden_size})); + current_embed_ = current_embed.tensor; + if (mem_saver_) { + ggml_set_input(current_embed_); + } + + auto fsq = + engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size, + config.scalar_quantization_latent_dim, true)) + .build(ctx, lm_hidden, proj.fsq_in_proj); + fsq = engine::core::wrap_tensor(ggml_tanh(ctx.ggml, fsq.tensor), fsq.shape, + GGML_TYPE_F32); + fsq = engine::core::wrap_tensor( + ggml_scale(ctx.ggml, fsq.tensor, + static_cast(config.scalar_quantization_scale)), + fsq.shape, GGML_TYPE_F32); + fsq = engine::core::wrap_tensor(ggml_round(ctx.ggml, fsq.tensor), fsq.shape, + GGML_TYPE_F32); + fsq = engine::core::wrap_tensor( + ggml_scale(ctx.ggml, fsq.tensor, + 1.0F / static_cast(config.scalar_quantization_scale)), + fsq.shape, GGML_TYPE_F32); + fsq = engine::modules::LinearModule( + binding::linear_config(config.scalar_quantization_latent_dim, + config.lm.hidden_size, true)) + .build(ctx, fsq, proj.fsq_out_proj); + fsq_hidden_output_ = fsq.tensor; + + // Check if fusion_concat_proj weight exists and was loaded (not synthesized) + // This matches VoxCPM.cpp behavior which checks weight existence + // For V1 models, synthesized weights (Xavier init) should not count as present + const bool has_fusion_proj = + proj.fusion_concat_proj.weight.tensor != nullptr && + config.architecture == "voxcpm2"; + + if (has_fusion_proj) { + // Concat + Linear (used by V2 and some V1 models trained with fusion) + auto current_residual_concat = + engine::modules::ConcatModule({1}).build(ctx, lm_hidden, current_embed); + auto current_residual_input = + engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size * 2, + config.lm.hidden_size, true)) + .build(ctx, current_residual_concat, proj.fusion_concat_proj); + current_residual_input_output_ = current_residual_input.tensor; + + auto residual_concat = + engine::modules::ConcatModule({1}).build(ctx, fsq, current_embed); + auto residual_input = + engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size * 2, + config.lm.hidden_size, true)) + .build(ctx, residual_concat, proj.fusion_concat_proj); + residual_input_output_ = residual_input.tensor; + } else { + // Simple ADD (true V1 without fusion_concat_proj) + current_residual_input_output_ = + engine::modules::AddModule() + .build(ctx, lm_hidden, current_embed) + .tensor; + residual_input_output_ = + engine::modules::AddModule().build(ctx, fsq, current_embed).tensor; + } + + auto current_lm_dit = + engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size, config.dit.hidden_dim, + true)) + .build(ctx, lm_hidden, proj.lm_to_dit_proj); + current_lm_dit_output_ = current_lm_dit.tensor; + + auto fsq_lm_dit = engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size, + config.dit.hidden_dim, true)) + .build(ctx, fsq, proj.lm_to_dit_proj); + fsq_lm_dit_output_ = fsq_lm_dit.tensor; + + auto residual_dit = engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size, + config.dit.hidden_dim, true)) + .build(ctx, residual_hidden, proj.res_to_dit_proj); + residual_dit_output_ = residual_dit.tensor; + + auto current_stop = engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size, + config.lm.hidden_size, true)) + .build(ctx, lm_hidden, proj.stop_proj); + current_stop = engine::modules::SiluModule{}.build(ctx, current_stop); + current_stop = engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size, 2, false)) + .build(ctx, current_stop, proj.stop_head); + current_stop_logits_output_ = current_stop.tensor; + + auto fsq_stop = engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size, + config.lm.hidden_size, true)) + .build(ctx, fsq, proj.stop_proj); + fsq_stop = engine::modules::SiluModule{}.build(ctx, fsq_stop); + fsq_stop = engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size, 2, false)) + .build(ctx, fsq_stop, proj.stop_head); + fsq_stop_logits_output_ = fsq_stop.tensor; + + graph_ = ggml_new_graph_custom(ctx_.get(), kDefaultGraphNodes, false); + ggml_set_output(fsq_hidden_output_); + if (mem_saver_ && fsq_hidden_output_->view_src != nullptr) { + ggml_set_output(fsq_hidden_output_->view_src); + } + ggml_set_output(current_residual_input_output_); + if (mem_saver_ && current_residual_input_output_->view_src != nullptr) { + ggml_set_output(current_residual_input_output_->view_src); + } + ggml_set_output(residual_input_output_); + if (mem_saver_ && residual_input_output_->view_src != nullptr) { + ggml_set_output(residual_input_output_->view_src); + } + ggml_set_output(current_lm_dit_output_); + if (mem_saver_ && current_lm_dit_output_->view_src != nullptr) { + ggml_set_output(current_lm_dit_output_->view_src); + } + ggml_set_output(fsq_lm_dit_output_); + if (mem_saver_ && fsq_lm_dit_output_->view_src != nullptr) { + ggml_set_output(fsq_lm_dit_output_->view_src); + } + ggml_set_output(residual_dit_output_); + if (mem_saver_ && residual_dit_output_->view_src != nullptr) { + ggml_set_output(residual_dit_output_->view_src); + } + ggml_set_output(current_stop_logits_output_); + if (mem_saver_ && current_stop_logits_output_->view_src != nullptr) { + ggml_set_output(current_stop_logits_output_->view_src); + } + ggml_set_output(fsq_stop_logits_output_); + if (mem_saver_ && fsq_stop_logits_output_->view_src != nullptr) { + ggml_set_output(fsq_stop_logits_output_->view_src); + } + ggml_build_forward_expand(graph_, fsq_hidden_output_); + ggml_build_forward_expand(graph_, current_residual_input_output_); + ggml_build_forward_expand(graph_, residual_input_output_); + ggml_build_forward_expand(graph_, current_lm_dit_output_); + ggml_build_forward_expand(graph_, fsq_lm_dit_output_); + ggml_build_forward_expand(graph_, residual_dit_output_); + ggml_build_forward_expand(graph_, current_stop_logits_output_); + ggml_build_forward_expand(graph_, fsq_stop_logits_output_); + if (mem_saver_) { + gallocr_ = ggml_gallocr_new( + ggml_backend_get_default_buffer_type(weights_->backend())); + if (gallocr_ == nullptr || !ggml_gallocr_reserve(gallocr_, graph_) || + !ggml_gallocr_alloc_graph(gallocr_, graph_)) { + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + throw std::runtime_error( + "failed to allocate VoxCPM1 step projection graph"); + } + return; + } + buffer_ = ggml_backend_alloc_ctx_tensors(ctx_.get(), weights_->backend()); + if (buffer_ == nullptr) { + throw std::runtime_error( + "failed to allocate VoxCPM1 step projection graph"); + } + } + + std::shared_ptr weights_; + bool mem_saver_ = false; + std::unique_ptr ctx_; + ggml_tensor *lm_hidden_ = nullptr; + ggml_tensor *residual_hidden_ = nullptr; + ggml_tensor *current_embed_ = nullptr; + ggml_tensor *fsq_hidden_output_ = nullptr; + ggml_tensor *current_residual_input_output_ = nullptr; + ggml_tensor *residual_input_output_ = nullptr; + ggml_tensor *current_lm_dit_output_ = nullptr; + ggml_tensor *fsq_lm_dit_output_ = nullptr; + ggml_tensor *residual_dit_output_ = nullptr; + ggml_tensor *current_stop_logits_output_ = nullptr; + ggml_tensor *fsq_stop_logits_output_ = nullptr; + ggml_cgraph *graph_ = nullptr; + ggml_backend_buffer_t buffer_ = nullptr; + ggml_gallocr_t gallocr_ = nullptr; +}; + +VoxCPM1StepProjectionRuntime::VoxCPM1StepProjectionRuntime( + std::shared_ptr weights, + size_t graph_context_bytes, bool mem_saver) + : impl_(std::make_unique(std::move(weights), graph_context_bytes, + mem_saver)) {} + +VoxCPM1StepProjectionRuntime::~VoxCPM1StepProjectionRuntime() = default; + +void VoxCPM1StepProjectionRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + +VoxCPM1StepProjectionOutput +VoxCPM1StepProjectionRuntime::run(const std::vector &lm_hidden, + const std::vector &residual_hidden, + const std::vector ¤t_embed) { + return impl_->run(lm_hidden, residual_hidden, current_embed); +} + +class VoxCPM1LocalEncoderRuntime::Impl { +public: + Impl(std::shared_ptr weights, + size_t graph_context_bytes, bool mem_saver) + : weights_(std::move(weights)), mem_saver_(mem_saver) { + if (weights_ == nullptr) { + throw std::runtime_error( + "VoxCPM1 local encoder runtime requires weights"); + } + build(graph_context_bytes); + } + + ~Impl() { release_graph(); } + + void release_runtime_memory() { release_graph(); } + + std::vector + encode_patch(const std::vector &patch_features) const { + const auto &config = weights_->assets().config; + const int64_t expected = config.patch_size * config.feat_dim; + if (static_cast(patch_features.size()) != expected) { + throw std::runtime_error( + "VoxCPM1 local encoder patch feature size mismatch"); + } + ggml_backend_tensor_set(input_, patch_features.data(), 0, + patch_features.size() * sizeof(float)); + engine::core::set_backend_threads(weights_->backend(), weights_->threads()); + const ggml_status status = + engine::core::compute_backend_graph(weights_->backend(), graph_); + ggml_backend_synchronize(weights_->backend()); + if (status != GGML_STATUS_SUCCESS) { + throw std::runtime_error("VoxCPM1 local encoder graph compute failed"); + } + std::vector output(static_cast(config.lm.hidden_size), 0.0F); + ggml_backend_tensor_get(output_, output.data(), 0, + output.size() * sizeof(float)); + return output; + } + +private: + void release_graph() { + if (graph_ != nullptr) { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + } + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + buffer_ = nullptr; + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + graph_ = nullptr; + input_ = nullptr; + positions_ = nullptr; + output_ = nullptr; + ctx_.reset(); + } + + void build(size_t graph_context_bytes) { + const auto &root_config = weights_->assets().config; + if (graph_context_bytes == 0) { + throw std::runtime_error( + "VoxCPM1 local encoder graph context bytes must be non-zero"); + } + ggml_init_params params{graph_context_bytes, nullptr, true}; + ctx_.reset(ggml_init(params)); + if (ctx_ == nullptr) { + throw std::runtime_error( + "failed to initialize VoxCPM1 local encoder graph context"); + } + engine::core::ModuleBuildContext ctx{ctx_.get(), "voxcpm1.local_encoder"}; + const auto &feat_weights = weights_->weights().feat_encoder; + auto x = engine::core::make_tensor( + ctx, GGML_TYPE_F32, + engine::core::TensorShape::from_dims( + {1, root_config.patch_size, root_config.feat_dim})); + input_ = x.tensor; + if (mem_saver_) { + ggml_set_input(input_); + } + x = engine::modules::LinearModule( + binding::linear_config(root_config.feat_dim, + root_config.encoder.hidden_dim, true)) + .build(ctx, x, feat_weights.in_proj); + auto special = engine::core::reshape_tensor( + ctx, feat_weights.special_token, + engine::core::TensorShape::from_dims( + {1, 1, root_config.encoder.hidden_dim})); + special = engine::core::wrap_tensor( + ggml_cast(ctx.ggml, special.tensor, GGML_TYPE_F32), special.shape, + GGML_TYPE_F32); + x = engine::modules::ConcatModule({1}).build(ctx, special, x); + positions_ = ggml_new_tensor_1d(ctx_.get(), GGML_TYPE_I32, + root_config.patch_size + 1); + if (mem_saver_) { + ggml_set_input(positions_); + ggml_set_output(positions_); + } + auto positions = engine::core::wrap_tensor( + positions_, + engine::core::TensorShape::from_dims({root_config.patch_size + 1}), + GGML_TYPE_I32); + x = minicpm_transformer(ctx, x, positions, feat_weights.encoder, false); + x = engine::modules::SliceModule({1, 0, 1}).build(ctx, x); + x = engine::modules::LinearModule( + binding::linear_config(root_config.encoder.hidden_dim, + root_config.lm.hidden_size, true)) + .build(ctx, x, weights_->weights().projections.enc_to_lm_proj); + output_ = x.tensor; + ggml_set_output(output_); + if (mem_saver_ && output_->view_src != nullptr) { + ggml_set_output(output_->view_src); + } + graph_ = ggml_new_graph_custom(ctx_.get(), kDefaultGraphNodes, false); + ggml_build_forward_expand(graph_, output_); + if (mem_saver_) { + gallocr_ = ggml_gallocr_new( + ggml_backend_get_default_buffer_type(weights_->backend())); + if (gallocr_ == nullptr || !ggml_gallocr_reserve(gallocr_, graph_) || + !ggml_gallocr_alloc_graph(gallocr_, graph_)) { + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + throw std::runtime_error( + "failed to allocate VoxCPM1 local encoder graph"); + } + } else { + buffer_ = ggml_backend_alloc_ctx_tensors(ctx_.get(), weights_->backend()); + } + if (!mem_saver_ && buffer_ == nullptr) { + throw std::runtime_error( + "failed to allocate VoxCPM1 local encoder graph"); + } + std::vector position_ids( + static_cast(root_config.patch_size + 1), 0); + for (int64_t i = 0; i < root_config.patch_size + 1; ++i) { + position_ids[static_cast(i)] = static_cast(i); + } + ggml_backend_tensor_set(positions_, position_ids.data(), 0, + position_ids.size() * sizeof(int32_t)); + } + + std::shared_ptr weights_; + bool mem_saver_ = false; + std::unique_ptr ctx_; + ggml_tensor *input_ = nullptr; + ggml_tensor *positions_ = nullptr; + ggml_tensor *output_ = nullptr; + ggml_cgraph *graph_ = nullptr; + ggml_backend_buffer_t buffer_ = nullptr; + ggml_gallocr_t gallocr_ = nullptr; +}; + +VoxCPM1LocalEncoderRuntime::VoxCPM1LocalEncoderRuntime( + std::shared_ptr weights, + size_t graph_context_bytes, bool mem_saver) + : impl_(std::make_unique(std::move(weights), graph_context_bytes, + mem_saver)) {} + +VoxCPM1LocalEncoderRuntime::~VoxCPM1LocalEncoderRuntime() = default; + +void VoxCPM1LocalEncoderRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + +std::vector VoxCPM1LocalEncoderRuntime::encode_patch( + const std::vector &patch_features) const { + return impl_->encode_patch(patch_features); +} + +class VoxCPM1DiTEstimatorRuntime::Impl { +public: + Impl(std::shared_ptr weights, + size_t graph_context_bytes, bool mem_saver) + : weights_(std::move(weights)), mem_saver_(mem_saver) { + if (weights_ == nullptr) { + throw std::runtime_error( + "VoxCPM1 DiT estimator runtime requires weights"); + } + build(graph_context_bytes); + } + + ~Impl() { release_graph(); } + + void release_runtime_memory() { release_graph(); } + + std::vector run(const std::vector &x, + const std::vector &mu, + const std::vector &cond, + const std::vector &time_embedding, + const std::vector &delta_time_embedding) { + const auto &config = weights_->assets().config; + // Check if fusion_concat_proj weight exists and was loaded (not synthesized) + // This matches VoxCPM.cpp behavior which checks weight existence + // For V1 models, synthesized weights (Xavier init) should not count as present + const bool has_fusion_proj = + weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && + config.architecture == "voxcpm2"; + const int64_t patch_elems = 2 * config.feat_dim * config.patch_size; + if (static_cast(x.size()) != patch_elems) { + throw std::runtime_error("VoxCPM1 DiT estimator x size mismatch"); + } + if (static_cast(cond.size()) != patch_elems) { + throw std::runtime_error("VoxCPM1 DiT estimator cond size mismatch"); + } + const int64_t expected_mu = + has_fusion_proj ? 2 * config.dit.hidden_dim * 2 : 2 * config.dit.hidden_dim; + if (static_cast(mu.size()) != expected_mu) { + throw std::runtime_error("VoxCPM1 DiT estimator mu size mismatch"); + } + if (static_cast(time_embedding.size()) != + 2 * config.dit.hidden_dim) { + throw std::runtime_error( + "VoxCPM1 DiT estimator time embedding size mismatch"); + } + if (static_cast(delta_time_embedding.size()) != + 2 * config.dit.hidden_dim) { + throw std::runtime_error( + "VoxCPM1 DiT estimator delta-time embedding size mismatch"); + } + ggml_backend_tensor_set(x_, x.data(), 0, x.size() * sizeof(float)); + ggml_backend_tensor_set(cond_, cond.data(), 0, cond.size() * sizeof(float)); + ggml_backend_tensor_set(mu_, mu.data(), 0, mu.size() * sizeof(float)); + ggml_backend_tensor_set(time_embedding_, time_embedding.data(), 0, + time_embedding.size() * sizeof(float)); + ggml_backend_tensor_set(delta_time_embedding_, delta_time_embedding.data(), + 0, delta_time_embedding.size() * sizeof(float)); + engine::core::set_backend_threads(weights_->backend(), weights_->threads()); + const ggml_status status = + engine::core::compute_backend_graph(weights_->backend(), graph_); + ggml_backend_synchronize(weights_->backend()); + if (status != GGML_STATUS_SUCCESS) { + throw std::runtime_error("VoxCPM1 DiT estimator graph compute failed"); + } + std::vector output(static_cast(patch_elems), 0.0F); + ggml_backend_tensor_get(output_, output.data(), 0, + output.size() * sizeof(float)); + if (std::getenv("VOXCPM_DUMP_NORM0") != nullptr) { + ggml_tensor *tn = ggml_get_tensor(ctx_.get(), "dump_norm0"); + if (tn != nullptr) { + const size_t nn = static_cast(ggml_nelements(tn)); + std::vector buf(nn, 0.0F); + ggml_backend_tensor_get(tn, buf.data(), 0, buf.size() * sizeof(float)); + FILE *f = std::fopen("/tmp/opencode/ours_norm0.bin", "wb"); + if (f != nullptr) { + std::fwrite(buf.data(), sizeof(float), std::min(nn, 5120), f); + std::fclose(f); + } + } + } + if (std::getenv("VOXCPM_DUMP_DECODER_LAYERS") != nullptr) { + for (int li = 0; li < 8; ++li) { + char name[32]; + snprintf(name, sizeof(name), "dump_layer_%d", li); + ggml_tensor *t = ggml_get_tensor(ctx_.get(), name); + if (t == nullptr) { + continue; + } + const size_t n = static_cast(ggml_nelements(t)); + std::vector buf(n, 0.0F); + ggml_backend_tensor_get(t, buf.data(), 0, buf.size() * sizeof(float)); + if (li == 0) { + FILE *f = std::fopen("/tmp/opencode/ours_branch0.bin", "wb"); + if (f != nullptr) { + std::fwrite(buf.data(), sizeof(float), std::min(n, 5120), f); + std::fclose(f); + } + } else if (li == 1) { + FILE *f = std::fopen("/tmp/opencode/ours_branch1.bin", "wb"); + if (f != nullptr) { + std::fwrite(buf.data(), sizeof(float), std::min(n, 5120), f); + std::fclose(f); + } + } + double s = 0.0, l2 = 0.0; + for (float v : buf) { + s += v; + l2 += static_cast(v) * v; + } + // batch-local stats for the branch-0 (first ne1*ne0 elements) + double s0 = 0.0, l20 = 0.0; + const size_t branch_elems = static_cast(t->ne[0]) * static_cast(t->ne[1]); + for (size_t i = 0; i < std::min(branch_elems, buf.size()); ++i) { + s0 += buf[i]; + l20 += static_cast(buf[i]) * buf[i]; + } + fprintf(stderr, + "[DEC_LAYER] input#%d ne0=%lld ne1=%lld ne2=%lld ne3=%lld " + "sum=%.6g l2=%.6g branch0_sum=%.6g branch0_l2=%.6g " + "first4=%.6g %.6g %.6g %.6g\n", + li, static_cast(t->ne[0]), + static_cast(t->ne[1]), + static_cast(t->ne[2]), + static_cast(t->ne[3]), s, std::sqrt(l2), s0, + std::sqrt(l20), + buf.empty() ? 0.0 : static_cast(buf[0]), + buf.size() < 2 ? 0.0 : static_cast(buf[1]), + buf.size() < 3 ? 0.0 : static_cast(buf[2]), + buf.size() < 4 ? 0.0 : static_cast(buf[3])); + } + } + if (std::getenv("VOXCPM_DUMP_LOCDIT_WEIGHTS") != nullptr) { + const auto &dw = weights_->weights().dit; + auto dump_w = [](const char *tag, ggml_tensor *t) { + if (t == nullptr) { + fprintf(stderr, "[LOCDIT_W] %s \n", tag); + return; + } + const size_t nbytes = static_cast(ggml_nbytes(t)); + const size_t nelems = static_cast(ggml_nelements(t)); + std::vector raw(nbytes, 0); + ggml_backend_tensor_get(t, raw.data(), 0, nbytes); + fprintf(stderr, "[LOCDIT_W] %s ne0=%lld ne1=%lld type=%d nbytes=%zu " + "v[0..7]=", + tag, static_cast(t->ne[0]), + static_cast(t->ne[1]), static_cast(t->type), + nbytes); + double vals[8]; + for (size_t i = 0; i < 8; ++i) { + if (t->type == GGML_TYPE_Q8_0) { + const size_t block = i / 32; + const size_t in_block = i % 32; + const float scale = + ggml_fp16_to_fp32( + *reinterpret_cast( + raw.data() + block * 34)); + vals[i] = static_cast( + scale * + static_cast( + *reinterpret_cast( + raw.data() + block * 34 + 2 + in_block))); + } else if (t->type == GGML_TYPE_F32) { + vals[i] = static_cast( + *reinterpret_cast(raw.data() + i * 4)); + } else if (t->type == GGML_TYPE_F16) { + vals[i] = static_cast(ggml_fp16_to_fp32( + *reinterpret_cast(raw.data() + i * 2))); + } else { + vals[i] = 0.0; + } + } + (void)nelems; + for (size_t i = 0; i < 8; ++i) { + fprintf(stderr, "%.6g ", vals[i]); + } + fprintf(stderr, "\n"); + }; + dump_w("in_proj", dw.in_proj.weight.tensor); + dump_w("cond_proj", dw.cond_proj.weight.tensor); + dump_w("out_proj", dw.out_proj.weight.tensor); + dump_w("time_mlp1", dw.time_mlp_1.weight.tensor); + dump_w("decoder.l0.q", dw.decoder.layers[0].q_proj.weight.tensor); + dump_w("decoder.l0.k", dw.decoder.layers[0].k_proj.weight.tensor); + dump_w("decoder.l0.o", dw.decoder.layers[0].o_proj.weight.tensor); + dump_w("decoder.norm", dw.decoder.norm.weight->tensor); + } + return output; + } + +private: + void release_graph() { + if (graph_ != nullptr) { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + } + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + buffer_ = nullptr; + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + graph_ = nullptr; + x_ = nullptr; + mu_ = nullptr; + cond_ = nullptr; + time_embedding_ = nullptr; + delta_time_embedding_ = nullptr; + positions_ = nullptr; + output_ = nullptr; + ctx_.reset(); + } + + void build(size_t graph_context_bytes) { + const auto &root_config = weights_->assets().config; + const auto &config = root_config.dit; + if (graph_context_bytes == 0) { + throw std::runtime_error( + "VoxCPM1 DiT estimator graph context bytes must be non-zero"); + } + ggml_init_params params{graph_context_bytes, nullptr, true}; + ctx_.reset(ggml_init(params)); + if (ctx_ == nullptr) { + throw std::runtime_error( + "failed to initialize VoxCPM1 DiT estimator graph context"); + } + engine::core::ModuleBuildContext ctx{ctx_.get(), "voxcpm1.dit.estimator"}; + const auto &weights = weights_->weights().dit; + x_ = engine::core::make_tensor( + ctx, GGML_TYPE_F32, + engine::core::TensorShape::from_dims( + {2, root_config.feat_dim, root_config.patch_size})) + .tensor; + if (mem_saver_) { + ggml_set_input(x_); + } + cond_ = engine::core::make_tensor( + ctx, GGML_TYPE_F32, + engine::core::TensorShape::from_dims( + {2, root_config.feat_dim, root_config.patch_size})) + .tensor; + if (mem_saver_) { + ggml_set_input(cond_); + } + // Check if fusion_concat_proj weight exists and was loaded (not synthesized) + // This matches VoxCPM.cpp behavior which checks weight existence + // For V1 models, synthesized weights (Xavier init) should not count as present + const bool has_fusion_proj = + weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && + root_config.architecture == "voxcpm2"; + mu_ = engine::core::make_tensor( + ctx, GGML_TYPE_F32, + has_fusion_proj + ? engine::core::TensorShape::from_dims( + {2, 2, config.hidden_dim}) + : engine::core::TensorShape::from_dims( + {2, config.hidden_dim})) + .tensor; + if (mem_saver_) { + ggml_set_input(mu_); + } + time_embedding_ = + engine::core::make_tensor( + ctx, GGML_TYPE_F32, + engine::core::TensorShape::from_dims({2, config.hidden_dim})) + .tensor; + if (mem_saver_) { + ggml_set_input(time_embedding_); + } + delta_time_embedding_ = + engine::core::make_tensor( + ctx, GGML_TYPE_F32, + engine::core::TensorShape::from_dims({2, config.hidden_dim})) + .tensor; + if (mem_saver_) { + ggml_set_input(delta_time_embedding_); + } + + auto x = engine::core::wrap_tensor( + x_, + engine::core::TensorShape::from_dims( + {2, root_config.feat_dim, root_config.patch_size}), + GGML_TYPE_F32); + x = engine::modules::TransposeModule({{0, 2, 1, 3}, 3}).build(ctx, x); + x = engine::modules::LinearModule( + binding::linear_config(root_config.feat_dim, config.hidden_dim, + true)) + .build(ctx, x, weights.in_proj); + + auto cond = engine::core::wrap_tensor( + cond_, + engine::core::TensorShape::from_dims( + {2, root_config.feat_dim, root_config.patch_size}), + GGML_TYPE_F32); + cond = engine::modules::TransposeModule({{0, 2, 1, 3}, 3}).build(ctx, cond); + cond = engine::modules::LinearModule( + binding::linear_config(root_config.feat_dim, config.hidden_dim, + true)) + .build(ctx, cond, weights.cond_proj); + + auto time = engine::core::wrap_tensor( + time_embedding_, + engine::core::TensorShape::from_dims({2, config.hidden_dim}), + GGML_TYPE_F32); + time = + engine::modules::LinearModule( + binding::linear_config(config.hidden_dim, config.hidden_dim, true)) + .build(ctx, time, weights.time_mlp_1); + time = engine::modules::SiluModule{}.build(ctx, time); + time = + engine::modules::LinearModule( + binding::linear_config(config.hidden_dim, config.hidden_dim, true)) + .build(ctx, time, weights.time_mlp_2); + + auto dt = engine::core::wrap_tensor( + delta_time_embedding_, + engine::core::TensorShape::from_dims({2, config.hidden_dim}), + GGML_TYPE_F32); + dt = engine::modules::LinearModule( + binding::linear_config(config.hidden_dim, config.hidden_dim, true)) + .build(ctx, dt, weights.delta_time_mlp_1); + dt = engine::modules::SiluModule{}.build(ctx, dt); + dt = engine::modules::LinearModule( + binding::linear_config(config.hidden_dim, config.hidden_dim, true)) + .build(ctx, dt, weights.delta_time_mlp_2); + time = engine::modules::AddModule{}.build(ctx, time, dt); + + const int64_t prefix_token_count = + has_fusion_proj ? 2 + 1 : 1; + auto hidden = time; + if (!has_fusion_proj) { + // True V1 (no fusion projection): the DiT conditioning mu is a single + // hidden vector that is ADDED into the timestep token. Batch 0 carries + // mu (conditioned branch); batch 1 carries zeros (unconditioned branch), + // mirroring LocDiTModel::forward_cfg_pair_projected with mu_tokens == 1. + auto mu = engine::core::wrap_tensor( + mu_, engine::core::TensorShape::from_dims({2, config.hidden_dim}), + GGML_TYPE_F32); + hidden = engine::modules::AddModule{}.build(ctx, hidden, mu); + } + hidden = engine::core::reshape_tensor( + ctx, hidden, + engine::core::TensorShape::from_dims({2, 1, config.hidden_dim})); + if (has_fusion_proj) { + // V2 (or V1 with fusion projection): mu is two hidden vectors concatenated as + // separate prefix tokens before the timestep token, matching + // LocDiTModel with mu_tokens == 2. + auto mu = engine::core::wrap_tensor( + mu_, engine::core::TensorShape::from_dims({2, 2, config.hidden_dim}), + GGML_TYPE_F32); + hidden = engine::modules::ConcatModule({1}).build(ctx, mu, hidden); + } + hidden = engine::modules::ConcatModule({1}).build(ctx, hidden, cond); + hidden = engine::modules::ConcatModule({1}).build(ctx, hidden, x); + + positions_ = ggml_new_tensor_1d( + ctx_.get(), GGML_TYPE_I32, + prefix_token_count + root_config.patch_size * 2); + if (mem_saver_) { + ggml_set_input(positions_); + ggml_set_output(positions_); + } + auto positions = + engine::core::wrap_tensor(positions_, + engine::core::TensorShape::from_dims( + {prefix_token_count + + root_config.patch_size * 2}), + GGML_TYPE_I32); + hidden = + minicpm_transformer(ctx, hidden, positions, weights.decoder, false); + hidden = engine::modules::SliceModule( + {1, prefix_token_count + root_config.patch_size, + root_config.patch_size}) + .build(ctx, hidden); + hidden = engine::modules::LinearModule( + binding::linear_config(config.hidden_dim, root_config.feat_dim, + true)) + .build(ctx, hidden, weights.out_proj); + hidden = + engine::modules::TransposeModule({{0, 2, 1, 3}, 3}).build(ctx, hidden); + hidden = ensure_contiguous(ctx, hidden); + output_ = hidden.tensor; + ggml_set_output(output_); + if (mem_saver_ && output_->view_src != nullptr) { + ggml_set_output(output_->view_src); + } + graph_ = ggml_new_graph_custom(ctx_.get(), kDefaultGraphNodes, false); + ggml_build_forward_expand(graph_, output_); + if (mem_saver_) { + gallocr_ = ggml_gallocr_new( + ggml_backend_get_default_buffer_type(weights_->backend())); + if (gallocr_ == nullptr || !ggml_gallocr_reserve(gallocr_, graph_) || + !ggml_gallocr_alloc_graph(gallocr_, graph_)) { + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + throw std::runtime_error( + "failed to allocate VoxCPM1 DiT estimator graph"); + } + } else { + buffer_ = ggml_backend_alloc_ctx_tensors(ctx_.get(), weights_->backend()); + } + if (!mem_saver_ && buffer_ == nullptr) { + throw std::runtime_error( + "failed to allocate VoxCPM1 DiT estimator graph"); + } + std::vector positions_data( + static_cast(prefix_token_count + root_config.patch_size * 2), + 0); + for (int64_t i = 0; i < static_cast(positions_data.size()); ++i) { + positions_data[static_cast(i)] = static_cast(i); + } + ggml_backend_tensor_set(positions_, positions_data.data(), 0, + positions_data.size() * sizeof(int32_t)); + } + + std::shared_ptr weights_; + bool mem_saver_ = false; + std::unique_ptr ctx_; + ggml_tensor *x_ = nullptr; + ggml_tensor *mu_ = nullptr; + ggml_tensor *cond_ = nullptr; + ggml_tensor *time_embedding_ = nullptr; + ggml_tensor *delta_time_embedding_ = nullptr; + ggml_tensor *positions_ = nullptr; + ggml_tensor *output_ = nullptr; + ggml_cgraph *graph_ = nullptr; + ggml_backend_buffer_t buffer_ = nullptr; + ggml_gallocr_t gallocr_ = nullptr; +}; + +VoxCPM1DiTEstimatorRuntime::VoxCPM1DiTEstimatorRuntime( + std::shared_ptr weights, + size_t graph_context_bytes, bool mem_saver) + : impl_(std::make_unique(std::move(weights), graph_context_bytes, + mem_saver)) {} + +VoxCPM1DiTEstimatorRuntime::~VoxCPM1DiTEstimatorRuntime() = default; + +void VoxCPM1DiTEstimatorRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + +std::vector VoxCPM1DiTEstimatorRuntime::run( + const std::vector &x, const std::vector &mu, + const std::vector &cond, const std::vector &time_embedding, + const std::vector &delta_time_embedding) { + return impl_->run(x, mu, cond, time_embedding, delta_time_embedding); +} + + +std::vector sinusoidal_time_embedding(float timestep, + int64_t hidden_size) { + if (hidden_size <= 0 || hidden_size % 2 != 0) { + throw std::runtime_error( + "VoxCPM1 sinusoidal time embedding requires even hidden size"); + } + const int64_t half = hidden_size / 2; + std::vector out(static_cast(hidden_size), 0.0F); + const double emb_scale = std::log(10000.0) / static_cast(half - 1); + for (int64_t index = 0; index < half; ++index) { + const double freq = std::exp(static_cast(index) * -emb_scale); + const double arg = 1000.0 * static_cast(timestep) * freq; + out[static_cast(index)] = static_cast(std::sin(arg)); + out[static_cast(half + index)] = static_cast(std::cos(arg)); + } + return out; +} + +class VoxCPM1CFMRuntime::Impl { +public: + Impl(std::shared_ptr weights, + size_t estimator_graph_context_bytes, bool mem_saver) + : weights_(std::move(weights)), + estimator_(weights_, estimator_graph_context_bytes, mem_saver) { + if (weights_ == nullptr) { + throw std::runtime_error("VoxCPM1 CFM runtime requires weights"); + } + } + + void release_runtime_memory() { estimator_.release_runtime_memory(); } + + std::vector generate_patch(const std::vector &mu, + const std::vector &cond_patch, + int64_t timesteps, float cfg_value, + uint64_t seed, uint64_t noise_start_index, + const std::string &noise_file, + float temperature) { + const auto &config = weights_->assets().config; + // Check if fusion_concat_proj weight exists and was loaded (not synthesized) + // This matches VoxCPM.cpp behavior which checks weight existence + // For V1 models, synthesized weights (Xavier init) should not count as present + const bool has_fusion_proj = + weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && + config.architecture == "voxcpm2"; + if (timesteps <= 0) { + throw std::runtime_error("VoxCPM1 CFM requires positive timesteps"); + } + if (!std::isfinite(cfg_value) || !std::isfinite(temperature)) { + throw std::runtime_error("VoxCPM1 CFM received non-finite scalar input"); + } + const int64_t patch_elems = config.feat_dim * config.patch_size; + const int64_t mu_dim = config.dit.hidden_dim * (has_fusion_proj ? 2 : 1); + if (static_cast(mu.size()) != mu_dim) { + throw std::runtime_error("VoxCPM1 CFM mu size mismatch"); + } + if (static_cast(cond_patch.size()) != patch_elems) { + throw std::runtime_error("VoxCPM1 CFM conditioning patch size mismatch"); + } + + std::vector x; + if (noise_file.empty()) { + x = engine::sampling::generate_torch_cuda_randn( + static_cast(patch_elems), seed, + engine::sampling::TorchRandnPrecision::Float32, noise_start_index); + } else { + if (noise_file_ != noise_file) { + noise_values_ = engine::io::read_f32_file(noise_file); + noise_file_ = noise_file; + } + const auto start = static_cast(noise_start_index); + const auto count = static_cast(patch_elems); + if (noise_values_.size() < start + count) { + throw std::runtime_error( + "VoxCPM1 CFM noise file is too short: expected at least " + + std::to_string(start + count) + " floats, got " + + std::to_string(noise_values_.size())); + } + x.assign(noise_values_.begin() + static_cast(start), + noise_values_.begin() + static_cast(start + count)); + } + for (float &value : x) { + value *= temperature; + } + x = patch_major_to_channel_major(x); + const std::vector cond = patch_major_to_channel_major(cond_patch); + std::vector x_in(static_cast(2 * patch_elems), 0.0F); + std::vector cond_in(static_cast(2 * patch_elems), 0.0F); + const int64_t mu_elements = + has_fusion_proj ? 4 * config.dit.hidden_dim : 2 * config.dit.hidden_dim; + std::vector mu_in(static_cast(mu_elements), 0.0F); + std::copy(mu.begin(), mu.end(), mu_in.begin()); + if (std::getenv("VOXCPM_TEST_BATCH_MU") != nullptr) { + std::copy(mu.begin(), mu.end(), + mu_in.begin() + static_cast(mu.size())); + } + std::copy(cond.begin(), cond.end(), cond_in.begin()); + std::copy(cond.begin(), cond.end(), + cond_in.begin() + static_cast(patch_elems)); + + std::vector t_span(static_cast(timesteps + 1), 0.0F); + constexpr double kHalfPi = 1.57079632679489661923; + for (int64_t i = 0; i <= timesteps; ++i) { + const double base = + 1.0 - static_cast(i) / static_cast(timesteps); + t_span[static_cast(i)] = + static_cast(base + (std::cos(kHalfPi * base) - 1.0 + base)); + } + + float t = t_span.front(); + float dt = t_span[0] - t_span[1]; + const int64_t zero_init_steps = + std::max(1, static_cast(t_span.size() * 0.04)); + for (int64_t step = 1; step < static_cast(t_span.size()); ++step) { + std::vector dphi(static_cast(patch_elems), 0.0F); + if (step > zero_init_steps) { + std::copy(x.begin(), x.end(), x_in.begin()); + std::copy(x.begin(), x.end(), + x_in.begin() + static_cast(patch_elems)); + const auto time_one = + sinusoidal_time_embedding(t, config.dit.hidden_dim); + const float dt_value = config.dit.mean_mode ? dt : 0.0F; + const auto dt_one = + sinusoidal_time_embedding(dt_value, config.dit.hidden_dim); + std::vector time_embedding( + static_cast(2 * config.dit.hidden_dim), 0.0F); + std::vector delta_embedding( + static_cast(2 * config.dit.hidden_dim), 0.0F); + std::copy(time_one.begin(), time_one.end(), time_embedding.begin()); + std::copy(time_one.begin(), time_one.end(), + time_embedding.begin() + + static_cast(config.dit.hidden_dim)); + std::copy(dt_one.begin(), dt_one.end(), delta_embedding.begin()); + std::copy(dt_one.begin(), dt_one.end(), + delta_embedding.begin() + + static_cast(config.dit.hidden_dim)); + + const auto estimator = estimator_.run(x_in, mu_in, cond_in, + time_embedding, delta_embedding); + const float scale = optimized_cfg_scale(estimator, patch_elems); + if (std::getenv("VOXCPM_DUMP_DPHI") != nullptr && + step == 2) { + double s0 = 0.0, s1 = 0.0, n0 = 0.0, n1 = 0.0; + std::vector combined(static_cast(patch_elems)); + double c2 = 0.0; + for (int64_t i = 0; i < patch_elems; ++i) { + const float p = estimator[static_cast(i)]; + const float m = estimator[static_cast(patch_elems + i)]; + const double d = static_cast(m) * scale + + cfg_value * (static_cast(p) - + static_cast(m) * scale); + combined[static_cast(i)] = d; + s0 += p; s1 += m; n0 += p * p; n1 += m * m; + c2 += d * d; + } + fprintf(stderr, + "[DUMP_DPHI] t=%.6f dt=%.6f pos_l2=%.6g neg_l2=%.6g " + "combined_l2=%.6g scale=%.6g combined[0..3]=%.6g %.6g %.6g %.6g\n", + static_cast(t), static_cast(dt), + std::sqrt(n0), std::sqrt(n1), std::sqrt(c2), + static_cast(scale), combined[0], combined[1], + combined[2], combined[3]); + } + for (int64_t i = 0; i < patch_elems; ++i) { + const size_t index = static_cast(i); + const float positive = estimator[index]; + const float negative = + estimator[static_cast(patch_elems + i)]; + dphi[index] = + negative * scale + cfg_value * (positive - negative * scale); + } + } + for (int64_t i = 0; i < patch_elems; ++i) { + x[static_cast(i)] -= dt * dphi[static_cast(i)]; + } + t -= dt; + if (step < static_cast(t_span.size()) - 1) { + dt = t - t_span[static_cast(step + 1)]; + } + } + return channel_major_to_patch_major(x); + } + +private: + std::vector + patch_major_to_channel_major(const std::vector &patch) const { + const auto &config = weights_->assets().config; + std::vector out(patch.size(), 0.0F); + for (int64_t p = 0; p < config.patch_size; ++p) { + for (int64_t d = 0; d < config.feat_dim; ++d) { + out[static_cast(d * config.patch_size + p)] = + patch[static_cast(p * config.feat_dim + d)]; + } + } + return out; + } + + std::vector + channel_major_to_patch_major(const std::vector &channel) const { + const auto &config = weights_->assets().config; + std::vector out(channel.size(), 0.0F); + for (int64_t p = 0; p < config.patch_size; ++p) { + for (int64_t d = 0; d < config.feat_dim; ++d) { + out[static_cast(p * config.feat_dim + d)] = + channel[static_cast(d * config.patch_size + p)]; + } + } + return out; + } + + float optimized_cfg_scale(const std::vector &estimator, + int64_t patch_elems) const { + double dot = 0.0; + double norm = 1.0e-8; + for (int64_t i = 0; i < patch_elems; ++i) { + const double positive = estimator[static_cast(i)]; + const double negative = estimator[static_cast(patch_elems + i)]; + dot += positive * negative; + norm += negative * negative; + } + return static_cast(dot / norm); + } + + std::shared_ptr weights_; + VoxCPM1DiTEstimatorRuntime estimator_; + std::string noise_file_; + std::vector noise_values_; +}; + +VoxCPM1CFMRuntime::VoxCPM1CFMRuntime( + std::shared_ptr weights, + size_t estimator_graph_context_bytes, + bool mem_saver) + : impl_(std::make_unique(std::move(weights), + estimator_graph_context_bytes, + mem_saver)) {} + +VoxCPM1CFMRuntime::~VoxCPM1CFMRuntime() = default; + +void VoxCPM1CFMRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + +std::vector VoxCPM1CFMRuntime::generate_patch( + const std::vector &mu, const std::vector &cond_patch, + int64_t timesteps, float cfg_value, uint64_t seed, + uint64_t noise_start_index, const std::string &noise_file, + float temperature) { + return impl_->generate_patch(mu, cond_patch, timesteps, cfg_value, seed, + noise_start_index, noise_file, temperature); +} + +class VoxCPM1FeatureGeneratorRuntime::Impl { +public: + Impl(std::shared_ptr assets, + engine::core::ExecutionContext &execution_context, + VoxCPM1FeatureGeneratorConfig config) + : assets_(require_assets(std::move(assets))), + weights_(std::make_shared( + assets_, execution_context, config.weight_context_bytes, + config.weight_storage_type)), + tokenizer_(assets_->gguf_tokenizer + ? VoxCPM1TokenizerWrapper(assets_->gguf_tokenizer) + : VoxCPM1TokenizerWrapper( + std::make_shared(assets_))), + text_embedding_(weights_, config.text_embedding_graph_context_bytes, + config.mem_saver), + prefill_(weights_, config.lm_step_graph_context_bytes, + config.mem_saver), + base_lm_(weights_, VoxCPM1MiniCPMKind::BaseLM, + assets_->config.max_length, + config.lm_step_graph_context_bytes), + residual_lm_(weights_, VoxCPM1MiniCPMKind::ResidualLM, + assets_->config.max_length, + config.lm_step_graph_context_bytes), + projection_(weights_, config.projection_graph_context_bytes, + config.mem_saver), + cfm_(weights_, config.dit_graph_context_bytes, config.mem_saver), + local_encoder_(weights_, config.local_encoder_graph_context_bytes, + config.mem_saver), + prompt_audio_embedding_cache_(config.prompt_cache_slots) {} + + VoxCPM1Result generate_zero_shot(const std::string &text, + const VoxCPM1GenerationOptions &options) { + return generate(text, nullptr, options); + } + + VoxCPM1Result generate(const std::string &text, + const VoxCPM1EncodedPrompt *prompt, + const VoxCPM1GenerationOptions &options) { + validate_generation_options(options); + const auto prefill = build_prefill_sequence(text, prompt); + + const int64_t max_tokens = + effective_max_tokens(options, prefill.target_text_tokens); + VoxCPM1Result last_result; + uint64_t retry_noise_start = 0; + for (int64_t attempt = 0; attempt < options.retry_badcase_max_times; + ++attempt) { + last_result = + generate_once(prefill, max_tokens, options, retry_noise_start); + retry_noise_start += static_cast(last_result.generated_patches * + assets_->config.patch_size * + assets_->config.feat_dim); + if (!options.retry_badcase || + static_cast(last_result.generated_patches) < + static_cast(prefill.target_text_tokens) * + options.retry_badcase_ratio_threshold) { + break; + } + } + return last_result; + } + + VoxCPM1StreamingResult + generate_streaming(const std::string &text, + const VoxCPM1EncodedPrompt *prompt, + const VoxCPM1GenerationOptions &options, + const std::function + &chunk_callback) { + validate_generation_options(options); + if (options.retry_badcase) { + fprintf(stderr, + "[VoxCPM1] warning: retry_badcase ignored in streaming " + "generation\n"); + } + const auto prefill = build_prefill_sequence(text, prompt); + const int64_t max_tokens = + effective_max_tokens(options, prefill.target_text_tokens); + VoxCPM1StreamingResult streaming; + auto *streaming_chunks = + chunk_callback ? nullptr : &streaming.chunks; + const auto result = generate_once(prefill, max_tokens, options, 0, + streaming_chunks, chunk_callback); + streaming.generated_patches = result.generated_patches; + return streaming; + } + + void release_runtime_memory() { + // Release every staged graph so a session can idle at weight-only + // VRAM. Each runtime lazily rebuilds its graph on the next use. + text_embedding_.release_runtime_memory(); + prefill_.release_runtime_memory(); + base_lm_.release_runtime_memory(); + residual_lm_.release_runtime_memory(); + projection_.release_runtime_memory(); + cfm_.release_runtime_memory(); + local_encoder_.release_runtime_memory(); + } + + void release_text_length_memory() { + // Only the prompt-prefill graph is sized by the request text/prompt + // length; the other generator graphs have fixed-size workspaces. Drop it + // after every request so a long-lived session does not retain buffers + // that scale with text length; the next request rebuilds it fresh. + prefill_.release_runtime_memory(); + } + +private: + struct PromptAudioEmbeddingCacheKey { + std::vector prompt_features; + int64_t prompt_patches = 0; + std::vector reference_features; + int64_t reference_patches = 0; + }; + + struct PromptAudioEmbeddingCacheKeyEqual { + bool operator()(const PromptAudioEmbeddingCacheKey &lhs, + const PromptAudioEmbeddingCacheKey &rhs) const { + return lhs.prompt_patches == rhs.prompt_patches && + lhs.reference_patches == rhs.reference_patches && + lhs.prompt_features == rhs.prompt_features && + lhs.reference_features == rhs.reference_features; + } + }; + + struct PromptAudioEmbeddingCacheEntry { + std::vector prompt_embeddings; + std::vector reference_embeddings; + }; + + PrefillSequence build_prefill_sequence(const std::string &target_text, + const VoxCPM1EncodedPrompt *prompt) { + const auto &config = assets_->config; + const int64_t patch_elems = config.patch_size * config.feat_dim; + const std::string normalized_target_text = + normalize_wrapper_text(target_text); + if (prompt != nullptr) { + validate_feature_block(prompt->prompt_features, prompt->prompt_patches, + patch_elems, "prompt"); + validate_feature_block(prompt->reference_features, + prompt->reference_patches, patch_elems, + "reference"); + if (prompt->prompt_patches > 0 && prompt->prompt_text.empty()) { + throw std::runtime_error( + "VoxCPM1 continuation prompt requires prompt text"); + } + } + + const bool use_prompt = has_prompt_audio(prompt); + const bool use_reference = has_reference_audio(prompt); + const PromptAudioEmbeddingCacheEntry *embedding_cache = + prompt != nullptr ? &cached_prompt_audio_embeddings(*prompt) : nullptr; + const std::string combined_text = + use_prompt ? prompt->prompt_text + normalized_target_text + : normalized_target_text; + const VoxCPM1TextPrompt text_prompt = + tokenizer_.build_prompt(combined_text); + const VoxCPM1TextPrompt target_prompt = + tokenizer_.build_prompt(normalized_target_text); + std::vector zero_patch(static_cast(patch_elems), 0.0F); + PrefillSequence sequence; + sequence.target_text_tokens = + static_cast(target_prompt.input_ids.size()); + + auto append_text = [&](int32_t token) { + PrefillRow row; + row.token = token; + row.feature = zero_patch; + row.text_mask = true; + sequence.rows.push_back(std::move(row)); + }; + auto append_audio = [&](const std::vector &features, + const std::vector &embeddings, + int64_t patch_index) { + PrefillRow row; + row.feature = feature_patch(features, patch_index, patch_elems); + row.embedding = hidden_patch(embeddings, patch_index, + config.lm.hidden_size); + row.audio_mask = true; + sequence.rows.push_back(std::move(row)); + }; + + if (use_reference) { + append_text(kRefAudioStartToken); + for (int64_t i = 0; i < prompt->reference_patches; ++i) { + append_audio(prompt->reference_features, + embedding_cache->reference_embeddings, i); + } + append_text(kRefAudioEndToken); + } + for (const int32_t token : text_prompt.input_ids) { + append_text(token); + } + append_text(tokenizer_.audio_start_token_id()); + if (use_prompt) { + for (int64_t i = 0; i < prompt->prompt_patches; ++i) { + append_audio(prompt->prompt_features, embedding_cache->prompt_embeddings, + i); + } + } + + if (sequence.rows.empty() || + static_cast(sequence.rows.size()) >= config.max_length) { + // Caller-controlled: the prompt audio/text decides how many rows this + // is. Report the numbers so the remedy is arithmetic, not guesswork. + throw engine::runtime::CapacityError( + "VoxCPM1 prompt exceeds the model cache length (" + + std::to_string(sequence.rows.size()) + " rows, limit " + + std::to_string(config.max_length) + "); shorten the prompt"); + } + return sequence; + } + + std::vector hidden_patch(const std::vector &embeddings, + int64_t index, int64_t hidden_size) const { + if (index < 0 || hidden_size <= 0 || + static_cast(embeddings.size()) < (index + 1) * hidden_size) { + throw std::runtime_error( + "VoxCPM1 prompt audio embedding cache size mismatch"); + } + const auto begin = + embeddings.begin() + static_cast(index * hidden_size); + return std::vector(begin, + begin + static_cast(hidden_size)); + } + + std::vector + encode_feature_embeddings(const std::vector &features, int64_t patches, + int64_t patch_elems) const { + std::vector embeddings; + embeddings.reserve(static_cast(patches * + assets_->config.lm.hidden_size)); + for (int64_t i = 0; i < patches; ++i) { + auto embedding = + local_encoder_.encode_patch(feature_patch(features, i, patch_elems)); + embeddings.insert(embeddings.end(), embedding.begin(), embedding.end()); + } + return embeddings; + } + + const PromptAudioEmbeddingCacheEntry & + cached_prompt_audio_embeddings(const VoxCPM1EncodedPrompt &prompt) { + const int64_t patch_elems = + assets_->config.patch_size * assets_->config.feat_dim; + PromptAudioEmbeddingCacheKey key; + key.prompt_features = prompt.prompt_features; + key.prompt_patches = prompt.prompt_patches; + key.reference_features = prompt.reference_features; + key.reference_patches = prompt.reference_patches; + if (auto *cached = prompt_audio_embedding_cache_.find(key)) { + debug::trace_log_scalar("voxcpm1.prompt_audio_embedding_cache.hit", 1); + debug::trace_log_scalar( + "voxcpm1.prompt_audio_embedding_cache.slots", + static_cast(prompt_audio_embedding_cache_.capacity())); + debug::trace_log_scalar( + "voxcpm1.prompt_audio_embedding_cache.entries", + static_cast(prompt_audio_embedding_cache_.size())); + debug::trace_log_scalar("voxcpm1.prompt_audio_embedding_cache.evicted", + 0); + debug::timing_log_scalar("voxcpm1.prompt_audio_embedding_ms", 0.0); + return *cached; + } + + const auto embedding_start = Clock::now(); + PromptAudioEmbeddingCacheEntry entry; + entry.prompt_embeddings = encode_feature_embeddings( + prompt.prompt_features, prompt.prompt_patches, patch_elems); + entry.reference_embeddings = encode_feature_embeddings( + prompt.reference_features, prompt.reference_patches, patch_elems); + const double embedding_ms = engine::debug::elapsed_ms(embedding_start); + if (prompt_audio_embedding_cache_.capacity() == 0) { + uncached_prompt_audio_embedding_ = std::move(entry); + debug::trace_log_scalar("voxcpm1.prompt_audio_embedding_cache.hit", 0); + debug::trace_log_scalar("voxcpm1.prompt_audio_embedding_cache.slots", 0); + debug::trace_log_scalar("voxcpm1.prompt_audio_embedding_cache.entries", + 0); + debug::trace_log_scalar("voxcpm1.prompt_audio_embedding_cache.evicted", + 0); + debug::timing_log_scalar("voxcpm1.prompt_audio_embedding_ms", + embedding_ms); + return *uncached_prompt_audio_embedding_; + } + const bool will_evict = prompt_audio_embedding_cache_.size() >= + prompt_audio_embedding_cache_.capacity(); + prompt_audio_embedding_cache_.put(std::move(key), std::move(entry)); + PromptAudioEmbeddingCacheKey lookup; + lookup.prompt_features = prompt.prompt_features; + lookup.prompt_patches = prompt.prompt_patches; + lookup.reference_features = prompt.reference_features; + lookup.reference_patches = prompt.reference_patches; + auto *cached = prompt_audio_embedding_cache_.find(lookup); + if (cached == nullptr) { + throw std::runtime_error( + "VoxCPM1 prompt audio embedding cache insert failed"); + } + debug::trace_log_scalar("voxcpm1.prompt_audio_embedding_cache.hit", 0); + debug::trace_log_scalar( + "voxcpm1.prompt_audio_embedding_cache.slots", + static_cast(prompt_audio_embedding_cache_.capacity())); + debug::trace_log_scalar( + "voxcpm1.prompt_audio_embedding_cache.entries", + static_cast(prompt_audio_embedding_cache_.size())); + debug::trace_log_scalar("voxcpm1.prompt_audio_embedding_cache.evicted", + will_evict ? 1 : 0); + debug::timing_log_scalar("voxcpm1.prompt_audio_embedding_ms", + embedding_ms); + return *cached; + } + + VoxCPM1Result + generate_once(const PrefillSequence &prefill, int64_t max_tokens, + const VoxCPM1GenerationOptions &options, + uint64_t noise_start_index, + std::vector *streaming_chunks = + nullptr, + const std::function + &streaming_chunk_callback = nullptr) { + const auto &config = assets_->config; + const int64_t hidden_size = config.lm.hidden_size; + const int64_t patch_elems = config.patch_size * config.feat_dim; + if (static_cast(prefill.rows.size()) + max_tokens > + config.max_length) { + // Same: prefill rows come from the input text, max_tokens from the + // request. Both are the caller's to reduce. + throw engine::runtime::CapacityError( + "VoxCPM1 generation exceeds the model cache length (" + + std::to_string(prefill.rows.size()) + " prefill rows + " + + std::to_string(max_tokens) + " requested tokens, limit " + + std::to_string(config.max_length) + "); shorten the input text"); + } + base_lm_.reset(); + residual_lm_.reset(); + + std::vector zero_hidden(static_cast(hidden_size), 0.0F); + std::vector zero_patch(static_cast(patch_elems), 0.0F); + VoxCPM1PromptPrefillInput prefill_input; + prefill_input.steps = static_cast(prefill.rows.size()); + prefill_input.input_embeddings.reserve( + static_cast(prefill_input.steps * hidden_size)); + prefill_input.current_embeddings.reserve( + static_cast(prefill_input.steps * hidden_size)); + prefill_input.text_mask.reserve(static_cast(prefill_input.steps)); + prefill_input.audio_mask.reserve(static_cast(prefill_input.steps)); + std::vector prefix_cond = zero_patch; + for (const auto &row : prefill.rows) { + if (row.text_mask == row.audio_mask) { + throw std::runtime_error("VoxCPM1 prefill row mask is invalid"); + } + std::vector input_embedding; + std::vector current_embed = zero_hidden; + if (row.text_mask) { + input_embedding = text_embedding_.embed_token(row.token); + } else { + current_embed = row.embedding; + if (static_cast(current_embed.size()) != hidden_size) { + throw std::runtime_error( + "VoxCPM1 prompt audio embedding size mismatch"); + } + input_embedding = current_embed; + } + if (row.audio_mask) { + prefix_cond = row.feature; + } + prefill_input.input_embeddings.insert(prefill_input.input_embeddings.end(), + input_embedding.begin(), + input_embedding.end()); + prefill_input.current_embeddings.insert( + prefill_input.current_embeddings.end(), current_embed.begin(), + current_embed.end()); + prefill_input.text_mask.push_back(row.text_mask ? 1.0F : 0.0F); + prefill_input.audio_mask.push_back(row.audio_mask ? 1.0F : 0.0F); + } + const auto prefill_output = prefill_.run(prefill_input); + if (std::getenv("VOXCPM_DUMP_PREFILL") != nullptr) { + auto dump_vec = [](const char *tag, const std::vector &v) { + double sum = 0.0; + double sum2 = 0.0; + for (float x : v) { + sum += x; + sum2 += static_cast(x) * x; + } + fprintf(stderr, "[DUMP_PREFILL] %s n=%zu sum=%.6g l2=%.6g", tag, + v.size(), sum, std::sqrt(sum2)); + for (size_t i = 0; i < std::min(8, v.size()); ++i) { + fprintf(stderr, " v[%zu]=%.6g", i, static_cast(v[i])); + } + fprintf(stderr, "\n"); + }; + dump_vec("lm_hidden", prefill_output.lm_hidden); + dump_vec("residual_hidden", prefill_output.residual_hidden); + } + base_lm_.import_state(prefill_output.base_state); + residual_lm_.import_state(prefill_output.residual_state); + std::vector lm_hidden = prefill_output.lm_hidden; + std::vector residual_hidden = prefill_output.residual_hidden; + // The prefill graph holds the largest sequence-shaped workspace; its + // outputs have been copied to host hiddens and its KV state imported + // into the step runtimes, so nothing below references it. Drop it now + // so the token loop runs against the much smaller step graphs. + prefill_.release_runtime_memory(); + + VoxCPM1Result result; + std::vector context_rows; + for (auto it = prefill.rows.rbegin(); it != prefill.rows.rend(); ++it) { + if (!it->audio_mask) { + break; + } + if (static_cast(context_rows.size()) >= + kStreamingPrefixLen - 1) { + break; + } + context_rows.push_back(&*it); + } + result.decode_trim_patches = static_cast(context_rows.size()); + for (auto it = context_rows.rbegin(); it != context_rows.rend(); ++it) { + append_patch(result.decode_features, (*it)->feature, patch_elems); + ++result.decode_patches; + } + uint64_t patch_noise_start = noise_start_index; + for (int64_t index = 0; index < max_tokens; ++index) { + const auto projected = + projection_.run(lm_hidden, residual_hidden, zero_hidden); + if (index == 0 && std::getenv("VOXCPM_DUMP_PREFILL") != nullptr) { + auto dump_vec = [](const char *tag, const std::vector &v) { + double sum = 0.0; + double sum2 = 0.0; + for (float x : v) { + sum += x; + sum2 += static_cast(x) * x; + } + fprintf(stderr, "[DUMP_PREFILL] %s n=%zu sum=%.6g l2=%.6g", tag, + v.size(), sum, std::sqrt(sum2)); + for (size_t i = 0; i < std::min(8, v.size()); ++i) { + fprintf(stderr, " v[%zu]=%.6g", i, static_cast(v[i])); + } + fprintf(stderr, "\n"); + }; + dump_vec("lm_to_dit", projected.current_lm_dit_hidden); + dump_vec("res_to_dit", projected.residual_dit_hidden); + } + // Check if fusion_concat_proj weight exists and was loaded (not synthesized) + // This matches VoxCPM.cpp behavior which checks weight existence + // For V1 models, synthesized weights (Xavier init) should not count as present + const bool has_fusion_proj = + weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && + config.architecture == "voxcpm2"; + const auto mu = has_fusion_proj + ? concat_dit_mu(projected.current_lm_dit_hidden, + projected.residual_dit_hidden) + : add_dit_mu(projected.current_lm_dit_hidden, + projected.residual_dit_hidden); + const auto patch = cfm_.generate_patch( + mu, prefix_cond, options.num_inference_steps, options.guidance_scale, + options.seed, patch_noise_start, options.cfm_noise_file); + if (const char *patch_dump_path = std::getenv("VOXCPM_DUMP_PATCH")) { + FILE *patch_file = std::fopen(patch_dump_path, "ab"); + if (patch_file != nullptr) { + std::fwrite(patch.data(), sizeof(float), patch.size(), patch_file); + std::fclose(patch_file); + } + } + patch_noise_start += static_cast(patch_elems); + append_patch(result.generated_features, patch, patch_elems); + ++result.generated_patches; + append_patch(result.decode_features, patch, patch_elems); + ++result.decode_patches; + if (streaming_chunks != nullptr || streaming_chunk_callback) { + VoxCPM1StreamingChunk chunk; + chunk.decode_features = patch; + chunk.decode_patches = 1; + chunk.generated_patches = result.generated_patches; + if (streaming_chunk_callback) { + streaming_chunk_callback(chunk); + } + if (streaming_chunks != nullptr) { + streaming_chunks->push_back(std::move(chunk)); + } + } + prefix_cond = patch; + + if (index > options.min_tokens && + stop_class(projected.current_stop_logits) == 1) { + break; + } + if (std::getenv("VOXCPM1_LOG_STOP") != nullptr) { + const auto &sl = projected.current_stop_logits; + fprintf(stderr, "[stop_logits] pos=%lld pre stop0=%.5f stop1=%.5f\n", + static_cast(index), + sl.empty() ? 0.0 : static_cast(sl[0]), + sl.size() < 2 ? 0.0 : static_cast(sl[1])); + } + + const auto curr_embed = local_encoder_.encode_patch(patch); + const auto next_lm = base_lm_.run_step(curr_embed).hidden; + const auto next_projected = + projection_.run(next_lm, residual_hidden, curr_embed); + lm_hidden = next_projected.fsq_hidden; + residual_hidden = + residual_lm_.run_step(next_projected.residual_input).hidden; + } + return result; + } + + std::shared_ptr assets_; + std::shared_ptr weights_; + VoxCPM1TokenizerWrapper tokenizer_; + VoxCPM1TextEmbeddingRuntime text_embedding_; + VoxCPM1PromptPrefillRuntime prefill_; + VoxCPM1MiniCPMStepRuntime base_lm_; + VoxCPM1MiniCPMStepRuntime residual_lm_; + VoxCPM1StepProjectionRuntime projection_; + VoxCPM1CFMRuntime cfm_; + VoxCPM1LocalEncoderRuntime local_encoder_; + engine::runtime::CacheSlots + prompt_audio_embedding_cache_; + std::optional + uncached_prompt_audio_embedding_; +}; + +VoxCPM1FeatureGeneratorRuntime::VoxCPM1FeatureGeneratorRuntime( + std::shared_ptr assets, + engine::core::ExecutionContext &execution_context, + VoxCPM1FeatureGeneratorConfig config) + : impl_(std::make_unique(std::move(assets), execution_context, + std::move(config))) {} + +VoxCPM1FeatureGeneratorRuntime::~VoxCPM1FeatureGeneratorRuntime() = default; + +VoxCPM1Result VoxCPM1FeatureGeneratorRuntime::generate_zero_shot( + const std::string &text, const VoxCPM1GenerationOptions &options) { + return impl_->generate_zero_shot(text, options); +} + +VoxCPM1Result VoxCPM1FeatureGeneratorRuntime::generate( + const std::string &text, const VoxCPM1EncodedPrompt *prompt, + const VoxCPM1GenerationOptions &options) { + return impl_->generate(text, prompt, options); +} + +VoxCPM1StreamingResult VoxCPM1FeatureGeneratorRuntime::generate_streaming( + const std::string &text, const VoxCPM1EncodedPrompt *prompt, + const VoxCPM1GenerationOptions &options, + const std::function &chunk_callback) { + return impl_->generate_streaming(text, prompt, options, chunk_callback); +} + +void VoxCPM1FeatureGeneratorRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + +void VoxCPM1FeatureGeneratorRuntime::release_text_length_memory() { + impl_->release_text_length_memory(); +} + +} // namespace engine::community_models::voxcpm1 diff --git a/src/models/voxcpm2/gguf_metadata.cpp b/src/community_models/voxcpm1/gguf_metadata.cpp similarity index 96% rename from src/models/voxcpm2/gguf_metadata.cpp rename to src/community_models/voxcpm1/gguf_metadata.cpp index da9ddcac..fc60c3c6 100644 --- a/src/models/voxcpm2/gguf_metadata.cpp +++ b/src/community_models/voxcpm1/gguf_metadata.cpp @@ -1,4 +1,4 @@ -#include "engine/models/voxcpm2/gguf_metadata.h" +#include "engine/community_models/voxcpm1/gguf_metadata.h" #include "engine/framework/assets/tensor_source.h" @@ -6,7 +6,7 @@ #include -namespace engine::models::voxcpm2 { +namespace engine::community_models::voxcpm1 { GgufMetadataReader::GgufMetadataReader(const engine::assets::TensorSource & source) { // Metadata-only open: no_alloc=true with no ggml context parses the GGUF @@ -144,4 +144,4 @@ std::vector GgufMetadataReader::require_i32_array(std::string_view key) throw std::runtime_error("GGUF metadata key not found: " + std::string(key)); } -} // namespace engine::models::voxcpm2 \ No newline at end of file +} // namespace engine::community_models::voxcpm1 \ No newline at end of file diff --git a/src/community_models/voxcpm1/minicpm.cpp b/src/community_models/voxcpm1/minicpm.cpp new file mode 100644 index 00000000..06b80344 --- /dev/null +++ b/src/community_models/voxcpm1/minicpm.cpp @@ -0,0 +1,1268 @@ +#include "engine/community_models/voxcpm1/minicpm.h" + +#include "minicpm_blocks.h" + +#include "engine/framework/core/execution_context.h" +#include "engine/framework/modules/activation_modules.h" +#include "engine/framework/modules/linear_module.h" +#include "engine/framework/modules/lookup_modules.h" +#include "engine/framework/modules/norm_modules.h" +#include "engine/framework/modules/optimizations/fast_kv_modules.h" +#include "engine/framework/modules/primitive_modules.h" +#include "engine/framework/modules/structural_modules.h" +#include "engine/framework/modules/weight_binding.h" + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace engine::community_models::voxcpm1 { + +namespace { + +namespace weight_binding = engine::modules::binding; + +VoxCPM1MiniCPMConfig residual_lm_config(const VoxCPM1Config &config) { + VoxCPM1MiniCPMConfig out = config.lm; + out.num_hidden_layers = config.residual_lm_num_layers; + out.vocab_size = 0; + out.no_rope = config.residual_lm_no_rope; + return out; +} + +VoxCPM1MiniCPMConfig +local_transformer_config(const VoxCPM1MiniCPMConfig &base, + const VoxCPM1LocalTransformerConfig &local) { + VoxCPM1MiniCPMConfig out = base; + out.hidden_size = local.hidden_dim; + out.intermediate_size = local.ffn_dim; + out.num_attention_heads = local.num_heads; + out.num_hidden_layers = local.num_layers; + out.num_key_value_heads = base.num_key_value_heads; + out.kv_channels = local.kv_channels; + out.vocab_size = 0; + return out; +} + +const std::vector & +active_rope_factors(const VoxCPM1MiniCPMConfig &config) { + if (config.max_position_embeddings > + config.rope_scaling.original_max_position_embeddings) { + return config.rope_scaling.long_factor; + } + return config.rope_scaling.short_factor; +} + +float rope_attn_factor(const VoxCPM1MiniCPMConfig &config) { + const auto original = + static_cast(config.rope_scaling.original_max_position_embeddings); + if (original <= 1.0F || + config.max_position_embeddings <= + config.rope_scaling.original_max_position_embeddings) { + return 1.0F; + } + const float scale = + static_cast(config.max_position_embeddings) / original; + return std::sqrt(1.0F + std::log(scale) / std::log(original)); +} + +engine::modules::LinearWeights +linear_weights(engine::core::BackendWeightStore &store, + const engine::assets::TensorSource &source, + const std::string &prefix, + engine::assets::TensorStorageType storage_type, + int64_t out_features, int64_t in_features, bool use_bias) { + return weight_binding::linear_from_source(store, source, prefix, storage_type, + out_features, in_features, use_bias); +} + +VoxCPM1MiniCPMWeights load_minicpm_weights( + engine::core::BackendWeightStore &store, + const engine::assets::TensorSource &source, const std::string &prefix, + const VoxCPM1MiniCPMConfig &config, + engine::assets::TensorStorageType storage_type, bool load_token_embedding) { + const int64_t dim = head_dim(config); + VoxCPM1MiniCPMWeights weights; + weights.config = config; + if (load_token_embedding) { + weights.token_embedding = + store.load_tensor(source, prefix + ".embed_tokens.weight", storage_type, + {config.vocab_size, config.hidden_size}); + } + if (!config.no_rope) { + const auto &factors = active_rope_factors(config); + if (static_cast(factors.size()) != dim / 2) { + throw std::runtime_error("VoxCPM1 MiniCPM RoPE factor shape mismatch"); + } + weights.rope_factors = + store.make_from_f32(engine::core::TensorShape::from_dims({dim / 2}), + engine::assets::TensorStorageType::F32, factors); + weights.rope_attn_factor = rope_attn_factor(config); + } + weights.layers.reserve(static_cast(config.num_hidden_layers)); + for (int64_t layer = 0; layer < config.num_hidden_layers; ++layer) { + const std::string layer_prefix = + prefix + ".layers." + std::to_string(layer); + VoxCPM1MiniCPMLayerWeights layer_weights; + layer_weights.input_norm = weight_binding::norm_weight_from_source( + store, source, layer_prefix + ".input_layernorm", config.hidden_size); + layer_weights.q_proj = linear_weights( + store, source, layer_prefix + ".self_attn.q_proj", storage_type, + config.num_attention_heads * dim, config.hidden_size, false); + layer_weights.k_proj = linear_weights( + store, source, layer_prefix + ".self_attn.k_proj", storage_type, + config.num_key_value_heads * dim, config.hidden_size, false); + layer_weights.v_proj = linear_weights( + store, source, layer_prefix + ".self_attn.v_proj", storage_type, + config.num_key_value_heads * dim, config.hidden_size, false); + layer_weights.o_proj = linear_weights( + store, source, layer_prefix + ".self_attn.o_proj", storage_type, + config.hidden_size, config.num_attention_heads * dim, false); + layer_weights.post_norm = weight_binding::norm_weight_from_source( + store, source, layer_prefix + ".post_attention_layernorm", + config.hidden_size); + layer_weights.gate_proj = linear_weights( + store, source, layer_prefix + ".mlp.gate_proj", storage_type, + config.intermediate_size, config.hidden_size, false); + layer_weights.up_proj = linear_weights( + store, source, layer_prefix + ".mlp.up_proj", storage_type, + config.intermediate_size, config.hidden_size, false); + layer_weights.down_proj = linear_weights( + store, source, layer_prefix + ".mlp.down_proj", storage_type, + config.hidden_size, config.intermediate_size, false); + weights.layers.push_back(std::move(layer_weights)); + } + weights.norm = weight_binding::norm_weight_from_source( + store, source, prefix + ".norm", config.hidden_size); + return weights; +} + +} // namespace + +int64_t head_dim(const VoxCPM1MiniCPMConfig &config) { + if (config.kv_channels <= 0 || config.num_attention_heads <= 0 || + config.num_key_value_heads <= 0) { + throw std::runtime_error("VoxCPM1 MiniCPM attention config is invalid"); + } + if (config.num_attention_heads % config.num_key_value_heads != 0) { + throw std::runtime_error( + "VoxCPM1 MiniCPM attention heads must be divisible by KV heads"); + } + return config.kv_channels; +} + +const VoxCPM1MiniCPMWeights & +select_minicpm_weights(const VoxCPM1ModelWeights &weights, + VoxCPM1MiniCPMKind kind) { + switch (kind) { + case VoxCPM1MiniCPMKind::BaseLM: + return weights.base_lm; + case VoxCPM1MiniCPMKind::ResidualLM: + return weights.residual_lm; + } + throw std::runtime_error( + "VoxCPM1 MiniCPM runtime received an unknown graph kind"); +} + +std::shared_ptr +load_model_weights(const VoxCPM1Assets &assets, + engine::core::ExecutionContext &execution_context, + size_t weight_context_bytes, + engine::assets::TensorStorageType storage_type) { + auto weights = std::make_shared(); + weights->store = std::make_shared( + execution_context.backend(), execution_context.backend_type(), + "voxcpm1.model.weights", weight_context_bytes); + auto &store = *weights->store; + const auto &source = *assets.model_weights; + weights->base_lm = load_minicpm_weights(store, source, "base_lm", + assets.config.lm, storage_type, true); + weights->residual_lm = load_minicpm_weights(store, source, "residual_lm", + residual_lm_config(assets.config), + storage_type, false); + + const auto encoder_config = + local_transformer_config(assets.config.lm, assets.config.encoder); + weights->feat_encoder.special_token = + store.load_tensor(source, "feat_encoder.special_token", storage_type, + {1, 1, 1, assets.config.encoder.hidden_dim}); + weights->feat_encoder.in_proj = linear_weights( + store, source, "feat_encoder.in_proj", storage_type, + assets.config.encoder.hidden_dim, assets.config.feat_dim, true); + weights->feat_encoder.encoder = + load_minicpm_weights(store, source, "feat_encoder.encoder", + encoder_config, storage_type, false); + + const auto dit_config = + local_transformer_config(assets.config.lm, assets.config.dit); + weights->dit.in_proj = linear_weights( + store, source, "feat_decoder.estimator.in_proj", storage_type, + assets.config.dit.hidden_dim, assets.config.feat_dim, true); + weights->dit.cond_proj = linear_weights( + store, source, "feat_decoder.estimator.cond_proj", storage_type, + assets.config.dit.hidden_dim, assets.config.feat_dim, true); + weights->dit.out_proj = linear_weights( + store, source, "feat_decoder.estimator.out_proj", storage_type, + assets.config.feat_dim, assets.config.dit.hidden_dim, true); + weights->dit.time_mlp_1 = linear_weights( + store, source, "feat_decoder.estimator.time_mlp.linear_1", storage_type, + assets.config.dit.hidden_dim, assets.config.dit.hidden_dim, true); + weights->dit.time_mlp_2 = linear_weights( + store, source, "feat_decoder.estimator.time_mlp.linear_2", storage_type, + assets.config.dit.hidden_dim, assets.config.dit.hidden_dim, true); + weights->dit.delta_time_mlp_1 = linear_weights( + store, source, "feat_decoder.estimator.delta_time_mlp.linear_1", + storage_type, assets.config.dit.hidden_dim, assets.config.dit.hidden_dim, + true); + weights->dit.delta_time_mlp_2 = linear_weights( + store, source, "feat_decoder.estimator.delta_time_mlp.linear_2", + storage_type, assets.config.dit.hidden_dim, assets.config.dit.hidden_dim, + true); + weights->dit.decoder = + load_minicpm_weights(store, source, "feat_decoder.estimator.decoder", + dit_config, storage_type, false); + + weights->projections.fsq_in_proj = + linear_weights(store, source, "fsq_layer.in_proj", storage_type, + assets.config.scalar_quantization_latent_dim, + assets.config.lm.hidden_size, true); + weights->projections.fsq_out_proj = + linear_weights(store, source, "fsq_layer.out_proj", storage_type, + assets.config.lm.hidden_size, + assets.config.scalar_quantization_latent_dim, true); + weights->projections.enc_to_lm_proj = linear_weights( + store, source, "enc_to_lm_proj", storage_type, + assets.config.lm.hidden_size, assets.config.encoder.hidden_dim, true); + weights->projections.lm_to_dit_proj = linear_weights( + store, source, "lm_to_dit_proj", storage_type, + assets.config.dit.hidden_dim, assets.config.lm.hidden_size, true); + weights->projections.res_to_dit_proj = linear_weights( + store, source, "res_to_dit_proj", storage_type, + assets.config.dit.hidden_dim, assets.config.lm.hidden_size, true); + weights->projections.fusion_concat_proj = linear_weights( + store, source, "fusion_concat_proj", storage_type, + assets.config.lm.hidden_size, assets.config.lm.hidden_size * 2, true); + weights->projections.stop_proj = linear_weights( + store, source, "stop_proj", storage_type, assets.config.lm.hidden_size, + assets.config.lm.hidden_size, true); + weights->projections.stop_head = + linear_weights(store, source, "stop_head", storage_type, 2, + assets.config.lm.hidden_size, false); + store.upload(); + return weights; +} + + + +class VoxCPM1WeightsRuntime::Impl { +public: + Impl(std::shared_ptr assets, + engine::core::ExecutionContext &execution_context, + size_t weight_context_bytes, + engine::assets::TensorStorageType weight_storage_type) + : assets_(std::move(assets)), execution_context_(execution_context) { + if (assets_ == nullptr) { + throw std::runtime_error("VoxCPM1 weights runtime requires assets"); + } + weights_ = load_model_weights(*assets_, execution_context_, + weight_context_bytes, weight_storage_type); + } + + const VoxCPM1Assets &assets() const noexcept { return *assets_; } + const VoxCPM1ModelWeights &weights() const noexcept { return *weights_; } + ggml_backend_t backend() const noexcept { + return execution_context_.backend(); + } + int threads() const noexcept { + return std::max(1, execution_context_.config().threads); + } + bool weights_uploaded() const noexcept { + return weights_ != nullptr && weights_->store != nullptr; + } + +private: + std::shared_ptr assets_; + engine::core::ExecutionContext &execution_context_; + std::shared_ptr weights_; +}; + +VoxCPM1WeightsRuntime::VoxCPM1WeightsRuntime( + std::shared_ptr assets, + engine::core::ExecutionContext &execution_context, + size_t weight_context_bytes, + engine::assets::TensorStorageType weight_storage_type) + : impl_(std::make_unique(std::move(assets), execution_context, + weight_context_bytes, weight_storage_type)) { +} + +VoxCPM1WeightsRuntime::~VoxCPM1WeightsRuntime() = default; + +const VoxCPM1Assets &VoxCPM1WeightsRuntime::assets() const noexcept { + return impl_->assets(); +} + +const VoxCPM1ModelWeights &VoxCPM1WeightsRuntime::weights() const noexcept { + return impl_->weights(); +} + +ggml_backend_t VoxCPM1WeightsRuntime::backend() const noexcept { + return impl_->backend(); +} + +int VoxCPM1WeightsRuntime::threads() const noexcept { return impl_->threads(); } + +bool VoxCPM1WeightsRuntime::weights_uploaded() const noexcept { + return impl_->weights_uploaded(); +} + +class VoxCPM1TextEmbeddingRuntime::Impl { +public: + Impl(std::shared_ptr weights, + size_t graph_context_bytes, bool mem_saver) + : weights_(std::move(weights)), mem_saver_(mem_saver) { + if (weights_ == nullptr) { + throw std::runtime_error( + "VoxCPM1 text embedding runtime requires weights"); + } + build(graph_context_bytes); + } + + ~Impl() { + release_graph(); + } + + void release_runtime_memory() { release_graph(); } + + std::vector embed_token(int32_t token_id) { + const auto &config = weights_->assets().config.lm; + if (token_id < 0 || token_id >= config.vocab_size) { + throw std::runtime_error( + "VoxCPM1 text embedding token id is out of range"); + } + ggml_backend_tensor_set(token_id_, &token_id, 0, sizeof(token_id)); + engine::core::set_backend_threads(weights_->backend(), weights_->threads()); + const ggml_status status = + engine::core::compute_backend_graph(weights_->backend(), graph_); + ggml_backend_synchronize(weights_->backend()); + if (status != GGML_STATUS_SUCCESS) { + throw std::runtime_error("VoxCPM1 text embedding graph compute failed"); + } + std::vector output(static_cast(config.hidden_size), 0.0F); + ggml_backend_tensor_get(output_, output.data(), 0, + output.size() * sizeof(float)); + return output; + } + +private: + void release_graph() { + if (graph_ != nullptr) { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + } + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + buffer_ = nullptr; + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + graph_ = nullptr; + token_id_ = nullptr; + output_ = nullptr; + ctx_.reset(); + } + + void build(size_t graph_context_bytes) { + const auto &config = weights_->assets().config.lm; + if (graph_context_bytes == 0) { + throw std::runtime_error( + "VoxCPM1 text embedding graph context bytes must be non-zero"); + } + if (!weights_->weights().base_lm.token_embedding.has_value()) { + throw std::runtime_error("VoxCPM1 text embedding weight is missing"); + } + ggml_init_params params{graph_context_bytes, nullptr, true}; + ctx_.reset(ggml_init(params)); + if (ctx_ == nullptr) { + throw std::runtime_error( + "failed to initialize VoxCPM1 text embedding graph context"); + } + engine::core::ModuleBuildContext ctx{ctx_.get(), "voxcpm1.text_embedding"}; + token_id_ = ggml_new_tensor_1d(ctx_.get(), GGML_TYPE_I32, 1); + if (mem_saver_) { + ggml_set_input(token_id_); + } + auto token = engine::core::wrap_tensor( + token_id_, engine::core::TensorShape::from_dims({1}), GGML_TYPE_I32); + auto embedding = + engine::modules::EmbeddingModule( + {config.vocab_size, config.hidden_size}) + .build(ctx, token, *weights_->weights().base_lm.token_embedding); + const float scale = + config.use_mup ? static_cast(config.scale_emb) : 1.0F; + embedding = scale_tensor(ctx, embedding, scale); + embedding = engine::core::reshape_tensor( + ctx, ensure_contiguous(ctx, embedding), + engine::core::TensorShape::from_dims({config.hidden_size})); + output_ = embedding.tensor; + ggml_set_output(output_); + if (mem_saver_ && output_->view_src != nullptr) { + ggml_set_output(output_->view_src); + } + graph_ = ggml_new_graph_custom(ctx_.get(), kDefaultGraphNodes, false); + ggml_build_forward_expand(graph_, output_); + if (mem_saver_) { + gallocr_ = ggml_gallocr_new( + ggml_backend_get_default_buffer_type(weights_->backend())); + if (gallocr_ == nullptr || !ggml_gallocr_reserve(gallocr_, graph_) || + !ggml_gallocr_alloc_graph(gallocr_, graph_)) { + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + throw std::runtime_error( + "failed to allocate VoxCPM1 text embedding graph"); + } + return; + } + buffer_ = ggml_backend_alloc_ctx_tensors(ctx_.get(), weights_->backend()); + if (buffer_ == nullptr) { + throw std::runtime_error( + "failed to allocate VoxCPM1 text embedding graph"); + } + } + + std::shared_ptr weights_; + bool mem_saver_ = false; + std::unique_ptr ctx_; + ggml_tensor *token_id_ = nullptr; + ggml_tensor *output_ = nullptr; + ggml_cgraph *graph_ = nullptr; + ggml_backend_buffer_t buffer_ = nullptr; + ggml_gallocr_t gallocr_ = nullptr; +}; + +VoxCPM1TextEmbeddingRuntime::VoxCPM1TextEmbeddingRuntime( + std::shared_ptr weights, + size_t graph_context_bytes, bool mem_saver) + : impl_(std::make_unique(std::move(weights), graph_context_bytes, + mem_saver)) {} + +VoxCPM1TextEmbeddingRuntime::~VoxCPM1TextEmbeddingRuntime() = default; + +std::vector VoxCPM1TextEmbeddingRuntime::embed_token(int32_t token_id) { + return impl_->embed_token(token_id); +} + +void VoxCPM1TextEmbeddingRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + +struct MiniCPMLayerWithCacheOutput { + engine::core::TensorValue output; + engine::core::TensorValue key; + engine::core::TensorValue value; +}; + +engine::core::TensorValue +mask_sequence(engine::core::ModuleBuildContext &ctx, + const engine::core::TensorValue &input, + const engine::core::TensorValue &mask) { + auto repeated = engine::core::wrap_tensor( + ggml_repeat(ctx.ggml, mask.tensor, input.tensor), input.shape, + GGML_TYPE_F32); + return engine::core::wrap_tensor( + ggml_mul(ctx.ggml, input.tensor, repeated.tensor), input.shape, + GGML_TYPE_F32); +} + +MiniCPMLayerWithCacheOutput +minicpm_prefill_layer(engine::core::ModuleBuildContext &ctx, + const engine::core::TensorValue &input, + const engine::core::TensorValue &positions, + const VoxCPM1MiniCPMLayerWeights &layer, + const VoxCPM1MiniCPMWeights &weights) { + const auto &config = weights.config; + const int64_t dim = head_dim(config); + const int64_t kv_repeats = + config.num_attention_heads / config.num_key_value_heads; + const engine::modules::AddModule add; + auto hidden = engine::modules::RMSNormModule( + {config.hidden_size, config.rms_norm_eps, true, false}) + .build(ctx, input, layer.input_norm); + auto q = engine::modules::LinearModule( + binding::linear_config(config.hidden_size, + config.num_attention_heads * dim, false)) + .build(ctx, hidden, layer.q_proj); + auto k = engine::modules::LinearModule( + binding::linear_config(config.hidden_size, + config.num_key_value_heads * dim, false)) + .build(ctx, hidden, layer.k_proj); + auto v = engine::modules::LinearModule( + binding::linear_config(config.hidden_size, + config.num_key_value_heads * dim, false)) + .build(ctx, hidden, layer.v_proj); + q = apply_minicpm_rope(ctx, + reshape_heads(ctx, q, config.num_attention_heads, dim), + positions, weights); + k = apply_minicpm_rope(ctx, + reshape_heads(ctx, k, config.num_key_value_heads, dim), + positions, weights); + v = reshape_heads(ctx, v, config.num_key_value_heads, dim); + auto q_heads = engine::modules::TransposeModule({{0, 2, 1, 3}, q.shape.rank}) + .build(ctx, q); + auto k_heads = repeat_kv_heads( + ctx, + engine::modules::TransposeModule({{0, 2, 1, 3}, k.shape.rank}) + .build(ctx, k), + kv_repeats); + auto v_heads = repeat_kv_heads( + ctx, + engine::modules::TransposeModule({{0, 2, 1, 3}, v.shape.rank}) + .build(ctx, v), + kv_repeats); + auto context = attention_from_heads(ctx, q_heads, k_heads, v_heads, dim, true); + context = engine::modules::TransposeModule({{0, 2, 1, 3}, context.shape.rank}) + .build(ctx, context); + context = engine::core::reshape_tensor( + ctx, ensure_contiguous(ctx, context), + engine::core::TensorShape::from_dims({input.shape.dims[0], + input.shape.dims[1], + config.num_attention_heads * dim})); + auto attn = engine::modules::LinearModule( + binding::linear_config(config.num_attention_heads * dim, + config.hidden_size, false)) + .build(ctx, context, layer.o_proj); + auto x = + add.build(ctx, input, + scale_tensor(ctx, attn, + config.use_mup ? config.scale_depth / + std::sqrt(static_cast( + config.num_hidden_layers)) + : 1.0F)); + + hidden = engine::modules::RMSNormModule( + {config.hidden_size, config.rms_norm_eps, true, false}) + .build(ctx, x, layer.post_norm); + auto gate = engine::modules::LinearModule( + binding::linear_config(config.hidden_size, + config.intermediate_size, false)) + .build(ctx, hidden, layer.gate_proj); + gate = engine::modules::SiluModule{}.build(ctx, gate); + auto up = engine::modules::LinearModule( + binding::linear_config(config.hidden_size, + config.intermediate_size, false)) + .build(ctx, hidden, layer.up_proj); + auto gated = engine::modules::MulModule{}.build(ctx, gate, up); + auto ff = engine::modules::LinearModule( + binding::linear_config(config.intermediate_size, + config.hidden_size, false)) + .build(ctx, gated, layer.down_proj); + auto output = add.build( + ctx, x, + scale_tensor(ctx, ff, + config.use_mup + ? config.scale_depth / std::sqrt(static_cast( + config.num_hidden_layers)) + : 1.0F)); + return {output, k, v}; +} + + +class VoxCPM1PromptPrefillRuntime::Impl { +public: + Impl(std::shared_ptr weights, + size_t graph_context_bytes, bool mem_saver) + : weights_(std::move(weights)), graph_context_bytes_(graph_context_bytes), + mem_saver_(mem_saver) { + if (weights_ == nullptr) { + throw std::runtime_error("VoxCPM1 prompt prefill runtime requires weights"); + } + if (graph_context_bytes_ == 0) { + throw std::runtime_error( + "VoxCPM1 prompt prefill graph context bytes must be non-zero"); + } + } + + ~Impl() { release_graph(); } + + void release_runtime_memory() { release_graph(); } + + VoxCPM1PromptPrefillOutput run(const VoxCPM1PromptPrefillInput &input) { + const auto &config = weights_->assets().config; + const int64_t hidden_size = config.lm.hidden_size; + if (input.steps <= 0) { + throw std::runtime_error("VoxCPM1 prompt prefill requires positive steps"); + } + if (static_cast(input.input_embeddings.size()) != + input.steps * hidden_size) { + throw std::runtime_error( + "VoxCPM1 prompt prefill input embedding size mismatch"); + } + if (static_cast(input.current_embeddings.size()) != + input.steps * hidden_size) { + throw std::runtime_error( + "VoxCPM1 prompt prefill current embedding size mismatch"); + } + if (static_cast(input.text_mask.size()) != input.steps || + static_cast(input.audio_mask.size()) != input.steps) { + throw std::runtime_error("VoxCPM1 prompt prefill mask size mismatch"); + } + if (sequence_steps_ != input.steps) { + build(input.steps); + } + ggml_backend_tensor_set(input_embeddings_, input.input_embeddings.data(), 0, + input.input_embeddings.size() * sizeof(float)); + ggml_backend_tensor_set(current_embeddings_, + input.current_embeddings.data(), 0, + input.current_embeddings.size() * sizeof(float)); + ggml_backend_tensor_set(text_mask_, input.text_mask.data(), 0, + input.text_mask.size() * sizeof(float)); + ggml_backend_tensor_set(audio_mask_, input.audio_mask.data(), 0, + input.audio_mask.size() * sizeof(float)); + engine::core::set_backend_threads(weights_->backend(), weights_->threads()); + const ggml_status status = + engine::core::compute_backend_graph(weights_->backend(), graph_); + ggml_backend_synchronize(weights_->backend()); + if (status != GGML_STATUS_SUCCESS) { + throw std::runtime_error("VoxCPM1 prompt prefill graph compute failed"); + } + + VoxCPM1PromptPrefillOutput output; + output.lm_hidden.resize(static_cast(hidden_size), 0.0F); + output.residual_hidden.resize(static_cast(hidden_size), 0.0F); + ggml_backend_tensor_get(lm_hidden_output_, output.lm_hidden.data(), 0, + output.lm_hidden.size() * sizeof(float)); + ggml_backend_tensor_get(residual_hidden_output_, + output.residual_hidden.data(), 0, + output.residual_hidden.size() * sizeof(float)); + output.base_state = read_state(base_keys_, base_values_, + config.lm.num_key_value_heads * + head_dim(config.lm)); + output.residual_state = read_state(residual_keys_, residual_values_, + config.residual_lm_num_layers > 0 + ? config.lm.num_key_value_heads * + head_dim(config.lm) + : 0); + return output; + } + +private: + engine::runtime::TransformerKVState + read_state(const std::vector &keys, + const std::vector &values, int64_t step_elems) { + if (keys.size() != values.size() || step_elems <= 0) { + throw std::runtime_error("VoxCPM1 prompt prefill KV state is invalid"); + } + engine::runtime::TransformerKVState state; + state.current_end = sequence_steps_; + state.layers.resize(keys.size()); + const size_t layer_values = + static_cast(sequence_steps_ * step_elems); + for (size_t layer = 0; layer < keys.size(); ++layer) { + auto &layer_state = state.layers[layer]; + layer_state.valid_steps = sequence_steps_; + layer_state.key.resize(layer_values); + layer_state.value.resize(layer_values); + ggml_backend_tensor_get(keys[layer], layer_state.key.data(), 0, + layer_state.key.size() * sizeof(float)); + ggml_backend_tensor_get(values[layer], layer_state.value.data(), 0, + layer_state.value.size() * sizeof(float)); + } + return state; + } + + void release_graph() { + if (graph_ != nullptr) { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + } + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + buffer_ = nullptr; + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + graph_ = nullptr; + input_embeddings_ = nullptr; + current_embeddings_ = nullptr; + text_mask_ = nullptr; + audio_mask_ = nullptr; + positions_ = nullptr; + lm_hidden_output_ = nullptr; + residual_hidden_output_ = nullptr; + base_keys_.clear(); + base_values_.clear(); + residual_keys_.clear(); + residual_values_.clear(); + ctx_.reset(); + sequence_steps_ = 0; + } + + void build(int64_t steps) { + const auto &config = weights_->assets().config; + const auto &model_weights = weights_->weights(); + release_graph(); + ggml_init_params params{graph_context_bytes_, nullptr, true}; + ctx_.reset(ggml_init(params)); + if (ctx_ == nullptr) { + throw std::runtime_error( + "failed to initialize VoxCPM1 prompt prefill graph context"); + } + engine::core::ModuleBuildContext ctx{ctx_.get(), "voxcpm1.prompt_prefill"}; + auto input_embeddings = engine::core::make_tensor( + ctx, GGML_TYPE_F32, + engine::core::TensorShape::from_dims( + {1, steps, config.lm.hidden_size})); + input_embeddings_ = input_embeddings.tensor; + if (mem_saver_) { + ggml_set_input(input_embeddings_); + } + auto current_embeddings = engine::core::make_tensor( + ctx, GGML_TYPE_F32, + engine::core::TensorShape::from_dims( + {1, steps, config.lm.hidden_size})); + current_embeddings_ = current_embeddings.tensor; + if (mem_saver_) { + ggml_set_input(current_embeddings_); + } + text_mask_ = ggml_new_tensor_3d(ctx_.get(), GGML_TYPE_F32, 1, steps, 1); + audio_mask_ = ggml_new_tensor_3d(ctx_.get(), GGML_TYPE_F32, 1, steps, 1); + if (mem_saver_) { + ggml_set_input(text_mask_); + ggml_set_input(audio_mask_); + } + auto text_mask = engine::core::wrap_tensor( + text_mask_, engine::core::TensorShape::from_dims({1, steps, 1}), + GGML_TYPE_F32); + auto audio_mask = engine::core::wrap_tensor( + audio_mask_, engine::core::TensorShape::from_dims({1, steps, 1}), + GGML_TYPE_F32); + positions_ = ggml_new_tensor_1d(ctx_.get(), GGML_TYPE_I32, steps); + if (mem_saver_) { + ggml_set_input(positions_); + } + auto positions = engine::core::wrap_tensor( + positions_, engine::core::TensorShape::from_dims({steps}), + GGML_TYPE_I32); + auto base_hidden = input_embeddings; + for (const auto &layer : model_weights.base_lm.layers) { + auto layer_out = minicpm_prefill_layer( + ctx, base_hidden, positions, layer, model_weights.base_lm); + base_hidden = layer_out.output; + base_keys_.push_back(layer_out.key.tensor); + base_values_.push_back(layer_out.value.tensor); + if (mem_saver_) { + ggml_set_output(base_keys_.back()); + if (base_keys_.back()->view_src != nullptr) { + ggml_set_output(base_keys_.back()->view_src); + } + ggml_set_output(base_values_.back()); + if (base_values_.back()->view_src != nullptr) { + ggml_set_output(base_values_.back()->view_src); + } + } + } + base_hidden = engine::modules::RMSNormModule( + {config.lm.hidden_size, config.lm.rms_norm_eps, true, + false}) + .build(ctx, base_hidden, model_weights.base_lm.norm); + + auto fsq = engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size, + config.scalar_quantization_latent_dim, + true)) + .build(ctx, base_hidden, + model_weights.projections.fsq_in_proj); + fsq = engine::core::wrap_tensor(ggml_tanh(ctx.ggml, fsq.tensor), fsq.shape, + GGML_TYPE_F32); + fsq = engine::core::wrap_tensor( + ggml_scale(ctx.ggml, fsq.tensor, + static_cast(config.scalar_quantization_scale)), + fsq.shape, GGML_TYPE_F32); + fsq = engine::core::wrap_tensor(ggml_round(ctx.ggml, fsq.tensor), fsq.shape, + GGML_TYPE_F32); + fsq = engine::core::wrap_tensor( + ggml_scale(ctx.ggml, fsq.tensor, + 1.0F / static_cast( + config.scalar_quantization_scale)), + fsq.shape, GGML_TYPE_F32); + fsq = engine::modules::LinearModule( + binding::linear_config(config.scalar_quantization_latent_dim, + config.lm.hidden_size, true)) + .build(ctx, fsq, model_weights.projections.fsq_out_proj); + auto masked_base = mask_sequence(ctx, base_hidden, text_mask); + auto masked_fsq = mask_sequence(ctx, fsq, audio_mask); + auto lm_hidden = + engine::modules::AddModule{}.build(ctx, masked_base, masked_fsq); + + auto masked_current = mask_sequence(ctx, current_embeddings, audio_mask); + auto residual_input = + config.v1 + ? engine::modules::AddModule{}.build(ctx, lm_hidden, masked_current) + : engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size * 2, + config.lm.hidden_size, true)) + .build(ctx, + engine::modules::ConcatModule({2}).build( + ctx, lm_hidden, masked_current), + model_weights.projections.fusion_concat_proj); + + auto residual_hidden = residual_input; + for (const auto &layer : model_weights.residual_lm.layers) { + auto layer_out = minicpm_prefill_layer( + ctx, residual_hidden, positions, layer, model_weights.residual_lm); + residual_hidden = layer_out.output; + residual_keys_.push_back(layer_out.key.tensor); + residual_values_.push_back(layer_out.value.tensor); + if (mem_saver_) { + ggml_set_output(residual_keys_.back()); + if (residual_keys_.back()->view_src != nullptr) { + ggml_set_output(residual_keys_.back()->view_src); + } + ggml_set_output(residual_values_.back()); + if (residual_values_.back()->view_src != nullptr) { + ggml_set_output(residual_values_.back()->view_src); + } + } + } + residual_hidden = engine::modules::RMSNormModule( + {config.lm.hidden_size, config.lm.rms_norm_eps, true, + false}) + .build(ctx, residual_hidden, + model_weights.residual_lm.norm); + + auto last_lm = engine::modules::SliceModule({1, steps - 1, 1}) + .build(ctx, lm_hidden); + last_lm = engine::core::reshape_tensor( + ctx, ensure_contiguous(ctx, last_lm), + engine::core::TensorShape::from_dims({config.lm.hidden_size})); + lm_hidden_output_ = last_lm.tensor; + auto last_residual = engine::modules::SliceModule({1, steps - 1, 1}) + .build(ctx, residual_hidden); + last_residual = engine::core::reshape_tensor( + ctx, ensure_contiguous(ctx, last_residual), + engine::core::TensorShape::from_dims({config.lm.hidden_size})); + residual_hidden_output_ = last_residual.tensor; + ggml_set_output(lm_hidden_output_); + if (mem_saver_ && lm_hidden_output_->view_src != nullptr) { + ggml_set_output(lm_hidden_output_->view_src); + } + ggml_set_output(residual_hidden_output_); + if (mem_saver_ && residual_hidden_output_->view_src != nullptr) { + ggml_set_output(residual_hidden_output_->view_src); + } + graph_ = ggml_new_graph_custom(ctx_.get(), kDefaultGraphNodes, false); + ggml_build_forward_expand(graph_, lm_hidden_output_); + ggml_build_forward_expand(graph_, residual_hidden_output_); + if (mem_saver_) { + gallocr_ = ggml_gallocr_new( + ggml_backend_get_default_buffer_type(weights_->backend())); + if (gallocr_ == nullptr || !ggml_gallocr_reserve(gallocr_, graph_) || + !ggml_gallocr_alloc_graph(gallocr_, graph_)) { + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + gallocr_ = nullptr; + } + release_graph(); + throw std::runtime_error( + "failed to allocate VoxCPM1 prompt prefill graph"); + } + } else { + buffer_ = ggml_backend_alloc_ctx_tensors(ctx_.get(), weights_->backend()); + } + if (!mem_saver_ && buffer_ == nullptr) { + throw std::runtime_error( + "failed to allocate VoxCPM1 prompt prefill graph"); + } + std::vector position_ids(static_cast(steps), 0); + for (int64_t i = 0; i < steps; ++i) { + position_ids[static_cast(i)] = static_cast(i); + } + ggml_backend_tensor_set(positions_, position_ids.data(), 0, + position_ids.size() * sizeof(int32_t)); + sequence_steps_ = steps; + } + + std::shared_ptr weights_; + size_t graph_context_bytes_ = 0; + bool mem_saver_ = false; + int64_t sequence_steps_ = 0; + std::unique_ptr ctx_; + ggml_tensor *input_embeddings_ = nullptr; + ggml_tensor *current_embeddings_ = nullptr; + ggml_tensor *text_mask_ = nullptr; + ggml_tensor *audio_mask_ = nullptr; + ggml_tensor *positions_ = nullptr; + ggml_tensor *lm_hidden_output_ = nullptr; + ggml_tensor *residual_hidden_output_ = nullptr; + std::vector base_keys_; + std::vector base_values_; + std::vector residual_keys_; + std::vector residual_values_; + ggml_cgraph *graph_ = nullptr; + ggml_backend_buffer_t buffer_ = nullptr; + ggml_gallocr_t gallocr_ = nullptr; +}; + +VoxCPM1PromptPrefillRuntime::VoxCPM1PromptPrefillRuntime( + std::shared_ptr weights, + size_t graph_context_bytes, bool mem_saver) + : impl_(std::make_unique(std::move(weights), graph_context_bytes, + mem_saver)) {} + +VoxCPM1PromptPrefillRuntime::~VoxCPM1PromptPrefillRuntime() = default; + +VoxCPM1PromptPrefillOutput +VoxCPM1PromptPrefillRuntime::run(const VoxCPM1PromptPrefillInput &input) { + return impl_->run(input); +} + +void VoxCPM1PromptPrefillRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + +engine::core::TensorValue +minicpm_layer_with_static_cache(engine::core::ModuleBuildContext &ctx, + const engine::core::TensorValue &input, + const engine::core::TensorValue &positions, + const engine::core::TensorValue &cache_slot, + const engine::core::TensorValue &attention_mask, + const engine::core::TensorValue &cache_key, + const engine::core::TensorValue &cache_value, + const VoxCPM1MiniCPMLayerWeights &layer, + const VoxCPM1MiniCPMWeights &weights) { + const auto &config = weights.config; + const int64_t dim = head_dim(config); + const engine::modules::AddModule add; + auto hidden = engine::modules::RMSNormModule( + {config.hidden_size, config.rms_norm_eps, true, false}) + .build(ctx, input, layer.input_norm); + auto q = engine::modules::LinearModule( + binding::linear_config(config.hidden_size, + config.num_attention_heads * dim, false)) + .build(ctx, hidden, layer.q_proj); + auto k = engine::modules::LinearModule( + binding::linear_config(config.hidden_size, + config.num_key_value_heads * dim, false)) + .build(ctx, hidden, layer.k_proj); + auto v = engine::modules::LinearModule( + binding::linear_config(config.hidden_size, + config.num_key_value_heads * dim, false)) + .build(ctx, hidden, layer.v_proj); + q = apply_minicpm_rope(ctx, + reshape_heads(ctx, q, config.num_attention_heads, dim), + positions, weights); + k = apply_minicpm_rope(ctx, + reshape_heads(ctx, k, config.num_key_value_heads, dim), + positions, weights); + v = reshape_heads(ctx, v, config.num_key_value_heads, dim); + + const engine::modules::FastKVSetRowsModule set_rows; + auto updated_key = set_rows.build(ctx, cache_key, k, cache_slot); + auto updated_value = set_rows.build(ctx, cache_value, v, cache_slot); + + auto q_heads = engine::modules::TransposeModule({{0, 2, 1, 3}, q.shape.rank}) + .build(ctx, q); + auto k_heads = + engine::modules::TransposeModule({{0, 2, 1, 3}, updated_key.shape.rank}) + .build(ctx, updated_key); + auto v_heads = + engine::modules::TransposeModule({{0, 2, 1, 3}, updated_value.shape.rank}) + .build(ctx, updated_value); + auto context = flash_attention_from_grouped_heads( + ctx, q_heads, k_heads, v_heads, dim, attention_mask); + context = engine::modules::TransposeModule({{0, 2, 1, 3}, context.shape.rank}) + .build(ctx, context); + context = engine::core::reshape_tensor( + ctx, ensure_contiguous(ctx, context), + engine::core::TensorShape::from_dims({input.shape.dims[0], + input.shape.dims[1], + config.num_attention_heads * dim})); + auto attn = engine::modules::LinearModule( + binding::linear_config(config.num_attention_heads * dim, + config.hidden_size, false)) + .build(ctx, context, layer.o_proj); + auto x = + add.build(ctx, input, + scale_tensor(ctx, attn, + config.use_mup ? config.scale_depth / + std::sqrt(static_cast( + config.num_hidden_layers)) + : 1.0F)); + + hidden = engine::modules::RMSNormModule( + {config.hidden_size, config.rms_norm_eps, true, false}) + .build(ctx, x, layer.post_norm); + auto gate = engine::modules::LinearModule( + binding::linear_config(config.hidden_size, + config.intermediate_size, false)) + .build(ctx, hidden, layer.gate_proj); + gate = engine::modules::SiluModule{}.build(ctx, gate); + auto up = engine::modules::LinearModule( + binding::linear_config(config.hidden_size, + config.intermediate_size, false)) + .build(ctx, hidden, layer.up_proj); + auto gated = engine::modules::MulModule{}.build(ctx, gate, up); + auto ff = engine::modules::LinearModule( + binding::linear_config(config.intermediate_size, + config.hidden_size, false)) + .build(ctx, gated, layer.down_proj); + return add.build( + ctx, x, + scale_tensor(ctx, ff, + config.use_mup + ? config.scale_depth / std::sqrt(static_cast( + config.num_hidden_layers)) + : 1.0F)); +} + +const char *minicpm_kind_name(VoxCPM1MiniCPMKind kind) { + switch (kind) { + case VoxCPM1MiniCPMKind::BaseLM: + return "base_lm"; + case VoxCPM1MiniCPMKind::ResidualLM: + return "residual_lm"; + } + throw std::runtime_error( + "VoxCPM1 MiniCPM runtime received an unknown graph kind"); +} + + +class VoxCPM1MiniCPMStepRuntime::Impl { +public: + Impl(std::shared_ptr weights, + VoxCPM1MiniCPMKind kind, int64_t cache_steps, size_t graph_context_bytes) + : weights_(std::move(weights)), kind_(kind), cache_steps_(cache_steps), + graph_context_bytes_(graph_context_bytes) { + if (weights_ == nullptr) { + throw std::runtime_error("VoxCPM1 MiniCPM step runtime requires weights"); + } + build(graph_context_bytes); + } + + ~Impl() { + release_runtime_memory(); + } + + void reset() { + ensure_graph(); + engine::runtime::TransformerKVState state; + state.current_end = 0; + state.layers.resize( + select_minicpm_weights(weights_->weights(), kind_).layers.size()); + step_cache_.import_state(state); + } + + void import_state(const engine::runtime::TransformerKVState &state) { + ensure_graph(); + step_cache_.import_state(state); + } + + engine::runtime::TransformerKVState export_state() const { + return step_cache_.export_state(); + } + + VoxCPM1MiniCPMStepOutput run_step(const std::vector &embedding) { + ensure_graph(); + const auto &config = + select_minicpm_weights(weights_->weights(), kind_).config; + if (static_cast(embedding.size()) != config.hidden_size) { + throw std::runtime_error("VoxCPM1 MiniCPM step embedding size mismatch"); + } + if (step_cache_.valid_steps() >= cache_steps_) { + throw std::runtime_error("VoxCPM1 MiniCPM step exceeds cache capacity"); + } + ggml_backend_tensor_set(input_, embedding.data(), 0, + embedding.size() * sizeof(float)); + const int32_t position = static_cast(step_cache_.current_end()); + ggml_backend_tensor_set(position_, &position, 0, sizeof(position)); + const int32_t cache_slot = static_cast(step_cache_.valid_steps()); + ggml_backend_tensor_set(cache_slot_, &cache_slot, 0, sizeof(cache_slot)); + std::fill(attention_mask_buffer_.begin(), attention_mask_buffer_.end(), + ggml_fp32_to_fp16(-INFINITY)); + for (int64_t i = 0; i < step_cache_.valid_steps(); ++i) { + attention_mask_buffer_[static_cast(i)] = ggml_fp32_to_fp16(0.0F); + } + attention_mask_buffer_[static_cast(cache_slot)] = + ggml_fp32_to_fp16(0.0F); + ggml_backend_tensor_set(attention_mask_, attention_mask_buffer_.data(), 0, + attention_mask_buffer_.size() * + sizeof(ggml_fp16_t)); + engine::core::set_backend_threads(weights_->backend(), weights_->threads()); + const ggml_status status = + engine::core::compute_backend_graph(weights_->backend(), graph_); + ggml_backend_synchronize(weights_->backend()); + if (status != GGML_STATUS_SUCCESS) { + throw std::runtime_error(std::string("VoxCPM1 MiniCPM ") + + minicpm_kind_name(kind_) + + " step graph compute failed"); + } + VoxCPM1MiniCPMStepOutput output; + output.position = step_cache_.current_end(); + output.hidden.resize(static_cast(config.hidden_size), 0.0F); + ggml_backend_tensor_get(hidden_output_, output.hidden.data(), 0, + output.hidden.size() * sizeof(float)); + step_cache_.advance_after_direct_append(1); + return output; + } + + void release_runtime_memory() { release_graph(); } + +private: + void ensure_graph() { + if (graph_ == nullptr) { + build(graph_context_bytes_); + } + } + + void release_graph() { + if (graph_ != nullptr) { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + } + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + buffer_ = nullptr; + } + graph_ = nullptr; + input_ = nullptr; + position_ = nullptr; + cache_slot_ = nullptr; + attention_mask_ = nullptr; + hidden_output_ = nullptr; + attention_mask_buffer_.clear(); + step_cache_ = engine::runtime::TransformerKVCache(); + ctx_.reset(); + } + + void build(size_t graph_context_bytes) { + if (cache_steps_ <= 0) { + throw std::runtime_error( + "VoxCPM1 MiniCPM step graph requires positive cache capacity"); + } + if (graph_context_bytes == 0) { + throw std::runtime_error( + "VoxCPM1 MiniCPM step graph context bytes must be non-zero"); + } + release_graph(); + ggml_init_params params{graph_context_bytes, nullptr, true}; + ctx_.reset(ggml_init(params)); + if (ctx_ == nullptr) { + throw std::runtime_error( + "failed to initialize VoxCPM1 MiniCPM step graph context"); + } + const auto &lm_weights = select_minicpm_weights(weights_->weights(), kind_); + const auto &config = lm_weights.config; + const int64_t dim = head_dim(config); + const std::string graph_name = + std::string("voxcpm1.") + minicpm_kind_name(kind_) + ".step"; + engine::core::ModuleBuildContext ctx{ctx_.get(), graph_name.c_str()}; + auto x = engine::core::make_tensor( + ctx, GGML_TYPE_F32, + engine::core::TensorShape::from_dims({1, 1, config.hidden_size})); + input_ = x.tensor; + position_ = ggml_new_tensor_1d(ctx_.get(), GGML_TYPE_I32, 1); + auto position = engine::core::wrap_tensor( + position_, engine::core::TensorShape::from_dims({1}), GGML_TYPE_I32); + cache_slot_ = ggml_new_tensor_1d(ctx_.get(), GGML_TYPE_I32, 1); + auto cache_slot = engine::core::wrap_tensor( + cache_slot_, engine::core::TensorShape::from_dims({1}), GGML_TYPE_I32); + attention_mask_ = + ggml_new_tensor_4d(ctx_.get(), GGML_TYPE_F16, cache_steps_, 1, 1, 1); + auto attention_mask = engine::core::wrap_tensor( + attention_mask_, + engine::core::TensorShape::from_dims({1, 1, 1, cache_steps_}), + GGML_TYPE_F16); + + std::vector cache_keys; + std::vector cache_values; + cache_keys.reserve(static_cast(config.num_hidden_layers)); + cache_values.reserve(static_cast(config.num_hidden_layers)); + for (const auto &layer : lm_weights.layers) { + cache_keys.push_back(engine::core::make_tensor( + ctx, GGML_TYPE_F32, + engine::core::TensorShape::from_dims( + {1, cache_steps_, config.num_key_value_heads, dim}))); + cache_values.push_back(engine::core::make_tensor( + ctx, GGML_TYPE_F32, + engine::core::TensorShape::from_dims( + {1, cache_steps_, config.num_key_value_heads, dim}))); + x = minicpm_layer_with_static_cache( + ctx, x, position, cache_slot, attention_mask, cache_keys.back(), + cache_values.back(), layer, lm_weights); + } + step_cache_ = engine::runtime::TransformerKVCache( + cache_steps_, config.num_key_value_heads * dim, std::move(cache_keys), + std::move(cache_values)); + x = engine::modules::RMSNormModule( + {config.hidden_size, config.rms_norm_eps, true, false}) + .build(ctx, x, lm_weights.norm); + hidden_output_ = x.tensor; + ggml_set_output(hidden_output_); + graph_ = ggml_new_graph_custom(ctx_.get(), kDefaultGraphNodes, false); + ggml_build_forward_expand(graph_, hidden_output_); + buffer_ = ggml_backend_alloc_ctx_tensors(ctx_.get(), weights_->backend()); + if (buffer_ == nullptr) { + throw std::runtime_error("failed to allocate VoxCPM1 MiniCPM step graph"); + } + attention_mask_buffer_.assign(static_cast(cache_steps_), + ggml_fp32_to_fp16(-INFINITY)); + } + + std::shared_ptr weights_; + VoxCPM1MiniCPMKind kind_ = VoxCPM1MiniCPMKind::BaseLM; + int64_t cache_steps_ = 0; + size_t graph_context_bytes_ = 0; + std::unique_ptr ctx_; + ggml_tensor *input_ = nullptr; + ggml_tensor *position_ = nullptr; + ggml_tensor *cache_slot_ = nullptr; + ggml_tensor *attention_mask_ = nullptr; + ggml_tensor *hidden_output_ = nullptr; + std::vector attention_mask_buffer_; + engine::runtime::TransformerKVCache step_cache_; + ggml_cgraph *graph_ = nullptr; + ggml_backend_buffer_t buffer_ = nullptr; +}; + +VoxCPM1MiniCPMStepRuntime::VoxCPM1MiniCPMStepRuntime( + std::shared_ptr weights, + VoxCPM1MiniCPMKind kind, int64_t cache_steps, size_t graph_context_bytes) + : impl_(std::make_unique(std::move(weights), kind, cache_steps, + graph_context_bytes)) {} + +VoxCPM1MiniCPMStepRuntime::~VoxCPM1MiniCPMStepRuntime() = default; + +void VoxCPM1MiniCPMStepRuntime::reset() { impl_->reset(); } + +void VoxCPM1MiniCPMStepRuntime::import_state( + const engine::runtime::TransformerKVState &state) { + impl_->import_state(state); +} + +engine::runtime::TransformerKVState +VoxCPM1MiniCPMStepRuntime::export_state() const { + return impl_->export_state(); +} + +VoxCPM1MiniCPMStepOutput +VoxCPM1MiniCPMStepRuntime::run_step(const std::vector &embedding) { + return impl_->run_step(embedding); +} + +void VoxCPM1MiniCPMStepRuntime::release_runtime_memory() { + impl_->release_runtime_memory(); +} + +} // namespace engine::community_models::voxcpm1 diff --git a/src/community_models/voxcpm1/minicpm_blocks.h b/src/community_models/voxcpm1/minicpm_blocks.h new file mode 100644 index 00000000..7d51b219 --- /dev/null +++ b/src/community_models/voxcpm1/minicpm_blocks.h @@ -0,0 +1,272 @@ +#pragma once + +#include "engine/community_models/voxcpm1/minicpm.h" + +#include "engine/framework/core/execution_context.h" +#include "engine/framework/modules/activation_modules.h" +#include "engine/framework/modules/linear_module.h" +#include "engine/framework/modules/lookup_modules.h" +#include "engine/framework/modules/norm_modules.h" +#include "engine/framework/modules/primitive_modules.h" +#include "engine/framework/modules/structural_modules.h" +#include "engine/framework/modules/weight_binding.h" + +#include + +#include +#include +#include +#include +#include + +namespace engine::community_models::voxcpm1 { +namespace { + +namespace binding = engine::modules::binding; + +constexpr size_t kDefaultGraphNodes = 65536; + +struct GgmlContextDeleter { + void operator()(ggml_context *ctx) const noexcept { + if (ctx != nullptr) { + ggml_free(ctx); + } + } +}; + +engine::core::TensorValue +ensure_contiguous(engine::core::ModuleBuildContext &ctx, + const engine::core::TensorValue &value) { + return engine::core::ensure_backend_addressable_layout(ctx, value); +} + +engine::core::TensorValue reshape_heads(engine::core::ModuleBuildContext &ctx, + const engine::core::TensorValue &input, + int64_t heads, int64_t dim) { + return engine::core::reshape_tensor( + ctx, ensure_contiguous(ctx, input), + engine::core::TensorShape::from_dims( + {input.shape.dims[0], input.shape.dims[1], heads, dim})); +} + +engine::core::TensorValue +repeat_kv_heads(engine::core::ModuleBuildContext &ctx, + const engine::core::TensorValue &input, int64_t repeats) { + if (repeats == 1) { + return input; + } + std::vector heads; + heads.reserve(static_cast(input.shape.dims[1] * repeats)); + for (int64_t head = 0; head < input.shape.dims[1]; ++head) { + auto one = engine::modules::SliceModule({1, head, 1}).build(ctx, input); + for (int64_t rep = 0; rep < repeats; ++rep) { + heads.push_back(one); + } + } + auto output = heads.front(); + for (size_t i = 1; i < heads.size(); ++i) { + output = engine::modules::ConcatModule({1}).build(ctx, output, heads[i]); + } + return output; +} + +engine::core::TensorValue scale_tensor(engine::core::ModuleBuildContext &ctx, + const engine::core::TensorValue &input, + float scale) { + if (scale == 1.0F) { + return input; + } + return engine::core::wrap_tensor(ggml_scale(ctx.ggml, input.tensor, scale), + input.shape, GGML_TYPE_F32); +} + +engine::core::TensorValue +apply_minicpm_rope(engine::core::ModuleBuildContext &ctx, + const engine::core::TensorValue &input, + const engine::core::TensorValue &positions, + const VoxCPM1MiniCPMWeights &weights) { + const auto &config = weights.config; + if (config.no_rope) { + return input; + } + if (!weights.rope_factors.has_value()) { + throw std::runtime_error("VoxCPM1 MiniCPM graph missing RoPE factors"); + } + const int64_t dim = head_dim(config); + return engine::core::wrap_tensor( + ggml_rope_ext(ctx.ggml, input.tensor, positions.tensor, + weights.rope_factors->tensor, static_cast(dim), + GGML_ROPE_TYPE_NEOX, + static_cast( + config.rope_scaling.original_max_position_embeddings), + config.rope_theta, 1.0F, 0.0F, weights.rope_attn_factor, + 0.0F, 0.0F), + input.shape, input.type); +} + +engine::core::TensorValue +attention_from_heads(engine::core::ModuleBuildContext &ctx, + const engine::core::TensorValue &q_heads, + const engine::core::TensorValue &k_heads, + const engine::core::TensorValue &v_heads, int64_t dim, + bool is_causal) { + const engine::modules::MatMulModule matmul; + auto scores = matmul.build( + ctx, q_heads, + engine::modules::TransposeModule({{0, 1, 3, 2}, k_heads.shape.rank}) + .build(ctx, k_heads)); + scores = engine::core::wrap_tensor( + ggml_scale(ctx.ggml, scores.tensor, + 1.0F / std::sqrt(static_cast(dim))), + scores.shape, GGML_TYPE_F32); + if (is_causal) { + scores = engine::core::wrap_tensor( + ggml_diag_mask_inf(ctx.ggml, scores.tensor, 0), scores.shape, + GGML_TYPE_F32); + } + scores = ensure_contiguous(ctx, scores); + auto attn = engine::core::wrap_tensor(ggml_soft_max(ctx.ggml, scores.tensor), + scores.shape, GGML_TYPE_F32); + return matmul.build(ctx, attn, v_heads); +} + +[[maybe_unused]] engine::core::TensorValue flash_attention_from_grouped_heads( + engine::core::ModuleBuildContext &ctx, + const engine::core::TensorValue &q_heads, + const engine::core::TensorValue &k_heads, + const engine::core::TensorValue &v_heads, int64_t dim, + const engine::core::TensorValue &attention_mask) { + const auto q = ensure_contiguous(ctx, q_heads); + const auto k = ensure_contiguous(ctx, k_heads); + const auto v = ensure_contiguous(ctx, v_heads); + auto *flash = ggml_flash_attn_ext( + ctx.ggml, q.tensor, k.tensor, v.tensor, attention_mask.tensor, + 1.0F / std::sqrt(static_cast(dim)), 0.0F, 0.0F); + ggml_flash_attn_ext_set_prec(flash, GGML_PREC_F32); + return engine::core::wrap_tensor( + flash, + engine::core::TensorShape::from_dims( + {q.shape.dims[0], q.shape.dims[2], q.shape.dims[1], dim}), + GGML_TYPE_F32); +} + +engine::core::TensorValue +minicpm_layer(engine::core::ModuleBuildContext &ctx, + const engine::core::TensorValue &input, + const engine::core::TensorValue &positions, + const VoxCPM1MiniCPMLayerWeights &layer, + const VoxCPM1MiniCPMWeights &weights, bool is_causal) { + const auto &config = weights.config; + const int64_t dim = head_dim(config); + const int64_t kv_repeats = + config.num_attention_heads / config.num_key_value_heads; + const engine::modules::AddModule add; + auto hidden = engine::modules::RMSNormModule( + {config.hidden_size, config.rms_norm_eps, true, false}) + .build(ctx, input, layer.input_norm); + if (std::getenv("VOXCPM_DUMP_NORM0") != nullptr && + ggml_nelements(hidden.tensor) == 10240) { + ggml_set_name(hidden.tensor, "dump_norm0"); + ggml_set_output(hidden.tensor); + } + auto q = engine::modules::LinearModule( + binding::linear_config(config.hidden_size, + config.num_attention_heads * dim, false)) + .build(ctx, hidden, layer.q_proj); + auto k = engine::modules::LinearModule( + binding::linear_config(config.hidden_size, + config.num_key_value_heads * dim, false)) + .build(ctx, hidden, layer.k_proj); + auto v = engine::modules::LinearModule( + binding::linear_config(config.hidden_size, + config.num_key_value_heads * dim, false)) + .build(ctx, hidden, layer.v_proj); + q = apply_minicpm_rope(ctx, + reshape_heads(ctx, q, config.num_attention_heads, dim), + positions, weights); + k = apply_minicpm_rope(ctx, + reshape_heads(ctx, k, config.num_key_value_heads, dim), + positions, weights); + v = reshape_heads(ctx, v, config.num_key_value_heads, dim); + auto q_heads = engine::modules::TransposeModule({{0, 2, 1, 3}, q.shape.rank}) + .build(ctx, q); + auto k_heads = repeat_kv_heads( + ctx, + engine::modules::TransposeModule({{0, 2, 1, 3}, k.shape.rank}) + .build(ctx, k), + kv_repeats); + auto v_heads = repeat_kv_heads( + ctx, + engine::modules::TransposeModule({{0, 2, 1, 3}, v.shape.rank}) + .build(ctx, v), + kv_repeats); + auto context = + attention_from_heads(ctx, q_heads, k_heads, v_heads, dim, is_causal); + context = engine::modules::TransposeModule({{0, 2, 1, 3}, context.shape.rank}) + .build(ctx, context); + context = engine::core::reshape_tensor( + ctx, ensure_contiguous(ctx, context), + engine::core::TensorShape::from_dims({input.shape.dims[0], + input.shape.dims[1], + config.num_attention_heads * dim})); + auto attn = engine::modules::LinearModule( + binding::linear_config(config.num_attention_heads * dim, + config.hidden_size, false)) + .build(ctx, context, layer.o_proj); + auto x = + add.build(ctx, input, + scale_tensor(ctx, attn, + config.use_mup ? config.scale_depth / + std::sqrt(static_cast( + config.num_hidden_layers)) + : 1.0F)); + + hidden = engine::modules::RMSNormModule( + {config.hidden_size, config.rms_norm_eps, true, false}) + .build(ctx, x, layer.post_norm); + auto gate = engine::modules::LinearModule( + binding::linear_config(config.hidden_size, + config.intermediate_size, false)) + .build(ctx, hidden, layer.gate_proj); + gate = engine::modules::SiluModule{}.build(ctx, gate); + auto up = engine::modules::LinearModule( + binding::linear_config(config.hidden_size, + config.intermediate_size, false)) + .build(ctx, hidden, layer.up_proj); + auto gated = engine::modules::MulModule{}.build(ctx, gate, up); + auto ff = engine::modules::LinearModule( + binding::linear_config(config.intermediate_size, + config.hidden_size, false)) + .build(ctx, gated, layer.down_proj); + return add.build( + ctx, x, + scale_tensor(ctx, ff, + config.use_mup + ? config.scale_depth / std::sqrt(static_cast( + config.num_hidden_layers)) + : 1.0F)); +} + +[[maybe_unused]] engine::core::TensorValue +minicpm_transformer(engine::core::ModuleBuildContext &ctx, + engine::core::TensorValue input, + const engine::core::TensorValue &positions, + const VoxCPM1MiniCPMWeights &weights, bool is_causal) { + for (size_t li = 0; li < weights.layers.size(); ++li) { + if (std::getenv("VOXCPM_DUMP_DECODER_LAYERS") != nullptr && li < 8) { + char name[32]; + snprintf(name, sizeof(name), "dump_layer_%zu", li); + ggml_set_name(input.tensor, name); + ggml_set_output(input.tensor); + } + input = minicpm_layer(ctx, input, positions, weights.layers[li], weights, + is_causal); + } + return engine::modules::RMSNormModule({weights.config.hidden_size, + weights.config.rms_norm_eps, true, + false}) + .build(ctx, input, weights.norm); +} + +} // namespace +} // namespace engine::community_models::voxcpm1 diff --git a/src/community_models/voxcpm1/session.cpp b/src/community_models/voxcpm1/session.cpp new file mode 100644 index 00000000..a7b87f5d --- /dev/null +++ b/src/community_models/voxcpm1/session.cpp @@ -0,0 +1,938 @@ +#include "engine/community_models/voxcpm1/session.h" + +#include "engine/framework/debug/profiler.h" +#include "engine/framework/model_spec/metadata.h" +#include "engine/framework/model_spec/package.h" +#include "engine/framework/runtime/options.h" +#include "engine/framework/text/chunking.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace engine::community_models::voxcpm1 { +namespace { + +using Clock = std::chrono::steady_clock; +constexpr int64_t kDefaultTextChunkSize = 2048; + +std::shared_ptr +require_assets(std::shared_ptr assets) { + if (assets == nullptr) { + throw std::runtime_error("VoxCPM1 session requires assets"); + } + return assets; +} + +void reject_enabled_denoise( + const std::unordered_map &options, + std::initializer_list keys) { + const auto match = runtime::find_option_match(options, keys); + if (match.has_value() && + runtime::parse_bool_option(match->value, match->key)) { + throw std::runtime_error( + "VoxCPM1 denoise is disabled in this implementation"); + } +} + +void reject_denoiser_option( + const std::unordered_map &options, + std::initializer_list keys) { + if (runtime::find_option_match(options, keys).has_value()) { + throw std::runtime_error( + "VoxCPM1 denoise is disabled in this implementation"); + } +} + +std::unordered_map normalize_v1_session_options( + std::unordered_map options) { + // This community family serves only VoxCPM1 and canonically advertises + // "voxcpm1.*" session options; accept legacy "voxcpm2.*" spellings by + // aliasing them to "voxcpm1.*" so both keep working. + std::unordered_map out; + out.reserve(options.size()); + for (auto &[key, value] : options) { + constexpr std::string_view kLegacyPrefix = "voxcpm2."; + if (key.rfind(kLegacyPrefix, 0) == 0) { + out[std::string("voxcpm1.") + + key.substr(kLegacyPrefix.size())] = std::move(value); + } else { + out[std::move(key)] = std::move(value); + } + } + return out; +} + +bool audio_buffer_equal(const runtime::AudioBuffer &lhs, + const runtime::AudioBuffer &rhs) { + return lhs.sample_rate == rhs.sample_rate && lhs.channels == rhs.channels && + lhs.samples == rhs.samples; +} + +bool optional_audio_equal(const std::optional &lhs, + const std::optional &rhs) { + if (lhs.has_value() != rhs.has_value()) { + return false; + } + return !lhs.has_value() || audio_buffer_equal(*lhs, *rhs); +} + +size_t prompt_cache_slots_from_options( + const std::unordered_map &options) { + constexpr int64_t kDefaultPromptCacheSlots = 1; + const int64_t slots = runtime::parse_i64_option( + options, {"voxcpm1.prompt_cache_slots", "voxcpm1.prompt_cache_slots"}) + .value_or(kDefaultPromptCacheSlots); + if (slots < 0) { + throw std::runtime_error("voxcpm1.prompt_cache_slots must be non-negative"); + } + return static_cast(slots); +} + +void validate_weight_storage(engine::assets::TensorStorageType storage_type, + const char *option_name) { + if (storage_type == engine::assets::TensorStorageType::Native || + storage_type == engine::assets::TensorStorageType::F32 || + storage_type == engine::assets::TensorStorageType::F16 || + storage_type == engine::assets::TensorStorageType::BF16 || + storage_type == engine::assets::TensorStorageType::Q8_0) { + return; + } + throw std::runtime_error(std::string(option_name) + + " supports only native, f32, f16, bf16, and q8_0"); +} + +void parse_weight_type( + const std::unordered_map &options, + const char *key, engine::assets::TensorStorageType &storage_type) { + const auto it = options.find(key); + if (it == options.end()) { + return; + } + storage_type = engine::assets::parse_tensor_storage_type(it->second); + validate_weight_storage(storage_type, key); +} + +void validate_session_options( + const std::unordered_map &options) { + for (const auto &[key, value] : options) { + (void)value; + if (key.rfind("voxcpm1.", 0) != 0) { + continue; + } + if (key == "voxcpm1.weight_context_mb" || + key == "voxcpm1.text_embedding_graph_context_mb" || + key == "voxcpm1.lm_step_graph_context_mb" || + key == "voxcpm1.projection_graph_context_mb" || + key == "voxcpm1.local_encoder_graph_context_mb" || + key == "voxcpm1.dit_graph_context_mb" || + key == "voxcpm1.audiovae_weight_context_mb" || + key == "voxcpm1.audiovae_graph_context_mb" || + key == "voxcpm1.audiovae_encoder_graph_context_mb" || + key == "voxcpm1.audiovae_latent_capacity" || + key == "voxcpm1.audiovae_encoder_sample_capacity" || + key == "voxcpm1.weight_type" || + key == "voxcpm1.audiovae_weight_type" || + key == "voxcpm1.prompt_cache_slots" || + key == "voxcpm1.mem_saver" || + key == "voxcpm1.denoise" || key == "voxcpm1.load_denoiser") { + continue; + } + throw std::runtime_error("unknown VoxCPM1 session option: " + key); + } +} + +int64_t product(const std::vector &values) { + int64_t out = 1; + for (const int64_t value : values) { + if (value <= 0) { + throw std::runtime_error("VoxCPM1 AudioVAE decoder rate is invalid"); + } + out *= value; + } + return out; +} + +} // namespace + +bool VoxCPM1SessionBase::EncodedPromptCacheKeyEqual::operator()( + const EncodedPromptCacheKey &lhs, + const EncodedPromptCacheKey &rhs) const { + return lhs.prompt_text == rhs.prompt_text && + optional_audio_equal(lhs.prompt_audio, rhs.prompt_audio) && + optional_audio_equal(lhs.reference_audio, rhs.reference_audio); +} + +VoxCPM1SessionBase::VoxCPM1SessionBase(runtime::TaskSpec task, + runtime::SessionOptions options, + std::shared_ptr assets) + : RuntimeSessionBase(options), task_(task), + assets_(require_assets(std::move(assets))), + encoded_prompt_cache_(prompt_cache_slots_from_options(options.options)) { + if (task_.mode != runtime::RunMode::Offline && + task_.mode != runtime::RunMode::Streaming) { + throw std::runtime_error( + std::string("VoxCPM1") + + " only supports offline and streaming sessions"); + } + if (task_.task != runtime::VoiceTaskKind::Tts) { + throw std::runtime_error( + std::string("VoxCPM1") + + " only supports the Tts task"); + } + + options.options = normalize_v1_session_options(std::move(options.options)); + + reject_enabled_denoise(options.options, {"voxcpm1.denoise"}); + reject_enabled_denoise(options.options, {"voxcpm1.load_denoiser"}); + reject_denoiser_option(options.options, {"voxcpm1.denoiser"}); + validate_session_options(options.options); + + generator_config_.weight_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm1.weight_context_mb"}, + generator_config_.weight_context_bytes); + generator_config_.text_embedding_graph_context_bytes = + runtime::parse_size_mb_option( + options.options, {"voxcpm1.text_embedding_graph_context_mb"}, + generator_config_.text_embedding_graph_context_bytes); + generator_config_.lm_step_graph_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm1.lm_step_graph_context_mb"}, + generator_config_.lm_step_graph_context_bytes); + generator_config_.projection_graph_context_bytes = + runtime::parse_size_mb_option( + options.options, {"voxcpm1.projection_graph_context_mb"}, + generator_config_.projection_graph_context_bytes); + generator_config_.local_encoder_graph_context_bytes = + runtime::parse_size_mb_option( + options.options, {"voxcpm1.local_encoder_graph_context_mb"}, + generator_config_.local_encoder_graph_context_bytes); + generator_config_.dit_graph_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm1.dit_graph_context_mb"}, + generator_config_.dit_graph_context_bytes); + generator_config_.prompt_cache_slots = encoded_prompt_cache_.capacity(); + decoder_config_.weight_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm1.audiovae_weight_context_mb"}, + decoder_config_.weight_context_bytes); + decoder_config_.graph_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm1.audiovae_graph_context_mb"}, + decoder_config_.graph_context_bytes); + decoder_config_.encoder_graph_context_bytes = runtime::parse_size_mb_option( + options.options, {"voxcpm1.audiovae_encoder_graph_context_mb"}, + decoder_config_.encoder_graph_context_bytes); + decoder_config_.latent_frame_capacity = runtime::parse_positive_i64_option( + options.options, {"voxcpm1.audiovae_latent_capacity"}, + decoder_config_.latent_frame_capacity); + decoder_config_.encoder_sample_capacity = runtime::parse_positive_i64_option( + options.options, {"voxcpm1.audiovae_encoder_sample_capacity"}, + decoder_config_.encoder_sample_capacity); + parse_weight_type(options.options, "voxcpm1.weight_type", + generator_config_.weight_storage_type); + parse_weight_type(options.options, "voxcpm1.audiovae_weight_type", + decoder_config_.weight_storage_type); + if (const auto mem_saver = + runtime::find_option(options.options, {"voxcpm1.mem_saver"})) { + generator_config_.mem_saver = + runtime::parse_bool_option(*mem_saver, "voxcpm1.mem_saver"); + } + + generator_ = std::make_unique( + assets_, execution_context(), generator_config_); + decoder_ = std::make_unique( + assets_, execution_context(), decoder_config_); +} + +VoxCPM1SessionBase::~VoxCPM1SessionBase() = default; + +std::string VoxCPM1SessionBase::family_impl() const { + return "voxcpm1"; +} + +runtime::VoiceTaskKind VoxCPM1SessionBase::task_kind_impl() const { return task_.task; } + +runtime::RunMode VoxCPM1SessionBase::run_mode_impl() const { return task_.mode; } + +void VoxCPM1SessionBase::prepare_impl( + const runtime::SessionPreparationRequest &request) { + (void)request; + mark_prepared(); +} + +runtime::TaskResult VoxCPM1SessionBase::run_offline_request(const runtime::TaskRequest &request) { + require_prepared("VoxCPM1 run"); + if (task_.mode != runtime::RunMode::Offline) { + throw std::runtime_error("VoxCPM1 run requires an offline session"); + } + validate_request(request); + auto release_runtime_memory = [this](VoxCPM1SessionBase *self) { + if (self != nullptr) { + self->release_request_runtime_memory(); + } + }; + std::unique_ptr + release_guard(this, release_runtime_memory); + + const auto wall_start = Clock::now(); + const int64_t text_chunk_size = + engine::text::parse_text_chunk_size_override(request.options).value_or(kDefaultTextChunkSize); + const auto text_chunk_mode = + engine::text::parse_text_chunk_mode_override(request.options) + .value_or(engine::text::TextChunkMode::TagAware); + const auto chunk_requests = + runtime::chunk_text_request(request, text_chunk_size, text_chunk_mode); + const auto generation_options = generation_options_from_request(request); + const auto prompt_text = + runtime::find_option(request.options, {"voxcpm1.prompt_text", + "voxcpm1.prompt_text", + "prompt_text", "reference_text"}) + .value_or(""); + std::optional reference_audio; + if (request.voice.has_value() && request.voice->speaker.has_value() && + request.voice->speaker->audio.has_value()) { + reference_audio = *request.voice->speaker->audio; + } + const VoxCPM1EncodedPrompt *prompt = + encoded_prompt_for_request(request.audio_input, prompt_text, + reference_audio); + // The encoded prompt (voice-clone conditioning) is cached as host-side + // vectors; the VAE encoder graph that produced it is not needed again until + // a different voice is encoded. Drop it before the generator runs so the + // generator and decoder phases never coexist with the encoder graph. + decoder_->release_encoder_graph(); + + runtime::TaskResult result; + double generator_ms = 0.0; + double decoder_ms = 0.0; + runtime::AudioBuffer merged_audio; + const bool mem_saver = generator_config_.mem_saver; + for (size_t chunk_index = 0; chunk_index < chunk_requests.size(); + ++chunk_index) { + const auto &chunk_request = chunk_requests[chunk_index]; + const auto generator_start = Clock::now(); + const auto generated = generator_->generate( + chunk_request.text_input->text, prompt, generation_options); + generator_ms += engine::debug::elapsed_ms(generator_start, Clock::now()); + + if (mem_saver && chunk_index + 1 == chunk_requests.size()) { + // Last chunk: free the generator graphs before the AudioVAE decode so + // the final decode peaks at weight + decoder graph instead of weight + + // generator + decoder. Graphs rebuild lazily on the next request. + generator_->release_runtime_memory(); + } + + const auto decoder_start = Clock::now(); + auto audio = decoder_->decode_features(generated.decode_features, + generated.decode_patches); + if (generated.decode_trim_patches > 0) { + const int64_t trim_samples = + generated.decode_trim_patches * assets_->config.patch_size * + product(assets_->config.audio_vae.decoder_rates); + if (trim_samples > static_cast(audio.samples.size())) { + throw std::runtime_error( + "VoxCPM1 decoded continuation trim exceeds audio length"); + } + audio.samples.erase( + audio.samples.begin(), + audio.samples.begin() + static_cast(trim_samples)); + } + decoder_ms += engine::debug::elapsed_ms(decoder_start, Clock::now()); + runtime::append_audio_buffer(merged_audio, audio); + } + result.audio_output = std::move(merged_audio); + + const auto wall_end = Clock::now(); + debug::trace_log_scalar("voxcpm1.text_chunk_size", text_chunk_size); + debug::trace_log_scalar("voxcpm1.text_chunk_mode", + engine::text::text_chunk_mode_name(text_chunk_mode)); + debug::trace_log_scalar("voxcpm1.text_chunk_count", + static_cast(chunk_requests.size())); + debug::timing_log_scalar("voxcpm1.generator_ms", generator_ms); + debug::timing_log_scalar("voxcpm1.audiovae_decoder_ms", decoder_ms); + debug::timing_log_scalar("session.wall_ms", + engine::debug::elapsed_ms(wall_start, wall_end)); + return result; +} + +runtime::TaskResult +VoxCPM1SessionBase::run_streaming_request( + const runtime::TaskRequest &request, + const runtime::StreamEventCallback &stream_event_sink) { + require_prepared("VoxCPM1 run_streaming"); + if (task_.mode != runtime::RunMode::Streaming) { + throw std::runtime_error( + "VoxCPM1 run_streaming requires a streaming session"); + } + validate_request(request); + auto release_runtime_memory = [this](VoxCPM1SessionBase *self) { + if (self != nullptr) { + self->release_request_runtime_memory(); + } + }; + std::unique_ptr + release_guard(this, release_runtime_memory); + + const auto wall_start = Clock::now(); + auto generation_options = generation_options_from_request(request); + const auto prompt_text = + runtime::find_option(request.options, {"voxcpm1.prompt_text", + "voxcpm1.prompt_text", + "prompt_text", "reference_text"}) + .value_or(""); + std::optional reference_audio; + if (request.voice.has_value() && request.voice->speaker.has_value() && + request.voice->speaker->audio.has_value()) { + reference_audio = *request.voice->speaker->audio; + } + const VoxCPM1EncodedPrompt *prompt = + encoded_prompt_for_request(request.audio_input, prompt_text, + reference_audio); + // Same host-side clone-conditioning cache invariant as the offline path: + // the encoder graph is only needed to produce the cached vectors, so free + // it before the streaming generation starts. + decoder_->release_encoder_graph(); + + runtime::TaskResult result; + runtime::AudioBuffer merged; + merged.sample_rate = assets_->config.audio_vae.output_sample_rate; + merged.channels = 1; + double decoder_ms = 0.0; + size_t emitted_chunks = 0; + auto emit_chunk = [&](const VoxCPM1StreamingChunk &chunk) { + const auto decoder_start = Clock::now(); + auto audio = decoder_->decode_features(chunk.decode_features, + chunk.decode_patches); + decoder_ms += engine::debug::elapsed_ms(decoder_start, Clock::now()); + if (emitted_chunks == 0) { + merged.sample_rate = audio.sample_rate; + merged.channels = audio.channels; + } else if (audio.sample_rate != merged.sample_rate || + audio.channels != merged.channels) { + throw std::runtime_error( + "VoxCPM1 streaming decoder chunk format changed"); + } + merged.samples.insert(merged.samples.end(), audio.samples.begin(), + audio.samples.end()); + runtime::NamedAudioBuffer named; + named.id = "chunk_" + std::to_string(emitted_chunks); + named.audio = std::move(audio); + named.meta.insert_or_assign( + "generated_patches", std::to_string(chunk.generated_patches)); + if (stream_event_sink) { + runtime::StreamEvent event; + event.named_audio_outputs.push_back(named); + stream_event_sink(event); + } + result.named_audio_outputs.push_back(std::move(named)); + ++emitted_chunks; + }; + + const auto generator_start = Clock::now(); + (void)generator_->generate_streaming(request.text_input->text, prompt, + generation_options, emit_chunk); + const auto generator_end = Clock::now(); + const double generator_with_callbacks_ms = + engine::debug::elapsed_ms(generator_start, generator_end); + + result.audio_output = std::move(merged); + + const auto wall_end = Clock::now(); + debug::timing_log_scalar( + "voxcpm1.generator_ms", + std::max(0.0, generator_with_callbacks_ms - decoder_ms)); + debug::timing_log_scalar("voxcpm1.generator_streaming_callbacks_ms", + generator_with_callbacks_ms); + debug::timing_log_scalar("voxcpm1.audiovae_decoder_ms", decoder_ms); + debug::timing_log_scalar("voxcpm1.streaming_chunks", + static_cast(emitted_chunks)); + debug::timing_log_scalar("session.wall_ms", + engine::debug::elapsed_ms(wall_start, wall_end)); + return result; +} + +void VoxCPM1SessionBase::release_request_runtime_memory() { + // Only the cloned voice is cached across requests (host-side encoded + // vectors in encoded_prompt_cache_). Every graph whose size follows the + // request text/audio length (prompt prefill, VAE encoder/decoder) is + // dropped so a long-lived server session returns to baseline VRAM and + // reallocates fresh buffers sized to the next request. + generator_->release_text_length_memory(); + decoder_->release_runtime_memory(); + if (generator_config_.mem_saver) { + // mem_saver additionally drops the fixed-size generator graphs so the + // session idles at weight-only VRAM. + generator_->release_runtime_memory(); + } +} + +const VoxCPM1EncodedPrompt *VoxCPM1SessionBase::encoded_prompt_for_request( + const std::optional &prompt_audio, + const std::string &prompt_text, + const std::optional &reference_audio) { + if (!prompt_audio.has_value() && !reference_audio.has_value()) { + return nullptr; + } + // VoxCPM1 clones only via prompt-continuation mode (golden VoxCPM.cpp + // uses --prompt-audio + --prompt-text); the V2 reference-mode path wraps + // audio in tokens 103/104, which the V1 LM was never trained on. Route a + // V1 reference audio through the prompt path so --voice-ref clones like + // --audio. + std::optional effective_prompt_audio = prompt_audio; + std::optional effective_reference_audio = reference_audio; + if (assets_->config.v1 && !effective_prompt_audio.has_value() && + effective_reference_audio.has_value()) { + effective_prompt_audio = effective_reference_audio; + effective_reference_audio.reset(); + } + EncodedPromptCacheKey key; + key.prompt_text = prompt_text; + key.prompt_audio = effective_prompt_audio; + key.reference_audio = effective_reference_audio; + if (auto *cached = encoded_prompt_cache_.find(key)) { + debug::trace_log_scalar("voxcpm1.prompt_cache.hit", 1); + debug::trace_log_scalar("voxcpm1.prompt_cache.slots", + static_cast( + encoded_prompt_cache_.capacity())); + debug::trace_log_scalar("voxcpm1.prompt_cache.entries", + static_cast(encoded_prompt_cache_.size())); + debug::trace_log_scalar("voxcpm1.prompt_cache.evicted", 0); + debug::timing_log_scalar("voxcpm1.prompt_encode_ms", 0.0); + return &cached->encoded; + } + + const auto encode_start = Clock::now(); + EncodedPromptCacheEntry entry; + entry.encoded = decoder_->encode_prompt_audio( + effective_prompt_audio, prompt_text, effective_reference_audio); + const double encode_ms = engine::debug::elapsed_ms(encode_start); + if (encoded_prompt_cache_.capacity() == 0) { + uncached_encoded_prompt_ = std::move(entry); + debug::trace_log_scalar("voxcpm1.prompt_cache.hit", 0); + debug::trace_log_scalar("voxcpm1.prompt_cache.slots", 0); + debug::trace_log_scalar("voxcpm1.prompt_cache.entries", 0); + debug::trace_log_scalar("voxcpm1.prompt_cache.evicted", 0); + debug::timing_log_scalar("voxcpm1.prompt_encode_ms", encode_ms); + return &uncached_encoded_prompt_->encoded; + } + const bool will_evict = + encoded_prompt_cache_.size() >= encoded_prompt_cache_.capacity(); + encoded_prompt_cache_.put(std::move(key), std::move(entry)); + EncodedPromptCacheKey lookup; + lookup.prompt_text = prompt_text; + lookup.prompt_audio = effective_prompt_audio; + lookup.reference_audio = effective_reference_audio; + auto *cached = encoded_prompt_cache_.find(lookup); + if (cached == nullptr) { + throw std::runtime_error("VoxCPM1 prompt cache insert failed"); + } + debug::trace_log_scalar("voxcpm1.prompt_cache.hit", 0); + debug::trace_log_scalar("voxcpm1.prompt_cache.slots", + static_cast( + encoded_prompt_cache_.capacity())); + debug::trace_log_scalar("voxcpm1.prompt_cache.entries", + static_cast(encoded_prompt_cache_.size())); + debug::trace_log_scalar("voxcpm1.prompt_cache.evicted", will_evict ? 1 : 0); + debug::timing_log_scalar("voxcpm1.prompt_encode_ms", + encode_ms); + return &cached->encoded; +} + +VoxCPM1GenerationOptions VoxCPM1SessionBase::generation_options_from_request( + const runtime::TaskRequest &request) const { + VoxCPM1GenerationOptions options; + bool min_tokens_explicit = false; + if (const auto value = runtime::parse_i64_option( + request.options, + {"voxcpm1.min_tokens", "voxcpm1.min_tokens", "min_tokens"})) { + options.min_tokens = *value; + min_tokens_explicit = true; + } + // Set V1-specific default min_tokens if not explicitly provided + if (!min_tokens_explicit && assets_->config.v1) { + // Reference VoxCPM.cpp uses kMinLen=2 (stop may fire from the 4th patch); + // the decode loop gates on `index > min_tokens`, which is the same check. + // A higher floor (e.g. 20) forces ~1.6 s of audio and pads short + // utterances with trailing silence after the stop predictor fires. + options.min_tokens = 2; + } + if (const auto value = runtime::parse_i64_option( + request.options, + {"max_tokens", "voxcpm1.max_tokens", "voxcpm1.max_tokens"})) { + options.max_tokens = *value; + } + if (const auto value = runtime::parse_i64_option( + request.options, + {"num_inference_steps", "voxcpm1.num_inference_steps", + "voxcpm1.num_inference_steps"})) { + options.num_inference_steps = *value; + } + if (const auto value = runtime::parse_finite_float_option( + request.options, + {"guidance_scale", "voxcpm1.guidance_scale", + "voxcpm1.guidance_scale"})) { + options.guidance_scale = *value; + } + bool retry_badcase_explicit = false; + if (const auto match = runtime::find_option_match( + request.options, + {"voxcpm1.retry_badcase", "voxcpm1.retry_badcase", + "retry_badcase"})) { + options.retry_badcase = + runtime::parse_bool_option(match->value, match->key); + retry_badcase_explicit = true; + } + // Streaming emits decoded chunks to the client in real time, so a bad-case + // retry (regenerate from scratch, discard earlier output) is impossible by + // construction. The struct default retry_badcase=true exists for the + // offline path; it must not leak into the streaming path and block every + // streaming request. Relax it to false unless the caller explicitly asked + // for retry, which the generator accepts and warns about. + if (!retry_badcase_explicit && task_.mode == runtime::RunMode::Streaming) { + options.retry_badcase = false; + } + if (const auto value = runtime::parse_i64_option( + request.options, + {"voxcpm1.retry_badcase_max_times", + "voxcpm1.retry_badcase_max_times", "retry_badcase_max_times"})) { + options.retry_badcase_max_times = *value; + } + if (const auto value = runtime::parse_finite_float_option( + request.options, + {"voxcpm1.retry_badcase_ratio_threshold", + "voxcpm1.retry_badcase_ratio_threshold", + "retry_badcase_ratio_threshold"})) { + options.retry_badcase_ratio_threshold = *value; + } + if (const auto value = runtime::parse_u32_option( + request.options, {"voxcpm1.seed", "voxcpm1.seed", "seed"})) { + options.seed = *value; + } + options.cfm_noise_file = + runtime::find_option(request.options, + {"voxcpm1.cfm_noise_file", "voxcpm1.cfm_noise_file", + "cfm_noise_file"}) + .value_or(""); + if (options.min_tokens < 0) { + throw std::runtime_error("VoxCPM1 min_tokens must be non-negative"); + } + if (options.max_tokens < 0) { + throw std::runtime_error("VoxCPM1 max_tokens must be non-negative"); + } + if (options.max_tokens == 0) { + options.max_tokens = assets_->config.max_length; + } + if (options.min_tokens > options.max_tokens) { + throw std::runtime_error("VoxCPM1 min_tokens must not exceed max_tokens"); + } + if (options.max_tokens > assets_->config.max_length) { + throw std::runtime_error( + "VoxCPM1 max_tokens exceeds model config max_length"); + } + if (options.num_inference_steps <= 0) { + throw std::runtime_error( + "VoxCPM1 num_inference_steps must be positive"); + } + if (options.guidance_scale < 0.0F) { + throw std::runtime_error("VoxCPM1 guidance_scale must be non-negative"); + } + if (options.retry_badcase_max_times <= 0) { + throw std::runtime_error( + "VoxCPM1 retry_badcase_max_times must be positive"); + } + if (options.retry_badcase_ratio_threshold <= 0.0F) { + throw std::runtime_error( + "VoxCPM1 retry_badcase_ratio_threshold must be positive"); + } + reject_enabled_denoise(request.options, + {"voxcpm1.denoise", "voxcpm1.denoise", "denoise"}); + reject_enabled_denoise(request.options, + {"voxcpm1.load_denoiser", "voxcpm1.load_denoiser", + "load_denoiser"}); + reject_denoiser_option(request.options, + {"voxcpm1.denoiser", "voxcpm1.denoiser", "denoiser"}); + return options; +} + +void VoxCPM1SessionBase::validate_request( + const runtime::TaskRequest &request) const { + if (!request.text_input.has_value()) { + throw std::runtime_error("VoxCPM1 requires text input"); + } + if (request.text_input->text.empty()) { + throw std::runtime_error("VoxCPM1 text input must not be empty"); + } + if (request.voice.has_value()) { + if (request.voice->style.has_value()) { + throw std::runtime_error( + "VoxCPM1 C++ session does not consume style conditions"); + } + if (request.voice->speaker.has_value()) { + const auto &speaker = *request.voice->speaker; + if (speaker.cached_voice_id.has_value()) { + throw std::runtime_error("VoxCPM1 C++ session requires speaker " + "reference audio, not a cached voice id"); + } + if (!speaker.audio.has_value()) { + throw std::runtime_error( + "VoxCPM1 C++ session speaker condition requires audio"); + } + } + } + if (!request.input_artifacts.empty()) { + throw std::runtime_error( + "VoxCPM1 C++ session does not consume input artifacts"); + } +} + +VoxCPM1OfflineSession::VoxCPM1OfflineSession( + runtime::TaskSpec task, + runtime::SessionOptions options, + std::shared_ptr assets) + : VoxCPM1SessionBase(task, std::move(options), std::move(assets)) {} + +std::string VoxCPM1OfflineSession::family() const { return family_impl(); } + +runtime::VoiceTaskKind VoxCPM1OfflineSession::task_kind() const { + return task_kind_impl(); +} + +runtime::RunMode VoxCPM1OfflineSession::run_mode() const { + return run_mode_impl(); +} + +void VoxCPM1OfflineSession::prepare( + const runtime::SessionPreparationRequest &request) { + prepare_impl(request); +} + +runtime::TaskResult +VoxCPM1OfflineSession::run(const runtime::TaskRequest &request) { + return run_offline_request(request); +} + +VoxCPM1StreamingSession::VoxCPM1StreamingSession( + runtime::TaskSpec task, + runtime::SessionOptions options, + std::shared_ptr assets) + : VoxCPM1SessionBase(task, std::move(options), std::move(assets)) {} + +std::string VoxCPM1StreamingSession::family() const { return family_impl(); } + +runtime::VoiceTaskKind VoxCPM1StreamingSession::task_kind() const { + return task_kind_impl(); +} + +runtime::RunMode VoxCPM1StreamingSession::run_mode() const { + return run_mode_impl(); +} + +void VoxCPM1StreamingSession::prepare( + const runtime::SessionPreparationRequest &request) { + prepare_impl(request); +} + +runtime::StreamingPolicy VoxCPM1StreamingSession::streaming_policy() const { + runtime::StreamingPolicy policy; + policy.input = runtime::StreamingInputKind::None; + policy.output = runtime::StreamingOutputKind::FinalResult; + return policy; +} + +void VoxCPM1StreamingSession::start_stream(const runtime::TaskRequest &request) { + reset(); + result_ = run_streaming_request(request, stream_event_sink_); + started_ = true; +} + +void VoxCPM1StreamingSession::set_stream_event_sink(runtime::StreamEventCallback sink) { + stream_event_sink_ = std::move(sink); +} + +std::optional VoxCPM1StreamingSession::next_stream_event() { + if (!started_) { + throw std::runtime_error("VoxCPM1 streaming has not been started"); + } + if (next_chunk_index_ >= result_.named_audio_outputs.size()) { + return std::nullopt; + } + const auto & named = result_.named_audio_outputs[next_chunk_index_++]; + runtime::StreamEvent event; + event.named_audio_outputs.push_back(named); + return event; +} + +runtime::TaskResult VoxCPM1StreamingSession::finish_stream() { + if (!started_) { + throw std::runtime_error("VoxCPM1 streaming has not been started"); + } + started_ = false; + next_chunk_index_ = 0; + return std::move(result_); +} + +void VoxCPM1StreamingSession::reset() { + result_ = runtime::TaskResult{}; + next_chunk_index_ = 0; + started_ = false; +} + +runtime::StreamEvent VoxCPM1StreamingSession::process_audio_chunk( + const runtime::AudioChunk &chunk) { + (void)chunk; + throw std::runtime_error("VoxCPM1 streaming does not consume audio chunks"); +} + +runtime::TaskResult VoxCPM1StreamingSession::finalize() { + return finish_stream(); +} + +namespace { + +runtime::ModelMetadata metadata_v1(const VoxCPM1Assets &assets) { + runtime::ModelMetadata out; + out.family = "voxcpm1"; + out.variant = assets.config.architecture; + out.description = "VoxCPM1 loaded from GGUF assets."; + return out; +} + +runtime::CapabilitySet capabilities_v1(const VoxCPM1Assets &) { + runtime::CapabilitySet out; + out.supported_tasks = { + {runtime::VoiceTaskKind::Tts, + {runtime::RunMode::Offline, runtime::RunMode::Streaming}}, + }; + out.languages = {"Auto"}; + out.supports_speaker_reference = true; + return out; +} + +runtime::ModelCliInterface cli_v1(const VoxCPM1Assets &) { + runtime::ModelCliInterface out; + out.request_options = { + {"text_chunk_mode", "default|tag_aware|japanese|endline", + "Text chunking mode; default tag_aware."}, + }; + out.session_options = { + {"voxcpm1.mem_saver", "true|false", + "Use tighter graph workspaces and release request runtime graphs; default false."}, + {"voxcpm1.prompt_cache_slots", "n", + "Prompt and prompt-audio embedding cache slots; default 1."}, + }; + return out; +} + +class VoxCPM1LoadedModel final : public runtime::ILoadedVoiceModel { +public: + VoxCPM1LoadedModel(runtime::ModelMetadata metadata, + runtime::CapabilitySet capabilities, + std::shared_ptr assets) + : metadata_(std::move(metadata)), + capabilities_(std::move(capabilities)), + assets_(std::move(assets)) {} + + const runtime::ModelMetadata &metadata() const noexcept override { + return metadata_; + } + + const runtime::CapabilitySet &capabilities() const noexcept override { + return capabilities_; + } + + std::unique_ptr create_task_session( + const runtime::TaskSpec &task, + const runtime::SessionOptions &options) const override { + if (task.task != runtime::VoiceTaskKind::Tts) { + throw std::runtime_error("VoxCPM1 only supports the Tts task"); + } + if (task.mode != runtime::RunMode::Offline && + task.mode != runtime::RunMode::Streaming) { + throw std::runtime_error( + "VoxCPM1 only supports offline and streaming sessions"); + } + if (task.mode == runtime::RunMode::Streaming) { + return std::make_unique( + task, options, assets_); + } + return std::make_unique(task, options, assets_); + } + +private: + runtime::ModelMetadata metadata_; + runtime::CapabilitySet capabilities_; + std::shared_ptr assets_; +}; + +class VoxCPM1Loader final : public runtime::IVoiceModelLoader { +public: + std::string family() const override { return "voxcpm1"; } + + runtime::CapabilitySet advertised_capabilities() const override { + runtime::CapabilitySet out; + out.supported_tasks = { + {runtime::VoiceTaskKind::Tts, + {runtime::RunMode::Offline, runtime::RunMode::Streaming}}, + }; + out.supports_speaker_reference = true; + return out; + } + + std::string advertised_instructions_policy() const override { + return "text_prefix"; + } + + bool can_load(const runtime::ModelLoadRequest &request) const override { + try { + (void)engine::model_spec::load_resource_bundle( + request.model_path, + engine::model_spec::default_spec_path(family())); + return !request.family_hint.has_value() || + *request.family_hint == family(); + } catch (...) { + return false; + } + } + + runtime::ModelInspection inspect( + const runtime::ModelLoadRequest &request) const override { + const auto assets = load_voxcpm1_assets(request.model_path); + runtime::ModelInspection inspection; + inspection.model_root = assets->resources.model_root(); + inspection.metadata = metadata_v1(*assets); + inspection.capabilities = capabilities_v1(*assets); + inspection.cli = cli_v1(*assets); + const auto spec_path = engine::model_spec::default_spec_path(family()); + inspection.discovered_configs = + runtime::discover_named_assets_from_package_spec( + request.model_path, + spec_path, + engine::model_spec::ResourceKind::Files); + inspection.discovered_weights = + runtime::discover_named_assets_from_package_spec( + request.model_path, + spec_path, + engine::model_spec::ResourceKind::Tensors); + return inspection; + } + + std::unique_ptr load( + const runtime::ModelLoadRequest &request) const override { + auto assets = load_voxcpm1_assets(request.model_path); + return std::make_unique( + metadata_v1(*assets), capabilities_v1(*assets), std::move(assets)); + } +}; + +} + +std::shared_ptr make_voxcpm1_loader() { + return std::make_shared(); +} + +} // namespace engine::community_models::voxcpm1 diff --git a/src/models/voxcpm2/tokenizer_gguf.cpp b/src/community_models/voxcpm1/tokenizer_gguf.cpp similarity index 95% rename from src/models/voxcpm2/tokenizer_gguf.cpp rename to src/community_models/voxcpm1/tokenizer_gguf.cpp index f3230653..be729dda 100644 --- a/src/models/voxcpm2/tokenizer_gguf.cpp +++ b/src/community_models/voxcpm1/tokenizer_gguf.cpp @@ -1,7 +1,7 @@ -#include "engine/models/voxcpm2/tokenizer_gguf.h" +#include "engine/community_models/voxcpm1/tokenizer_gguf.h" #include "engine/framework/assets/tensor_source.h" -#include "engine/models/voxcpm2/gguf_metadata.h" +#include "engine/community_models/voxcpm1/gguf_metadata.h" #include #include @@ -12,13 +12,13 @@ #include #include -namespace engine::models::voxcpm2 { +namespace engine::community_models::voxcpm1 { namespace { // UTF-8 handling functions (copied from tokenizer_text.cpp) uint32_t next_utf8_codepoint(std::string_view text, size_t & offset) { if (offset >= text.size()) { - throw std::runtime_error("VoxCPM2 tokenizer UTF-8 offset is out of range"); + throw std::runtime_error("VoxCPM1 tokenizer UTF-8 offset is out of range"); } const unsigned char first = static_cast(text[offset]); uint32_t codepoint = 0; @@ -35,15 +35,15 @@ uint32_t next_utf8_codepoint(std::string_view text, size_t & offset) { len = 4; codepoint = first & 0x07U; } else { - throw std::runtime_error("VoxCPM2 tokenizer encountered invalid UTF-8"); + throw std::runtime_error("VoxCPM1 tokenizer encountered invalid UTF-8"); } if (offset + len > text.size()) { - throw std::runtime_error("VoxCPM2 tokenizer encountered truncated UTF-8"); + throw std::runtime_error("VoxCPM1 tokenizer encountered truncated UTF-8"); } for (size_t i = 1; i < len; ++i) { const unsigned char ch = static_cast(text[offset + i]); if ((ch & 0xC0U) != 0x80U) { - throw std::runtime_error("VoxCPM2 tokenizer encountered invalid UTF-8 continuation"); + throw std::runtime_error("VoxCPM1 tokenizer encountered invalid UTF-8 continuation"); } codepoint = (codepoint << 6U) | (ch & 0x3FU); } @@ -313,11 +313,11 @@ std::vector VoxCPM1GgufTokenizer::encode(const std::string & text) cons return ids; } -VoxCPM2TextPrompt VoxCPM1GgufTokenizer::build_prompt(const std::string & text) const { +VoxCPM1TextPrompt VoxCPM1GgufTokenizer::build_prompt(const std::string & text) const { if (text.empty()) { throw std::runtime_error("VoxCPM1 requires non-empty text input"); } - VoxCPM2TextPrompt prompt; + VoxCPM1TextPrompt prompt; prompt.text = text; prompt.input_ids = encode(text); if (prompt.input_ids.empty()) { @@ -361,4 +361,4 @@ bool VoxCPM1GgufTokenizer::has_tokenizer_metadata(const engine::assets::TensorSo metadata.optional_string_array("tokenizer.ggml.merges").has_value(); } -} // namespace engine::models::voxcpm2 \ No newline at end of file +} // namespace engine::community_models::voxcpm1 \ No newline at end of file diff --git a/src/community_models/voxcpm1/tokenizer_text.cpp b/src/community_models/voxcpm1/tokenizer_text.cpp new file mode 100644 index 00000000..b7f789dc --- /dev/null +++ b/src/community_models/voxcpm1/tokenizer_text.cpp @@ -0,0 +1,353 @@ +#include "engine/community_models/voxcpm1/tokenizer_text.h" +#include "engine/community_models/voxcpm1/assets.h" + +#include "engine/framework/io/json.h" + +#include +#include +#include +#include +#include +#include +#include + +namespace engine::community_models::voxcpm1 { +namespace { + +uint32_t next_utf8_codepoint(std::string_view text, size_t & offset) { + if (offset >= text.size()) { + throw std::runtime_error("VoxCPM1 tokenizer UTF-8 offset is out of range"); + } + const unsigned char first = static_cast(text[offset]); + uint32_t codepoint = 0; + size_t len = 1; + if ((first & 0x80U) == 0) { + codepoint = first; + } else if ((first & 0xE0U) == 0xC0U) { + len = 2; + codepoint = first & 0x1FU; + } else if ((first & 0xF0U) == 0xE0U) { + len = 3; + codepoint = first & 0x0FU; + } else if ((first & 0xF8U) == 0xF0U) { + len = 4; + codepoint = first & 0x07U; + } else { + throw std::runtime_error("VoxCPM1 tokenizer encountered invalid UTF-8"); + } + if (offset + len > text.size()) { + throw std::runtime_error("VoxCPM1 tokenizer encountered truncated UTF-8"); + } + for (size_t i = 1; i < len; ++i) { + const unsigned char ch = static_cast(text[offset + i]); + if ((ch & 0xC0U) != 0x80U) { + throw std::runtime_error("VoxCPM1 tokenizer encountered invalid UTF-8 continuation"); + } + codepoint = (codepoint << 6U) | (ch & 0x3FU); + } + offset += len; + return codepoint; +} + +std::vector utf8_codepoints(std::string_view text) { + std::vector out; + for (size_t offset = 0; offset < text.size();) { + const size_t start = offset; + (void) next_utf8_codepoint(text, offset); + out.emplace_back(text.substr(start, offset - start)); + } + return out; +} + +std::string normalize_text(std::string_view text) { + const std::string space = "\xE2\x96\x81"; + std::string out = space; + for (char ch : text) { + if (ch == ' ') { + out += space; + } else { + out.push_back(ch); + } + } + return out; +} + +std::string byte_fallback_token(unsigned char byte) { + constexpr char kHex[] = "0123456789ABCDEF"; + std::string out = "<0x"; + out.push_back(kHex[(byte >> 4U) & 0x0FU]); + out.push_back(kHex[byte & 0x0FU]); + out.push_back('>'); + return out; +} + +std::vector bpe_initial_pieces( + std::string_view normalized_text, + const std::unordered_map & vocab) { + std::vector pieces; + for (size_t offset = 0; offset < normalized_text.size();) { + const size_t start = offset; + (void) next_utf8_codepoint(normalized_text, offset); + std::string piece(normalized_text.substr(start, offset - start)); + if (vocab.find(piece) != vocab.end()) { + pieces.push_back(std::move(piece)); + continue; + } + for (size_t i = start; i < offset; ++i) { + pieces.push_back(byte_fallback_token(static_cast(normalized_text[i]))); + } + } + return pieces; +} + +bool is_cjk_codepoint(uint32_t codepoint) { + return (codepoint >= 0x4E00 && codepoint <= 0x9FFF) || + (codepoint >= 0x3400 && codepoint <= 0x4DBF) || + (codepoint >= 0xF900 && codepoint <= 0xFAFF) || + (codepoint >= 0x20000 && codepoint <= 0x2A6DF); +} + +bool is_pure_multichar_cjk(std::string_view text) { + size_t count = 0; + for (size_t offset = 0; offset < text.size();) { + if (!is_cjk_codepoint(next_utf8_codepoint(text, offset))) { + return false; + } + ++count; + } + return count >= 2; +} + +std::string strip_sentencepiece_prefix(std::string token) { + const std::string prefix = "\xE2\x96\x81"; + size_t pos = 0; + while ((pos = token.find(prefix, pos)) != std::string::npos) { + token.erase(pos, prefix.size()); + } + return token; +} + +bool starts_with_at(std::string_view text, size_t pos, std::string_view prefix) { + return pos + prefix.size() <= text.size() && text.substr(pos, prefix.size()) == prefix; +} + +std::string pair_key(const std::string & left, const std::string & right) { + std::string key = left; + key.push_back('\0'); + key += right; + return key; +} + +int32_t require_token_id( + const std::unordered_map & vocab, + const std::string & token) { + const auto it = vocab.find(token); + if (it == vocab.end()) { + throw std::runtime_error("VoxCPM1 tokenizer missing token: " + token); + } + return it->second; +} + +} // namespace + +struct VoxCPM1TextTokenizer::Impl { + std::unordered_map vocab; + std::unordered_map id_to_token; + std::unordered_map special_tokens; + std::unordered_map merge_ranks; + std::unordered_map> cjk_split_map; + int32_t audio_start_token_id = 101; + int32_t audio_end_token_id = 102; + int32_t reference_audio_start_token_id = 103; + int32_t reference_audio_end_token_id = 104; + + std::vector bpe(std::string_view normalized_text) const { + std::vector word = bpe_initial_pieces(normalized_text, vocab); + if (word.size() <= 1) { + return word; + } + while (true) { + int32_t best_rank = std::numeric_limits::max(); + size_t best_index = word.size(); + for (size_t i = 0; i + 1 < word.size(); ++i) { + const auto it = merge_ranks.find(pair_key(word[i], word[i + 1])); + if (it != merge_ranks.end() && it->second < best_rank) { + best_rank = it->second; + best_index = i; + } + } + if (best_index == word.size()) { + break; + } + word[best_index] += word[best_index + 1]; + word.erase(word.begin() + static_cast(best_index + 1)); + if (word.size() <= 1) { + break; + } + } + return word; + } + + void append_expanded_id(std::vector & ids, int32_t id) const { + const auto split = cjk_split_map.find(id); + if (split == cjk_split_map.end()) { + ids.push_back(id); + return; + } + ids.insert(ids.end(), split->second.begin(), split->second.end()); + } +}; + +namespace { + +void load_tokenizer_json(const std::filesystem::path & path, VoxCPM1TextTokenizer::Impl & impl) { + const auto root = engine::io::json::parse_file(path); + const auto & model = root.require("model"); + if (model.require("type").as_string() != "BPE") { + throw std::runtime_error("VoxCPM1 tokenizer expects BPE tokenizer.json"); + } + const auto & vocab = model.require("vocab").as_object(); + for (const auto & [token, id_value] : vocab) { + const int32_t id = static_cast(id_value.as_i64()); + impl.vocab.emplace(token, id); + impl.id_to_token.emplace(id, token); + } + const auto & merges = model.require("merges").as_array(); + int32_t rank = 0; + for (const auto & merge_value : merges) { + const std::string merge = merge_value.as_string(); + const size_t split = merge.find(' '); + if (split == std::string::npos) { + throw std::runtime_error("invalid VoxCPM1 tokenizer merge: " + merge); + } + impl.merge_ranks.emplace(pair_key(merge.substr(0, split), merge.substr(split + 1)), rank); + ++rank; + } +} + +void load_special_tokens(const std::filesystem::path & path, VoxCPM1TextTokenizer::Impl & impl) { + const auto root = engine::io::json::parse_file(path); + const auto * added = root.find("added_tokens_decoder"); + if (added == nullptr) { + throw std::runtime_error("VoxCPM1 tokenizer_config missing added_tokens_decoder"); + } + for (const auto & [id_text, token_config] : added->as_object()) { + const auto * content = token_config.find("content"); + if (content == nullptr || !content->is_string()) { + continue; + } + const int32_t id = static_cast(std::stoll(id_text)); + const std::string token = content->as_string(); + impl.vocab[token] = id; + impl.id_to_token[id] = token; + impl.special_tokens[token] = id; + } + impl.audio_start_token_id = require_token_id(impl.vocab, "<|audio_start|>"); + impl.audio_end_token_id = require_token_id(impl.vocab, "<|audio_end|>"); + impl.reference_audio_start_token_id = require_token_id(impl.vocab, "<|audio_prompt_start|>"); + impl.reference_audio_end_token_id = require_token_id(impl.vocab, "<|audio_prompt_end|>"); +} + +void build_cjk_split_map(VoxCPM1TextTokenizer::Impl & impl) { + for (const auto & [id, token] : impl.id_to_token) { + const std::string clean = strip_sentencepiece_prefix(token); + if (!is_pure_multichar_cjk(clean)) { + continue; + } + std::vector char_ids; + for (const auto & ch : utf8_codepoints(clean)) { + const auto it = impl.vocab.find(ch); + if (it == impl.vocab.end()) { + char_ids.clear(); + break; + } + char_ids.push_back(it->second); + } + if (!char_ids.empty()) { + impl.cjk_split_map.emplace(id, std::move(char_ids)); + } + } +} + +std::shared_ptr load_impl(const VoxCPM1Assets & assets) { + auto impl = std::make_shared(); + load_tokenizer_json(assets.resources.require_file("tokenizer_json"), *impl); + load_special_tokens(assets.resources.require_file("tokenizer_config"), *impl); + build_cjk_split_map(*impl); + return impl; +} + +} // namespace + +VoxCPM1TextTokenizer::VoxCPM1TextTokenizer(std::shared_ptr assets) { + if (assets == nullptr) { + throw std::runtime_error("VoxCPM1 text tokenizer requires assets"); + } + impl_ = load_impl(*assets); +} + +std::vector VoxCPM1TextTokenizer::encode(const std::string & text) const { + std::vector ids; + for (size_t i = 0; i < text.size();) { + const auto special_it = std::find_if( + impl_->special_tokens.begin(), + impl_->special_tokens.end(), + [&](const auto & item) { return starts_with_at(text, i, item.first); }); + if (special_it != impl_->special_tokens.end()) { + impl_->append_expanded_id(ids, special_it->second); + i += special_it->first.size(); + continue; + } + + size_t next_special = text.size(); + for (const auto & [special, _] : impl_->special_tokens) { + const size_t pos = text.find(special, i); + if (pos != std::string::npos) { + next_special = std::min(next_special, pos); + } + } + const std::string normalized = normalize_text(std::string_view( + text.data() + static_cast(i), + next_special - i)); + for (const auto & bpe_token : impl_->bpe(normalized)) { + const auto vocab_it = impl_->vocab.find(bpe_token); + if (vocab_it == impl_->vocab.end()) { + throw std::runtime_error("VoxCPM1 tokenizer produced token not present in vocab: " + bpe_token); + } + impl_->append_expanded_id(ids, vocab_it->second); + } + i = next_special; + } + return ids; +} + +VoxCPM1TextPrompt VoxCPM1TextTokenizer::build_prompt(const std::string & text) const { + if (text.empty()) { + throw std::runtime_error("VoxCPM1 requires non-empty text input"); + } + VoxCPM1TextPrompt prompt; + prompt.text = text; + prompt.input_ids = encode(text); + if (prompt.input_ids.empty()) { + throw std::runtime_error("VoxCPM1 tokenizer produced no tokens"); + } + return prompt; +} + +int32_t VoxCPM1TextTokenizer::audio_start_token_id() const noexcept { + return impl_->audio_start_token_id; +} + +int32_t VoxCPM1TextTokenizer::audio_end_token_id() const noexcept { + return impl_->audio_end_token_id; +} + +int32_t VoxCPM1TextTokenizer::reference_audio_start_token_id() const noexcept { + return impl_->reference_audio_start_token_id; +} + +int32_t VoxCPM1TextTokenizer::reference_audio_end_token_id() const noexcept { + return impl_->reference_audio_end_token_id; +} + +} // namespace engine::community_models::voxcpm1 diff --git a/src/models/voxcpm2/assets.cpp b/src/models/voxcpm2/assets.cpp index 08b98c07..7d4ae2c4 100644 --- a/src/models/voxcpm2/assets.cpp +++ b/src/models/voxcpm2/assets.cpp @@ -1,21 +1,12 @@ #include "engine/models/voxcpm2/assets.h" -#include "engine/models/voxcpm2/tokenizer_gguf.h" -#include "engine/models/voxcpm2/config_gguf.h" #include "engine/framework/model_spec/package.h" #include "engine/framework/assets/resource_bundle.h" -#include "engine/framework/assets/tensor_source.h" #include "engine/framework/io/config.h" #include "engine/framework/io/json.h" #include #include -#include -#include -#include -#include -#include -#include namespace engine::models::voxcpm2 { namespace json = engine::io::json; @@ -147,8 +138,8 @@ VoxCPM2Config parse_config(const assets::ResourceBundle & resources) { const auto root = resources.parse_json("config"); VoxCPM2Config config; config.architecture = json::require_string(root, "architecture"); - if (config.architecture != "voxcpm2" && config.architecture != "voxcpm") { - throw std::runtime_error("VoxCPM config architecture mismatch: " + config.architecture); + if (config.architecture != "voxcpm2") { + throw std::runtime_error("VoxCPM2 config architecture mismatch: " + config.architecture); } config.lm = parse_lm_config(root.require("lm_config")); config.patch_size = json::optional_i64(root, "patch_size", config.patch_size); @@ -181,686 +172,6 @@ VoxCPM2Config parse_config(const assets::ResourceBundle & resources) { return config; } -namespace assets = engine::assets; - -namespace { -core::TensorShape make_tensor_shape(const std::vector & dims) { - if (dims.empty() || dims.size() > core::kMaxTensorRank) { - throw std::runtime_error("tensor rank must be between 1 and 4"); - } - switch (dims.size()) { - case 1: - return core::TensorShape::from_dims({dims[0]}); - case 2: - return core::TensorShape::from_dims({dims[0], dims[1]}); - case 3: - return core::TensorShape::from_dims({dims[0], dims[1], dims[2]}); - case 4: - return core::TensorShape::from_dims({dims[0], dims[1], dims[2], dims[3]}); - default: - throw std::runtime_error("unsupported tensor rank"); - } -} -} // namespace - -class TransformingTensorSource final : public assets::TensorSource { -public: - TransformingTensorSource( - std::shared_ptr source, - const VoxCPM2Config & config, - bool is_v1) - : source_(std::move(source)), config_(config), is_v1_(is_v1) { - build_routes(); - } - - const std::filesystem::path & source_path() const noexcept override { - return source_->source_path(); - } - - bool has_tensor(std::string_view name) const noexcept override { - const std::string key{std::string(name)}; - if (routes_.find(key) != routes_.end() || - synthesized_tensors_.find(key) != synthesized_tensors_.end()) { - return true; - } - if (is_v1_) { - // v1 GGUF stores folded AudioVAE conv weights; the loader asks for - // decomposed weight_v/weight_g names which we synthesize from the - // folded tensors on demand. - const auto base = folded_base_name(key); - if (!base.empty() && folded_convs_.find(base) != folded_convs_.end()) { - return true; - } - } - return false; - } - - assets::TensorMetadata require_metadata(std::string_view name) const override { - const auto it = synthesized_tensors_.find(std::string(name)); - if (it != synthesized_tensors_.end()) { - return it->second; - } - const std::string key{std::string(name)}; - if (is_v1_) { - const auto base = folded_base_name(key); - if (!base.empty()) { - const auto folded_it = folded_convs_.find(base); - if (folded_it != folded_convs_.end()) { - auto metadata = source_->require_metadata(folded_it->second); - metadata.name = key; - if (has_suffix(key, ".weight_g") && !metadata.shape.empty()) { - metadata.shape = {metadata.shape.front(), 1, 1}; - } - return metadata; - } - } - } - const auto route_it = routes_.find(key); - if (route_it == routes_.end()) { - throw std::runtime_error("missing tensor: " + std::string(name)); - } - auto metadata = source_->require_metadata(route_it->second); - metadata.name = key; - // Apply shape transformations if needed - if (reshape_map_.find(key) != reshape_map_.end()) { - metadata.shape = reshape_map_.at(key); - } - return metadata; - } - - std::vector tensors() const override { - std::vector out; - out.reserve(routes_.size() + synthesized_tensors_.size()); - for (const auto & [name, route] : routes_) { - out.push_back(require_metadata(name)); - } - for (const auto & [name, metadata] : synthesized_tensors_) { - out.push_back(metadata); - } - std::sort(out.begin(), out.end(), - [](const assets::TensorMetadata & lhs, const assets::TensorMetadata & rhs) { - return lhs.name < rhs.name; - }); - return out; - } - - void release_storage() const override { source_->release_storage(); } - - assets::RawTensorData require_tensor_data(std::string_view name) const override { - const auto it = synthesized_tensors_.find(std::string(name)); - if (it != synthesized_tensors_.end()) { - return generate_synthesized_tensor(name); - } - const std::string key{std::string(name)}; - if (is_v1_) { - const auto base = folded_base_name(key); - if (!base.empty() && folded_convs_.find(base) != folded_convs_.end()) { - auto data = source_->require_tensor_data(folded_convs_.at(base)); - data.metadata.name = key; - return data; - } - } - const auto route_it = routes_.find(key); - if (route_it == routes_.end()) { - throw std::runtime_error("missing tensor: " + std::string(name)); - } - auto data = source_->require_tensor_data(route_it->second); - data.metadata.name = key; - // Apply transformations - if (reshape_map_.find(key) != reshape_map_.end()) { - const auto & target_shape = reshape_map_.at(key); - if (data.metadata.shape != target_shape) { - // Reshape the data - data = reshape_tensor_data(data, target_shape); - } - } - return data; - } - - std::vector require_f32( - std::string_view name, - const std::optional> & expected_shape) const override { - const auto it = synthesized_tensors_.find(std::string(name)); - if (it != synthesized_tensors_.end()) { - return generate_synthesized_f32(name); - } - const std::string key{std::string(name)}; - if (is_v1_) { - const auto base = folded_base_name(key); - if (!base.empty()) { - const auto folded_it = folded_convs_.find(base); - if (folded_it != folded_convs_.end()) { - const auto folded = source_->require_f32(folded_it->second, std::nullopt); - if (has_suffix(key, ".weight_g")) { - return folded_weight_g(folded, folded_it->second, expected_shape); - } - return folded; - } - } - } - const auto route_it = routes_.find(key); - if (route_it == routes_.end()) { - throw std::runtime_error("missing tensor: " + std::string(name)); - } - if (is_v1_ && expected_shape.has_value()) { - const auto meta = source_->require_metadata(route_it->second); - const int64_t expected_elems = checked_element_count("expected", *expected_shape); - const int64_t actual_elems = checked_element_count(route_it->second, meta.shape); - if (expected_elems == actual_elems && meta.shape != *expected_shape) { - return source_->require_f32(route_it->second, std::nullopt); - } - } - // Check if we need to reshape - if (reshape_map_.find(key) != reshape_map_.end()) { - const auto & target_shape = reshape_map_.at(key); - if (expected_shape.has_value() && *expected_shape != target_shape) { - // We'll fetch with target shape and then it will be validated - } - return source_->require_f32(route_it->second, target_shape); - } - return source_->require_f32(route_it->second, expected_shape); - } - - std::optional> optional_f32( - std::string_view name, - const std::optional> & expected_shape) const override { - if (!has_tensor(name)) return std::nullopt; - return require_f32(name, expected_shape); - } - - void set_backend_tensor( - ggml_tensor * tensor, - std::string_view name, - assets::TensorStorageType storage_type, - const std::vector & expected_shape) const override { - const auto it = synthesized_tensors_.find(std::string(name)); - if (it != synthesized_tensors_.end()) { - const auto values = generate_synthesized_f32(name); - engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, values, - make_tensor_shape(expected_shape), - engine::assets::ggml_type_for_tensor_storage(storage_type)); - return; - } - const auto route_it = routes_.find(std::string(name)); - if (route_it == routes_.end()) { - throw std::runtime_error("missing tensor: " + std::string(name)); - } - // Check for weight norm decomposition (weight_v + weight_g) - const std::string logical_name = std::string(name); - if (weight_norm_map_.find(logical_name) != weight_norm_map_.end()) { - const auto & wn = weight_norm_map_.at(logical_name); - const auto weight_v = source_->require_f32(wn.weight_v_name, wn.weight_v_shape); - const auto weight_g = source_->require_f32(wn.weight_g_name, wn.weight_g_shape); - const auto folded = fold_weight_norm(weight_v, weight_g, wn.out_channels, wn.in_channels, wn.kernel_size); - const auto shape = make_tensor_shape(expected_shape); - const ggml_type type = engine::assets::ggml_type_for_tensor_storage( - engine::assets::resolve_tensor_storage_type(*this, name, storage_type)); - engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, folded, shape, type); - return; - } - // Check for reshape - if (reshape_map_.find(logical_name) != reshape_map_.end()) { - const auto & target_shape = reshape_map_.at(logical_name); - const auto values = source_->require_f32(route_it->second, target_shape); - const auto shape = make_tensor_shape(expected_shape); - const ggml_type type = engine::assets::ggml_type_for_tensor_storage( - engine::assets::resolve_tensor_storage_type(*this, name, storage_type)); - engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, values, shape, type); - return; - } - // Special handling for V1 embedding weight: token_embd.weight is transposed in GGUF - // V1 GGUF stores [hidden_size, vocab_size] but we need [vocab_size, hidden_size] - if (is_v1_ && logical_name == "base_lm.embed_tokens.weight") { - const auto source_values = source_->require_f32(route_it->second, std::nullopt); - const auto source_meta = source_->require_metadata(route_it->second); - if (source_meta.shape.size() == 2) { - const int64_t src_rows = source_meta.shape[0]; - const int64_t src_cols = source_meta.shape[1]; - const int64_t dst_rows = expected_shape.size() > 0 ? expected_shape[0] : src_cols; - const int64_t dst_cols = expected_shape.size() > 1 ? expected_shape[1] : src_rows; - if (src_rows == dst_cols && src_cols == dst_rows) { - // Transpose the weight matrix - std::vector transposed(static_cast(dst_rows * dst_cols)); - for (int64_t i = 0; i < src_rows; ++i) { - for (int64_t j = 0; j < src_cols; ++j) { - transposed[static_cast(j * dst_rows + i)] = source_values[static_cast(i * src_cols + j)]; - } - } - const auto shape = make_tensor_shape(expected_shape); - const ggml_type type = engine::assets::ggml_type_for_tensor_storage( - engine::assets::resolve_tensor_storage_type(*this, name, storage_type)); - engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, transposed, shape, type); - return; - } - } - } - // V1 relaxed rank: if expected element count matches actual but shapes differ, - // fetch data without expected_shape and set manually - if (is_v1_) { - const auto source_meta = source_->require_metadata(route_it->second); - int64_t expected_elems = 1; - for (const int64_t dim : expected_shape) expected_elems *= dim; - int64_t actual_elems = 1; - for (const int64_t dim : source_meta.shape) actual_elems *= dim; - if (expected_elems == actual_elems && source_meta.shape != expected_shape) { - const auto values = source_->require_f32(route_it->second, std::nullopt); - const auto shape = make_tensor_shape(expected_shape); - const ggml_type type = engine::assets::ggml_type_for_tensor_storage( - engine::assets::resolve_tensor_storage_type(*this, name, storage_type)); - engine::assets::set_backend_tensor_from_f32_parallel(tensor, name, values, shape, type); - return; - } - } - source_->set_backend_tensor(tensor, route_it->second, storage_type, expected_shape); - } - - void set_backend_f32_tensor( - ggml_tensor * tensor, - std::string_view name, - const std::vector & expected_shape) const override { - set_backend_tensor(tensor, name, assets::TensorStorageType::F32, expected_shape); - } - - int64_t require_i64_scalar(std::string_view name) const override { - return source_->require_i64_scalar(name); - } - -private: - struct WeightNormInfo { - std::string weight_v_name; - std::string weight_g_name; - std::vector weight_v_shape; - std::vector weight_g_shape; - int64_t out_channels = 0; - int64_t in_channels = 0; - int64_t kernel_size = 0; - }; - - void build_routes() { - // V1 -> V2 tensor name mapping - std::unordered_map rename_map = { - // LM embeddings - {"token_embd.weight", "base_lm.embed_tokens.weight"}, - // LM blocks - {"blk.", "base_lm.layers."}, - {"attn_q.weight", "self_attn.q_proj.weight"}, - {"attn_k.weight", "self_attn.k_proj.weight"}, - {"attn_v.weight", "self_attn.v_proj.weight"}, - {"attn_norm.weight", "input_layernorm.weight"}, - {"attn_output.weight", "self_attn.o_proj.weight"}, - {"ffn_norm.weight", "post_attention_layernorm.weight"}, - {"ffn_gate.weight", "mlp.gate_proj.weight"}, - {"ffn_up.weight", "mlp.up_proj.weight"}, - {"ffn_down.weight", "mlp.down_proj.weight"}, - // Output norm - {"output_norm.weight", "base_lm.norm.weight"}, - // Residual LM - {"residual_lm.blk.", "residual_lm.layers."}, - {"residual_lm.output_norm.weight", "residual_lm.norm.weight"}, - // Local encoder (feat_encoder) - {"locenc.in_proj.weight", "feat_encoder.in_proj.weight"}, - {"locenc.in_proj.bias", "feat_encoder.in_proj.bias"}, - {"locenc.special_token", "feat_encoder.special_token"}, - {"locenc.blk.", "feat_encoder.encoder.layers."}, - {"locenc.output_norm.weight", "feat_encoder.encoder.norm.weight"}, - // Local DiT (feat_decoder) - {"locdit.in_proj.weight", "feat_decoder.estimator.in_proj.weight"}, - {"locdit.in_proj.bias", "feat_decoder.estimator.in_proj.bias"}, - {"locdit.cond_proj.weight", "feat_decoder.estimator.cond_proj.weight"}, - {"locdit.cond_proj.bias", "feat_decoder.estimator.cond_proj.bias"}, - {"locdit.out_proj.weight", "feat_decoder.estimator.out_proj.weight"}, - {"locdit.out_proj.bias", "feat_decoder.estimator.out_proj.bias"}, - {"locdit.time_mlp.linear_1.weight", "feat_decoder.estimator.time_mlp.linear_1.weight"}, - {"locdit.time_mlp.linear_1.bias", "feat_decoder.estimator.time_mlp.linear_1.bias"}, - {"locdit.time_mlp.linear_2.weight", "feat_decoder.estimator.time_mlp.linear_2.weight"}, - {"locdit.time_mlp.linear_2.bias", "feat_decoder.estimator.time_mlp.linear_2.bias"}, - {"locdit.delta_time_mlp.linear_1.weight", "feat_decoder.estimator.delta_time_mlp.linear_1.weight"}, - {"locdit.delta_time_mlp.linear_1.bias", "feat_decoder.estimator.delta_time_mlp.linear_1.bias"}, - {"locdit.delta_time_mlp.linear_2.weight", "feat_decoder.estimator.delta_time_mlp.linear_2.weight"}, - {"locdit.delta_time_mlp.linear_2.bias", "feat_decoder.estimator.delta_time_mlp.linear_2.bias"}, - {"locdit.output_norm.weight", "feat_decoder.estimator.decoder.norm.weight"}, - {"locdit.blk.", "feat_decoder.estimator.decoder.layers."}, - // Projections - {"proj.enc_to_lm.weight", "enc_to_lm_proj.weight"}, - {"proj.enc_to_lm.bias", "enc_to_lm_proj.bias"}, - {"proj.lm_to_dit.weight", "lm_to_dit_proj.weight"}, - {"proj.lm_to_dit.bias", "lm_to_dit_proj.bias"}, - {"proj.res_to_dit.weight", "res_to_dit_proj.weight"}, - {"proj.res_to_dit.bias", "res_to_dit_proj.bias"}, - // V1→V2 mapping for fusion_concat_proj (critical for V1 models with fusion) - {"proj.fusion_concat.weight", "fusion_concat_proj.weight"}, - {"proj.fusion_concat.bias", "fusion_concat_proj.bias"}, - {"fusion_concat_proj.weight", "fusion_concat_proj.weight"}, - {"stop.stop_proj.weight", "stop_proj.weight"}, - {"stop.stop_proj.bias", "stop_proj.bias"}, - {"stop.stop_head.weight", "stop_head.weight"}, - // FSQ - {"fsq.in_proj.weight", "fsq_layer.in_proj.weight"}, - {"fsq.in_proj.bias", "fsq_layer.in_proj.bias"}, - {"fsq.out_proj.weight", "fsq_layer.out_proj.weight"}, - {"fsq.out_proj.bias", "fsq_layer.out_proj.bias"}, - // Audio VAE (prefixed with audio_vae.) - {"audio_vae.encoder.block.", "encoder.block."}, - {"audio_vae.encoder.fc_mu", "encoder.fc_mu"}, - {"audio_vae.decoder.model.", "decoder.model."}, - {"audio_vae.decoder.sr_cond_model.", "decoder.sr_cond_model."}, - }; - - // Build routes by scanning source tensors - for (const auto & tensor : source_->tensors()) { - std::string v1_name = tensor.name; - std::string v2_name = v1_name; - - // Apply prefix replacements - for (const auto & [from, to] : rename_map) { - if (v2_name.rfind(from, 0) == 0) { - v2_name = to + v2_name.substr(from.size()); - break; - } - } - - // Handle blk.N.* -> layers.N.* (base LM, residual LM, locenc, locdit) - constexpr std::string_view kBlk = "blk."; - const size_t blk_pos = v1_name.find(kBlk); - if (blk_pos != std::string::npos) { - const size_t layer_start = blk_pos + kBlk.size(); - const size_t dot = v1_name.find('.', layer_start); - if (dot != std::string::npos) { - const std::string layer_idx = v1_name.substr(layer_start, dot - layer_start); - const std::string rest = v1_name.substr(dot + 1); - if (v1_name.rfind("residual_lm.", 0) == 0) { - v2_name = "residual_lm.layers." + layer_idx + "." + rest; - } else if (v1_name.rfind("locenc.", 0) == 0) { - v2_name = "feat_encoder.encoder.layers." + layer_idx + "." + rest; - } else if (v1_name.rfind("locdit.", 0) == 0) { - v2_name = "feat_decoder.estimator.decoder.layers." + layer_idx + "." + rest; - } else { - v2_name = "base_lm.layers." + layer_idx + "." + rest; - } - // Further sub-replacements - for (const auto & [from, to] : rename_map) { - size_t pos = v2_name.find(from); - if (pos != std::string::npos) { - v2_name.replace(pos, from.size(), to); - } - } - } - } - - routes_[v2_name] = v1_name; - } - - // Reshape map - reshape_map_ = { - // feat_quant: {N, F} -> {N, F, 1} - // merge: {N, D} -> {N, D, 1} - // downsample/upsample: {out, in} -> {out, in, k, k} (k=3 for 3x3) - // V1 embedding: token_embd.weight [hidden, vocab] -> base_lm.embed_tokens.weight [vocab, hidden] - {"base_lm.embed_tokens.weight", {config_.lm.vocab_size, config_.lm.hidden_size}}, - }; - - // Folded AudioVAE conv weights: v1 GGUF stores weight-norm weights - // already folded into a single `.weight` tensor, while the v2 loader - // requests decomposed `.weight_v`/`.weight_g` names. Register every - // audio_vae conv weight so those logical names resolve to the folded - // data (weight_v) and its per-channel row norms (weight_g), which makes - // the loader's fold_weight_norm an exact identity. - if (is_v1_) { - std::vector> folded; - for (const auto & [logical, source] : routes_) { - if (source.rfind("audio_vae.", 0) == 0 && has_suffix(logical, ".weight")) { - folded.emplace_back( - logical.substr(0, logical.size() - 7), source); - } - } - for (const auto & [base, source] : folded) { - folded_convs_[base] = source; - } - } - - // Synthesized tensors for V1 - const int64_t encoder_hidden = config_.encoder.hidden_dim; - const int64_t feat_dim = config_.feat_dim; - const int64_t lm_hidden = config_.lm.hidden_size; - - // feat_encoder.scale_embed (identity buckets) - synthesized_tensors_["feat_encoder.scale_embed.weight"] = - assets::TensorMetadata{"feat_encoder.scale_embed.weight", "F32", {32, encoder_hidden}}; - synthesized_tensors_["feat_encoder.bias_embed.weight"] = - assets::TensorMetadata{"feat_encoder.bias_embed.weight", "F32", {32, encoder_hidden}}; - - // feat_encoder.fc_logvar (zeros) - synthesized_tensors_["feat_encoder.fc_logvar.weight"] = - assets::TensorMetadata{"feat_encoder.fc_logvar.weight", "F32", {feat_dim, encoder_hidden}}; - - // feat_encoder.diag (identity) - synthesized_tensors_["feat_encoder.diag"] = - assets::TensorMetadata{"feat_encoder.diag", "F32", {feat_dim}}; - - // feat_encoder.special_token: V1 GGUF stores as 1D [1024], model code handles reshaping - // Only synthesize if not present in GGUF - if (routes_.find("feat_encoder.special_token") == routes_.end()) { - synthesized_tensors_["feat_encoder.special_token"] = - assets::TensorMetadata{"feat_encoder.special_token", "F32", {encoder_hidden}}; - } - - // token_embd.extra_bias (from logit_scale or zeros) - if (routes_.find("token_embd.extra_bias") == routes_.end()) { - synthesized_tensors_["token_embd.extra_bias"] = - assets::TensorMetadata{"token_embd.extra_bias", "F32", {lm_hidden}}; - } - - // feat_encoder.merge (zeros) - if (routes_.find("feat_encoder.merge.weight") == routes_.end()) { - synthesized_tensors_["feat_encoder.merge.weight"] = - assets::TensorMetadata{"feat_encoder.merge.weight", "F32", {encoder_hidden, feat_dim}}; - } - - // Identity SR-condition embeddings for V1 decoder blocks. VoxCPM1 - // GGUFs contain no sr_cond_model tensors (no SR conditioning), but the - // shared decoder loader requires scale_embed/bias_embed. - { - const auto & vae = config_.audio_vae; - const size_t num_blocks = vae.decoder_rates.size(); - for (size_t i = 0; i < num_blocks; ++i) { - const int64_t input_channels = - vae.decoder_dim / (int64_t{1} << static_cast(i)); - const std::string prefix = - "decoder.sr_cond_model." + std::to_string(i + 2) + "."; - if (routes_.find(prefix + "scale_embed.weight") == routes_.end()) { - synthesized_tensors_[prefix + "scale_embed.weight"] = - assets::TensorMetadata{prefix + "scale_embed.weight", "F32", {1, input_channels}}; - } - if (routes_.find(prefix + "bias_embed.weight") == routes_.end()) { - synthesized_tensors_[prefix + "bias_embed.weight"] = - assets::TensorMetadata{prefix + "bias_embed.weight", "F32", {1, input_channels}}; - } - } - } - - // Missing projection weights for V1 (not in VoxCPM1 GGUF) - if (routes_.find("fusion_concat_proj.weight") == routes_.end()) { - synthesized_tensors_["fusion_concat_proj.weight"] = - assets::TensorMetadata{"fusion_concat_proj.weight", "F32", {lm_hidden, lm_hidden * 2}}; - } - if (routes_.find("fusion_concat_proj.bias") == routes_.end()) { - synthesized_tensors_["fusion_concat_proj.bias"] = - assets::TensorMetadata{"fusion_concat_proj.bias", "F32", {lm_hidden}}; - } - } - - std::vector fold_weight_norm( - const std::vector & weight_v, - const std::vector & weight_g, - int64_t out_channels, int64_t in_channels, int64_t kernel_size) const { - if (static_cast(weight_v.size()) != out_channels * in_channels * kernel_size || - static_cast(weight_g.size()) != out_channels) { - throw std::runtime_error("VoxCPM1 weight-norm shape mismatch"); - } - std::vector out(weight_v.size(), 0.0F); - for (int64_t d0 = 0; d0 < out_channels; ++d0) { - const size_t base = static_cast(d0 * in_channels * kernel_size); - double norm_sq = 0.0; - for (int64_t i = 0; i < in_channels * kernel_size; ++i) { - const double value = weight_v[base + static_cast(i)]; - norm_sq += value * value; - } - const float scale = weight_g[static_cast(d0)] / - static_cast(std::sqrt(norm_sq + 1e-8)); - for (int64_t i = 0; i < in_channels * kernel_size; ++i) { - out[base + static_cast(i)] = weight_v[base + static_cast(i)] * scale; - } - } - return out; - } - - assets::RawTensorData reshape_tensor_data(const assets::RawTensorData & data, - const std::vector &) const { - // For now, just return the data as-is (validation happens elsewhere) - // The actual reshape happens in require_f32 - return data; - } - - assets::RawTensorData generate_synthesized_tensor(std::string_view name) const { - const auto it = synthesized_tensors_.find(std::string(name)); - if (it == synthesized_tensors_.end()) { - throw std::runtime_error("no synthesized tensor: " + std::string(name)); - } - const auto & metadata = it->second; - const int64_t num_elements = std::accumulate(metadata.shape.begin(), metadata.shape.end(), 1, std::multiplies()); - std::vector bytes(num_elements * sizeof(float)); - std::memset(bytes.data(), 0, bytes.size()); - return {metadata, std::move(bytes)}; - } - - std::vector generate_synthesized_f32(std::string_view name) const { - const auto it = synthesized_tensors_.find(std::string(name)); - if (it == synthesized_tensors_.end()) { - throw std::runtime_error("no synthesized tensor: " + std::string(name)); - } - const auto & metadata = it->second; - const int64_t num_elements = std::accumulate(metadata.shape.begin(), metadata.shape.end(), 1, std::multiplies()); - if (name == "feat_encoder.diag") { - std::vector out(num_elements, 1.0F); - return out; - } - if (std::string_view prefix = "decoder.sr_cond_model."; - name.rfind(prefix, 0) == 0 && has_suffix(name, ".scale_embed.weight")) { - return std::vector(num_elements, 1.0F); - } - if (name == "feat_encoder.scale_embed.weight" || name == "feat_encoder.bias_embed.weight") { - // Identity-like initialization - std::vector out(num_elements, 0.0F); - // Fill with small values - for (size_t i = 0; i < out.size(); ++i) { - out[i] = 0.01F; - } - return out; - } - if (name == "fusion_concat_proj.weight") { - // Xavier/Glorot initialization for fusion_concat_proj weight - // shape is [lm_hidden, lm_hidden * 2] - std::vector out(num_elements); - const float scale = std::sqrt(2.0f / (config_.lm.hidden_size + config_.lm.hidden_size * 2)); - for (size_t i = 0; i < out.size(); ++i) { - // Simple uniform distribution in [-scale, scale] - out[i] = (static_cast(std::rand()) / RAND_MAX * 2.0f - 1.0f) * scale; - } - return out; - } - return std::vector(num_elements, 0.0F); - } - - static bool has_suffix(std::string_view value, std::string_view suffix) { - return value.size() >= suffix.size() && - value.substr(value.size() - suffix.size()) == suffix; - } - - std::string folded_base_name(const std::string & key) const { - constexpr std::string_view kWeightV = ".weight_v"; - constexpr std::string_view kWeightG = ".weight_g"; - if (has_suffix(key, kWeightV)) { - return key.substr(0, key.size() - kWeightV.size()); - } - if (has_suffix(key, kWeightG)) { - return key.substr(0, key.size() - kWeightG.size()); - } - return ""; - } - - std::vector folded_weight_g( - const std::vector & folded, - const std::string & folded_source_name, - const std::optional> & expected_shape) const { - const auto meta = source_->require_metadata(folded_source_name); - const int64_t groups = expected_shape.has_value() && !expected_shape->empty() - ? expected_shape->front() - : (meta.shape.empty() ? 0 : meta.shape.front()); - const int64_t rows = checked_element_count(folded_source_name, meta.shape); - if (groups <= 0 || rows == 0 || rows % groups != 0) { - throw std::runtime_error("folded weight_g shape mismatch: " + folded_source_name); - } - const int64_t inner = rows / groups; - std::vector out(static_cast(groups), 0.0F); - for (int64_t g = 0; g < groups; ++g) { - double norm_sq = 0.0; - for (int64_t i = 0; i < inner; ++i) { - const float value = folded[static_cast(g * inner + i)]; - norm_sq += static_cast(value) * static_cast(value); - } - out[static_cast(g)] = static_cast(std::sqrt(norm_sq)); - } - return out; - } - - static int64_t checked_element_count(std::string_view name, const std::vector & shape) { - int64_t count = 1; - for (const int64_t dim : shape) { - if (dim <= 0) { - throw std::runtime_error("tensor shape contains a non-positive dimension: " + std::string(name)); - } - if (count > std::numeric_limits::max() / dim) { - throw std::runtime_error("tensor element count overflow: " + std::string(name)); - } - count *= dim; - } - return count; - } - - std::shared_ptr source_; - VoxCPM2Config config_; - bool is_v1_; - std::unordered_map routes_; - std::unordered_map> reshape_map_; - std::unordered_map weight_norm_map_; - std::unordered_map synthesized_tensors_; - std::unordered_map folded_convs_; -}; - -void require_vae_weight_v_shape(const assets::TensorSource & source, - std::string_view name, - const std::vector & expected_shape, - bool relaxed_rank) { - const auto metadata = source.require_metadata(name); - if (metadata.shape == expected_shape) { - return; - } - if (!relaxed_rank) { - throw std::runtime_error("tensor shape mismatch for " + std::string(name)); - } - int64_t expected_elems = 1; - for (const int64_t dim : expected_shape) { - expected_elems *= dim; - } - int64_t actual_elems = 1; - for (const int64_t dim : metadata.shape) { - actual_elems *= dim; - } - if (actual_elems != expected_elems) { - throw std::runtime_error("tensor element count mismatch for " + std::string(name)); - } -} - void validate_weight_anchors(const VoxCPM2Assets & assets) { const auto & config = assets.config; const auto & weights = *assets.model_weights; @@ -871,7 +182,7 @@ void validate_weight_anchors(const VoxCPM2Assets & assets) { {config.lm.num_key_value_heads * config.lm.kv_channels, config.lm.hidden_size}); assets::require_tensor_shape(weights, "base_lm.layers.0.mlp.gate_proj.weight", {config.lm.intermediate_size, config.lm.hidden_size}); assets::require_tensor_shape(weights, "residual_lm.norm.weight", {config.lm.hidden_size}); - require_vae_weight_v_shape(weights, "feat_encoder.special_token", {1, 1, 1, config.encoder.hidden_dim}, config.v1); + assets::require_tensor_shape(weights, "feat_encoder.special_token", {1, 1, 1, config.encoder.hidden_dim}); assets::require_tensor_shape(weights, "feat_encoder.in_proj.weight", {config.encoder.hidden_dim, config.feat_dim}); assets::require_tensor_shape(weights, "feat_encoder.encoder.norm.weight", {config.encoder.hidden_dim}); assets::require_tensor_shape(weights, "feat_decoder.estimator.in_proj.weight", {config.dit.hidden_dim, config.feat_dim}); @@ -888,58 +199,22 @@ void validate_weight_anchors(const VoxCPM2Assets & assets) { assets::require_tensor_shape(weights, "stop_head.weight", {2, config.lm.hidden_size}); const auto & vae = *assets.audiovae_weights; - int64_t encoder_in_channels = config.audio_vae.encoder_dim; - for (size_t i = 0; i < config.audio_vae.encoder_rates.size(); ++i) { - encoder_in_channels *= 2; - } - require_vae_weight_v_shape(vae, "encoder.fc_mu.weight_v", {config.audio_vae.latent_dim, encoder_in_channels, 3}, config.v1); + assets::require_tensor_shape(vae, "encoder.fc_mu.weight_v", {config.audio_vae.latent_dim, config.audio_vae.decoder_dim, 3}); assets::require_tensor_shape(vae, "encoder.fc_mu.bias", {config.audio_vae.latent_dim}); - require_vae_weight_v_shape(vae, "decoder.model.0.weight_v", {config.audio_vae.latent_dim, 1, 7}, config.v1); - require_vae_weight_v_shape(vae, "decoder.model.1.weight_v", {config.audio_vae.decoder_dim, config.audio_vae.latent_dim, 1}, config.v1); + assets::require_tensor_shape(vae, "decoder.model.0.weight_v", {config.audio_vae.latent_dim, 1, 7}); + assets::require_tensor_shape(vae, "decoder.model.1.weight_v", {config.audio_vae.decoder_dim, config.audio_vae.latent_dim, 1}); } } -std::shared_ptr load_voxcpm2_assets(const std::filesystem::path & model_path, bool is_v1) { +std::shared_ptr load_voxcpm2_assets(const std::filesystem::path & model_path) { auto out = std::make_shared(); out->resources = engine::model_spec::load_resource_bundle( model_path, - engine::model_spec::default_spec_path(is_v1 ? "voxcpm1" : "voxcpm2")); - - // For VoxCPM1, try to load config and tokenizer from GGUF metadata - if (is_v1) { - auto raw_model_weights = out->resources.open_tensor_source("weights"); - - // Check if GGUF has tokenizer metadata - bool has_tokenizer = VoxCPM1GgufTokenizer::has_tokenizer_metadata(*raw_model_weights); - bool has_config = has_voxcpm1_config_metadata(*raw_model_weights); - - if (has_tokenizer && has_config) { - // Load config from GGUF metadata - out->config = load_voxcpm1_config_from_gguf(*raw_model_weights); - out->config.v1 = true; - - // Create GGUF-native tokenizer - out->gguf_tokenizer = std::make_shared(raw_model_weights); - } else { - // Fall back to external files - out->config = parse_config(out->resources); - out->config.v1 = true; - } - } else { - out->config = parse_config(out->resources); - out->config.v1 = false; - } - - auto raw_model_weights = out->resources.open_tensor_source("weights"); - auto raw_audiovae_weights = out->resources.open_tensor_source("audiovae_weights"); - if (is_v1) { - out->model_weights = std::make_shared(raw_model_weights, out->config, true); - out->audiovae_weights = std::make_shared(raw_audiovae_weights, out->config, true); - } else { - out->model_weights = raw_model_weights; - out->audiovae_weights = raw_audiovae_weights; - } + engine::model_spec::default_spec_path("voxcpm2")); + out->config = parse_config(out->resources); + out->model_weights = out->resources.open_tensor_source("weights"); + out->audiovae_weights = out->resources.open_tensor_source("audiovae_weights"); validate_weight_anchors(*out); return out; } diff --git a/src/models/voxcpm2/audiovae.cpp b/src/models/voxcpm2/audiovae.cpp index f59fe280..2d890f5e 100644 --- a/src/models/voxcpm2/audiovae.cpp +++ b/src/models/voxcpm2/audiovae.cpp @@ -40,78 +40,6 @@ using Clock = std::chrono::steady_clock; constexpr int64_t kResidualKernel = 7; -enum class PaddingMode { Left, Right }; - -std::vector trim_audio_silence_vad(const std::vector& input, - int sample_rate, - float max_silence_ms = 100.0f, - float top_db = 30.0f) { - if (input.empty() || sample_rate <= 0) { - return input; - } - - constexpr int kFrameLength = 2048; - constexpr int kHopLength = 512; - const float ref = *std::max_element(input.begin(), input.end(), [](float a, float b) { - return std::fabs(a) < std::fabs(b); - }); - if (std::fabs(ref) <= 0.0f) { - return input; - } - - const float threshold = std::fabs(ref) * std::pow(10.0f, -top_db / 20.0f); - const size_t n = input.size(); - int first_voice_frame = -1; - int last_voice_frame = -1; - - for (size_t idx = 0, frame = 0; idx < n; idx += kHopLength, ++frame) { - const size_t frame_end = std::min(idx + static_cast(kFrameLength), n); - const size_t frame_size = frame_end - idx; - if (frame_size == 0) { - break; - } - double energy = 0.0; - for (size_t i = idx; i < frame_end; ++i) { - energy += static_cast(input[i]) * static_cast(input[i]); - } - const float rms = static_cast(std::sqrt(energy / static_cast(frame_size))); - if (rms >= threshold) { - if (first_voice_frame < 0) { - first_voice_frame = static_cast(frame); - } - last_voice_frame = static_cast(frame); - } - if (frame_end == n) { - break; - } - } - - if (first_voice_frame < 0 || last_voice_frame < 0) { - return input; - } - - const int max_silence_samples = std::max(0, static_cast(std::lround(max_silence_ms * sample_rate / 1000.0f))); - const int start = std::max(0, first_voice_frame * kHopLength - max_silence_samples); - const int end = std::min(static_cast(n), - (last_voice_frame + 1) * kHopLength + (kFrameLength - kHopLength) + max_silence_samples); - if (start >= end) { - return input; - } - return std::vector(input.begin() + start, input.begin() + end); -} - -void pad_audio_for_patch_alignment(std::vector& audio, size_t patch_len, PaddingMode mode) { - if (patch_len == 0 || audio.empty() || (audio.size() % patch_len) == 0) { - return; - } - const size_t padding = patch_len - (audio.size() % patch_len); - if (mode == PaddingMode::Left) { - audio.insert(audio.begin(), padding, 0.0f); - } else { - audio.insert(audio.end(), padding, 0.0f); - } -} - struct GgmlContextDeleter { void operator()(ggml_context *ctx) const noexcept { if (ctx != nullptr) { @@ -597,8 +525,9 @@ core::TensorValue encoder_block(core::ModuleBuildContext &ctx, hidden = residual_unit(ctx, hidden, weights.residual_units[2], 9); hidden = snake_exact(ctx, hidden, weights.snake, weights.input_channels); const int padding = static_cast((weights.stride + 1) / 2); + const int output_padding = weights.stride % 2; return causal_conv1d(ctx, hidden, weights.downsample, weights.stride, padding, - 1); + 1, output_padding); } core::TensorValue decoder_block(core::ModuleBuildContext &ctx, @@ -710,11 +639,9 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { void release_runtime_memory() { release_decoder_graph(); - release_encoder_graph_impl(); + release_encoder_graph(); } - void release_encoder_graph() { release_encoder_graph_impl(); } - private: struct EncodedFeatures { std::vector features; @@ -741,26 +668,13 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { mono = engine::audio::resample_mono_soxr_or_linear( mono, audio.sample_rate, vae.sample_rate, options); } - // VAD trim silence (match VoxCPM.cpp server_common.cpp:842/878) - mono = trim_audio_silence_vad(mono, vae.sample_rate); - if (const char *dump_path = std::getenv("VOXCPM_DUMP_REF_MONO")) { - FILE *f = std::fopen(dump_path, "wb"); - if (f != nullptr) { - std::fwrite(mono.data(), sizeof(float), mono.size(), f); - std::fclose(f); - } - } - // Patch-aligned padding (Left for prompt, Right for reference) const int64_t patch_samples = assets_->config.patch_size * encoder_stride_; - pad_audio_for_patch_alignment(mono, static_cast(patch_samples), - left_pad ? PaddingMode::Left : PaddingMode::Right); - // Final padding to encoder_sample_capacity - const int64_t sample_count = static_cast(mono.size()); - const int64_t padded_samples = - ((sample_count + patch_samples - 1) / patch_samples) * patch_samples; if (patch_samples <= 0) { throw std::runtime_error("VoxCPM2 AudioVAE patch sample size is invalid"); } + const int64_t sample_count = static_cast(mono.size()); + const int64_t padded_samples = + ((sample_count + patch_samples - 1) / patch_samples) * patch_samples; if (padded_samples > config_.encoder_sample_capacity) { throw std::runtime_error( "VoxCPM2 AudioVAE encoder sample capacity exceeded"); @@ -780,19 +694,6 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { if (status != GGML_STATUS_SUCCESS) { throw std::runtime_error("VoxCPM2 AudioVAE encoder graph compute failed"); } - if (const char *stage_path = std::getenv("VOXCPM_DUMP_ENC_STAGE")) { - const std::string dir(stage_path); - for (size_t i = 0; i < encoder_stages_.size(); ++i) { - ggml_tensor *stage = encoder_stages_[i]; - std::vector buf(static_cast(ggml_nelements(stage)), 0.0F); - ggml_backend_tensor_get(stage, buf.data(), 0, buf.size() * sizeof(float)); - FILE *f = std::fopen((dir + "/stage_" + std::to_string(i) + ".bin").c_str(), "wb"); - if (f != nullptr) { - std::fwrite(buf.data(), sizeof(float), buf.size(), f); - std::fclose(f); - } - } - } const int64_t latent_frames = padded_samples / encoder_stride_; const int64_t expected_capacity_frames = @@ -815,14 +716,6 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { full[static_cast(c * expected_capacity_frames + t)]; } } - if (const char *dump_path = std::getenv("VOXCPM_DUMP_REF_FEAT")) { - FILE *f = std::fopen(dump_path, "wb"); - if (f != nullptr) { - std::fwrite(encoded.features.data(), sizeof(float), - encoded.features.size(), f); - std::fclose(f); - } - } return encoded; } @@ -939,7 +832,7 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { decoder_latent_frame_capacity_ = latent_frame_capacity; } - void release_encoder_graph_impl() { + void release_encoder_graph() { if (encoder_graph_ != nullptr) { core::release_backend_graph_resources(execution_context_.backend(), encoder_graph_); @@ -979,31 +872,15 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { core::TensorShape::from_dims({1, 1, config_.encoder_sample_capacity})); encoder_input_ = hidden.tensor; ggml_set_input(encoder_input_); - encoder_stages_.clear(); - if (std::getenv("VOXCPM_DUMP_ENC_STAGE") != nullptr) { - encoder_stages_.push_back(encoder_input_); - } hidden = causal_conv1d(ctx, hidden, weights_.encoder_first, 1, 3, 1); - if (std::getenv("VOXCPM_DUMP_ENC_STAGE") != nullptr) { - encoder_stages_.push_back(hidden.tensor); - } for (const auto &block : weights_.encoder_blocks) { hidden = encoder_block(ctx, hidden, block); - if (std::getenv("VOXCPM_DUMP_ENC_STAGE") != nullptr) { - encoder_stages_.push_back(hidden.tensor); - } } hidden = causal_conv1d(ctx, hidden, weights_.encoder_fc_mu, 1, 1, 1); encoder_output_ = hidden.tensor; ggml_set_output(encoder_output_); - for (ggml_tensor *stage : encoder_stages_) { - ggml_set_output(stage); - } encoder_graph_ = ggml_new_graph_custom(encoder_ctx_.get(), 65536, false); ggml_build_forward_expand(encoder_graph_, encoder_output_); - for (ggml_tensor *stage : encoder_stages_) { - ggml_build_forward_expand(encoder_graph_, stage); - } encoder_gallocr_ = ggml_gallocr_new( ggml_backend_get_default_buffer_type(execution_context_.backend())); if (encoder_gallocr_ == nullptr || @@ -1025,7 +902,6 @@ class VoxCPM2AudioVAEDecoderRuntime::Impl { ggml_tensor *output_ = nullptr; ggml_tensor *encoder_input_ = nullptr; ggml_tensor *encoder_output_ = nullptr; - std::vector encoder_stages_; ggml_cgraph *graph_ = nullptr; ggml_cgraph *encoder_graph_ = nullptr; ggml_gallocr_t gallocr_ = nullptr; @@ -1061,8 +937,4 @@ void VoxCPM2AudioVAEDecoderRuntime::release_runtime_memory() { impl_->release_runtime_memory(); } -void VoxCPM2AudioVAEDecoderRuntime::release_encoder_graph() { - impl_->release_encoder_graph(); -} - } // namespace engine::models::voxcpm2 diff --git a/src/models/voxcpm2/generator.cpp b/src/models/voxcpm2/generator.cpp index 7d74924b..3186a142 100644 --- a/src/models/voxcpm2/generator.cpp +++ b/src/models/voxcpm2/generator.cpp @@ -16,7 +16,6 @@ #include "engine/models/voxcpm2/assets.h" #include "engine/models/voxcpm2/minicpm.h" #include "engine/models/voxcpm2/tokenizer_text.h" -#include "engine/models/voxcpm2/tokenizer_wrapper.h" #include #include @@ -117,18 +116,6 @@ std::vector concat_dit_mu(const std::vector &lm, return out; } -std::vector add_dit_mu(const std::vector &lm, - const std::vector &residual) { - if (lm.size() != residual.size()) { - throw std::runtime_error("VoxCPM1 dit mu inputs must have equal size"); - } - std::vector out(lm.size(), 0.0F); - for (size_t i = 0; i < lm.size(); ++i) { - out[i] = lm[i] + residual[i]; - } - return out; -} - void append_patch(std::vector &features, const std::vector &patch, int64_t expected_size) { if (static_cast(patch.size()) != expected_size) { @@ -206,7 +193,6 @@ class VoxCPM2StepProjectionRuntime final { VoxCPM2StepProjectionOutput run(const std::vector &lm_hidden, const std::vector &residual_hidden, const std::vector ¤t_embed); - void release_runtime_memory(); private: class Impl; @@ -222,7 +208,6 @@ class VoxCPM2LocalEncoderRuntime final { std::vector encode_patch(const std::vector &patch_features) const; - void release_runtime_memory(); private: class Impl; @@ -241,7 +226,6 @@ class VoxCPM2DiTEstimatorRuntime final { const std::vector &cond, const std::vector &time_embedding, const std::vector &delta_time_embedding); - void release_runtime_memory(); private: class Impl; @@ -262,7 +246,6 @@ class VoxCPM2CFMRuntime final { uint64_t noise_start_index = 0, const std::string &noise_file = {}, float temperature = 1.0F); - void release_runtime_memory(); private: class Impl; @@ -281,9 +264,15 @@ class VoxCPM2StepProjectionRuntime::Impl { build(graph_context_bytes); } - ~Impl() { release_graph(); } - - void release_runtime_memory() { release_graph(); } + ~Impl() { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + } + } VoxCPM2StepProjectionOutput run(const std::vector &lm_hidden, const std::vector &residual_hidden, @@ -354,33 +343,6 @@ class VoxCPM2StepProjectionRuntime::Impl { } private: - void release_graph() { - if (graph_ != nullptr) { - engine::core::release_backend_graph_resources(weights_->backend(), graph_); - } - if (buffer_ != nullptr) { - ggml_backend_buffer_free(buffer_); - buffer_ = nullptr; - } - if (gallocr_ != nullptr) { - ggml_gallocr_free(gallocr_); - gallocr_ = nullptr; - } - graph_ = nullptr; - lm_hidden_ = nullptr; - residual_hidden_ = nullptr; - current_embed_ = nullptr; - fsq_hidden_output_ = nullptr; - current_residual_input_output_ = nullptr; - residual_input_output_ = nullptr; - current_lm_dit_output_ = nullptr; - fsq_lm_dit_output_ = nullptr; - residual_dit_output_ = nullptr; - current_stop_logits_output_ = nullptr; - fsq_stop_logits_output_ = nullptr; - ctx_.reset(); - } - void build(size_t graph_context_bytes) { const auto &config = weights_->assets().config; if (graph_context_bytes == 0) { @@ -440,41 +402,23 @@ class VoxCPM2StepProjectionRuntime::Impl { .build(ctx, fsq, proj.fsq_out_proj); fsq_hidden_output_ = fsq.tensor; - // Check if fusion_concat_proj weight exists and was loaded (not synthesized) - // This matches VoxCPM.cpp behavior which checks weight existence - // For V1 models, synthesized weights (Xavier init) should not count as present - const bool has_fusion_proj = - proj.fusion_concat_proj.weight.tensor != nullptr && - config.architecture == "voxcpm2"; - - if (has_fusion_proj) { - // Concat + Linear (used by V2 and some V1 models trained with fusion) - auto current_residual_concat = - engine::modules::ConcatModule({1}).build(ctx, lm_hidden, current_embed); - auto current_residual_input = - engine::modules::LinearModule( - binding::linear_config(config.lm.hidden_size * 2, - config.lm.hidden_size, true)) - .build(ctx, current_residual_concat, proj.fusion_concat_proj); - current_residual_input_output_ = current_residual_input.tensor; - - auto residual_concat = - engine::modules::ConcatModule({1}).build(ctx, fsq, current_embed); - auto residual_input = - engine::modules::LinearModule( - binding::linear_config(config.lm.hidden_size * 2, - config.lm.hidden_size, true)) - .build(ctx, residual_concat, proj.fusion_concat_proj); - residual_input_output_ = residual_input.tensor; - } else { - // Simple ADD (true V1 without fusion_concat_proj) - current_residual_input_output_ = - engine::modules::AddModule() - .build(ctx, lm_hidden, current_embed) - .tensor; - residual_input_output_ = - engine::modules::AddModule().build(ctx, fsq, current_embed).tensor; - } + auto current_residual_concat = + engine::modules::ConcatModule({1}).build(ctx, lm_hidden, current_embed); + auto current_residual_input = + engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size * 2, + config.lm.hidden_size, true)) + .build(ctx, current_residual_concat, proj.fusion_concat_proj); + current_residual_input_output_ = current_residual_input.tensor; + + auto residual_concat = + engine::modules::ConcatModule({1}).build(ctx, fsq, current_embed); + auto residual_input = + engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size * 2, + config.lm.hidden_size, true)) + .build(ctx, residual_concat, proj.fusion_concat_proj); + residual_input_output_ = residual_input.tensor; auto current_lm_dit = engine::modules::LinearModule( @@ -604,10 +548,6 @@ VoxCPM2StepProjectionRuntime::VoxCPM2StepProjectionRuntime( VoxCPM2StepProjectionRuntime::~VoxCPM2StepProjectionRuntime() = default; -void VoxCPM2StepProjectionRuntime::release_runtime_memory() { - impl_->release_runtime_memory(); -} - VoxCPM2StepProjectionOutput VoxCPM2StepProjectionRuntime::run(const std::vector &lm_hidden, const std::vector &residual_hidden, @@ -627,9 +567,15 @@ class VoxCPM2LocalEncoderRuntime::Impl { build(graph_context_bytes); } - ~Impl() { release_graph(); } - - void release_runtime_memory() { release_graph(); } + ~Impl() { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + } + } std::vector encode_patch(const std::vector &patch_features) const { @@ -655,25 +601,6 @@ class VoxCPM2LocalEncoderRuntime::Impl { } private: - void release_graph() { - if (graph_ != nullptr) { - engine::core::release_backend_graph_resources(weights_->backend(), graph_); - } - if (buffer_ != nullptr) { - ggml_backend_buffer_free(buffer_); - buffer_ = nullptr; - } - if (gallocr_ != nullptr) { - ggml_gallocr_free(gallocr_); - gallocr_ = nullptr; - } - graph_ = nullptr; - input_ = nullptr; - positions_ = nullptr; - output_ = nullptr; - ctx_.reset(); - } - void build(size_t graph_context_bytes) { const auto &root_config = weights_->assets().config; if (graph_context_bytes == 0) { @@ -778,10 +705,6 @@ VoxCPM2LocalEncoderRuntime::VoxCPM2LocalEncoderRuntime( VoxCPM2LocalEncoderRuntime::~VoxCPM2LocalEncoderRuntime() = default; -void VoxCPM2LocalEncoderRuntime::release_runtime_memory() { - impl_->release_runtime_memory(); -} - std::vector VoxCPM2LocalEncoderRuntime::encode_patch( const std::vector &patch_features) const { return impl_->encode_patch(patch_features); @@ -799,9 +722,15 @@ class VoxCPM2DiTEstimatorRuntime::Impl { build(graph_context_bytes); } - ~Impl() { release_graph(); } - - void release_runtime_memory() { release_graph(); } + ~Impl() { + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + } + } std::vector run(const std::vector &x, const std::vector &mu, @@ -809,12 +738,6 @@ class VoxCPM2DiTEstimatorRuntime::Impl { const std::vector &time_embedding, const std::vector &delta_time_embedding) { const auto &config = weights_->assets().config; - // Check if fusion_concat_proj weight exists and was loaded (not synthesized) - // This matches VoxCPM.cpp behavior which checks weight existence - // For V1 models, synthesized weights (Xavier init) should not count as present - const bool has_fusion_proj = - weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && - config.architecture == "voxcpm2"; const int64_t patch_elems = 2 * config.feat_dim * config.patch_size; if (static_cast(x.size()) != patch_elems) { throw std::runtime_error("VoxCPM2 DiT estimator x size mismatch"); @@ -822,9 +745,7 @@ class VoxCPM2DiTEstimatorRuntime::Impl { if (static_cast(cond.size()) != patch_elems) { throw std::runtime_error("VoxCPM2 DiT estimator cond size mismatch"); } - const int64_t expected_mu = - has_fusion_proj ? 2 * config.dit.hidden_dim * 2 : 2 * config.dit.hidden_dim; - if (static_cast(mu.size()) != expected_mu) { + if (static_cast(mu.size()) != 2 * config.dit.hidden_dim * 2) { throw std::runtime_error("VoxCPM2 DiT estimator mu size mismatch"); } if (static_cast(time_embedding.size()) != @@ -854,152 +775,10 @@ class VoxCPM2DiTEstimatorRuntime::Impl { std::vector output(static_cast(patch_elems), 0.0F); ggml_backend_tensor_get(output_, output.data(), 0, output.size() * sizeof(float)); - if (std::getenv("VOXCPM_DUMP_NORM0") != nullptr) { - ggml_tensor *tn = ggml_get_tensor(ctx_.get(), "dump_norm0"); - if (tn != nullptr) { - const size_t nn = static_cast(ggml_nelements(tn)); - std::vector buf(nn, 0.0F); - ggml_backend_tensor_get(tn, buf.data(), 0, buf.size() * sizeof(float)); - FILE *f = std::fopen("/tmp/opencode/ours_norm0.bin", "wb"); - if (f != nullptr) { - std::fwrite(buf.data(), sizeof(float), std::min(nn, 5120), f); - std::fclose(f); - } - } - } - if (std::getenv("VOXCPM_DUMP_DECODER_LAYERS") != nullptr) { - for (int li = 0; li < 8; ++li) { - char name[32]; - snprintf(name, sizeof(name), "dump_layer_%d", li); - ggml_tensor *t = ggml_get_tensor(ctx_.get(), name); - if (t == nullptr) { - continue; - } - const size_t n = static_cast(ggml_nelements(t)); - std::vector buf(n, 0.0F); - ggml_backend_tensor_get(t, buf.data(), 0, buf.size() * sizeof(float)); - if (li == 0) { - FILE *f = std::fopen("/tmp/opencode/ours_branch0.bin", "wb"); - if (f != nullptr) { - std::fwrite(buf.data(), sizeof(float), std::min(n, 5120), f); - std::fclose(f); - } - } else if (li == 1) { - FILE *f = std::fopen("/tmp/opencode/ours_branch1.bin", "wb"); - if (f != nullptr) { - std::fwrite(buf.data(), sizeof(float), std::min(n, 5120), f); - std::fclose(f); - } - } - double s = 0.0, l2 = 0.0; - for (float v : buf) { - s += v; - l2 += static_cast(v) * v; - } - // batch-local stats for the branch-0 (first ne1*ne0 elements) - double s0 = 0.0, l20 = 0.0; - const size_t branch_elems = static_cast(t->ne[0]) * static_cast(t->ne[1]); - for (size_t i = 0; i < std::min(branch_elems, buf.size()); ++i) { - s0 += buf[i]; - l20 += static_cast(buf[i]) * buf[i]; - } - fprintf(stderr, - "[DEC_LAYER] input#%d ne0=%lld ne1=%lld ne2=%lld ne3=%lld " - "sum=%.6g l2=%.6g branch0_sum=%.6g branch0_l2=%.6g " - "first4=%.6g %.6g %.6g %.6g\n", - li, static_cast(t->ne[0]), - static_cast(t->ne[1]), - static_cast(t->ne[2]), - static_cast(t->ne[3]), s, std::sqrt(l2), s0, - std::sqrt(l20), - buf.empty() ? 0.0 : static_cast(buf[0]), - buf.size() < 2 ? 0.0 : static_cast(buf[1]), - buf.size() < 3 ? 0.0 : static_cast(buf[2]), - buf.size() < 4 ? 0.0 : static_cast(buf[3])); - } - } - if (std::getenv("VOXCPM_DUMP_LOCDIT_WEIGHTS") != nullptr) { - const auto &dw = weights_->weights().dit; - auto dump_w = [](const char *tag, ggml_tensor *t) { - if (t == nullptr) { - fprintf(stderr, "[LOCDIT_W] %s \n", tag); - return; - } - const size_t nbytes = static_cast(ggml_nbytes(t)); - const size_t nelems = static_cast(ggml_nelements(t)); - std::vector raw(nbytes, 0); - ggml_backend_tensor_get(t, raw.data(), 0, nbytes); - fprintf(stderr, "[LOCDIT_W] %s ne0=%lld ne1=%lld type=%d nbytes=%zu " - "v[0..7]=", - tag, static_cast(t->ne[0]), - static_cast(t->ne[1]), static_cast(t->type), - nbytes); - double vals[8]; - for (size_t i = 0; i < 8; ++i) { - if (t->type == GGML_TYPE_Q8_0) { - const size_t block = i / 32; - const size_t in_block = i % 32; - const float scale = - ggml_fp16_to_fp32( - *reinterpret_cast( - raw.data() + block * 34)); - vals[i] = static_cast( - scale * - static_cast( - *reinterpret_cast( - raw.data() + block * 34 + 2 + in_block))); - } else if (t->type == GGML_TYPE_F32) { - vals[i] = static_cast( - *reinterpret_cast(raw.data() + i * 4)); - } else if (t->type == GGML_TYPE_F16) { - vals[i] = static_cast(ggml_fp16_to_fp32( - *reinterpret_cast(raw.data() + i * 2))); - } else { - vals[i] = 0.0; - } - } - (void)nelems; - for (size_t i = 0; i < 8; ++i) { - fprintf(stderr, "%.6g ", vals[i]); - } - fprintf(stderr, "\n"); - }; - dump_w("in_proj", dw.in_proj.weight.tensor); - dump_w("cond_proj", dw.cond_proj.weight.tensor); - dump_w("out_proj", dw.out_proj.weight.tensor); - dump_w("time_mlp1", dw.time_mlp_1.weight.tensor); - dump_w("decoder.l0.q", dw.decoder.layers[0].q_proj.weight.tensor); - dump_w("decoder.l0.k", dw.decoder.layers[0].k_proj.weight.tensor); - dump_w("decoder.l0.o", dw.decoder.layers[0].o_proj.weight.tensor); - dump_w("decoder.norm", dw.decoder.norm.weight->tensor); - } return output; } private: - void release_graph() { - if (graph_ != nullptr) { - engine::core::release_backend_graph_resources(weights_->backend(), graph_); - } - if (buffer_ != nullptr) { - ggml_backend_buffer_free(buffer_); - buffer_ = nullptr; - } - if (gallocr_ != nullptr) { - ggml_gallocr_free(gallocr_); - gallocr_ = nullptr; - } - graph_ = nullptr; - x_ = nullptr; - mu_ = nullptr; - cond_ = nullptr; - time_embedding_ = nullptr; - delta_time_embedding_ = nullptr; - positions_ = nullptr; - output_ = nullptr; - ctx_.reset(); - } - void build(size_t graph_context_bytes) { const auto &root_config = weights_->assets().config; const auto &config = root_config.dit; @@ -1031,19 +810,9 @@ class VoxCPM2DiTEstimatorRuntime::Impl { if (mem_saver_) { ggml_set_input(cond_); } - // Check if fusion_concat_proj weight exists and was loaded (not synthesized) - // This matches VoxCPM.cpp behavior which checks weight existence - // For V1 models, synthesized weights (Xavier init) should not count as present - const bool has_fusion_proj = - weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && - root_config.architecture == "voxcpm2"; mu_ = engine::core::make_tensor( ctx, GGML_TYPE_F32, - has_fusion_proj - ? engine::core::TensorShape::from_dims( - {2, 2, config.hidden_dim}) - : engine::core::TensorShape::from_dims( - {2, config.hidden_dim})) + engine::core::TensorShape::from_dims({2, 2, config.hidden_dim})) .tensor; if (mem_saver_) { ggml_set_input(mu_); @@ -1113,38 +882,19 @@ class VoxCPM2DiTEstimatorRuntime::Impl { binding::linear_config(config.hidden_dim, config.hidden_dim, true)) .build(ctx, dt, weights.delta_time_mlp_2); time = engine::modules::AddModule{}.build(ctx, time, dt); - - const int64_t prefix_token_count = - has_fusion_proj ? 2 + 1 : 1; - auto hidden = time; - if (!has_fusion_proj) { - // True V1 (no fusion projection): the DiT conditioning mu is a single - // hidden vector that is ADDED into the timestep token. Batch 0 carries - // mu (conditioned branch); batch 1 carries zeros (unconditioned branch), - // mirroring LocDiTModel::forward_cfg_pair_projected with mu_tokens == 1. - auto mu = engine::core::wrap_tensor( - mu_, engine::core::TensorShape::from_dims({2, config.hidden_dim}), - GGML_TYPE_F32); - hidden = engine::modules::AddModule{}.build(ctx, hidden, mu); - } - hidden = engine::core::reshape_tensor( - ctx, hidden, + time = engine::core::reshape_tensor( + ctx, time, engine::core::TensorShape::from_dims({2, 1, config.hidden_dim})); - if (has_fusion_proj) { - // V2 (or V1 with fusion projection): mu is two hidden vectors concatenated as - // separate prefix tokens before the timestep token, matching - // LocDiTModel with mu_tokens == 2. - auto mu = engine::core::wrap_tensor( - mu_, engine::core::TensorShape::from_dims({2, 2, config.hidden_dim}), - GGML_TYPE_F32); - hidden = engine::modules::ConcatModule({1}).build(ctx, mu, hidden); - } + + auto mu = engine::core::wrap_tensor( + mu_, engine::core::TensorShape::from_dims({2, 2, config.hidden_dim}), + GGML_TYPE_F32); + auto hidden = engine::modules::ConcatModule({1}).build(ctx, mu, time); hidden = engine::modules::ConcatModule({1}).build(ctx, hidden, cond); hidden = engine::modules::ConcatModule({1}).build(ctx, hidden, x); - positions_ = ggml_new_tensor_1d( - ctx_.get(), GGML_TYPE_I32, - prefix_token_count + root_config.patch_size * 2); + positions_ = ggml_new_tensor_1d(ctx_.get(), GGML_TYPE_I32, + 2 + 1 + root_config.patch_size * 2); if (mem_saver_) { ggml_set_input(positions_); ggml_set_output(positions_); @@ -1152,14 +902,12 @@ class VoxCPM2DiTEstimatorRuntime::Impl { auto positions = engine::core::wrap_tensor(positions_, engine::core::TensorShape::from_dims( - {prefix_token_count + - root_config.patch_size * 2}), + {2 + 1 + root_config.patch_size * 2}), GGML_TYPE_I32); hidden = minicpm_transformer(ctx, hidden, positions, weights.decoder, false); hidden = engine::modules::SliceModule( - {1, prefix_token_count + root_config.patch_size, - root_config.patch_size}) + {1, 2 + 1 + root_config.patch_size, root_config.patch_size}) .build(ctx, hidden); hidden = engine::modules::LinearModule( binding::linear_config(config.hidden_dim, root_config.feat_dim, @@ -1195,8 +943,7 @@ class VoxCPM2DiTEstimatorRuntime::Impl { "failed to allocate VoxCPM2 DiT estimator graph"); } std::vector positions_data( - static_cast(prefix_token_count + root_config.patch_size * 2), - 0); + static_cast(2 + 1 + root_config.patch_size * 2), 0); for (int64_t i = 0; i < static_cast(positions_data.size()); ++i) { positions_data[static_cast(i)] = static_cast(i); } @@ -1227,10 +974,6 @@ VoxCPM2DiTEstimatorRuntime::VoxCPM2DiTEstimatorRuntime( VoxCPM2DiTEstimatorRuntime::~VoxCPM2DiTEstimatorRuntime() = default; -void VoxCPM2DiTEstimatorRuntime::release_runtime_memory() { - impl_->release_runtime_memory(); -} - std::vector VoxCPM2DiTEstimatorRuntime::run( const std::vector &x, const std::vector &mu, const std::vector &cond, const std::vector &time_embedding, @@ -1268,8 +1011,6 @@ class VoxCPM2CFMRuntime::Impl { } } - void release_runtime_memory() { estimator_.release_runtime_memory(); } - std::vector generate_patch(const std::vector &mu, const std::vector &cond_patch, int64_t timesteps, float cfg_value, @@ -1277,12 +1018,6 @@ class VoxCPM2CFMRuntime::Impl { const std::string &noise_file, float temperature) { const auto &config = weights_->assets().config; - // Check if fusion_concat_proj weight exists and was loaded (not synthesized) - // This matches VoxCPM.cpp behavior which checks weight existence - // For V1 models, synthesized weights (Xavier init) should not count as present - const bool has_fusion_proj = - weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && - config.architecture == "voxcpm2"; if (timesteps <= 0) { throw std::runtime_error("VoxCPM2 CFM requires positive timesteps"); } @@ -1290,8 +1025,7 @@ class VoxCPM2CFMRuntime::Impl { throw std::runtime_error("VoxCPM2 CFM received non-finite scalar input"); } const int64_t patch_elems = config.feat_dim * config.patch_size; - const int64_t mu_dim = config.dit.hidden_dim * (has_fusion_proj ? 2 : 1); - if (static_cast(mu.size()) != mu_dim) { + if (static_cast(mu.size()) != config.dit.hidden_dim * 2) { throw std::runtime_error("VoxCPM2 CFM mu size mismatch"); } if (static_cast(cond_patch.size()) != patch_elems) { @@ -1322,18 +1056,12 @@ class VoxCPM2CFMRuntime::Impl { for (float &value : x) { value *= temperature; } - x = patch_major_to_channel_major(x); const std::vector cond = patch_major_to_channel_major(cond_patch); std::vector x_in(static_cast(2 * patch_elems), 0.0F); std::vector cond_in(static_cast(2 * patch_elems), 0.0F); - const int64_t mu_elements = - has_fusion_proj ? 4 * config.dit.hidden_dim : 2 * config.dit.hidden_dim; - std::vector mu_in(static_cast(mu_elements), 0.0F); + std::vector mu_in(static_cast(4 * config.dit.hidden_dim), + 0.0F); std::copy(mu.begin(), mu.end(), mu_in.begin()); - if (std::getenv("VOXCPM_TEST_BATCH_MU") != nullptr) { - std::copy(mu.begin(), mu.end(), - mu_in.begin() + static_cast(mu.size())); - } std::copy(cond.begin(), cond.end(), cond_in.begin()); std::copy(cond.begin(), cond.end(), cond_in.begin() + static_cast(patch_elems)); @@ -1378,29 +1106,6 @@ class VoxCPM2CFMRuntime::Impl { const auto estimator = estimator_.run(x_in, mu_in, cond_in, time_embedding, delta_embedding); const float scale = optimized_cfg_scale(estimator, patch_elems); - if (std::getenv("VOXCPM_DUMP_DPHI") != nullptr && - step == 2) { - double s0 = 0.0, s1 = 0.0, n0 = 0.0, n1 = 0.0; - std::vector combined(static_cast(patch_elems)); - double c2 = 0.0; - for (int64_t i = 0; i < patch_elems; ++i) { - const float p = estimator[static_cast(i)]; - const float m = estimator[static_cast(patch_elems + i)]; - const double d = static_cast(m) * scale + - cfg_value * (static_cast(p) - - static_cast(m) * scale); - combined[static_cast(i)] = d; - s0 += p; s1 += m; n0 += p * p; n1 += m * m; - c2 += d * d; - } - fprintf(stderr, - "[DUMP_DPHI] t=%.6f dt=%.6f pos_l2=%.6g neg_l2=%.6g " - "combined_l2=%.6g scale=%.6g combined[0..3]=%.6g %.6g %.6g %.6g\n", - static_cast(t), static_cast(dt), - std::sqrt(n0), std::sqrt(n1), std::sqrt(c2), - static_cast(scale), combined[0], combined[1], - combined[2], combined[3]); - } for (int64_t i = 0; i < patch_elems; ++i) { const size_t index = static_cast(i); const float positive = estimator[index]; @@ -1477,10 +1182,6 @@ VoxCPM2CFMRuntime::VoxCPM2CFMRuntime( VoxCPM2CFMRuntime::~VoxCPM2CFMRuntime() = default; -void VoxCPM2CFMRuntime::release_runtime_memory() { - impl_->release_runtime_memory(); -} - std::vector VoxCPM2CFMRuntime::generate_patch( const std::vector &mu, const std::vector &cond_patch, int64_t timesteps, float cfg_value, uint64_t seed, @@ -1499,10 +1200,7 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { weights_(std::make_shared( assets_, execution_context, config.weight_context_bytes, config.weight_storage_type)), - tokenizer_(assets_->gguf_tokenizer - ? VoxCPM2TokenizerWrapper(assets_->gguf_tokenizer) - : VoxCPM2TokenizerWrapper( - std::make_shared(assets_))), + tokenizer_(assets_), text_embedding_(weights_, config.text_embedding_graph_context_bytes, config.mem_saver), prefill_(weights_, config.lm_step_graph_context_bytes, @@ -1560,9 +1258,8 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { &chunk_callback) { validate_generation_options(options); if (options.retry_badcase) { - fprintf(stderr, - "[VoxCPM2] warning: retry_badcase ignored in streaming " - "generation\n"); + throw std::runtime_error( + "VoxCPM2 streaming generation requires retry_badcase=false"); } const auto prefill = build_prefill_sequence(text, prompt); const int64_t max_tokens = @@ -1577,23 +1274,8 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { } void release_runtime_memory() { - // Release every staged graph so a session can idle at weight-only - // VRAM. Each runtime lazily rebuilds its graph on the next use. - text_embedding_.release_runtime_memory(); - prefill_.release_runtime_memory(); base_lm_.release_runtime_memory(); residual_lm_.release_runtime_memory(); - projection_.release_runtime_memory(); - cfm_.release_runtime_memory(); - local_encoder_.release_runtime_memory(); - } - - void release_text_length_memory() { - // Only the prompt-prefill graph is sized by the request text/prompt - // length; the other generator graphs have fixed-size workspaces. Drop it - // after every request so a long-lived session does not retain buffers - // that scale with text length; the next request rebuilds it fresh. - prefill_.release_runtime_memory(); } private: @@ -1849,9 +1531,7 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { } input_embedding = current_embed; } - if (row.audio_mask) { - prefix_cond = row.feature; - } + prefix_cond = row.feature; prefill_input.input_embeddings.insert(prefill_input.input_embeddings.end(), input_embedding.begin(), input_embedding.end()); @@ -1862,33 +1542,10 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { prefill_input.audio_mask.push_back(row.audio_mask ? 1.0F : 0.0F); } const auto prefill_output = prefill_.run(prefill_input); - if (std::getenv("VOXCPM_DUMP_PREFILL") != nullptr) { - auto dump_vec = [](const char *tag, const std::vector &v) { - double sum = 0.0; - double sum2 = 0.0; - for (float x : v) { - sum += x; - sum2 += static_cast(x) * x; - } - fprintf(stderr, "[DUMP_PREFILL] %s n=%zu sum=%.6g l2=%.6g", tag, - v.size(), sum, std::sqrt(sum2)); - for (size_t i = 0; i < std::min(8, v.size()); ++i) { - fprintf(stderr, " v[%zu]=%.6g", i, static_cast(v[i])); - } - fprintf(stderr, "\n"); - }; - dump_vec("lm_hidden", prefill_output.lm_hidden); - dump_vec("residual_hidden", prefill_output.residual_hidden); - } base_lm_.import_state(prefill_output.base_state); residual_lm_.import_state(prefill_output.residual_state); std::vector lm_hidden = prefill_output.lm_hidden; std::vector residual_hidden = prefill_output.residual_hidden; - // The prefill graph holds the largest sequence-shaped workspace; its - // outputs have been copied to host hiddens and its KV state imported - // into the step runtimes, so nothing below references it. Drop it now - // so the token loop runs against the much smaller step graphs. - prefill_.release_runtime_memory(); VoxCPM2Result result; std::vector context_rows; @@ -1911,45 +1568,11 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { for (int64_t index = 0; index < max_tokens; ++index) { const auto projected = projection_.run(lm_hidden, residual_hidden, zero_hidden); - if (index == 0 && std::getenv("VOXCPM_DUMP_PREFILL") != nullptr) { - auto dump_vec = [](const char *tag, const std::vector &v) { - double sum = 0.0; - double sum2 = 0.0; - for (float x : v) { - sum += x; - sum2 += static_cast(x) * x; - } - fprintf(stderr, "[DUMP_PREFILL] %s n=%zu sum=%.6g l2=%.6g", tag, - v.size(), sum, std::sqrt(sum2)); - for (size_t i = 0; i < std::min(8, v.size()); ++i) { - fprintf(stderr, " v[%zu]=%.6g", i, static_cast(v[i])); - } - fprintf(stderr, "\n"); - }; - dump_vec("lm_to_dit", projected.current_lm_dit_hidden); - dump_vec("res_to_dit", projected.residual_dit_hidden); - } - // Check if fusion_concat_proj weight exists and was loaded (not synthesized) - // This matches VoxCPM.cpp behavior which checks weight existence - // For V1 models, synthesized weights (Xavier init) should not count as present - const bool has_fusion_proj = - weights_->weights().projections.fusion_concat_proj.weight.tensor != nullptr && - config.architecture == "voxcpm2"; - const auto mu = has_fusion_proj - ? concat_dit_mu(projected.current_lm_dit_hidden, - projected.residual_dit_hidden) - : add_dit_mu(projected.current_lm_dit_hidden, - projected.residual_dit_hidden); + const auto mu = concat_dit_mu(projected.current_lm_dit_hidden, + projected.residual_dit_hidden); const auto patch = cfm_.generate_patch( mu, prefix_cond, options.num_inference_steps, options.guidance_scale, options.seed, patch_noise_start, options.cfm_noise_file); - if (const char *patch_dump_path = std::getenv("VOXCPM_DUMP_PATCH")) { - FILE *patch_file = std::fopen(patch_dump_path, "ab"); - if (patch_file != nullptr) { - std::fwrite(patch.data(), sizeof(float), patch.size(), patch_file); - std::fclose(patch_file); - } - } patch_noise_start += static_cast(patch_elems); append_patch(result.generated_features, patch, patch_elems); ++result.generated_patches; @@ -1973,13 +1596,6 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { stop_class(projected.current_stop_logits) == 1) { break; } - if (std::getenv("VOXCPM1_LOG_STOP") != nullptr) { - const auto &sl = projected.current_stop_logits; - fprintf(stderr, "[stop_logits] pos=%lld pre stop0=%.5f stop1=%.5f\n", - static_cast(index), - sl.empty() ? 0.0 : static_cast(sl[0]), - sl.size() < 2 ? 0.0 : static_cast(sl[1])); - } const auto curr_embed = local_encoder_.encode_patch(patch); const auto next_lm = base_lm_.run_step(curr_embed).hidden; @@ -1994,7 +1610,7 @@ class VoxCPM2FeatureGeneratorRuntime::Impl { std::shared_ptr assets_; std::shared_ptr weights_; - VoxCPM2TokenizerWrapper tokenizer_; + VoxCPM2TextTokenizer tokenizer_; VoxCPM2TextEmbeddingRuntime text_embedding_; VoxCPM2PromptPrefillRuntime prefill_; VoxCPM2MiniCPMStepRuntime base_lm_; @@ -2041,8 +1657,4 @@ void VoxCPM2FeatureGeneratorRuntime::release_runtime_memory() { impl_->release_runtime_memory(); } -void VoxCPM2FeatureGeneratorRuntime::release_text_length_memory() { - impl_->release_text_length_memory(); -} - } // namespace engine::models::voxcpm2 diff --git a/src/models/voxcpm2/loader.cpp b/src/models/voxcpm2/loader.cpp index 3754b0a6..3821dba7 100644 --- a/src/models/voxcpm2/loader.cpp +++ b/src/models/voxcpm2/loader.cpp @@ -1,7 +1,6 @@ #include "engine/models/voxcpm2/loader.h" #include "engine/framework/model_spec/package.h" -#include "engine/models/voxcpm2/assets.h" #include "engine/models/voxcpm2/session.h" #include @@ -76,7 +75,7 @@ class VoxCPM2Loader final : public runtime::IVoiceModelLoader { runtime::ModelInspection inspect(const runtime::ModelLoadRequest &request) const override { const auto assets = - load_voxcpm2_assets(request.model_path, false); + load_voxcpm2_assets(request.model_path); runtime::ModelInspection inspection; inspection.model_root = assets->resources.model_root(); inspection.metadata = metadata(*assets); @@ -100,104 +99,6 @@ class VoxCPM2Loader final : public runtime::IVoiceModelLoader { } }; -// VoxCPM1 Loader -runtime::CapabilitySet capabilities_v1(const VoxCPM2Assets &) { - runtime::CapabilitySet out; - out.supported_tasks = { - {runtime::VoiceTaskKind::Tts, - {runtime::RunMode::Offline, runtime::RunMode::Streaming}}, - }; - out.languages = {"Auto"}; - out.supports_speaker_reference = true; - return out; -} - -runtime::ModelMetadata metadata_v1(const VoxCPM2Assets &assets) { - runtime::ModelMetadata out; - out.family = "voxcpm1"; - out.variant = assets.config.architecture; - out.description = "VoxCPM1 loaded from GGUF assets."; - return out; -} - -runtime::ModelCliInterface cli_v1(const VoxCPM2Assets &) { - runtime::ModelCliInterface out; - out.request_options = { - {"text_chunk_mode", "default|tag_aware|japanese|endline", - "Text chunking mode; default tag_aware."}, - }; - out.session_options = { - {"voxcpm1.mem_saver", "true|false", - "Use tighter graph workspaces and release request runtime graphs; default false."}, - {"voxcpm1.prompt_cache_slots", "n", - "Prompt and prompt-audio embedding cache slots; default 1."}, - }; - return out; -} - -std::unique_ptr -load_voxcpm1_model(const std::filesystem::path &model_path) { - auto assets = load_voxcpm2_assets(model_path, true); - return std::make_unique( - metadata_v1(*assets), capabilities_v1(*assets), std::move(assets)); -} - -class VoxCPM1Loader final : public runtime::IVoiceModelLoader { -public: - std::string family() const override { return "voxcpm1"; } - - runtime::CapabilitySet advertised_capabilities() const override { - runtime::CapabilitySet out; - out.supported_tasks = { - {runtime::VoiceTaskKind::Tts, - {runtime::RunMode::Offline, runtime::RunMode::Streaming}}, - }; - out.supports_speaker_reference = true; - return out; - } - - std::string advertised_instructions_policy() const override { - return "text_prefix"; - } - - bool can_load(const runtime::ModelLoadRequest &request) const override { - try { - (void)engine::model_spec::load_resource_bundle( - request.model_path, - engine::model_spec::default_spec_path(family())); - return !request.family_hint.has_value() || *request.family_hint == family(); - } catch (...) { - return false; - } - } - - runtime::ModelInspection - inspect(const runtime::ModelLoadRequest &request) const override { - const auto assets = - load_voxcpm2_assets(request.model_path, true); - runtime::ModelInspection inspection; - inspection.model_root = assets->resources.model_root(); - inspection.metadata = metadata_v1(*assets); - inspection.capabilities = capabilities_v1(*assets); - inspection.cli = cli_v1(*assets); - const auto spec_path = engine::model_spec::default_spec_path(family()); - inspection.discovered_configs = runtime::discover_named_assets_from_package_spec( - request.model_path, - spec_path, - engine::model_spec::ResourceKind::Files); - inspection.discovered_weights = runtime::discover_named_assets_from_package_spec( - request.model_path, - spec_path, - engine::model_spec::ResourceKind::Tensors); - return inspection; - } - - std::unique_ptr - load(const runtime::ModelLoadRequest &request) const override { - return load_voxcpm1_model(request.model_path); - } -}; - } // namespace VoxCPM2LoadedModel::VoxCPM2LoadedModel( @@ -219,14 +120,13 @@ std::unique_ptr VoxCPM2LoadedModel::create_task_session( const runtime::TaskSpec &task, const runtime::SessionOptions &options) const { - const std::string family_label = metadata_.family == "voxcpm1" ? "VoxCPM1" : "VoxCPM2"; if (task.mode != runtime::RunMode::Offline && task.mode != runtime::RunMode::Streaming) { - throw std::runtime_error(family_label + - " only supports offline and streaming sessions"); + throw std::runtime_error( + "VoxCPM2 only supports offline and streaming sessions"); } if (task.task != runtime::VoiceTaskKind::Tts) { - throw std::runtime_error(family_label + " only supports the Tts task"); + throw std::runtime_error("VoxCPM2 only supports the Tts task"); } if (task.mode == runtime::RunMode::Streaming) { return std::make_unique(task, options, assets_); @@ -236,7 +136,7 @@ VoxCPM2LoadedModel::create_task_session( std::unique_ptr load_voxcpm2_model(const std::filesystem::path &model_path) { - auto assets = load_voxcpm2_assets(model_path, false); + auto assets = load_voxcpm2_assets(model_path); return std::make_unique( metadata(*assets), capabilities(*assets), std::move(assets)); } @@ -245,16 +145,4 @@ std::shared_ptr make_voxcpm2_loader() { return std::make_shared(); } -// VoxCPM1 model loading -std::unique_ptr -load_voxcpm1_model(const std::filesystem::path &model_path) { - auto assets = load_voxcpm2_assets(model_path, true); - return std::make_unique( - metadata_v1(*assets), capabilities_v1(*assets), std::move(assets)); -} - -std::shared_ptr make_voxcpm1_loader() { - return std::make_shared(); -} - } // namespace engine::models::voxcpm2 diff --git a/src/models/voxcpm2/minicpm.cpp b/src/models/voxcpm2/minicpm.cpp index 409aec91..5830c29e 100644 --- a/src/models/voxcpm2/minicpm.cpp +++ b/src/models/voxcpm2/minicpm.cpp @@ -338,11 +338,15 @@ class VoxCPM2TextEmbeddingRuntime::Impl { } ~Impl() { - release_graph(); + engine::core::release_backend_graph_resources(weights_->backend(), graph_); + if (buffer_ != nullptr) { + ggml_backend_buffer_free(buffer_); + } + if (gallocr_ != nullptr) { + ggml_gallocr_free(gallocr_); + } } - void release_runtime_memory() { release_graph(); } - std::vector embed_token(int32_t token_id) { const auto &config = weights_->assets().config.lm; if (token_id < 0 || token_id >= config.vocab_size) { @@ -364,24 +368,6 @@ class VoxCPM2TextEmbeddingRuntime::Impl { } private: - void release_graph() { - if (graph_ != nullptr) { - engine::core::release_backend_graph_resources(weights_->backend(), graph_); - } - if (buffer_ != nullptr) { - ggml_backend_buffer_free(buffer_); - buffer_ = nullptr; - } - if (gallocr_ != nullptr) { - ggml_gallocr_free(gallocr_); - gallocr_ = nullptr; - } - graph_ = nullptr; - token_id_ = nullptr; - output_ = nullptr; - ctx_.reset(); - } - void build(size_t graph_context_bytes) { const auto &config = weights_->assets().config.lm; if (graph_context_bytes == 0) { @@ -464,10 +450,6 @@ std::vector VoxCPM2TextEmbeddingRuntime::embed_token(int32_t token_id) { return impl_->embed_token(token_id); } -void VoxCPM2TextEmbeddingRuntime::release_runtime_memory() { - impl_->release_runtime_memory(); -} - struct MiniCPMLayerWithCacheOutput { engine::core::TensorValue output; engine::core::TensorValue key; @@ -596,8 +578,6 @@ class VoxCPM2PromptPrefillRuntime::Impl { ~Impl() { release_graph(); } - void release_runtime_memory() { release_graph(); } - VoxCPM2PromptPrefillOutput run(const VoxCPM2PromptPrefillInput &input) { const auto &config = weights_->assets().config; const int64_t hidden_size = config.lm.hidden_size; @@ -809,15 +789,13 @@ class VoxCPM2PromptPrefillRuntime::Impl { auto masked_current = mask_sequence(ctx, current_embeddings, audio_mask); auto residual_input = - config.v1 - ? engine::modules::AddModule{}.build(ctx, lm_hidden, masked_current) - : engine::modules::LinearModule( - binding::linear_config(config.lm.hidden_size * 2, - config.lm.hidden_size, true)) - .build(ctx, - engine::modules::ConcatModule({2}).build( - ctx, lm_hidden, masked_current), - model_weights.projections.fusion_concat_proj); + engine::modules::ConcatModule({2}).build(ctx, lm_hidden, masked_current); + residual_input = + engine::modules::LinearModule( + binding::linear_config(config.lm.hidden_size * 2, + config.lm.hidden_size, true)) + .build(ctx, residual_input, + model_weights.projections.fusion_concat_proj); auto residual_hidden = residual_input; for (const auto &layer : model_weights.residual_lm.layers) { @@ -929,10 +907,6 @@ VoxCPM2PromptPrefillRuntime::run(const VoxCPM2PromptPrefillInput &input) { return impl_->run(input); } -void VoxCPM2PromptPrefillRuntime::release_runtime_memory() { - impl_->release_runtime_memory(); -} - engine::core::TensorValue minicpm_layer_with_static_cache(engine::core::ModuleBuildContext &ctx, const engine::core::TensorValue &input, diff --git a/src/models/voxcpm2/minicpm_blocks.h b/src/models/voxcpm2/minicpm_blocks.h index 5a951822..08ae4445 100644 --- a/src/models/voxcpm2/minicpm_blocks.h +++ b/src/models/voxcpm2/minicpm_blocks.h @@ -164,11 +164,6 @@ minicpm_layer(engine::core::ModuleBuildContext &ctx, auto hidden = engine::modules::RMSNormModule( {config.hidden_size, config.rms_norm_eps, true, false}) .build(ctx, input, layer.input_norm); - if (std::getenv("VOXCPM_DUMP_NORM0") != nullptr && - ggml_nelements(hidden.tensor) == 10240) { - ggml_set_name(hidden.tensor, "dump_norm0"); - ggml_set_output(hidden.tensor); - } auto q = engine::modules::LinearModule( binding::linear_config(config.hidden_size, config.num_attention_heads * dim, false)) @@ -252,15 +247,8 @@ minicpm_transformer(engine::core::ModuleBuildContext &ctx, engine::core::TensorValue input, const engine::core::TensorValue &positions, const VoxCPM2MiniCPMWeights &weights, bool is_causal) { - for (size_t li = 0; li < weights.layers.size(); ++li) { - if (std::getenv("VOXCPM_DUMP_DECODER_LAYERS") != nullptr && li < 8) { - char name[32]; - snprintf(name, sizeof(name), "dump_layer_%zu", li); - ggml_set_name(input.tensor, name); - ggml_set_output(input.tensor); - } - input = minicpm_layer(ctx, input, positions, weights.layers[li], weights, - is_causal); + for (const auto &layer : weights.layers) { + input = minicpm_layer(ctx, input, positions, layer, weights, is_causal); } return engine::modules::RMSNormModule({weights.config.hidden_size, weights.config.rms_norm_eps, true, diff --git a/src/models/voxcpm2/session.cpp b/src/models/voxcpm2/session.cpp index 021ae765..780ae8e9 100644 --- a/src/models/voxcpm2/session.cpp +++ b/src/models/voxcpm2/session.cpp @@ -50,24 +50,6 @@ void reject_denoiser_option( } } -std::unordered_map normalize_v1_session_options( - std::unordered_map options) { - // V1 sessions share this runtime but advertise "voxcpm1.*" options; alias - // them to "voxcpm2.*" so the shared parsing below accepts both spellings. - std::unordered_map out; - out.reserve(options.size()); - for (auto &[key, value] : options) { - constexpr std::string_view kV1Prefix = "voxcpm1."; - if (key.rfind(kV1Prefix, 0) == 0) { - out[std::string("voxcpm2.") + - key.substr(kV1Prefix.size())] = std::move(value); - } else { - out[std::move(key)] = std::move(value); - } - } - return out; -} - bool audio_buffer_equal(const runtime::AudioBuffer &lhs, const runtime::AudioBuffer &rhs) { return lhs.sample_rate == rhs.sample_rate && lhs.channels == rhs.channels && @@ -85,9 +67,9 @@ bool optional_audio_equal(const std::optional &lhs, size_t prompt_cache_slots_from_options( const std::unordered_map &options) { constexpr int64_t kDefaultPromptCacheSlots = 1; - const int64_t slots = runtime::parse_i64_option( - options, {"voxcpm2.prompt_cache_slots", "voxcpm1.prompt_cache_slots"}) - .value_or(kDefaultPromptCacheSlots); + const int64_t slots = + runtime::parse_i64_option(options, {"voxcpm2.prompt_cache_slots"}) + .value_or(kDefaultPromptCacheSlots); if (slots < 0) { throw std::runtime_error("voxcpm2.prompt_cache_slots must be non-negative"); } @@ -177,17 +159,12 @@ VoxCPM2SessionBase::VoxCPM2SessionBase(runtime::TaskSpec task, if (task_.mode != runtime::RunMode::Offline && task_.mode != runtime::RunMode::Streaming) { throw std::runtime_error( - std::string(assets_->config.v1 ? "VoxCPM1" : "VoxCPM2") + - " only supports offline and streaming sessions"); + "VoxCPM2 only supports offline and streaming sessions"); } if (task_.task != runtime::VoiceTaskKind::Tts) { - throw std::runtime_error( - std::string(assets_->config.v1 ? "VoxCPM1" : "VoxCPM2") + - " only supports the Tts task"); + throw std::runtime_error("VoxCPM2 only supports the Tts task"); } - options.options = normalize_v1_session_options(std::move(options.options)); - reject_enabled_denoise(options.options, {"voxcpm2.denoise"}); reject_enabled_denoise(options.options, {"voxcpm2.load_denoiser"}); reject_denoiser_option(options.options, {"voxcpm2.denoiser"}); @@ -248,9 +225,7 @@ VoxCPM2SessionBase::VoxCPM2SessionBase(runtime::TaskSpec task, VoxCPM2SessionBase::~VoxCPM2SessionBase() = default; -std::string VoxCPM2SessionBase::family_impl() const { - return assets_->config.v1 ? "voxcpm1" : "voxcpm2"; -} +std::string VoxCPM2SessionBase::family_impl() const { return "voxcpm2"; } runtime::VoiceTaskKind VoxCPM2SessionBase::task_kind_impl() const { return task_.task; } @@ -274,7 +249,8 @@ runtime::TaskResult VoxCPM2SessionBase::run_offline_request(const runtime::TaskR } }; std::unique_ptr - release_guard(this, release_runtime_memory); + release_guard(generator_config_.mem_saver ? this : nullptr, + release_runtime_memory); const auto wall_start = Clock::now(); const int64_t text_chunk_size = @@ -287,7 +263,6 @@ runtime::TaskResult VoxCPM2SessionBase::run_offline_request(const runtime::TaskR const auto generation_options = generation_options_from_request(request); const auto prompt_text = runtime::find_option(request.options, {"voxcpm2.prompt_text", - "voxcpm1.prompt_text", "prompt_text", "reference_text"}) .value_or(""); std::optional reference_audio; @@ -298,32 +273,17 @@ runtime::TaskResult VoxCPM2SessionBase::run_offline_request(const runtime::TaskR const VoxCPM2EncodedPrompt *prompt = encoded_prompt_for_request(request.audio_input, prompt_text, reference_audio); - // The encoded prompt (voice-clone conditioning) is cached as host-side - // vectors; the VAE encoder graph that produced it is not needed again until - // a different voice is encoded. Drop it before the generator runs so the - // generator and decoder phases never coexist with the encoder graph. - decoder_->release_encoder_graph(); runtime::TaskResult result; double generator_ms = 0.0; double decoder_ms = 0.0; runtime::AudioBuffer merged_audio; - const bool mem_saver = generator_config_.mem_saver; - for (size_t chunk_index = 0; chunk_index < chunk_requests.size(); - ++chunk_index) { - const auto &chunk_request = chunk_requests[chunk_index]; + for (const auto & chunk_request : chunk_requests) { const auto generator_start = Clock::now(); const auto generated = generator_->generate( chunk_request.text_input->text, prompt, generation_options); generator_ms += engine::debug::elapsed_ms(generator_start, Clock::now()); - if (mem_saver && chunk_index + 1 == chunk_requests.size()) { - // Last chunk: free the generator graphs before the AudioVAE decode so - // the final decode peaks at weight + decoder graph instead of weight + - // generator + decoder. Graphs rebuild lazily on the next request. - generator_->release_runtime_memory(); - } - const auto decoder_start = Clock::now(); auto audio = decoder_->decode_features(generated.decode_features, generated.decode_patches); @@ -373,13 +333,13 @@ VoxCPM2SessionBase::run_streaming_request( } }; std::unique_ptr - release_guard(this, release_runtime_memory); + release_guard(generator_config_.mem_saver ? this : nullptr, + release_runtime_memory); const auto wall_start = Clock::now(); auto generation_options = generation_options_from_request(request); const auto prompt_text = runtime::find_option(request.options, {"voxcpm2.prompt_text", - "voxcpm1.prompt_text", "prompt_text", "reference_text"}) .value_or(""); std::optional reference_audio; @@ -390,10 +350,6 @@ VoxCPM2SessionBase::run_streaming_request( const VoxCPM2EncodedPrompt *prompt = encoded_prompt_for_request(request.audio_input, prompt_text, reference_audio); - // Same host-side clone-conditioning cache invariant as the offline path: - // the encoder graph is only needed to produce the cached vectors, so free - // it before the streaming generation starts. - decoder_->release_encoder_graph(); runtime::TaskResult result; runtime::AudioBuffer merged; @@ -454,18 +410,11 @@ VoxCPM2SessionBase::run_streaming_request( } void VoxCPM2SessionBase::release_request_runtime_memory() { - // Only the cloned voice is cached across requests (host-side encoded - // vectors in encoded_prompt_cache_). Every graph whose size follows the - // request text/audio length (prompt prefill, VAE encoder/decoder) is - // dropped so a long-lived server session returns to baseline VRAM and - // reallocates fresh buffers sized to the next request. - generator_->release_text_length_memory(); - decoder_->release_runtime_memory(); - if (generator_config_.mem_saver) { - // mem_saver additionally drops the fixed-size generator graphs so the - // session idles at weight-only VRAM. - generator_->release_runtime_memory(); + if (!generator_config_.mem_saver) { + return; } + generator_->release_runtime_memory(); + decoder_->release_runtime_memory(); } const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( @@ -475,22 +424,10 @@ const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( if (!prompt_audio.has_value() && !reference_audio.has_value()) { return nullptr; } - // VoxCPM1 clones only via prompt-continuation mode (golden VoxCPM.cpp - // uses --prompt-audio + --prompt-text); the V2 reference-mode path wraps - // audio in tokens 103/104, which the V1 LM was never trained on. Route a - // V1 reference audio through the prompt path so --voice-ref clones like - // --audio. - std::optional effective_prompt_audio = prompt_audio; - std::optional effective_reference_audio = reference_audio; - if (assets_->config.v1 && !effective_prompt_audio.has_value() && - effective_reference_audio.has_value()) { - effective_prompt_audio = effective_reference_audio; - effective_reference_audio.reset(); - } EncodedPromptCacheKey key; key.prompt_text = prompt_text; - key.prompt_audio = effective_prompt_audio; - key.reference_audio = effective_reference_audio; + key.prompt_audio = prompt_audio; + key.reference_audio = reference_audio; if (auto *cached = encoded_prompt_cache_.find(key)) { debug::trace_log_scalar("voxcpm2.prompt_cache.hit", 1); debug::trace_log_scalar("voxcpm2.prompt_cache.slots", @@ -505,8 +442,8 @@ const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( const auto encode_start = Clock::now(); EncodedPromptCacheEntry entry; - entry.encoded = decoder_->encode_prompt_audio( - effective_prompt_audio, prompt_text, effective_reference_audio); + entry.encoded = + decoder_->encode_prompt_audio(prompt_audio, prompt_text, reference_audio); const double encode_ms = engine::debug::elapsed_ms(encode_start); if (encoded_prompt_cache_.capacity() == 0) { uncached_encoded_prompt_ = std::move(entry); @@ -522,8 +459,8 @@ const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( encoded_prompt_cache_.put(std::move(key), std::move(entry)); EncodedPromptCacheKey lookup; lookup.prompt_text = prompt_text; - lookup.prompt_audio = effective_prompt_audio; - lookup.reference_audio = effective_reference_audio; + lookup.prompt_audio = prompt_audio; + lookup.reference_audio = reference_audio; auto *cached = encoded_prompt_cache_.find(lookup); if (cached == nullptr) { throw std::runtime_error("VoxCPM2 prompt cache insert failed"); @@ -543,77 +480,45 @@ const VoxCPM2EncodedPrompt *VoxCPM2SessionBase::encoded_prompt_for_request( VoxCPM2GenerationOptions VoxCPM2SessionBase::generation_options_from_request( const runtime::TaskRequest &request) const { VoxCPM2GenerationOptions options; - bool min_tokens_explicit = false; if (const auto value = runtime::parse_i64_option( - request.options, - {"voxcpm2.min_tokens", "voxcpm1.min_tokens", "min_tokens"})) { + request.options, {"voxcpm2.min_tokens", "min_tokens"})) { options.min_tokens = *value; - min_tokens_explicit = true; - } - // Set V1-specific default min_tokens if not explicitly provided - if (!min_tokens_explicit && assets_->config.v1) { - // Reference VoxCPM.cpp uses kMinLen=2 (stop may fire from the 4th patch); - // the decode loop gates on `index > min_tokens`, which is the same check. - // A higher floor (e.g. 20) forces ~1.6 s of audio and pads short - // utterances with trailing silence after the stop predictor fires. - options.min_tokens = 2; } if (const auto value = runtime::parse_i64_option( - request.options, - {"max_tokens", "voxcpm2.max_tokens", "voxcpm1.max_tokens"})) { + request.options, {"max_tokens", "voxcpm2.max_tokens"})) { options.max_tokens = *value; } if (const auto value = runtime::parse_i64_option( request.options, - {"num_inference_steps", "voxcpm2.num_inference_steps", - "voxcpm1.num_inference_steps"})) { + {"num_inference_steps", "voxcpm2.num_inference_steps"})) { options.num_inference_steps = *value; } if (const auto value = runtime::parse_finite_float_option( - request.options, - {"guidance_scale", "voxcpm2.guidance_scale", - "voxcpm1.guidance_scale"})) { + request.options, {"guidance_scale", "voxcpm2.guidance_scale"})) { options.guidance_scale = *value; } - bool retry_badcase_explicit = false; if (const auto match = runtime::find_option_match( - request.options, - {"voxcpm2.retry_badcase", "voxcpm1.retry_badcase", - "retry_badcase"})) { + request.options, {"voxcpm2.retry_badcase", "retry_badcase"})) { options.retry_badcase = runtime::parse_bool_option(match->value, match->key); - retry_badcase_explicit = true; - } - // Streaming emits decoded chunks to the client in real time, so a bad-case - // retry (regenerate from scratch, discard earlier output) is impossible by - // construction. The struct default retry_badcase=true exists for the - // offline path; it must not leak into the streaming path and block every - // streaming request. Relax it to false unless the caller explicitly asked - // for retry, which the generator accepts and warns about. - if (!retry_badcase_explicit && task_.mode == runtime::RunMode::Streaming) { - options.retry_badcase = false; } if (const auto value = runtime::parse_i64_option( request.options, - {"voxcpm2.retry_badcase_max_times", - "voxcpm1.retry_badcase_max_times", "retry_badcase_max_times"})) { + {"voxcpm2.retry_badcase_max_times", "retry_badcase_max_times"})) { options.retry_badcase_max_times = *value; } if (const auto value = runtime::parse_finite_float_option( - request.options, - {"voxcpm2.retry_badcase_ratio_threshold", - "voxcpm1.retry_badcase_ratio_threshold", - "retry_badcase_ratio_threshold"})) { + request.options, {"voxcpm2.retry_badcase_ratio_threshold", + "retry_badcase_ratio_threshold"})) { options.retry_badcase_ratio_threshold = *value; } - if (const auto value = runtime::parse_u32_option( - request.options, {"voxcpm2.seed", "voxcpm1.seed", "seed"})) { + if (const auto value = runtime::parse_u32_option(request.options, + {"voxcpm2.seed", "seed"})) { options.seed = *value; } options.cfm_noise_file = runtime::find_option(request.options, - {"voxcpm2.cfm_noise_file", "voxcpm1.cfm_noise_file", - "cfm_noise_file"}) + {"voxcpm2.cfm_noise_file", "cfm_noise_file"}) .value_or(""); if (options.min_tokens < 0) { throw std::runtime_error("VoxCPM2 min_tokens must be non-negative"); @@ -646,13 +551,10 @@ VoxCPM2GenerationOptions VoxCPM2SessionBase::generation_options_from_request( throw std::runtime_error( "VoxCPM2 retry_badcase_ratio_threshold must be positive"); } + reject_enabled_denoise(request.options, {"voxcpm2.denoise", "denoise"}); reject_enabled_denoise(request.options, - {"voxcpm2.denoise", "voxcpm1.denoise", "denoise"}); - reject_enabled_denoise(request.options, - {"voxcpm2.load_denoiser", "voxcpm1.load_denoiser", - "load_denoiser"}); - reject_denoiser_option(request.options, - {"voxcpm2.denoiser", "voxcpm1.denoiser", "denoiser"}); + {"voxcpm2.load_denoiser", "load_denoiser"}); + reject_denoiser_option(request.options, {"voxcpm2.denoiser", "denoiser"}); return options; } diff --git a/src/models/voxcpm2/tokenizer_text.cpp b/src/models/voxcpm2/tokenizer_text.cpp index ecdc62e6..8f49c619 100644 --- a/src/models/voxcpm2/tokenizer_text.cpp +++ b/src/models/voxcpm2/tokenizer_text.cpp @@ -1,5 +1,4 @@ #include "engine/models/voxcpm2/tokenizer_text.h" -#include "engine/models/voxcpm2/assets.h" #include "engine/framework/io/json.h" diff --git a/tools/audiocpp_cli/audiocpp_cli_path_cases.json b/tools/audiocpp_cli/audiocpp_cli_path_cases.json index c1bc3293..f522a666 100644 --- a/tools/audiocpp_cli/audiocpp_cli_path_cases.json +++ b/tools/audiocpp_cli/audiocpp_cli_path_cases.json @@ -805,116 +805,6 @@ } ] }, - { - "id": "voxcpm1_1.5b_q4k_load", - "coverage": "VoxCPM1 1.5B q4_k GGUF load and minimal offline tts sanity", - "family": "voxcpm1", - "model": "models/VoxCPM1.5-GGUF/voxcpm1.5-q4_k-audiovae-f16.gguf", - "task": "tts", - "mode": "offline", - "outputs": [ - "audio" - ], - "requests": [ - { - "id": "q4k_load", - "text": "This minimal q4k test loads the VoxCPM1 1.5B quantized variant.", - "seed": 1234, - "max_tokens": 160, - "guidance_scale": 2.0, - "num_inference_steps": 10 - } - ] - }, - { - "id": "voxcpm1_1.5b_q4k_tts", - "coverage": "VoxCPM1 1.5B q4_k text-to-speech path with MiniCPM generation, diffusion feature generation, and AudioVAE decode", - "family": "voxcpm1", - "model": "models/VoxCPM1.5-GGUF/voxcpm1.5-q4_k-audiovae-f16.gguf", - "task": "tts", - "mode": "offline", - "outputs": [ - "audio" - ], - "requests": [ - { - "id": "q4k_tts", - "text": "This VoxCPM1 path test checks the text-to-speech interface through AudioCPP CLI.", - "seed": 1234, - "max_tokens": 160, - "guidance_scale": 2.0, - "num_inference_steps": 10 - } - ] - }, - { - "id": "voxcpm1_1.5b_tts", - "coverage": "VoxCPM1 1.5B text-to-speech path with MiniCPM generation, diffusion feature generation, and AudioVAE decode", - "family": "voxcpm1", - "model": "models/VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf", - "task": "tts", - "mode": "offline", - "outputs": [ - "audio" - ], - "requests": [ - { - "id": "tts", - "text": "This VoxCPM1 path test checks the text-to-speech interface through AudioCPP CLI.", - "seed": 1234, - "max_tokens": 160, - "guidance_scale": 2.0, - "num_inference_steps": 10 - } - ] - }, - { - "id": "voxcpm1_1.5b_voice_clone", - "coverage": "VoxCPM1 1.5B voice clone path with reference audio encoding, MiniCPM generation, diffusion feature generation, and AudioVAE decode", - "family": "voxcpm1", - "model": "models/VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf", - "task": "tts", - "mode": "offline", - "outputs": [ - "audio" - ], - "requests": [ - { - "id": "clone", - "text": "This VoxCPM1 path test clones the reference speaker for a short review sentence.", - "voice_ref": "resources/sample.wav", - "seed": 1234, - "max_tokens": 160, - "guidance_scale": 2.0, - "num_inference_steps": 10 - } - ] - }, - { - "id": "voxcpm1_1.5b_streaming_tts", - "coverage": "VoxCPM1 1.5B streaming text-to-speech path with MiniCPM streaming generation, diffusion feature generation, and AudioVAE chunk decode", - "family": "voxcpm1", - "model": "models/VoxCPM1.5-GGUF/voxcpm1.5-q8_0.gguf", - "task": "tts", - "mode": "streaming", - "chunk_size": 512, - "outputs": [ - "audio" - ], - "requests": [ - { - "id": "streaming_tts", - "text": "This VoxCPM1 streaming path test checks that the CLI can emit audio chunks for a longer request while preserving a steady speaking style.", - "seed": 1234, - "max_tokens": 160, - "guidance_scale": 2.0, - "num_inference_steps": 10, - "options": { - "retry_badcase": false - } - } - ] - }, { "id": "higgs_audio_tts_voice_clone_chunked", "coverage": "Higgs Audio v3 voice clone path with framework text chunking, AR generation, and codec decode", diff --git a/webui/configs/models_catalog.json b/webui/configs/models_catalog.json index 9d8784ae..3a0d939f 100644 --- a/webui/configs/models_catalog.json +++ b/webui/configs/models_catalog.json @@ -17,8 +17,6 @@ { "id": "miotts", "display_name": "MioTTS 1.7B (tts; needs MioCodec)", "family": "miotts", "path": "models/MioTTS-1.7B", "task": "tts", "mode": "offline", "download_id": "miotts_1_7b", "min_vram_gb": 8 }, { "id": "voxcpm2", "display_name": "VoxCPM2 (tts)", "family": "voxcpm2", "path": "models/VoxCPM2", "task": "tts", "mode": "offline", "download_id": "voxcpm2", "session_options": { "voxcpm2.weight_type": "q8_0" }, "min_vram_gb": 6 }, { "id": "voxcpm1", "display_name": "VoxCPM1 0.5B (tts + clone)", "family": "voxcpm1", "path": "models/VoxCPM1-GGUF", "task": "tts", "mode": "offline", "download_id": "voxcpm1_0.5b_q8_0", "min_vram_gb": 4 }, - { "id": "voxcpm1_1_5b_q8_0", "display_name": "VoxCPM1 1.5B Q8_0 (tts + clone)", "family": "voxcpm1", "path": "models/VoxCPM1.5-GGUF", "task": "tts", "mode": "offline", "download_id": "voxcpm1_1.5b_q8_0", "min_vram_gb": 8 }, - { "id": "voxcpm1_1_5b_q4_k", "display_name": "VoxCPM1 1.5B Q4_K (tts + clone)", "family": "voxcpm1", "path": "models/VoxCPM1.5-GGUF", "task": "tts", "mode": "offline", "download_id": "voxcpm1_1.5b_q4_k", "min_vram_gb": 6 }, { "id": "vibevoice", "display_name": "VibeVoice 1.5B (tts, long-form/multi-speaker)", "family": "vibevoice", "path": "models/VibeVoice-1.5B", "task": "tts", "mode": "offline", "download_id": "vibevoice_1_5b", "min_vram_gb": 7 }, { "id": "index-tts2", "display_name": "IndexTTS2 (tts 中英克隆+情感)", "display_name_en": "IndexTTS2 (tts, zh/en clone + emotion)", "family": "index_tts2", "path": "models/IndexTTS-2", "task": "tts", "mode": "offline", "download_id": "index_tts2", "min_vram_gb": 8 }, { "id": "index-tts2.5", "display_name": "IndexTTS2.5 (tts 多语种克隆+情感, GGUF Q8)", "display_name_en": "IndexTTS2.5 (tts, zh/en/ja/es/ar clone + emotion, GGUF Q8)", "family": "index_tts2", "path": "models/IndexTTS2.5-GGUF", "task": "tts", "mode": "offline", "download_id": "index_tts2_5_q8_0", "min_vram_gb": 8, From 5ac840df4334a3996a1c1f8abc2378c657ed9047 Mon Sep 17 00:00:00 2001 From: Jason Chen Date: Fri, 21 Aug 2026 13:25:50 +0200 Subject: [PATCH 23/23] update webuil for voxcpm1 --- webui/native/dist/index.html | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/webui/native/dist/index.html b/webui/native/dist/index.html index 90309fc1..f6d81c7b 100644 --- a/webui/native/dist/index.html +++ b/webui/native/dist/index.html @@ -5,7 +5,7 @@ - + @@ -13,20 +13,20 @@