fix(tts/kokoro-ane): throw instead of trapping on non-finite PostAlbert durations#816
Conversation
…rt durations
On iOS 27 betas the new E5 CoreML runtime mis-executes the dynamic-shape
Kokoro stages ("MIL program has non-constant (dynamic) shapes ... E5
function will be produced with unknown input shapes") and PostAlbert
returns NaN for its 'duration' output. The pred_dur conversion then hit
Int32(Float.nan), a Swift runtime trap that crashed the host app
(issue #738, report from iOS 27 public beta).
Extract the rounding into predictedDurations(from:), which now:
- throws KokoroAneError.nonFiniteModelOutput on NaN/inf instead of
trapping, so host apps get a catchable error naming the likely cause
- clamps finite values to [1, maxAcousticFrames] so huge garbage values
cannot trap Int32() either; absurd totals still surface via the
existing acousticFramesExceedCap check
Behavior for well-formed outputs is unchanged (durations are small
positive floats, far below the cap).
Adds pure-function unit tests for rounding, clamping, and the NaN/inf
throw path (no models required).
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 1m34s • 07/23/2026, 06:34 PM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 120.5s processing • Test runtime: 2m 18s • 07/23/2026, 06:36 PM EST |
Supertonic3 Smoke Test ✅
Runtime: 0m42s Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf. |
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 50.0s diarization time • Test runtime: 2m 39s • 07/23/2026, 06:39 PM EST |
PocketTTS Smoke Test ✅
Runtime: 1m58s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon. |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 10m25s • 07/23/2026, 06:43 PM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 3m 21s • 2026-07-23T22:48:50.156Z |
1 participant
Summary
Fixes the hard crash reported in #738 (comment): on iOS 27 public beta, Kokoro synthesis kills the host app with
at
KokoroAneSynthesizer.swift:74(d=NaN).Root cause
iOS 27's new E5 CoreML runtime flags the dynamic-shape Kokoro stages at load time (
MIL program has non-constant (dynamic) shapes for external input but FlexibleShapeInformation attribute is missing) and then mis-executes them — PostAlbert returns NaN for itsdurationoutput. Thepred_durconversion didInt32(Float(d).rounded()), which is a Swift runtime trap on NaN, so the whole host app crashed instead of receiving an error. Same class of on-device dynamic-shape misbehavior as #739 (there the symptom was zeros; here NaNs).Change
KokoroAneSynthesizer.predictedDurations(from:):KokoroAneError.nonFiniteModelOutput(stage:output:)on NaN/±inf, with a message pointing at the iOS 27 runtime and MacOS 27 Beta 2 Release Notes mentions ios 27 locking down ANE #738[1, maxAcousticFrames]so huge garbage values can't trapInt32()either; absurd totals still surface through the existingacousticFramesExceedCapcheckThis makes the failure catchable; it does not make Kokoro work on iOS 27 betas. The model-side fix (attaching flexible-shape info so E5 compiles the dynamic stages correctly) is separate follow-up work. Affected users can try
KokoroAneComputeUnits.cpuAndGpuas a possible workaround.Testing
swift buildclean,swift format lintcleanKokoroAnePredictedDurationTests(run on CI; no model downloads required)🤖 Generated with Claude Code