Luxtts Coreml

LuxTTS CoreML

CoreML conversion of YatharthS/LuxTTS for on-device inference on Apple platforms.

LuxTTS is a 48 kHz zero-shot voice-cloning TTS (~123M params): give it a short reference clip plus its transcript, and it speaks new text in that voice. It is based on ZipVoice (k2-fsa, Xiaomi) — a flow-matching Zipformer decoder run with a 4-step solver — with a dual-head Vocos vocoder from LinaCodec (ysharma3501).

Models

Path	Model	Size	Notes
gpu/TextEncoder.mlmodelc	Text encoder (256 tokens)	9.5 MB	fp16
gpu/FmDecoder.mlmodelc	Flow-matching decoder, 1024-frame bucket	256 MB	fp16, original graph — macOS GPU path
gpu-6bit/FmDecoder.mlmodelc	6-bit palettized decoder, 1024-frame bucket	115 MB	full GPU speed, -0.45 dB, transcript-transparent
gpu-long/TextEncoder.mlmodelc	Text encoder (512 tokens)	13 MB	fp16
gpu-long/FmDecoder.mlmodelc	Decoder, 2048-frame bucket	349 MB	long-form macOS (~16.8 s per bucket)
ane/TextEncoder.mlmodelc	Text encoder (256 tokens)	9.5 MB	fp16
ane/FmDecoder.mlmodelc	ANE-canonical decoder, 1024-frame bucket	313 MB	100% ANE placement — iPhone path
vocoder/Vocoder282.mlmodelc	Vocos vocoder, 282 gen frames (~3.0 s)	33 MB	ISTFT + 24k→48k resample + crossover in-graph
vocoder/Vocoder555.mlmodelc	Vocos vocoder, 555 gen frames (~5.9 s)	33 MB	matches the full 1024-frame bucket
tokens.txt / config.json	Phoneme token table + model config	—	from upstream LuxTTS

Two decoder graphs — pick by compute unit

• gpu/ — the original graph layout. Fastest on Mac GPU. Do not run it on the ANE: the seq-first rel-pos attention path loses precision there and produces corrupted audio.
• ane/ — ANE-canonical rewrite (channels-first (1,C,1,S), 1×1-conv projections, constant rel-pos basis, split depthwise kernels). 100% ANE op placement, zero CPU fallback. This is the iPhone path; it is slower than the original graph on GPU, so keep both.

All decoder inputs/outputs are identical across graphs (t, x, text_condition, speech_condition, guidance_scale, padding_mask → v).

Performance (M5 Pro, macOS)

path	fm step	core RTFx (full bucket)	steady footprint
gpu/ fp16, GPU	14.7 ms	92x (~5.9 s gen)	996 MB
gpu-6bit/, GPU	15.0 ms	~92x	419 MB
ane/ fp16, ANE	54.6 ms	27x	25.5 MB

• Vocoder: 2.34 ms on GPU (282 frames); end-to-end ~64 ms for 3 s of audio (~47x realtime).
• Quality vs PyTorch reference: log-mel cosine 0.99925 (GPU path); whisper-base transcripts word-identical for all shipped variants.
• Footprints are phys_footprint from a Swift host; the ANE graph keeps weights and activations in ANE-managed memory, hence the 25.5 MB jetsam-visible footprint.

Fixed-shape buckets

CoreML graphs are compiled at fixed shapes. Frame budget covers prompt + generated speech (86.13 frames/s):

• gpu/, ane/: 1024 frames (~5.9 s generated after a 5 s prompt), 256 text tokens
• gpu-long/: 2048 frames (~16.8 s generated), 512 text tokens
• vocoder: fixed at 282 or 555 generated frames

Chunk long text at sentence boundaries for the 1024-frame bucket on iPhone.

Usage notes

The CoreML graphs cover the text encoder, flow-matching decoder, and vocoder (ISTFT, 24k→48k resample, and the Linkwitz crossover are all in-graph). The host must provide:

• Phoneme tokenization against tokens.txt (espeak-compatible token set) — the frontend is phoneme-token based, not raw text.
• Duration expansion of encoder output into the frame grid (upstream pad_labels keeps S+1 token rows; the pad slot's embedding fills remainder frames).
• The 4-step anchor-Euler flow solver with t_shift = 0.5 and guidance_scale = 3.0.
• Prompt trimming at a speech pause (VAD boundary), not a fixed duration — a prompt hard-cut mid-phrase makes the model elide sentence-initial words.
• speed default 1.0 (upstream's generate() default of 1.3 clips sentence onsets).

Swift integration in FluidAudio is upcoming.

Known limitations

• Fixed-shape buckets only (no enumerated/flexible shapes yet).
• An int8-linear variant exists but crashes MPSGraph on macOS 26.5 GPU — not shipped.
• The ANE graph is ~0.5 dB softer than the GPU graph (fp16 accumulation over 16 layers × 4 solver steps); transcripts are unaffected.

Attribution & license

• LuxTTS by YatharthS — Apache-2.0.
• Based on ZipVoice by k2-fsa (Xiaomi) — Apache-2.0.
• Vocoder from LinaCodec (ysharma3501).
• CoreML conversion by FluidInference; conversion source lives in the mobius repo (models/tts/zipvoice, PR #75).

References

• https://huggingface.co/YatharthS/LuxTTS
• https://github.com/k2-fsa/ZipVoice
• https://github.com/FluidInference/FluidAudio