Chatterbox Turbo LoRA

--- license: mit library_name: onnx tags: - text-to-speech - chatterbox - lora - onnx - mobile base_model: ResembleAI/chatterbox --- # chatterbox-turbo-lora LoRA-aware ONNX export of the Chatterbox-Turbo language-model subgraph, quantized to **q4f16** for on-device inference (Android / iOS via ORT). This is the base LM only — the full Chatterbox-Turbo TTS pipeline still needs the conditional-encoder, speech-token, and S3-Gen subgraphs from the upstream Chatterbox repo. ## What's new in v2.0.0 v1.0.0 of the language-model subgraph had no LoRA input ports — adapter weights could not be bound at runtime. v2.0.0 wraps every `Conv1D` projection inside the transformer (`attn.c_attn`, `attn.c_proj`, `mlp.c_fc`, `mlp.c_proj`) in a `LoraConv1D` that exposes two extra input tensors per layer: ``` lora.layers.{N}.{attn_c_attn|attn_c_proj|mlp_c_fc|mlp_c_proj}.{A|B} ``` → **24 layers × 4 projections × 2 tensors = 192 extra graph inputs.** At inference time the host runtime binds a PEFT-trained LoRA adapter (rank, scaling baked in) to these inputs via the ONNX Runtime `OrtLoraAdapter` API. Unbound inputs default to zero-sized initializers so the graph still runs without any adapter loaded — the base voice synthesizes as-is. ## Files | File | Size | Purpose | |---|---|---| | `language_model_q4f16.onnx` | ~0.8 MB | Graph protobuf (no weights) | | `language_model_q4f16.onnx_data` | ~222 MB | External weight data (q4f16) | Both files must be downloaded together. The `.onnx` file references the `.onnx_data` companion by relative filename. ## Quantization details - **Format:** q4f16 (4-bit asymmetric weights + fp16 scales) - **Block size:** 32 (matches Chatterbox's original 1.0.0 layout) - **Quantizer:** `onnxruntime.quantization.MatMulNBitsQuantizer` - **Excluded from quantization:** all 192 LoRA-path `MatMul`/`Add` nodes (kept fp32 so adapter weights can be hot-swapped without retraining scales) - The 96 GPT-2 projection layers are first rewritten from `Gemm` → `MatMul + Add` so they fall inside the quantizer's coverage; without this step the file would stay at ~1.2 GB. Embedding tables (`wte`, `wpe`) remain fp32 because they are read via `Gather`, not `MatMul`, and `MatMulNBitsQuantizer` does not touch them. ## License MIT, inherited from the upstream Chatterbox project.