Parakeet Tdt 0.6b v2 Mlx 8bit

--- license: cc-by-4.0 language: - en library_name: mlx tags: - automatic-speech-recognition - parakeet - mlx - quantized - apple-silicon base_model: nvidia/parakeet-tdt-0.6b-v2 pipeline_tag: automatic-speech-recognition --- # parakeet-tdt-0.6b-v2-mlx-8bit 8-bit quantized MLX weights for **NVIDIA Parakeet TDT 0.6B v2**, ready to load on Apple Silicon without any quantization step at runtime. Published by [CosmicFusion Labs](https://whisperbud.com) for [WhisperBud](https://whisperbud.com), an on-device dictation app for macOS. ## Why this exists The upstream MLX weights (`mlx-community/parakeet-tdt-0.6b-v2`) are full precision: **2,472 MB** to download, then quantized on each machine at load time. These weights are the result of that same quantization (`mlx.nn.quantize`, `bits=8`, `group_size=64`), saved once: | | upstream | this repo | |---|---|---| | download | 2,472 MB | **933 MB** | | model load | ~1.7 s | **~0.05 s** | | transcription output | reference | **identical** | Verified byte-identical transcripts against the quantize-at-load path on real speech clips (English, 5 to 45 seconds). ## Usage `parakeet_mlx.from_pretrained()` does NOT load quantized weights: it builds a full-precision model and loads the tensors into it. Build the quantized structure first, then load: ```python import json import mlx.nn as nn from huggingface_hub import hf_hub_download from parakeet_mlx.utils import from_config repo = "cosmicfusionlabs/parakeet-tdt-0.6b-v2-mlx-8bit" config = json.load(open(hf_hub_download(repo, "config.json"))) quant = config.pop("quantization") # {"bits": 8, "group_size": 64} model = from_config(config) nn.quantize(model, bits=quant["bits"], group_size=quant["group_size"]) model.load_weights(hf_hub_download(repo, "model.safetensors")) model.eval() result = model.transcribe("audio.wav") print(result.text) ``` ## Accuracy Same weights as quantizing the upstream model at load time, so accuracy is unchanged from the 8-bit path. In our own benchmark on real (non-synthetic) English speech, 8-bit Parakeet TDT 0.6B v2 measured 11.6% mean WER, between `whisper-small.en` (13.5%) and `whisper-medium.en` (8.2%), while running 2 to 4x faster than `medium.en` on an M1 Pro. 4-bit was measured too and degraded accuracy (13.3% WER), which is why this repo ships 8-bit. ## Credits and license - Original model: [nvidia/parakeet-tdt-0.6b-v2](https://h...

Source context: 12 downloads · 0 likes · Pipeline automatic-speech-recognition · Library mlx · Repo cosmicfusionlabs/parakeet-tdt-0.6b-v2-mlx-8bit