Granite Speech 4.1 2b Mlx 5bit

--- license: apache-2.0 library_name: mlx pipeline_tag: automatic-speech-recognition base_model: ibm-granite/granite-speech-4.1-2b base_model_relation: quantized language: - multilingual - en - fr - de - es - pt - ja tags: - mlx - mlx-audio - automatic-speech-recognition - speech-to-text - speech-translation - granite - granite-speech - autoregressive - quantized - 5-bit --- # Granite Speech 4.1 2B MLX 5-bit Quality-oriented mixed-precision MLX conversion of [`ibm-granite/granite-speech-4.1-2b`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b) for Apple Silicon. This is the **autoregressive Granite Speech model**, not the separate NAR variant. The model runs with [`mlx-audio`](https://github.com/Blaizzy/mlx-audio) and supports multilingual automatic speech recognition (ASR), speech translation (AST), punctuation and capitalization, and keyword-biased transcription. ## Quantization This conversion uses post-training weight quantization only. No training, fine-tuning, calibration dataset, or importance matrix was used. | Component | Precision | |---|---| | 16-layer Conformer speech encoder | BF16 | | 2-layer Q-Former speech projector | BF16 | | Granite language model, ordinary eligible internal linear layers | MLX affine 5-bit, group size 64 | | All `v_proj` and `down_proj` layers | MLX affine 6-bit, group size 64 | | Token embedding | BF16 | | Language-model output head | BF16 | | Norms, biases, and unsupported tensors | BF16 | Value and down projections receive extra precision because mixed-tensor quantizers commonly protect these sensitive paths. The acoustic stack, token embedding, and output head remain BF16. This is conceptually similar to medium mixed-tensor quantizations such as GGUF Q5_K_M, but the file format and numerical representation are MLX affine quantization rather than GGUF K-quants. - Source revision: `de575db64086f84fdc79da4932d1076e965bc546` - Effective average reported by MLX: 9.691 bits per weight - `model.safetensors`: approximately 2.5 GB ## Requirements - Apple Silicon Mac - macOS 14 or later - `mlx-audio >= 0.4.5` ```bash pip install -U mlx-audio ``` ## Usage ```bash python -m mlx_audio.stt.generate \ --model divydeep/granite-speech-4.1-2b-mlx-5bit \ --audio audio.wav \ --output-path transcript \ --format txt \ --prompt "transcribe the speech with proper punctuation and capitalization."...