Granite Speech 4.1 2b Mlx 6bit

--- license: apache-2.0 library_name: mlx pipeline_tag: automatic-speech-recognition base_model: ibm-granite/granite-speech-4.1-2b base_model_relation: quantized language: - multilingual - en - fr - de - es - pt - ja tags: - mlx - mlx-audio - automatic-speech-recognition - speech-to-text - speech-translation - granite - granite-speech - autoregressive - quantized - 6-bit --- # Granite Speech 4.1 2B MLX 6-bit Quality-oriented MLX conversion of [`ibm-granite/granite-speech-4.1-2b`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b) for Apple Silicon. This is the **autoregressive Granite Speech model**, not the separate NAR variant. The model runs with [`mlx-audio`](https://github.com/Blaizzy/mlx-audio) and supports multilingual automatic speech recognition (ASR), speech translation (AST), punctuation and capitalization, and keyword-biased transcription. ## Quantization This conversion uses post-training weight quantization only. No training, fine-tuning, calibration dataset, or importance matrix was used. | Component | Precision | |---|---| | 16-layer Conformer speech encoder | BF16 | | 2-layer Q-Former speech projector | BF16 | | Granite language model, eligible internal linear layers | MLX affine 6-bit, group size 64 | | Token embedding | BF16 | | Language-model output head | BF16 | | Norms, biases, and unsupported tensors | BF16 | The acoustic stack is kept in BF16 because quantization errors can compound through the Conformer and alter the embeddings supplied to the language model. The token embedding and output head are also kept in BF16 to preserve input and output token precision. This follows the general multimodal quantization practice of protecting encoders, projectors, embeddings, and output tensors while compressing the large internal language-model projections. - Source revision: `de575db64086f84fdc79da4932d1076e965bc546` - Effective average reported by MLX: 10.144 bits per weight - `model.safetensors`: approximately 2.7 GB ## Requirements - Apple Silicon Mac - macOS 14 or later - `mlx-audio >= 0.4.5` ```bash pip install -U mlx-audio ``` ## Usage ```bash python -m mlx_audio.stt.generate \ --model divydeep/granite-speech-4.1-2b-mlx-6bit \ --audio audio.wav \ --output-path transcript \ --format txt \ --prompt "transcribe the speech with proper punctuation and capitalization." ``` Python: ```python from...