Diffusion Gemma Asr Small

--- license: apache-2.0 language: [en, de, fr, es, hi, zh] library_name: transformers pipeline_tag: automatic-speech-recognition tags: - asr - speech-to-text - diffusion - diffusiongemma - gemma base_model: - google/diffusiongemma-26B-A4B-it - openai/whisper-small --- # diffusion-gemma-asr-small 📝 **Links:** [Blog](https://interfaze.ai/blog/the-first-open-source-diffusion-audio-asr-model) · [Demo Space](https://huggingface.co/spaces/interfaze-ai/diffusion-gemma-asr-demo) · [Code](https://github.com/JigsawStack/diffusion-gemma-asr) **Audio-native, multilingual speech recognition that transcribes through DiffusionGemma's own discrete-diffusion decoder** — not autoregressive, not an external ASR decoder. Audio is projected directly into the Gemma embedding space, and the transcript is produced by parallel diffusion denoising (~8–16 steps), giving real-time-plus throughput where cost is set by the number of denoising steps, not the length of the transcript. This repo ships the **trained adapter only** (projector + LoRA, ~42M params — 0.16% of the model). The frozen 26B DiffusionGemma backbone and the frozen whisper-small encoder load from their own repos. ## How it works ``` raw audio ─► whisper-small encoder (frozen) ─► projector (trained, ~19M) ─► scatter into <audio> token slots of DiffusionGemma's encoder ─► DiffusionGemma decoder denoises a 192-token canvas (bidirectional, cross-attends audio) ─► transcript ``` - **Backbone:** `google/diffusiongemma-26B-A4B-it` — frozen, small LoRA adapters on encoder/decoder attention. - **Audio frontend:** `openai/whisper-small` encoder — frozen feature extractor (NOT a decoder). - **Grounding:** trained with three losses — uniform-diffusion (the generator), an AR auxiliary, and a **CTC loss on the projector via the frozen `lm_head`** (the key unlock that makes the audio embeddings transcript-predictive). ## Usage **Install** ```bash pip install torch peft soundfile librosa huggingface_hub \ "transformers @ git+https://github.com/huggingface/transformers.git" # DiffusionGemma support ``` **Transcribe in Python** ```python import sys, soundfile as sf from huggingface_hub import snapshot_download repo = snapshot_download("interfaze-ai/diffusion-gemma-asr-small") # this adapter (~170 MB) sys.path.insert(0, repo) from inference import load, transcribe # bundled in this repo # Loads frozen Diffus...

Source context: 0 downloads · 11 likes · Pipeline automatic-speech-recognition · Library transformers · Repo interfaze-ai/diffusion-gemma-asr-small