--- license: mit language: - en datasets: - speechbrain/LoquaciousSet base_model: - zai-org/GLM-ASR-Nano-2512 - Qwen/Qwen3-0.6B pipeline_tag: automatic-speech-recognition tags: - asr - speech-recognition - audio - qwen - glm-asr library_name: transformers --- # Tiny Audio A speech recognition model trained in 24 hours on a single GPU for ~$12. Built with [Tiny Audio](https://github.com/alexkroman/tiny-audio)—a minimal, hackable ASR framework. ## Quick Start ```python from transformers import pipeline pipe = pipeline("automatic-speech-recognition", model="mazesmazes/tiny-audio", trust_remote_code=True) result = pipe("audio.wav") print(result["text"]) ``` ## Usage Examples ### Basic Transcription ```python from transformers import pipeline pipe = pipeline("automatic-speech-recognition", model="mazesmazes/tiny-audio", trust_remote_code=True) # From file result = pipe("audio.wav") print(result["text"]) # From URL result = pipe("https://example.com/audio.mp3") # From numpy array (must be 16kHz) import numpy as np audio = np.random.randn(16000).astype(np.float32) # 1 second result = pipe(audio) ``` ### Batch Processing ```python # Process multiple files files = ["audio1.wav", "audio2.wav", "audio3.wav"] results = pipe(files, batch_size=4) for r in results: print(r["text"]) ``` ### Word-Level Timestamps ```python result = pipe("audio.wav", return_timestamps="word") # Returns: # { # "text": "hello world", # "chunks": [ # {"text": "hello", "timestamp": (0.0, 0.5)}, # {"text": "world", "timestamp": (0.6, 1.0)} # ] # } ``` ### Streaming Inference ```python from tiny_audio import ASRModel, ASRProcessor import torch model = ASRModel.from_pretrained("mazesmazes/tiny-audio") processor = ASRProcessor.from_pretrained("mazesmazes/tiny-audio") # Load and process audio import librosa audio, sr = librosa.load("audio.wav", sr=16000) inputs = processor(audio, sampling_rate=16000, return_tensors="pt") # Stream tokens for token in model.generate_streaming(inputs["input_features"]): print(token, end="", flush=True) ``` ### Using with torch directly ```python from tiny_audio import ASRModel, ASRProcessor import torch import librosa # Load model and processor model = ASRModel.from_pretrained("mazesmazes/tiny-audio") processor = ASRProcessor.from_pretrained("mazesmazes/tiny-audio") # Load audio (16kHz) audio, sr = librosa.load("audio.wav", sr=16000) # Process inputs = processor(audio, sampling_rate=16000, return_tensors="pt") # Generate with torch.no_grad(): output = model.generate( input_features=inputs["input_features"], attention_mask=inputs["attention_mask"], max_new_tokens=256 ) # Decode text = processor.batch_decode(output, skip_special_tokens=True)[0] print(text) ``` ### GPU Inference ```python import torch pipe = pipeline( "automatic-speech-recognition", model="mazesmazes/tiny-audio", trust_remote_code=True, device="cuda" # or device=0 ) ``` ### Half Precision ```python pipe = pipeline( "automatic-speech-recognition", model="mazesmazes/tiny-audio", trust_remote_code=True, torch_dtype=torch.float16, device="cuda" ) ``` ## Architecture ``` Audio (16kHz) → GLM-ASR Encoder (frozen) → MLP Projector (trained) → Qwen3 (frozen) → Text ``` Only the projector is trained (~12M params). The encoder and decoder remain frozen, leveraging their pretrained knowledge. | Component | Model | Parameters | Status | |-----------|-------|------------|--------| | Audio Encoder | GLM-ASR-Nano-2512 | ~600M | Frozen | | Projector | 2-layer MLP | ~12M | Trained | | Language Model | Qwen3-0.6B | ~600M | Frozen | ### How It Works 1. **Audio Encoder**: GLM-ASR converts 16kHz audio into frame-level embeddings (768-dim) 2. **Projector**: A 2-layer MLP with frame stacking bridges the audio and text embedding spaces 3. **Language Model**: Qwen3 generates text autoregressively, conditioned on the projected audio The projector reduces sequence length via frame stacking: `output_len = (input_len - 5) // 5 + 1` ## Model Specifications | Specification | Value | |---------------|-------| | Input | Audio (16kHz mono) | | Output | Text transcription | | Max Audio Length | ~30 seconds (limited by encoder) | | Vocabulary | Qwen3 tokenizer | | Languages | English only | | Generation | Greedy decoding (num_beams=1, do_sample=False) | ## Training Details | | | |---|---| | **Dataset** | LoquaciousSet (25,000 hours) | | **Hardware** | Single NVIDIA A40 | | **Time** | ~24 hours | | **Cost** | ~$12 | | **Optimizer** | AdamW | | **Learning Rate** | 1e-4 | | **Batch Size** | 4 | | **Steps** | 50,000 | ## Limitations - **English only**: Not trained on other languages - **Sample rate**: Expects 16kHz audio (other rates resampled automatically) - **Audio length**: Best for clips under 30 seconds - **Accuracy**: May degrade on: - Heavily accented speech - Noisy or low-quality audio - Domain-specific terminology - Overlapping speakers - **No punctuation**: Output is lowercase without punctuation by default ## Requirements ``` transformers>=4.40.0 torch>=2.0.0 torchaudio>=2.0.0 ``` Optional for streaming: ``` librosa soundfile ``` ## Files | File | Description | |------|-------------| | `config.json` | Model configuration | | `model.safetensors` | Projector weights (~48MB) | | `preprocessor_config.json` | Audio preprocessing config | | `tokenizer.json` | Tokenizer | | `tokenizer_config.json` | Tokenizer config | | `special_tokens_map.json` | Special tokens | Note: Only the projector weights are stored. The encoder (GLM-ASR) and decoder (Qwen3) are loaded from their respective HuggingFace repos. ## Citation If you use this model, please cite: ```bibtex @misc{tinyaudio2024, author = {Alex Kroman}, title = {Tiny Audio: Minimal ASR Training}, year = {2024}, publisher = {GitHub}, url = {https://github.com/alexkroman/tiny-audio} } ``` ## Links - [GitHub Repository](https://github.com/alexkroman/tiny-audio) - Train your own model - [Free 3.5-hour Course](https://github.com/alexkroman/tiny-audio/blob/main/docs/course/0-course-overview.md) - Learn ASR from scratch - [Live Demo](https://huggingface.co/spaces/mazesmazes/tiny-audio) - Try it in your browser ## Acknowledgments - [GLM-ASR](https://huggingface.co/zai-org/GLM-ASR-Nano-2512) for the audio encoder - [Qwen3](https://huggingface.co/Qwen/Qwen3-0.6B) for the language model - [LoquaciousSet](https://huggingface.co/datasets/speechbrain/LoquaciousSet) for training data ## License MIT
