Ultravox Qwen3.6 27b Base

A speech-language model built on the Ultravox architecture, using Qwen 3.6-27B as the language backbone and Whisper-large-v3-turbo as the audio encoder. Pretrained by Quantum Desk LTD for real-time voice agent applications.

Unlike a cascaded ASR → LLM pipeline, this model consumes raw audio embeddings directly into the LLM's token stream — eliminating the transcription hop and cutting 200-400 ms of latency per turn.

Model Details

Training

• Loss: KL divergence to teacher (temperature 2.0)
• Steps: 10 000 (loss converged near step ~500, remaining steps refined the projection)
• Effective batch: 16 (8 × 2 gradient accumulation)
• LR: 5e-4 cosine → 5e-5, 500-step warmup
• Optimizer: AdamW
• Hardware: 1 × NVIDIA B200 (180 GB HBM3e), single-GPU, no FSDP
• Wall-clock: ~6 h 30 min
• Whisper adaptation: LoRA rank 8 on Whisper k_proj, q_proj, linear_k, linear_q

Training data

English audio-transcription and audio-continuation mix (~10 datasets):

• fixie-ai/librispeech_asr (clean + other, continuation + transcription)
• fixie-ai/peoplespeech-clean
• fixie-ai/gigaspeech-xl
• fixie-ai/commonvoice-en

This is an English-only base checkpoint. Multilingual and phone-quality audio training were deliberately excluded to accelerate iteration and reduce cost. That broader training (Phase 1B telephony mix + Phase 2 domain data) has since shipped as ultravox-qwen3.6-27b-v2.

Evaluation

Training-time metrics on fixie-ai/librispeech-clean-transcription (validation, 256 samples):

For reference, Fixie's ultravox-v0_6-qwen-3-32b achieves 2.88 WER on librispeech (multilingual training, 32B base).

Full WER benchmarks were measured on the successor checkpoint — v2 reaches 2.08% LibriSpeech test-clean / 9.67% Switchboard WER — and are published on its card. This base checkpoint was not separately benchmarked.

Usage

With vLLM (recommended for serving)

vllm serve QuantumDesk-AI/ultravox-qwen3.6-27b-base \
  --served-model-name ultravox \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.85 \
  --port 8000

With transformers (for research)

from transformers import AutoModel, AutoProcessor
import torchaudio

processor = AutoProcessor.from_pretrained(
    "QuantumDesk-AI/ultravox-qwen3.6-27b-base",
    trust_remote_code=True,
)
model = AutoModel.from_pretrained(
    "QuantumDesk-AI/ultravox-qwen3.6-27b-base",
    trust_remote_code=True,
    torch_dtype="bfloat16",
).to("cuda")

audio, sr = torchaudio.load("hello.wav")
messages = [
    {"role": "user", "content": [
        {"type": "audio", "audio": audio[0].numpy(), "sampling_rate": sr},
        {"type": "text", "text": "What did I say?"},
    ]},
]
inputs = processor.apply_chat_template(messages, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(outputs[0], skip_special_tokens=True))

Serving hardware

FP8 base + bf16 adapter fits on any Blackwell (RTX PRO 6000, B200) or Hopper (H100 80/94GB) GPU:

*Compute-bound (KV cache is not the limit). Assumes ~2000 tokens context per session and voice-typical 5-6 tok/s per session sustained rate.

Limitations

• English only — do not use for other languages; expect degraded quality even for accented English.
• Clean audio only — trained without phone-codec, noise, or music-background data. Real-world telephony will underperform relative to studio audio.
• No chat / instruction tuning — this is a base checkpoint. For interactive voice agents, fine-tune on your domain (see [Phase 2 discussion in repo]).
• Reasoning under audio input — some capacity is spent decoding audio; expect small (2-5 %) degradation vs pure-text prompting on reasoning-heavy tasks. Use text cascade for complex queries.

License

Apache 2.0. Model weights are Quantum Desk LTD's contribution. See individual base-model licenses for Qwen 3.6-27B (Apache 2.0) and Whisper-large-v3-turbo (MIT).

Acknowledgements

Built on:

• Fixie AI's Ultravox — architecture and training code
• Qwen team — 3.6-27B base
• OpenAI — Whisper encoder

Citation

@misc{quantumdesk_ultravox_qwen36_2026,
  author = {Quantum Desk LTD},
  title = {Ultravox-Qwen3.6-27B-Base},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/QuantumDesk-AI/ultravox-qwen3.6-27b-base},
}