Ultravox v0 6 Llama 3 1 8b

--- language: - ar - be - bg - bn - cs - cy - da - de - el - en - es - et - fa - fi - fr - gl - hi - hu - it - ja - ka - lt - lv - mk - mr - nl - pl - pt - ro - ru - sk - sl - sr - sv - sw - ta - th - tr - uk - ur - vi - zh license: mit library_name: transformers metrics: - bleu pipeline_tag: audio-text-to-text --- # Model Card for Ultravox Ultravox is a multimodal Speech LLM built around a pretrained LLM (Llama, Gemma, Qwen, etc) and a speech encoder ([whisper-large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo)) backbone. See https://ultravox.ai for the GitHub repo and more information. ## Model Details ### Model Description Ultravox is a multimodal model that can consume both speech and text as input (e.g., a text system prompt and voice user message). The input to the model is given as a text prompt with a special `<|audio|>` pseudo-token, and the model processor will replace this magic token with embeddings derived from the input audio. Using the merged embeddings as input, the model will then generate output text as usual. In v0.6 series, ultravox models are trained on expanded Hindi speech data, resulting in significantly improved speech understanding performance on Hindi and modest degradation on other languages. Additionally, the v0.6 models are also trained on noise datasets for improved noise robustness and the ability to output a special string ``((noise))`` if the input audio is too noisy or doesn't contain clear speech. In a future revision of Ultravox, we plan to expand the token vocabulary to support generation of semantic and acoustic audio tokens, which can then be fed to a vocoder to produce voice output. No preference tuning has been applied to this revision of the model. - **Developed by:** Fixie.ai - **License:** MIT ### Model Sources - **Repository:** https://ultravox.ai - **Demo:** See repo ## Usage Think of the model as an LLM that can also hear and understand speech. As such, it can be used as a voice agent, and also to do speech-to-speech translation, analysis of spoken audio, etc. To use the model, try the following: ```python # pip install transformers peft librosa import transformers import numpy as np import librosa pipe = transformers.pipeline(model='fixie-ai/ultravox-v0_6-llama-3_1-8b', trust_remote_code=True) path = "<path-to-input-audio>" # TODO: pass the audio here audio, sr = librosa.l...