Shuka 1

--- library_name: transformers pipeline_tag: audio-text-to-text license: llama3 language: - en - hi --- `Shuka v1` is a language model which natively understands audio in Indic languages. It is an encoder-decoder model built by combining two models: - Our state-of-the-art, in-house, audio encoder: Saaras v1 - Meta’s Llama3-8B-Instruct as the decoder The encoder and decoder are connected by a small projector with ~60M parameters. During training, only the projector weights are finetuned while the rest of the network is frozen. Following our tradition of training models frugally, we train `Shuka v1` on less than 100 hours of audio. Though we only finetune the projector on English and Hindi data, the multilingual nature of our encoder makes `Shuka v1` perform well on zero-shot QA in other Indic languages as well. We have tested on the model on Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, and Telugu. See what `Shuka v1` can do in this [demo video](https://www.youtube.com/watch?v=VgJhjCPbORs), and get started by using huggingface pipeline, as follows: ``` # install libraries # pip install transformers==4.41.2 peft==0.11.1 librosa==0.10.2 import transformers import librosa # load the model pipeline on gpu:0 pipe = transformers.pipeline(model='sarvamai/shuka_v1', trust_remote_code=True, device=0, torch_dtype='bfloat16') # get a sample audio # wget https://huggingface.co/sarvamai/shuka_v1/resolve/main/hi-question.webm audio, sr = librosa.load("./hi-question.webm", sr=16000) turns = [ {'role': 'system', 'content': 'Respond naturally and informatively.'}, {'role': 'user', 'content': '<|audio|>'} ] pipe({'audio': audio, 'turns': turns, 'sampling_rate': sr}, max_new_tokens=512) ``` For more details, please see our [blog](https://www.sarvam.ai/blogs/shuka-v1).

Source context: 269 downloads · 92 likes · Pipeline audio-text-to-text · Library transformers · Repo sarvamai/shuka-1