MuQ MuLan Large

--- license: cc-by-nc-4.0 language: - en - zh pipeline_tag: audio-classification tags: - music --- # MuQ & MuQ-MuLan <div> <a href='#'><img alt="Static Badge" src="https://img.shields.io/badge/Python-3.8%2B-blue?logo=python&logoColor=white"></a> <a href='https://arxiv.org/abs/2501.01108'><img alt="Static Badge" src="https://img.shields.io/badge/arXiv-2501.01108-%23b31b1b?logo=arxiv&link=https%3A%2F%2Farxiv.org%2F"></a> <a href='https://huggingface.co/OpenMuQ'><img alt="Static Badge" src="https://img.shields.io/badge/huggingface-OpenMuQ-%23FFD21E?logo=huggingface&link=https%3A%2F%2Fhuggingface.co%2FOpenMuQ"></a> <a href='https://pytorch.org/'><img alt="Static Badge" src="https://img.shields.io/badge/framework-PyTorch-%23EE4C2C?logo=pytorch"></a> <a href='https://pypi.org/project/muq'><img alt="Static Badge" src="https://img.shields.io/badge/pip%20install-muq-green?logo=PyPI&logoColor=white&link=https%3A%2F%2Fpypi.org%2Fproject%2Fmuq"></a> </div> This is the official repository for the paper *"**MuQ**: Self-Supervised **Mu**sic Representation Learning with Mel Residual Vector **Q**uantization"*. For more detailed information, we strongly recommend referring to https://github.com/tencent-ailab/MuQ and the [paper]((https://arxiv.org/abs/2501.01108)). In this repo, the following models are released: - **MuQ**(see [this link](https://huggingface.co/OpenMuQ/MuQ-large-msd-iter)): A large music foundation model pre-trained via Self-Supervised Learning (SSL), achieving SOTA in various MIR tasks. - **MuQ-MuLan**(see [this link](https://huggingface.co/OpenMuQ/MuQ-MuLan-large)): A music-text joint embedding model trained via contrastive learning, supporting both English and Chinese texts. ## Usage To begin with, please use pip to install the official `muq` lib, and ensure that your `python>=3.8`: ```bash pip3 install muq ``` Using **MuQ-MuLan** to extract the music and text embeddings and calculate the similarity: ```python import torch, librosa from muq import MuQMuLan # This will automatically fetch checkpoints from huggingface device = 'cuda' mulan = MuQMuLan.from_pretrained("OpenMuQ/MuQ-MuLan-large") mulan = mulan.to(device).eval() # Extract music embeddings wav, sr = librosa.load("path/to/music_audio.wav", sr = 24000) wavs = torch.tensor(wav).unsqueeze(0).to(device) with torch.no_grad(): audio_embeds = mulan(wavs = wavs) # Extract text embedding...

Source context: 3 downloads · 0 likes · Pipeline audio-classification · Repo PapaMoth/MuQ-MuLan-large