Mlx Rmvpe

--- license: mit tags: - mlx - audio - pitch-estimation - f0 - voice-conversion - rvc - apple-silicon library_name: mlx pipeline_tag: audio-to-audio --- # MLX-RMVPE MLX implementation of [RMVPE](https://arxiv.org/abs/2306.15412) (Robust Model for Vocal Pitch Estimation) for Apple Silicon. ## Model Description RMVPE extracts **fundamental frequency (F0)** from audio, essential for preserving pitch/melody in voice conversion. Unlike simpler methods (CREPE, pYIN), RMVPE is specifically designed for **polyphonic music**, making it ideal for singing voice conversion where background music may be present. - **Architecture**: Deep U-Net with BiGRU layers - **Parameters**: ~15.4M - **Input**: 16kHz audio - **Output**: F0 in Hz at 100fps (hop_length=160) - **Pitch range**: ~32 Hz to ~1975 Hz (360 bins) ## Usage ```bash pip install mlx-rmvpe ``` ```python import librosa from mlx_rmvpe import RMVPE # Load model (auto-downloads weights) model = RMVPE.from_pretrained() # Load audio at 16kHz audio, sr = librosa.load("singing.wav", sr=16000, mono=True) # Extract F0 f0 = model.infer_from_audio(audio) print(f"F0 shape: {f0.shape} at 100fps") print(f"Pitch range: {f0[f0 > 0].min():.1f} - {f0[f0 > 0].max():.1f} Hz") ``` ## Manual Loading ```python from huggingface_hub import hf_hub_download from mlx_rmvpe import RMVPE weights_path = hf_hub_download( repo_id="lexandstuff/mlx-rmvpe", filename="rmvpe.safetensors" ) model = RMVPE() model.load_weights(weights_path) model.eval() ``` ## Technical Details This implementation is converted from the PyTorch weights and produces numerically similar outputs: | Metric | Value | |--------|-------| | Mean F0 difference | 1.29 Hz | | Correlation | >0.99 | See the [GitHub repository](https://github.com/lexandstuff/mlx-rmvpe) for implementation details and the full API reference. ## Citation ```bibtex @inproceedings{wei2023rmvpe, title={RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Music}, author={Wei, Yongmao and others}, booktitle={ISMIR}, year={2023} } ``` ## License MIT ## Acknowledgments - [RMVPE](https://github.com/Dream-High/RMVPE) - Original implementation - [RVC](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI) - Voice conversion pipeline - [MLX](https://github.com/ml-explore/mlx) - Apple's machine learning framework

Source context: 0 downloads · 0 likes · Pipeline audio-to-audio · Library mlx · Repo lexandstuff/mlx-rmvpe