Realtime REUSE Whisper Finetune

--- license: other track_downloads: true pipeline_tag: audio-to-audio library_name: mamba-ssm tags: - speech-enhancement - universal-speech-enhancement - real-time-speech-enhancement - whisper-finetuned - multiple-input-sampling-rates - language-agnostic --- # Realtime RE-USE — Whisper Finetune A **fine-tuned** version of [NVIDIA Real-time RE-USE](https://huggingface.co/nvidia/Real-time_RE-USE) optimised for enhancing whispered speech and noisy recordings in ASR pipelines. ## Usage ```python import torch from Realtime_REUSE.models.streaming_generator_SEMamba_time_d1_random_layer_ahead_sep_conv import ( SEMamba_decoder_list, ) model = SEMamba_decoder_list.from_pretrained("{repo_id}") model.eval() ``` Or via the CLI: ```bash python -m scripts.inference_realtime_reuse \ --input ./noisy_audio/ \ --output ./enhanced_audio/ \ --checkpoint {repo_id}/pytorch_model.bin ``` ## Model Architecture | Property | Value | | --- | --- | | Base model | NVIDIA Real-time RE-USE (SEMamba, 12 layers, 3.7 M params) | | Architecture | Conv encoder + Mamba blocks + Conv decoders | | Fine-tuning data | Whisper-format speech (normal + whispered + background noise) | ## Training Details - **Base checkpoint**: NVIDIA Real-time RE-USE `USEMamba_12x1_lr_00002_norm_05_vq_067_nfft_320_hop_160_NRIR_012_pha_0005_com_04_early_005_release_random_layer_GAN_longer_1k` - **Optimiser**: Optuna-tuned learning rate, early stopping - **Exit layer**: 8 (configurable at inference time) ## Input / Output - **Input**: Mono-channel audio, 8 kHz – 48 kHz (.wav, .flac) - **Output**: Enhanced mono-channel audio (.flac) ## License This model is derived from NVIDIA Real-time RE-USE, released under the [NVIDIA One-Way Noncommercial License (NSCLv1)](https://github.com/NVlabs/HMAR/blob/main/LICENSE). ## References - [Rethinking Training Targets, Architectures and Data Quality for Universal Speech Enhancement](https://arxiv.org/abs/2603.02641) - [One Model, Many Latencies: Universal Speech Enhancement for Diverse Real-Time Applications](https://arxiv.org/abs/2606.25621)

Source context: 10 downloads · 1 likes · Pipeline audio-to-audio · Library mamba-ssm · Repo Malfaro43/Realtime-REUSE-whisper-finetune

Source context: 402 downloads · 0 likes · Pipeline audio-to-audio · Library mamba-ssm · Repo Malfaro43/Realtime-REUSE-whisper-finetune