SE NAC

--- license: mit datasets: - westbrook/LibriMix language: - en tags: - speech - SE - Neural-Audio-Codec pipeline_tag: audio-to-audio --- # Modeling strategies for speech enhancement in the latent space of a neural audio codec This repository provides the official model checkpoints for the paper *[Modeling strategies for speech enhancement in the latent space of a neural audio codec](https://arxiv.org/abs/2510.26299)* authored by Sofiene Kammoun, Xavier Alameda-Pineda, and Simon Leglaive, and published at IEEE ICASSP 2026. We explore different modeling strategies (autoregressive vs. non-autoregressive) and representation spaces (discrete vs. continuous) for speech enhancement using neural audio codecs and Conformer-based architectures. [arXiv](https://arxiv.org/abs/2510.26299) | [Code and Audio examples](https://sofienekammoun.github.io/SE-NAC-25/) | [Bibtex](#citation) ## Overview Our work introduces and compares a family of speech enhancement models that systematically vary along two main axes: - **Representation Type** - Discrete tokens - Continuous latent vectors - **Modeling Strategy** - Autoregressive (AR): Sequential prediction of clean speech representation - Non-Autoregressive (NAR): Parallel prediction of clean speech representation The current release includes the following models: | Model Name | Modeling Strategy | Input Representation | Output Representation | Model Checkpoint | |-------------|------|----------------|----------------|----------------| | **D-AR** | Autoregressive | Discrete |Discrete | `D-AR_ckpt_300.pt` | | **D-NAR** | Non-Autoregressive | Discrete |Discrete | `D-NAR_ckpt_300.pt` | | **D-NAR*** | Non-Autoregressive | Continuous |Discrete | `D-NAR_star_ckpt_300.pt` | | **C-AR** | Autoregressive | Continuous | Continuous | `C-AR_ckpt_300.pt` | | **C-NAR** | Non-Autoregressive | Continuous | Continuous | `C-NAR_ckpt_300.pt` | Additional models: - **C-FT** (`C-FT-encoder_ckpt_300.pt`) and **D-FT** (`D-FT-encoder_ckpt_300.pt`), where we only finetune the NAC's encoder with an MSE loss and a cross-entropy loss, respectively. - **STFT-NAR** (`STFT_NAR_Mask_ckpt_300.pt`), where instead of the embeddings of the NAC, we work with STFT representations, and we train the model to output an STFT mask.

Source context: 0 downloads · 2 likes · Pipeline audio-to-audio · Repo SofieneK/SE-NAC

Source context: 0 downloads · 1 likes · Pipeline audio-to-audio · Repo SofieneK/SE-NAC