ESPERNet

--- license: apache-2.0 pipeline_tag: audio-to-audio --- # ESPERNet ESPERNet is a set of AI models for audio-to-audio speech processing. The versions available here have been trained on the Mozilla CommonVoice dataset. **This model is still in development! Weights will be uploaded as soon as hyperparameter tuning and training are complete.** ESPERNet is built from three models: an encoder, a decoder, and a classifier. Together, they form a VLGAN architecture. ***(Lee, Je-Yeol & Choi, Sang-Il. (2020). Improvement of Learning Stability of Generative Adversarial Network Using Variational Learning. Applied Sciences. 10. 4528. 10.3390/app10134528. )*** The VAE latent space is split into two parts: a low-dimensional, time-dependent phoneme space and a larger, time-constant style-and-speaker space. This difference in dimensionality enables the encoder to separate speech content from speaker identity. ***(Xie, Yuying, et al. "Speaker and style disentanglement of speech based on contrastive predictive coding supported factorized variational autoencoder." 2024 32nd European Signal Processing Conference (EUSIPCO). IEEE, 2024.)*** Paired with the decoder, which benefits from adversarial training to produce high-quality outputs, the model can be used for speech transfer from one speaker to another. Unlike most audio-processing networks, ESPERNet uses the ESPER format instead of MEL spectrograms. That format was designed specifically for speech processing and enables high-quality speech reconstruction without an auxiliary network, as well as a range of effects, augmentations and transforms through [libESPER-V2](https://github.com/CdrSonan/libESPER-V2). Training and other related scripts are [available on GitHub](https://github.com/CdrSonan/ESPERNet).

Source context: 0 downloads · 0 likes · Pipeline audio-to-audio · Repo jklatt/ESPERNet