Soundstream Impl

--- library_name: transformers tags: - audio - soundstream license: mit language: - en pipeline_tag: audio-to-audio --- # SoundStream A PyTorch implementation of the [SoundStream](https://arxiv.org/abs/2107.03312) neural audio codec. Accepts only 16 kHz audio. Encodes speech into discrete tokens (8 codebooks × 80 tokens/sec) and decodes them back to audio. ## Metrics Evaluated on LibriSpeech test-clean: - **STOI**: 0.804 - **NISQA**: 2.276 ## Architecture - **Encoder**: Causal convolutions with residual units and strided downsampling (2, 4, 5, 5 = 200x compression) - **Quantizer**: Residual Vector Quantizer with 8 codebooks of 1024 entries each - **Decoder**: Mirrored encoder with transposed convolutions - **Discriminator** (training only): 3 multi-scale waveform discriminators + 1 STFT-based discriminator **Model parameters**: 16 kHz, 32 channels, latent dim 512, codebook size 1024, 8 quantizers, 200x downsampling ## Usage ```python import torchaudio from transformers import AutoModel # Load model model = AutoModel.from_pretrained("timofeiiz/soundstream-impl", trust_remote_code=True) model.eval() waveform, sr = torchaudio.load("audio.wav") assert sr == 16000 # Only 16 kHz sample rate is supported # Encode to discrete tokens indices = model.encode(waveform.unsqueeze(0)) # (1, 8, T) # Decode back to audio reconstructed = model.decode(indices, original_length=waveform.size(-1)) torchaudio.save("reconstructed.wav", reconstructed.squeeze(0).cpu(), 16000) ``` ## License MIT

Source context: 6 downloads · 0 likes · Pipeline audio-to-audio · Library transformers · Repo timofeiiz/soundstream-impl