Piano Vocoder

--- tags: - audio - hifigan - vocoder - speech-synthesis - piano - music - generated-audio license: apache-2.0 --- # Piano-Vocoder A HiFi-GAN vocoder trained from scratch on piano music, designed for audio reconstruction tasks. **Training**: Trained from scratch (random initialization) - NOT fine-tuned from pretrained ## Model Overview - **Architecture**: HiFi-GAN (Mel-spectrogram vocoder) - **Dataset**: Piano music from YouTube - **Best Validation Mel Error**: 0.2267 (step 34000) - **Training Steps**: 66,000+ steps ## Quick Start ### Requirements ```bash pip install torch torchaudio librosa scipy ``` ### Inference Example ```python import torch import torchaudio from scipy.io.wavfile import write # Load model from env import AttrDict from models import Generator import json config = AttrDict(json.load(open('config.json'))) generator = Generator(config).cuda() checkpoint = torch.load('generator_best.pt', map_location='cuda') generator.load_state_dict(checkpoint['generator']) generator.eval() generator.remove_weight_norm() # Load and process audio wav, sr = torchaudio.load('your_audio.wav') if sr != 22050: wav = torchaudio.functional.resample(wav, sr, 22050) wav = wav.mean(dim=0) # stereo to mono # Generate mel spectrogram from meldataset import mel_spectrogram mel = mel_spectrogram(wav.unsqueeze(0).cuda(), config.n_fft, config.num_mels, config.sampling_rate, config.hop_size, config.win_size, config.fmin, config.fmax) # Generate audio with torch.no_grad(): generated = generator(mel) audio = generated.squeeze().cpu().numpy() # Save write('output.wav', 22050, (audio * 32768).astype('int16')) ``` ## Training Details ### Data - Piano music from YouTube - 15-second chunks at 22050 Hz mono ### Architecture ```json { "upsample_rates": [8, 8, 2, 2], "upsample_initial_channel": 512, "resblock_kernel_sizes": [3, 7, 11], "num_mels": 80, "n_fft": 1024, "hop_size": 256, "sampling_rate": 22050 } ``` ### Training Config - Batch size: 8 - Learning rate: 0.0002 - Adam betas: (0.8, 0.99) - LR decay: 0.999 per epoch - Checkpoint interval: 500 steps ## Limitations - Trained on piano music only - May not generalize to other instruments or voice - Not fine-tuned from pretrained - started from random init ## Files - `config.json` - Model architecture configuration - `generator_best.pt` - Generator checkpoint (best validation error at step 34000)...