I'm building a text-to-speech (TTS) pipeline and I've reached the final stage: converting mel-spectrograms into actual audio. I’ve heard that older models like WaveNet are too slow for real-time use, and I'm looking at newer GAN-based or Diffusion-based options. Specifically, how do I implement a vocoder like HiFi-GAN or BigVGAN to ensure the audio doesn't sound "metallic" or have robotic artifacts? Also, what are the best practices for training a universal vocoder that works across different speakers and even singing voices?
3 answers
In 2026, the industry has largely shifted toward GAN-based architectures for real-time applications and Diffusion/Flow-matching for ultimate studio quality. To implement a neural vocoder effectively, you need to understand that its job is Upsampling: it takes a low-resolution mel-spectrogram (e.g., 80 bins) and reconstructs a high-resolution waveform (e.g., 22.05kHz or 44.1kHz).
For most real-time needs, HiFi-GAN is the baseline. It uses a Multi-Period Discriminator (MPD) and a Multi-Scale Discriminator (MSD) to look at different periodic patterns in the audio, which is why it handles human speech so well without the robotic artifacts of the past. If you need a "Universal Vocoder" that handles music or unseen speakers, BigVGAN is the current king—it introduces periodic activation functions (Snake) that provide the model with a better "inductive bias" for rhythmic and harmonic sounds.
Sarah’s spot on about the architectures. However, the most common implementation mistake is mismatched preprocessing. If your acoustic model (like FastSpeech 2) generates mel-spectrograms with a 256-hop length but your vocoder was trained with a 512-hop length, your audio will sound like high-pitched static or gibberish.
If you have the compute budget, I’d suggest looking into the new Flow-matching vocoders like WaveFM. They are much more stable to train than GANs (which can suffer from mode collapse) and can generate 44.1kHz "CD-quality" audio in a single inference step using consistency distillation.
I agree with Mark. Diffusion-based vocoders were too slow in 2024, but with Consistency Distillation, they’ve reached real-time parity with GANs while having fewer phase artifacts. One final tip: use Fréchet Audio Distance (FAD) instead of just listening tests to objectively measure how close your generated audio distribution is to your training data.
I second this! Always check your config.json. Also, for training, don't just rely on the standard L1 loss between spectrograms. You need a Multi-Resolution STFT loss. This forces the model to match the spectral convergence across various window sizes, which is what gives the audio that "crisp" high-end frequency. In my experience, adding a small amount of Gaussian noise during training also helps the vocoder generalize better to "imperfect" spectrograms produced by AI models.