I’m working on a speech recognition (ASR) system for a low-resource language with very little transcribed audio. I’ve seen that wav2vec 2.0 can theoretically work with just 10 minutes of data, but my initial results are full of "hallucinated" words and high Word Error Rates (WER).
I'm currently using the facebook/wav2vec2-base checkpoint. Should I be freezing more layers during fine-tuning? Does Data Augmentation (like SpecAugment) help or hurt when the dataset is this small? Also, has anyone successfully used pseudo-labeling or Self-Training to bridge the gap? I need a strategy that doesn't just overfit to the 100 sentences I actually have labeled.
3 answers
Kimberly is spot on about XLSR. However, if you are truly at the 10-minute mark, you need to implement Partial Layer Freezing.
In 2026, the consensus for "Low-Resource ASR" is that Pre-training is everything. If your labeled data is under 1 hour, you shouldn't start with a monolingual English model. Instead, use a Multilingual checkpoint (like wav2vec2-large-xlsr-53). These models have already learned a universal "phonetic alphabet" from dozens of languages, making it much easier to map your specific language's sounds to text.
The "Golden Rule" for fine-tuning with limited data: Freeze the Feature Encoder. The CNN layers that extract raw features are usually robust enough from pre-training. By freezing them, you focus your limited gradients on the Transformer layers and the Language Modeling head, which prevents the model from "forgetting" how to hear basic audio patterns while trying to learn your specific vocabulary.
Don't overlook the power of Self-Training (Pseudo-Labeling). If you have 10 minutes of labeled data but 10 hours of unlabeled audio in the same language, you can use a "Teacher-Student" setup.
Exactly, Steven. We use our 10-minute model (the Teacher) to transcribe the 10 hours of unlabeled audio. We then filter for "High Confidence" transcriptions and use those as pseudo-labels to train a second model (the Student). In 2026, we also integrate a 4-gram or Transformer-based Language Model during decoding. This can lower your WER by 10-15% simply by "correcting" the acoustic model's spelling based on common language patterns it hasn't seen in the training data yet.
Research shows that in wav2vec 2.0, the lower transformer layers capture acoustics, while the top layers capture language/semantics. When data is scarce, try freezing the first 6–8 transformer blocks and only fine-tune the top 4. This acts as a powerful regularizer. Also, SpecAugment is your best friend—masking time-steps and frequency channels during training forces the model to learn the "context" of a word rather than just memorizing a specific audio snippet. Without it, a 10-minute dataset will overfit in less than 50 epochs.