I’ve successfully trained several independent adapters for sentiment analysis, NER, and question answering using a frozen BERT backbone. Now, I want to leverage all of them simultaneously using AdapterFusion to see if the shared knowledge improves my target task performance. I’m a bit confused about the "two-stage" learning process—specifically, how the fusion layer weights are learned without causing catastrophic forgetting or interference between the pre-trained adapters. Has anyone successfully implemented this in a production environment?
3 answers
In my workflow, I use AdapterFusion to combine a general language adapter with a domain-specific one (like medical text). It’s much more efficient than full fine-tuning.
Implementing AdapterFusion is actually a very elegant way to handle multi-tasking. The key is understanding that it’s a non-destructive composition. In the first stage, you train your adapters (like your NER and QA ones) independently. In the second stage, you freeze both the pre-trained model and those adapters, and only train the "Fusion" layer. This layer uses an attention mechanism where the query is the hidden state of the transformer, and the keys/values are the outputs of your task-specific adapters. This allows the model to dynamically decide which adapter is most relevant for a specific input at each layer. I found that using the AdapterHub library simplifies this greatly since they have built-in classes for the Fuse composition block, which handles the key-query-value mapping automatically.
That makes sense for the setup, but how do you handle the data balancing during that second stage? If you are fusing three adapters but your target task dataset is much smaller than the others, doesn't the fusion layer just end up heavily favoring the most "over-trained" adapter?
That’s a common pitfall, Karen. To avoid this, you should use a separate "fusion dataset" or a balanced subset of all your tasks to train those fusion weights. From my experience, the fusion layer is remarkably good at identifying linguistic features; for example, if an input looks like a question, it will naturally weigh the QA adapter higher even if the sentiment adapter was trained on more data. Just ensure your learning rate for the fusion parameters is small, typically around 5e-5, to keep the training stable.
I agree with Brian. The modularity is the best part. I’ve even seen cases where you can swap out one of the fused adapters later for a better-performing one without having to retrain the entire fusion stack from scratch, which is a huge time-saver for DevOps.