We are about to deploy a fine-tuned LLM for our customer support portal, but we are terrified of the model generating biased or inappropriate responses. What are the best industry practices for "Alignment" and safety filtering? Are there specific datasets or Reinforcement Learning from Human Feedback (RLHF) techniques that a medium-sized company can realistically implement without a massive budget?
3 answers
Alignment is a major challenge. I recommend starting with "Constitutional AI" principles—giving the model a set of rules it must follow during the fine-tuning phase. For a medium budget, you don't necessarily need full RLHF. You can use "Supervised Fine-Tuning" (SFT) on a high-quality, curated dataset of "gold standard" customer interactions. Additionally, implement a "guardrail" layer using a secondary, smaller model that checks the output for toxicity before it ever reaches the customer. Tools like NeMo Guardrails can help you set up these boundaries effectively without needing a team of PhDs to manage it.
Are you planning to use a closed-source model like OpenAI's API which has built-in filters, or are you running an open-source model like Llama-3 on your own private servers?
You should definitely look into "Red Teaming" your model. Have your team intentionally try to break it or trick it into saying something bad before you go live.
Red teaming is essential, Thomas. It’s the only way to find those weird edge cases where the model might fail in a way you never expected during standard testing.
James, we are using Llama-3 for data privacy reasons, so we are responsible for all the filtering ourselves. That's why I'm so focused on finding robust open-source libraries. We need to make sure the model doesn't accidentally hallucinate fake return policies or use language that could be seen as discriminatory. Michelle’s suggestion about NeMo Guardrails seems like exactly the kind of middleware we were looking for to sit between the model and the user.