Our team is building an internal assistant for our Cloud Technology stack. We have thousands of pages of proprietary documentation that changes weekly. Should we go with a RAG approach to ensure the model always sees the latest docs, or is fine-tuning better for understanding the complex relationships between our microservices?
3 answers
For technical documentation that changes weekly, RAG is the clear winner. Fine-tuning a model every week would be an absolute nightmare in terms of compute costs and version control. With a retrieval setup, you just need a script that updates your vector index whenever a document is edited. This ensures that when an engineer asks about a new API parameter released yesterday, the model actually finds that specific line in the docs. Fine-tuning is really only necessary if the model struggles to understand your specific coding syntax or internal jargon.
Brenda makes a strong case for retrieval, but what if the documentation is highly fragmented? Can RAG actually connect the dots between five different files to solve a complex architectural problem?
We tried both and found that a hybrid approach worked best. We fine-tuned for our specific nomenclature and used RAG for the actual factual retrieval of data.
I’ve heard many companies are moving to that hybrid model, Katherine. It gives you the "vibe" and language of your company with the surgical accuracy of a search engine.
Ronald, that's where "Agentic RAG" comes in. Instead of a simple search, you use an agent that can perform multiple searches and synthesize the findings. For complex cloud architectures, the agent might look up the load balancer config first, then the security group rules, and finally the VPC peering docs. It mimics how a human architect would jump between different tabs in a browser. It’s a bit more complex to build, but it solves the fragmentation issue effectively for specialized cloud environments.