We are seeing a trend where companies stop retraining their models and instead use time retrieval (RAG) to provide context. Does this mean the era of massive fine-tuning is over, or is there still a specific need for training that RAG just can't touch yet?
3 answers
RAG is definitely the smarter play for 90% of enterprises because it allows the model to stay "current" without a single dollar spent on GPU training hours. However, retraining or fine-tuning is still necessary for changing the "behavior" or the "style" of the model. If you need a model to follow a very specific medical or legal jargon format that it wasn't exposed to during the base training, RAG won't help as much as a targeted fine-tune. That said, the industry is moving toward a "Small Base + Big RAG" architecture because it is far more sustainable.
Don't you think the increased context window size in newer models will eventually make the vector database part of RAG redundant?
Fine-tuning is for skills; RAG is for facts. Most businesses just need the facts, so they are pivoting hard toward optimizing the retrieval process.
Totally agree. Our recent implementation showed that refining the search query before inference improved accuracy more than any fine-tuning ever did.
Austin, while a 1M token window is impressive, feeding that much data into every prompt makes the inference cost explode. Vector databases allow us to be surgical, only pulling in the 500 relevant tokens. It’s not just about what the model can "see," it’s about what you can afford to have it "read" every time.