We want to integrate AI into our app, but the API costs for the most popular generative AI models are eating our budget. Should we try hosting open-source versions like Llama 3 locally, or is the infrastructure overhead even more expensive than just paying the subscription fees?
3 answers
For a startup, I recommend starting with quantized versions of open-source generative AI models. You can run a 7B or 13B parameter model on a single high-end GPU, which significantly reduces per-token costs compared to proprietary APIs. However, don't overlook the "hidden" costs of engineering time for setup and maintenance. If your traffic is sporadic, serverless inference providers might be the best middle ground. They allow you to pay only for the compute time used while giving you the flexibility of open-source models without the headache of managing raw hardware.
Have you calculated your "break-even" point where the monthly cloud GPU rental becomes cheaper than the token costs for proprietary generative AI models?
I suggest using a multi-model approach where you use smaller, cheaper generative AI models for easy tasks and only call the expensive ones for complex logic.
This "cascading" strategy is brilliant, Danielle. It drastically cuts costs while maintaining high quality for the most important user interactions.
We are at about 2 million tokens a day right now. At this volume, I think a dedicated instance might actually save us a few hundred dollars a month.