Our monthly cloud bill for GKE and EKS is growing 20% faster than our user base. Between over-provisioned pods and idle "zombie" namespaces, we are bleeding money. Is anyone successfully implementing FinOps for Kubernetes? Specifically, are you using Kubecost or CAST AI to automate rightsizing, and how do you convince developers to actually lower their resource requests?
3 answers
We made "Cost Visibility" a part of our engineering culture. We use Kubecost to send a weekly "Efficiency Report" to every team lead showing exactly how much they spent on idle resources. This "showback" model worked wonders—once developers saw their names next to a $2,000 waste figure, they were much more willing to tune their HPA (Horizontal Pod Autoscaler) settings. We also automated the deletion of any namespace with the tag 'temporary' that hasn't seen a deployment in 48 hours. This alone saved us $5,000 a month in our dev environment.
Patricia, do you find that developers start "under-provisioning" and causing OOMKilled errors just to keep their cost reports looking good?
We moved all of our non-critical batch jobs to Spot Instances using an orchestrator. It cut our compute costs by almost 60% with very little impact on reliability.
Spot instances are the ultimate "cheat code" for cost. As long as you have good Pod Disruption Budgets (PDBs), there’s almost no reason not to use them for dev/test.
Brian, that was a concern initially! We countered it by using Vertical Pod Autoscaler (VPA) in "Recommendation mode." We tell devs they don't have to guess; they just have to accept the VPA's recommendation. Since the VPA is data-driven, it provides enough headroom to avoid OOM issues while still cutting the fat that human "guessing" usually leaves behind.