Our team has spent years perfecting our training pipelines, but now latency is our biggest headache. It feels like the industry is hitting diminishing returns on pre-training scaling laws. Are other MLOps professionals seeing a shift where the "production" side of machine learning is now more complex and vital than the actual model development?
3 answers
We transitioned our focus to inference optimization about a year ago. The reality is that training happens in a controlled, batch-oriented environment, but inference has to survive the "wild" with unpredictable traffic and strict P99 latency requirements. We’ve found that techniques like quantization and KV caching are now more critical for our bottom line than finding a slightly better learning rate. The industry is moving from "how do we teach it" to "how do we serve it at scale" without going bankrupt on GPU costs.
Are you finding that specific hardware, like NPUs or custom ASICs, is making a bigger difference than software-level optimizations?
Definitely. Training is the prep work, but inference is the actual game day where performance counts for the business.
Exactly, and with more companies using pre-trained models, the competitive advantage is moving to who can run them most efficiently.
Hardware is huge, Brandon, but software optimizations like speculative decoding are currently the real game-changers for LLMs. They allow us to get much higher throughput on existing hardware, which is vital since the demand for real-time tokens is growing exponentially compared to our training schedules.