We are seeing massive investments in chips that literally cannot train models but are elite at tasks. Does this mean we will eventually see separate "Training Clouds" and "Serving Clouds" in the next few years?
3 answers
This split is already happening. Large providers are building "inference clusters" that use high-bandwidth memory but lower compute power compared to the H100 training monsters. It makes sense from a utility perspective; you don't need a Ferrari to drive to the grocery store. By separating the two, providers can offer "Inference-as-a-Service" at much lower prices because the hardware is cheaper to run and cool. This specialization will likely lead to a marketplace where you train on one cloud but deploy your weights on a hyper-optimized "serving" cloud.
Will this specialization make it harder for small startups to switch providers if their weights are optimized for one specific type of inference chip?
Efficiency is king. If inference-only chips can cut my monthly API bill in half, most CTOs won't care about the architecture behind the curtain.
Exactly, Jeffrey. In our current digital marketing stack, the only thing that matters is the cost-per-thousand-tokens, and specialized hardware is the only way to drive that down.
Franklin, that is a major concern. "Vendor lock-in" could move from the software layer to the hardware layer. If your model is compiled specifically for a proprietary inference engine's instruction set, moving it to a general GPU cloud might result in a massive performance hit. Open standards will be vital here.