Our team is scaling up to use 70B parameter models and we are curious about the framework. Does it offer native support for tensor parallelism across multiple GPUs using Ray? We need a solution that can handle massive models without manually partitioning the weights across different cards during the deployment phase.
3 answers
Yes, it is designed from the ground up to support distributed inference via Ray. When you initialize the engine, you can specify the 'tensor_parallel_size' to match the number of GPUs you have available. The framework automatically handles the sharding of the model weights and manages the communication between the devices. This is particularly useful for models like Llama or Mixtral that exceed the memory capacity of a single A100 or H100. It abstracts away the complexity of torch.distributed, allowing you to focus on the application logic rather than the underlying hardware.
Does the performance scale linearly as you add more GPUs, or is there a significant communication overhead?
It works great. We used it to deploy a Falcon 180B model across a pod of 8 GPUs with very little configuration effort.
Ray as the backbone makes it incredibly stable for production. It’s a huge step up from trying to manually script the model sharding in basic PyTorch.
Scaling is quite efficient, though not perfectly linear due to the All-Reduce operations required for tensor parallelism. However, for large models, the bottleneck is usually memory bandwidth rather than communication. In my experience, going from 2 to 4 GPUs provides a massive jump in responsiveness for high-parameter models because of the increased total memory bandwidth available for the KV cache.