I'm looking for ways to automate our model evaluation. Can I use vLLM as a backend to run automated benchmarks every time we update our fine-tuned weights? I need a way to programmatically capture throughput and latency metrics to ensure no performance regressions occur before we push to production.
3 answers
This is a great idea for a MLOps pipeline. You can definitely run the server in a headless Docker container as part of a Jenkins or GitHub Actions runner. There is a benchmark_throughput.py script included in the official repository that is perfect for this. It allows you to simulate various request rates and record the results in JSON format. We use it to compare every new iteration of our internal models against a baseline. If the tokens-per-second drops by more than 5%, the build fails. It has helped us catch several inefficient tokenization issues before they hit the customers.
Does the benchmarking script support realistic workloads, like varying prompt lengths, or does it just use a static input for the entire test?
We use it with Prometheus and Grafana. The server exports metrics natively, so you can track performance in real-time during your stress tests.
The Prometheus integration is vital. Visualizing the memory usage during those automated tests gives much more insight than just a simple pass/fail log
You can provide a dataset file (like ShareGPT) to the script. This makes the benchmark much more realistic because it processes prompts of different lengths, which tests the PagedAttention's ability to handle dynamic memory allocation effectively.