We are worried about the "Time to First Token" (TTFT) for our new AI assistant. Can the library handle streaming responses to the frontend while simultaneously processing new incoming prompts? We need a very low latency experience for our users, and I'm wondering if the engine's scheduling logic supports this without lag.
3 answers
Streaming is one of the strongest features of the OpenAI-compatible server provided by the library. It uses an asynchronous generator to yield tokens as they are produced by the GPU kernels. Because of the continuous batching mechanism, the TTFT is significantly lower than other frameworks because the engine doesn't wait for a full batch to be ready. It can start prefilling a new prompt while the previous ones are still in the decoding phase. For a typical 7B model, we are seeing first-token latencies under 50ms, which feels instantaneous to a human user.
Does the streaming performance degrade when the server is at 90% GPU utilization?
We implemented the streaming API with a React frontend and the user experience is incredibly smooth.
The Server-Sent Events (SSE) implementation is very standard, so it works perfectly with most modern JavaScript libraries without needing custom wrappers.
While any system slows down under extreme load, the scheduling logic here is very fair. It uses a First-Come-First-Served approach for the prefill stage, ensuring that new users aren't left waiting too long even if the decoding queue is long. The PagedAttention memory management ensures that we don't run out of memory during these peaks, so instead of crashing, the system gracefully handles the throughput by slightly increasing the latency for everyone equally.