We are building a creative writing assistant and need fine-grained control over generation. How does the server handle sampling parameters like frequency penalty, presence penalty, and top-p? Are these passed through the standard API, and do they affect the batching efficiency of the PagedAttention engine during high traffic?
3 answers
The engine supports a wide array of sampling parameters that are fully compatible with the OpenAI API format. You can define a 'SamplingParams' object containing temperature, top_p, presence_penalty, and more. One of the best things about this specific architecture is that it uses "continuous batching." Unlike static batching where every request in a batch must finish before a new one starts, this engine allows new requests to join the batch as soon as a slot opens up, regardless of the sampling complexity of individual requests.
If I use a very high temperature for one request, does it slow down the deterministic requests in the same batch?
The SamplingParams class is very well documented in the Python SDK. It makes the transition from OpenAI very simple.
I found that the frequency penalty works even better here than in some other libraries because of how the logits are handled in the PagedAttention kernels.
No, it doesn't. The engine computes the logits for the entire batch in parallel, and then applies the individual sampling logic per sequence. The actual overhead of the sampling calculation is negligible compared to the forward pass of the model. So, you can have a mix of diverse creative outputs and strict deterministic ones without any performance cross-talk between them.