We are running Kafka on a K8s cluster using the Strimzi operator. Our producers are currently hitting a bottleneck and we aren't seeing the throughput we expected. Should we increase the batch size or adjust the linger.ms settings to better utilize the network bandwidth in our cloud environment?
3 answers
Increasing the batch size worked for us. We also found that using the LZ4 compression codec gave us the best balance between CPU usage and throughput.
In a Kubernetes environment, network overhead is a factor. To increase throughput, try increasing batch.size to 64KB or even 128KB and set linger.ms to 10-20ms. This forces the producer to wait slightly longer to group more messages into a single request, reducing the number of I/O operations. You should also check your buffer.memory to ensure the producer doesn't block when sending high volumes. Also, ensure your K8s nodes have sufficient CPU resources, as compression (like Snappy or LZ4) is CPU-intensive but greatly improves throughput by reducing payload size.
Are you monitoring your 'request-latency-avg' and 'produce-throttle-time-max' metrics to see if the brokers are actually the ones slowing down the producers?
That's a smart check, Charles. If the throttle time is high, it means the brokers are hitting quota limits or disk I/O caps. You might need to scale your brokers or upgrade your cloud storage volumes.
I’ve had the same experience, Mary. LZ4 is incredibly fast for the amount of compression it provides, especially with typical JSON or string data.