I have a sentiment analysis model trained in Python, and I want to deploy it as part of a Kafka Streams topology to process incoming tweets in real-time. Should I wrap the model in a REST API and call it from the stream, or is it possible to embed the model directly using ONNX or TensorFlow Java? What are the pros and cons for scaling?
3 answers
For high-throughput AI pipelines, embedding the model is almost always better. If you call a REST API for every message in Kafka, you'll create a massive bottleneck and introduce network jitter. By using the ONNX Runtime Java API or TensorFlow Java, you can load the model directly into the Kafka Streams processor. This allows for "in-process" inference, which is significantly faster. The downside is that your Kafka pods will require much more memory and CPU (especially if using GPUs), and model updates require a redeployment of the streaming application rather than just updating an API endpoint.
If you embed the model, how do you handle A/B testing? Can you run two different versions of a model on the same Kafka topic and compare the outputs programmatically?
We tried the REST API approach initially but hit severe latency issues. Switching to an embedded ONNX model reduced our processing time from 50ms per message to under 4ms.
Those numbers speak for themselves, Laura. In the world of real-time AI, every millisecond saved in the pipeline is more time available for complex model computation.
Jeremy, you can achieve this by using a "Branch" or "Processor" in Kafka Streams. You can send the same input stream to two different model beans within your topology. Each model writes its prediction to a separate "evaluation" topic along with a unique correlation ID. Then, a downstream consumer can aggregate those results to compare accuracy. It’s more complex than switching an API flag, but it gives you incredibly detailed real-time performance data without any external network overhead.