There are so many new tools popping up. Is AgentOps the clear winner, or should we be looking at LangSmith or Arize Phoenix? What are the key features that make an observability tool "the Datadog" of this new space?
3 answers
The "winner" usually depends on your tech stack. If you are deeply integrated with LangChain, LangSmith is a natural fit. However, for framework-agnostic agents, AgentOps is gaining traction because of its focus on the "agentic" lifecycle rather than just the chain. The key features to look for are real-time session recording, multi-modal support, and the ability to link costs directly to specific agent actions. A "well-designed" observability tool should provide a unified view of performance, cost, and reliability without forcing you to change how you write your core agent logic.
Patrick here. Does the ability to "fine-tune" based on the logged data exist in any of these platforms yet?
I think the industry is still too young for a single "Datadog" to exist, but the competition is definitely driving rapid innovation.
Martha is right. It's a crowded space, but that just means we get better features every month. I'd recommend testing two or three with a small pilot project first.
Patrick, some platforms are beginning to offer "curation" features. You can flag high-quality interactions and export them as a JSONL file for fine-tuning. It’s a very powerful workflow because you are training your next model version on actual successful production traces rather than synthetic data.