There is a lot of hype about AI-driven monitoring tools that claim to predict outages. Does this actually work in a real-world DevOps & SRE setting, or is it just another layer of noisy alerts that our engineers have to filter through manually?
3 answers
AIOps is best at "alert correlation," not necessarily "prediction." In 2023, we used an ML-based tool to group 500 individual alerts into a single incident. This saved our on-call person from getting 500 Slack notifications for the same database lag. While it hasn't replaced our DevOps & SRE experts, it has significantly reduced the cognitive load. You still need a human to verify the "root cause," but the AI helps you find the needle in the haystack much faster than a manual search through logs.
Have you checked if your current logging data is clean enough for an AI model to actually learn from, or is there too much "garbage" in your telemetry?
Don't let the AI take actions yet. Use it for "suggestions" first. Let the engineer click a button to execute the AI's proposed fix until you build up enough trust.
This "human-in-the-loop" approach Larry mentions is the safest way to transition. It builds confidence in the tool without risking an automated catastrophe.
Edward, that is exactly our concern. Our current logs are quite messy and lack a consistent schema across different microservices. We are worried that if we feed poor quality data into an AIOps tool, we'll just end up with "automated hallucinations" rather than useful insights. We're currently working on a log standardization project to ensure that every service outputs structured JSON with a common set of required fields.