We've just moved to a Kubernetes-based microservices setup, and manual monitoring is becoming impossible. We need an automation tool that can handle distributed tracing and provide real-time alerts when latency spikes between services. Should we go with Prometheus/Grafana or a full-stack APM like Datadog?
3 answers
The choice depends on your budget and how much "setup" work you want to do. Prometheus and Grafana are excellent, open-source, and the standard for Kubernetes metrics, but they require significant effort to set up distributed tracing (usually requiring Jeager or Tempo). If you have a small team and need "instant" visibility into your service mesh, a tool like Datadog or New Relic is superior. They offer auto-instrumentation that can trace a request from the frontend all the way through your backend services without you writing custom code for every endpoint. The automation of the "map" of your services is worth the cost if you're scaling fast.
Are you more concerned about the cost of data ingestion in Datadog, or the operational overhead of managing your own Prometheus scaling and long-term storage?
Don't forget about "Alertmanager." Automating your notifications so they only fire during actual anomalies (using 3-sigma rules) is the only way to prevent alert fatigue in a large cluster.
Absolutely, Alice. Alert fatigue is real. Fine-tuning those thresholds is just as important as the monitoring itself, otherwise, everyone just starts ignoring the Slack notifications.
Ryan, the cost is the real killer with Datadog. To answer your concern about overhead: we actually compromised by using "Managed Prometheus" from our cloud provider. It gave us the familiar open-source API but took away the headache of managing the underlying storage nodes. We then coupled it with "OpenTelemetry" for our tracing. This approach is "vendor-neutral," meaning if we ever want to switch tools, we don't have to rewrite our instrumentation, which is the ultimate form of future-proofing our automation strategy.