Debugging in a distributed environment is proving to be a nightmare for our DevOps team. We need recommendations for a tech stack that provides deep visibility into request flows across multiple services. What are you using for distributed tracing, log aggregation, and real-time monitoring?
3 answers
The gold standard right now is the LGTM stack (Loki, Grafana, Tempo, Mimir). For tracing specifically, OpenTelemetry is a must-adopt standard because it prevents vendor lock-in. We implemented Jaeger for tracing in late 2023 and it transformed how we root-cause performance bottlenecks. You need a way to correlate your logs with your traces; otherwise, you're just looking at disconnected data points. Make sure your developers are consistently passing trace IDs through every single header in your inter-service calls.
How do you manage the storage costs associated with high-cardinality data in Prometheus? We found that tracking every single request detail started to bloat our cloud storage bill quite rapidly.
We use ELK stack for logging and Prometheus for metrics. It’s a classic combo that works well if you have a dedicated team to manage the Elasticsearch cluster.
Agreed, Laura. While ELK is heavy, its search capabilities are still superior to almost everything else when you need to dig deep into raw text logs during an outage.
Steven, the trick is to use sampling. You don't need to trace 100% of successful requests. Trace all errors, but maybe only 5% of successful calls. Also, use Prometheus recording rules to pre-aggregate data before storage.