3 answers
Have you already established a "steady state" for your application metrics so you can actually tell when a chaos experiment has caused a significant deviation?
Start small with network latency injection. It’s the most common cause of distributed system failure and usually exposes bugs in your retry logic very quickly.
I agree with Thomas. Simulating slow dependencies is often much more revealing than just killing a process, as it tests your timeouts and circuit breakers.
It isn't "mandatory" for everyone, but for high-scale systems, it is the only way to find "unknown unknowns." We started with simple "Game Days" in 2023 where we manually shut down a node during office hours. It taught us more about our recovery time (MTTR) than any documentation ever could. A true DevOps & SRE culture thrives on this because it proves that your monitoring and alerting actually work under stress, rather than just assuming they will work when a real disaster strikes at 3 AM.
Steven, that's a great point. We have baseline metrics for latency and throughput, but we haven't formally defined a "steady state hypothesis" for our experiments yet. I suppose we should probably start by defining exactly what "normal" looks like under various load conditions before we start injecting faults, otherwise, we won't know if the system is actually self-healing or just struggling to stay afloat.