We recently had a production incident where a bad deployment went out at 2 AM. It took us 30 minutes to manually revert the changes. I want our CI/CD pipeline to detect a failure (like high 500 error rates) and automatically roll back to the previous stable version. What tools or metrics should I be looking at to automate this?
3 answers
Automated rollbacks require a tight integration between your deployment tool and your monitoring system (like Prometheus or Datadog). When you deploy, you should trigger a "Verification Phase." During this 5-10 minute window, the pipeline monitors specific Service Level Indicators (SLIs). If the error rate exceeds 1% or p99 latency spikes by 50%, the pipeline should automatically trigger a "Rollback Action." If you are using Kubernetes, this is built-in with kubectl rollout undo. For other systems, you might need a script that re-tags the "previous" Docker image as "latest" and redeploys. It’s better to have a false-positive rollback than a 30-minute outage.
Is it safer to use "Feature Flags" to kill a bad feature instead of rolling back the entire application build?
Always make sure your rollback script is tested just as much as your deployment script. You don't want to find out the rollback is broken during an actual crisis.
So true, Sandra. I’ve seen teams try to roll back only to realize the "previous" version wasn't actually saved in the registry!
That is actually the pro-level move, Kenneth. Using a tool like LaunchDarkly or a self-hosted flag system allows you to decouple "Deployment" from "Release." You deploy the code (it’s off), and then you toggle it on for 1% of users. If the metrics look bad, you just flip the switch off. The code stays in production, but it's not being executed. This is much faster and less risky than a full re-deployment, which involves shifting traffic and potentially restarting servers. It allows your "Rollback" to happen in milliseconds rather than minutes.