I want to make our deployment process more resilient. Currently, if a production release fails, we have to manually trigger a redeploy of the previous version. I’ve heard about using "Deployment Gates" and "Release Triggers" for automated rollbacks in Azure DevOps. Can someone explain how to configure a gate that monitors Azure Monitor alerts and triggers a revert if it detects high error rates?
3 answers
Automated rollbacks are best handled using a combination of 'Post-deployment gates' and the 'Auto-redeploy' feature in the Release Pipeline. You can set up a gate to query Azure Monitor alerts. If the alerts stay active for a 5-minute sampling period, the gate fails. In the 'Post-deployment' settings, you can then enable the option to "Auto-redeploy last successful release" upon failure. This ensures that your production environment stays stable without manual intervention. Just be careful with your alert thresholds to avoid "flapping" rollbacks triggered by transient network blips.
Does your application support "Blue-Green" deployment or "Canary" releases? If you are using Azure App Service, using deployment slots might be a cleaner way to handle rollbacks than a full pipeline revert. Wouldn't it be easier to just swap the slots back if the health check fails in the production slot?
You should look into "Post-deployment approvals" combined with the gate. It gives you a human safety net before the automated rollback triggers, just in case the alert is a false positive.
I agree with Elizabeth. Sometimes the "high error rate" is just a spike during startup. Having a 10-minute delay or a manual "abort rollback" button is a lifesaver in real-world production.
Using slots is definitely faster for web apps, William. However, for our microservices running on AKS, we need a more robust logic. I’m trying to figure out if Azure Pipelines can natively handle the 'kubectl rollout undo' command as part of a failure hook in the YAML definition, or if I have to script that logic manually within a 'deployment' job type under the 'on: failure:' hook.