We are moving our microservices architecture to Kubernetes and want to implement Blue-Green deployments to minimize downtime. However, I am confused about the rollback strategy if a bug is detected post-deployment. If the 'Green' environment is live and fails, what is the fastest way to point the traffic back to 'Blue' without causing a service interruption? Also, how do you handle database schema changes during this process, as the DB is usually shared between both environments?
3 answers
In Kubernetes, the fastest way to handle a Blue-Green rollback is by updating the Service selector. Your 'Service' object points to a specific set of Pod labels. To roll back, you simply use kubectl apply or a Helm chart update to change the selector back to the 'Blue' version labels. This change happens almost instantaneously at the networking layer. Regarding databases, the industry standard is "Expand and Contract" migration. First, apply schema changes that are backward compatible with the old code. Once the Green version is stable, you can then remove the old, unused columns or tables.
Are you currently using an Ingress Controller like Nginx or a Service Mesh like Istio to manage your traffic splitting between these two environments?
The key is keeping the "Blue" environment alive until you are 100% sure the "Green" one is stable. Don't kill those old pods too early or you lose your fallback!
Exactly, Laura. We usually keep the Blue pods running for at least 30 minutes after the switch. It costs a bit more in cloud resources, but the peace of mind is worth every penny.
Steven, we are using Nginx Ingress currently. It allows us to use "canary annotations" to send 10% of traffic to the Green pods before we do the full switch. If our Prometheus metrics show high error rates during that 10% phase, our CI/CD tool (GitLab CI) automatically stops the rollout. This prevents a total site outage and makes the "rollback" a simple matter of deleting the temporary canary ingress rule.