Our team is moving away from manual "all-at-once" deployments toward a more mature DevOps approach. We want to implement Canary Releases in our AWS EKS cluster to minimize the blast radius of new updates. I am specifically looking for advice on how to handle traffic shifting—should we use a Service Mesh like Istio, or is an Ingress controller enough? Also, how do you automate the "rollback" trigger if the canary version shows a spike in 5xx errors? What tools or metrics should we prioritize to ensure the release is safe?
3 answers
Implementing a Canary Release in the cloud is most effective when you use a "Progressive Delivery" controller like Argo Rollouts or Flagger. These tools extend Kubernetes to handle the complex logic of shifting traffic incrementally (e.g., 5%, 10%, 25%). For the traffic shifting itself, while an Ingress controller (like NGINX or ALB) can do basic weighted routing, a Service Mesh like Istio or Linkerd gives you much finer control and better observability. The "secret sauce" is the analysis phase: you connect your rollout controller to Prometheus or Datadog. You define a "success criteria" (like an error rate below 1%), and the controller automatically queries these metrics. If the canary fails the check, the traffic is instantly routed back to the stable version without any manual intervention.
That’s a solid architecture, Jennifer. But how do you handle "sticky sessions" during a canary rollout? If a user is routed to the canary version (5%) and then refreshes their page, is there a risk they’ll be bounced back to the stable version, leading to a confusing or broken UI experience
In my production setup, we use Spinnaker for canary analysis. It has a great "Judge" feature that compares the canary metrics against a baseline rather than just fixed thresholds, which reduces false positives.
I agree with Mark. Comparing against a "baseline" (a fresh deploy of the current version) is much more accurate than comparing against the "production" environment, which might have different cache states or aged connections. This isolation is what makes the canary signal truly reliable.
That is a critical point, Jennifer. To prevent "UI jumping," you need to implement session affinity at the load balancer or service mesh level. In Istio, you can use "Destination Rules" with consistent hashing based on a cookie or header. This ensures that once a user is part of the "canary group," they stay there for the duration of their session. In my experience, if you don't pin the user, you'll see a lot of strange transient errors, especially if your new version includes database schema changes that aren't perfectly backward compatible.