Every time we push an update to our cluster, the rollout completely stalls, forcing us to manually roll back to the previous stable revision. Why exactly do these critical errors happen during live rollouts, and what is the optimal sequence of system admin commands to quickly pinpoint whether the issue is network or resource-driven?
3 answers
When your rollout stalls mid-way, it is almost always because the new pods are failing their readiness probes, preventing the deployment controller from scaling down the older, healthy instances. When in this specific manner, the system protects your traffic by halting the progressive rollout. You need to inspect the container logs to verify if the application is successfully starting up and listening on the designated container port. If the application is perfectly fine, double-check your initial delay seconds in the probe configurations.
Are you utilizing specific resource limits within your deployment manifests, or are you letting the pods scale without strict boundaries? Often, unexpected resource pressure inside a specific namespace blocks scheduling.
When mid-rollout, check your maxSurge and maxUnavailable settings. Misconfiguring these parameters can freeze the cluster deployment controller entirely.
Spot on, Evelyn! If maxSurge is set to zero and maxUnavailable is also zero, the cluster deployment controller enters a logical deadlock where it cannot create new pods or delete old ones, causing the whole process to stall.
We do use resource quotas across our namespaces to manage boundary allocations. After reviewing the node conditions following your point, we discovered that our worker nodes were heavily overcommitted, causing new pods to remain in a perpetual scheduling state because no single node could satisfy the requested CPU capacity. We either need to add another node to our pool or optimize the current CPU requests.