I keep hitting the ImagePullBackOff error when trying to deploy my microservices to a managed Kubernetes cluster. I have verified that the image exists in my private registry, but the pods just won't start. Is this typically a service account issue or a missing imagePullSecret? How do I properly debug the authentication handshake between the K8s node and a private container repository?
3 answers
The most frequent cause for this is a missing or misconfigured imagePullSecret. Even if you have the secret created in your cluster, it must be explicitly linked to your deployment manifest or the default service account in that specific namespace. I dealt with this last year during a migration; we found that the secret was in the 'dev' namespace but the deployment was in 'staging'. Kubernetes secrets are namespace-scoped, so you must recreate the secret in every namespace where you plan to pull that private image. Always run 'kubectl describe pod [name]' to see the exact error message from the kubelet.
Have you double-checked if the nodes themselves have the correct IAM permissions to pull from the registry, especially if you are using a cloud-native provider like ECR or GCR?
I highly suggest using 'docker login' locally to test the credentials first. If they work there, then use those same credentials to generate your K8s secret.
I agree with Sarah. Many people forget that the config.json generated by Docker needs to be encoded correctly into the secret. This simple check saves hours of debugging time.
Matthew, that is a great point for managed services. If the worker node's instance profile lacks the 'read' permission for the container registry, the pull will fail even if the K8s secret is correct. I usually recommend testing with a public image first to rule out networking issues. If the public one pulls but the private one fails, you know it is strictly an identity or credential problem.