We have recently implemented GitOps using ArgoCD for our Kubernetes clusters, but we keep running into issues where developers make manual changes via kubectl. These changes cause a "drift" between the live state and our Git repository. Does ArgoCD have a built-in mechanism to automatically overwrite these manual changes to ensure Git remains the single source of truth? Also, is there a way to receive alerts when such a drift is detected before the auto-sync kicks in?
3 answers
To maintain the integrity of your GitOps workflow, you should enable the 'Self-Heal' and 'Prune' options within your ArgoCD Application manifest. When 'Self-Heal' is active, ArgoCD constantly monitors the cluster; if it detects that the live state deviates from the desired state in Git (like a manual kubectl edit), it will automatically re-apply the Git configuration to overwrite the change. For alerting, you can integrate ArgoCD Notifications with Slack or Email. This allows you to set up triggers that fire whenever an application's health status changes or when a 'OutOfSync' phase is detected, giving you visibility into who might be bypassing the official pipeline.
Are you currently using "App-of-Apps" patterns to manage these configurations, or are you defining each microservice as a standalone Argo application?
The best practice is to remove 'write' access to the cluster for developers. If they can't use kubectl to change things, the drift problem disappears at the source.
I totally agree with Jennifer. Moving to a "Pull" based GitOps model works best when you strictly enforce RBAC, allowing only the GitOps controller to make changes.
Steven, we are using the App-of-Apps pattern. It makes it easier to manage the drift settings globally across all services. We found that by setting the syncPolicy at the root application level, we can enforce 'Self-Heal' across the entire cluster. This effectively locks down the environment, making manual interventions futile because ArgoCD reverts them within seconds. It has definitely improved our compliance and security posture significantly by ensuring that no "shadow IT" changes stick around for long.