Our infrastructure is split between on-prem servers and AWS. We want to use ArgoCD for GitOps, but I'm worried about secret management and cross-environment consistency. How does a standard DevOps & SRE team handle state synchronization across such a complex landscape?
3 answers
Are you planning to use a single centralized GitOps controller for both environments, or will you deploy separate instances of ArgoCD in each cluster to reduce cross-cloud latency?
You should definitely look into Crossplane if you want to manage non-Kubernetes resources through the same GitOps flow. It makes the whole infrastructure feel unified.
Kevin is right on the money. Crossplane combined with a GitOps approach allows you to treat your cloud providers exactly like your local K8s clusters.
In hybrid environments, the "Source of Truth" must be strictly controlled via Git. We utilized Sealed Secrets to ensure that sensitive data could be safely stored in the repository without exposure. Managing a DevOps & SRE workflow this way requires a very high level of discipline regarding pull requests. When we moved to this model in late 2024, it reduced our configuration drift by nearly 70% because manual "hotfixes" on the servers were strictly forbidden by the policy and the automated sync.
Daniel, we are leaning towards a hub-and-spoke model where one central management cluster pushes configurations to the edge locations. This seems easier for auditing, though I am concerned about what happens if the main connection to the control plane drops during a critical update period. We need to ensure that the remote clusters can still maintain their desired state even if they are temporarily isolated.