We are expanding to a multi-region setup (US-East and EU-West) for data sovereignty. Managing Kubernetes Secrets manually is becoming a security risk. Are you guys using HashiCorp Vault with an injector, or are you leaning into cloud-native solutions like AWS Secrets Manager CSI driver? I’m worried about the latency of fetching secrets from a central vault during a pod restart.
3 answers
We use HashiCorp Vault with the Agent Injector. To solve the latency issue, we deployed Vault in a "Performance Replication" mode across both regions. When a pod in EU-West restarts, it talks to a local Vault replica, so the secret injection happens in milliseconds. It’s a bit more overhead to manage the Vault cluster itself, but the centralized audit logs and the ability to use "Dynamic Secrets" (like temporary DB credentials that expire after an hour) make it significantly more secure than standard Kubernetes secrets.
Sharon, how do you handle the "initial secret" problem—basically, how does the pod authenticate with Vault to get its actual secrets without having a hardcoded token?
We use the External Secrets Operator. It synchronizes secrets from AWS Secrets Manager directly into K8s Secrets. It’s much simpler if you are already all-in on a single cloud provider.
External Secrets Operator is the way to go for simplicity. It keeps the "native" K8s feel while ensuring your actual source of truth is a secure, managed cloud service.
Philip, we use Kubernetes Auth Method in Vault. The pod sends its signed ServiceAccount JWT to Vault. Vault verifies the token with the K8s API server and then issues a short-lived Vault token to the pod. It’s completely "zero-trust" because there are no long-lived passwords or tokens stored anywhere in the repo or the pod spec.