We recently had a security incident where a developer accidentally hardcoded a cloud provider API key into a Dockerfile. Even though the key was deleted in a later commit, it stayed in the Docker layer history and was discovered by an attacker. How do we set up an automated "Secret Scanner" that looks at the entire Git history and the final container image? Is it better to block the commit locally on the dev's machine or scan it during the CI build?
3 answers
The short answer is: you must do both. Use a tool like 'ggshield' or 'trufflehog' as a "pre-commit hook" to stop the secret from ever leaving the developer's laptop. However, since hooks can be bypassed, you also need a "failsafe" scan in your CI pipeline. For Docker specifically, use 'Trivy' or 'Clair' to scan the final image layers. If a secret is found in a hidden layer, these tools will flag the image as non-compliant. To fix your current issue, you'll need to rotate the leaked key immediately and use a tool like 'BFG Repo-Cleaner' to scrub the secret from your entire Git history before it is further exposed.
How are you managing your secrets now? If you move to a "Dynamic Secret" model with HashiCorp Vault, the leaked keys would only be valid for a few minutes anyway, which drastically reduces the risk.
Never trust the developer to remember to run the scanner. If it isn't "Enforced" in the CI/CD gateway, it doesn't exist. Automation is the only way to ensure 100% coverage.
100% agree. We made the secret scan a "Hard Failure" in our pipeline. If 'trufflehog' finds so much as a suspicious string, the build stops and the security team gets paged.
George, moving to Vault was the best decision we ever made. Instead of hardcoding keys, our apps now use a "Sidecar" to fetch a temporary token from Vault at startup. This means our Docker images contain zero secrets. Even if an attacker pulls the image from our registry, they find nothing but code. We also enabled "Secret Scanning" in GitHub, which automatically alerts us if any of our known AWS or Stripe keys appear in a public repo. This "Defense in Depth" approach has made our development process much more resilient to human error.