I'm setting up a new production pipeline and I'm worried about security. Currently, I have some API keys hardcoded in my Python scripts, which I know is bad practice. How does Apache Airflow handle secrets management? Should I use the built-in Connections UI, or is it better to integrate with something like AWS Secrets Manager or HashiCorp Vault for better compliance?
3 answers
For a production-grade environment, you should definitely avoid hardcoding and even consider moving beyond the local metadata database for secrets. While the Airflow Connections UI is fine for development, it stores encrypted secrets in its own DB. The "Gold Standard" is using a Secret Backend. You can configure Airflow to automatically check AWS Secrets Manager or Vault before looking at its local DB. This centralizes your security and ensures that rotation of keys happens in one place. You just need to set the backend property in your airflow.cfg and provide the necessary permissions to the Airflow role.
Does your organization already use a specific cloud provider's secret vault? Integrating Airflow with an existing secret manager is usually easier than setting up a new one from scratch for just one tool.
Use the Variable.get() and Connection.get() methods in your code. They are designed to interface seamlessly with whatever backend you choose to configure later.
Samuel has a point; as long as you use those standard methods, you can switch from the UI to AWS Secrets Manager later without changing a single line of your actual DAG code.
We are heavily invested in AWS, so Secrets Manager would be the path of least resistance. I was concerned about the latency of fetching secrets during every task execution, but I read that Airflow can cache these lookups. Brenda's suggestion about the backend property seems straightforward to implement via environment variables. This will definitely satisfy our security audit requirements for the upcoming quarter without requiring us to rewrite our entire task logic.