We are using Terraform for our Infrastructure as Code (IaC), but we keep running into issues with "State Drift" in our multi-region AWS setup. Sometimes manual changes are made in the console, and our 'terraform plan' becomes a nightmare to reconcile. What is the most reliable way to enforce state consistency and ensure that our S3 backend with DynamoDB locking is scaling correctly?
3 answers
State drift is the silent killer of IaC. The best way to combat this in 2024 is to implement a "Drift Detection" job in your CI/CD pipeline that runs 'terraform plan' on a schedule (e.g., every 4 hours) and alerts the team if any discrepancies are found. For multi-region setups, ensure you are using a modular approach where each region has its own state file to limit the blast radius. Also, strictly enforce "No Manual Changes" via IAM policies, allowing only the Terraform service role to modify resources in production.
Have you considered using a wrapper like Terragrunt to manage your remote state configurations more dryly? It can simplify the management of multiple state files across different regions and accounts.
We solved this by using Terraform Cloud. The built-in drift detection and the workspace-per-region model made it much easier to track changes without managing our own S3 backends.
Richard makes a great point. While the S3/DynamoDB approach is free, the managed features of Terraform Cloud or Spacelift save a lot of operational overhead for growing teams.
William, I've heard of Terragrunt but was worried about the learning curve for the team. Does it handle the DynamoDB locking automatically across different AWS profiles, or do we still need to manually define the backend for every single sub-module? We are looking for something that reduces our boilerplate code while keeping the state secure.