We recently migrated our AWS infrastructure management to Terraform. We are using an S3 bucket for the state file and a DynamoDB table for state locking to prevent concurrent writes. However, we keep encountering 'Error: Error acquiring the state lock' even when no other pipeline is running. Has anyone found a reliable way to force-unlock or prevent these stale locks in a CI/CD environment?
3 answers
Stale locks usually happen if a CodeBuild project is interrupted or times out before Terraform can finish its cleanup. To fix this manually, you can use the command 'terraform force-unlock <LOCK_ID>', but you must ensure nobody else is actually running a plan. To prevent this in the future, I suggest wrapping your Terraform commands in a script that checks for existing locks or setting a shorter timeout in your DynamoDB configuration. We also found that increasing the 'retry_lock' parameter in the backend configuration helps the pipeline wait out transient issues without failing immediately.
Are you sure the IAM role used by your CI/CD pipeline has the 'dynamodb:DeleteItem' permission on the specific lock table? If it can create the lock but lacks permission to delete it after completion, you'll be stuck with a permanent stale lock every single time.
Always use versioning on your S3 bucket. If a lock failure leads to state corruption, you can revert to a previous version of the .tfstate file quickly.
I agree with Steven. S3 versioning is your ultimate safety net. It’s saved us more than once when a pipeline crash corrupted the state file during a complex resource migration.
That was exactly it, Robert! I checked our IAM policy and we only had PutItem and GetItem allowed for the DynamoDB table. After adding DeleteItem to the policy, the locks started clearing automatically after every 'terraform apply'. It’s a subtle permission that’s easy to miss when you're setting up a new environment from scratch.