I have a background job that takes about 20 minutes to process large images. I know Lambda has a 15-minute timeout, so that’s out. I’m looking at AWS Fargate for this, but I’m not sure how to trigger these tasks efficiently. Is Fargate more expensive to run for these types of bursty workloads, and how do I manage the underlying storage requirements?
3 answers
For a 20-minute job, Fargate is the middle ground between Lambda and EC2. Trigger Fargate tasks from an S3 event using an AWS Step Function. The Step Function can start the task and monitor the container until completion. Regarding costs, Fargate is billed by vCPU and memory per second, so for bursty workloads, it remains efficient since you aren't paying for idle time. For storage, mount an Amazon EFS to your task. This provides a persistent storage layer that multiple tasks can access simultaneously, which is better than trying to fit large files into the limited local ephemeral storage provided by the container.
Since your workload is bursty, have you looked into "Fargate Spot"? It can save you up to 70% on costs if your image processing jobs are capable of being restarted after an interruption?
Using Amazon ECR for your container images ensures that Fargate can pull them quickly and securely within the AWS network, reducing your task startup latency significantly.
ECR is definitely the way to go. The integration with IAM makes the security aspect seamless, and the image scanning features help us catch vulnerabilities before they reach production.
We recently switched our non-critical batch processing to Fargate Spot and the savings were immediate. As long as you have a retry mechanism in your Step Function, the rare interruptions aren't a problem. It turned a $500 monthly bill into $150. However, for the images that have strict client SLAs, we keep those on the regular Fargate "On-Demand" tier to ensure they aren't interrupted mid-process. It's all about classifying your workload's priority levels correctly to maximize your budget.