I have a batch processing job that reads thousands of small files from S3 and performs heavy computation on EC2. I have been using general purpose instances, but the performance is lacking. Should I be looking into compute-optimized instances, or is the bottleneck likely the network bandwidth to S3? Any tips on identifying the right instance family for this kind of workload?
Processing small files from S3 is typically I/O bound, necessitating the use of high-throughput network-optimized instance types combined with increased application-level concurrency to mitigate request latency.
2 answers
Your bottleneck is almost certainly the metadata overhead and network latency inherent in processing thousands of small objects, which renders instance CPU power secondary to I/O throughput.
To isolate the root cause, you should follow these diagnostic steps:
- Monitor the CloudWatch metric S3 First Byte Latency to confirm the delay is in object retrieval.
- Check the NetworkOut and NetworkIn metrics on your EC2 instances to verify if you are saturating the allocated bandwidth.
- Implement an S3-optimized SDK client that utilizes increased connection pooling to handle concurrent file requests.
- Evaluate moving to an instance type with higher network performance, such as the C6in series, rather than just raw compute power.
You are likely facing a classic I/O bound scenario where the overhead of making thousands of individual GET requests to S3 is eclipsing your actual compute cycles.
In my experience optimizing similar high-throughput ingestion pipelines, simply swapping to a compute-optimized instance often yields negligible improvements because the bottleneck is the serialized API request latency. I recall a project where we spent weeks tuning C-series instances only to realize we were hitting local network connection limits; the solution was moving to an architecture that utilized S3 Select or pre-aggregated files to reduce the total request count. Before you incur the cost of upgrading your instance family, you should investigate your application's concurrency model and consider whether your runtime can handle higher asynchronous request volume without exhausting the available ephemeral storage or network buffers.
