I have a long list of items (e.g., 1,000 entries) that I need to process in batches of 10. I’m looking for a clean, efficient way to "chunk" this list into a list of lists or an iterator of sub-lists. I’ve tried using standard for-loops with manual index slicing, but it feels clunky and prone to "off-by-one" errors. Is there a built-in function in itertools or a concise list comprehension that can handle this? Ideally, the solution should also handle cases where the final chunk is smaller than the specified batch size.
3 answers
If you are already doing data analysis and have NumPy installed, you can use np.array_split(). The advantage here is that you can specify the number of chunks you want rather than the size of the chunks, which is very useful for parallel processing across a fixed number of CPU cores.
The most common "Pythonic" way to do this without external libraries is using a list comprehension with slicing. You can iterate through the list using a range that steps by your desired chunk size n:
chunks = [my_list[i : i + n] for i in range(0, len(my_list), n)]
This is highly readable and efficiently handles the "remainder" if your list length isn't perfectly divisible by n. If you are working with extremely large datasets and want to save memory, you should use an iterator approach. Since Python 3.12, the itertools module includes batched(), which is specifically designed for this:
import itertools
for batch in itertools.batched(my_list, n):
process(batch)
This avoids creating all sub-lists in memory at once.
For those stuck on older versions of Python (before 3.12) but still wanting an iterator, the "grouper" recipe from the itertools documentation is a classic. It uses zip(*[iter(s)]*n), though it fills missing values with None. If you don't want the padding, the list comprehension in Answer 1 is usually the safest bet.
I agree with Alex. That zip trick is clever but can be a nightmare for beginners to read and debug. I almost always stick to the list comprehension unless I have a massive file that requires the memory efficiency of itertools.batched.
Sarah, that's a great tip for heavy lifting! However, just a heads-up for others: np.array_split() returns a list of arrays, so if your next step strictly requires standard Python lists, you might have to convert them back, which adds a bit of overhead.