I am currently designing a data cleansing job that needs to touch roughly 5 million records. I’m worried about hitting the 10,000ms CPU timeout and the SOQL query limits. What are the best patterns for "Chaining" batches or using the "Iterable" interface to stay within the multi-tenant constraints?
3 answers
To handle 5 million records, you must ensure your start() method is highly efficient. Using a QueryLocator is usually better than an Iterable for large sets because it can bypass the 50,000 record limit (up to 50 million). In the execute() method, keep your batch size small—around 200 is standard, but you can drop it to 50 if your logic is heavy. To stay under the CPU limit, move any non-essential logic to @future methods or Queueable Apex if it doesn't need to be part of the atomic transaction. If you need to "chain" batches, remember that you can only start one new batch from the finish() method of another. This "sequential batching" pattern is the safest way to avoid the "Too many concurrent batches" error.
Does using the "Database.Stateful" interface significantly increase the execution time due to the serialization overhead?
Always remember to use "Aggregate Queries" if you only need sums or counts to keep the SOQL count low.
True, Mary. Leveraging the database engine for math instead of doing it in Apex loops is a lifesaver for CPU time.
It does add some overhead, Richard, but for 5 million records, the impact is usually negligible compared to the benefit of tracking progress or errors across segments. Just be careful not to store massive collections in your variables; only keep primitive counters or small maps to minimize the state size being passed between executions.