We are trying to push about 50,000 records daily into Salesforce from an external ERP, and we keep hitting API governor limits. We are currently using basic REST API calls, but the architecture is failing under load.
What are the best practices for scaling integrations? Should we look into Platform Events or change our Batch Apex logic entirely? I need advice on how to handle error logging and record locking when things go sideways during high-traffic windows.
High-volume data integrations in Salesforce should utilize the Bulk API 2.0 to asynchronously process records in batches, decoupling data ingestion from synchronous API calls to prevent governor limit exhaustion and record locking contention.
6 answers
To scale your integration effectively, you should implement the following architectural changes to minimize governor limit impact:
- Use the Bulk API 2.0 instead of REST for high volume data loads.
- Implement a staging object to decouple the integration from your core business logic.
- Adopt a platform event pattern for error notifications rather than immediate retry logic.
- Optimize your record update sequences to reduce row-level locking contention.
Stop using the standard REST API for 50,000 records daily, as it was never designed for bulk processing. Switch immediately to the Bulk API 2.0 to handle these batches asynchronously and avoid hitting your governor limits on every single request.
I remember walking into a massive retail client a few years back where the developers were convinced that custom REST endpoints were the golden ticket. They had a daily job crashing every afternoon because they were trying to sync product catalogs in real-time, creating a recursive locking nightmare that brought their entire org to its knees.
We ended up scrapping their entire synchronous loop and moving to a staging object pattern that processed files in chunks after hours. It was not pretty, but it was the only way to stabilize the environment and stop the constant locking errors that were plaguing the sales teams.
You are choosing between synchronous and asynchronous processing, and the trade-off here is strictly between latency and system stability. If you force your integration to be real-time via REST, you will inevitably hit concurrency limits during spikes, which is why asynchronous Bulk API 2.0 is superior for your 50,000 record threshold. Platform events are excellent for decoupling systems and handling orchestration but they introduce a secondary layer of complexity that you might not need if you are just doing a batch dump. Stick to Bulk API for data movement and use Platform Events only if you need to trigger downstream logic in response to those changes, as relying on them for massive record ingestion is an anti-pattern that creates unnecessary event bus overhead.
Bulk API 2.0 is the correct tool for this volume; stop using REST. Trying to push 50,000 records via REST is just asking for concurrency issues.
Stop throwing more code at a systemic architecture failure. You are dealing with a classic concurrency bottleneck because REST requires synchronous processing of record locks. When 50,000 records hit the org, the database locks rows, and your API calls timeout while waiting for those locks to release.
You need an asynchronous staging pattern. Push your ERP data to a flat staging object first. Use a platform-native tool or a simple Batch Apex job to process these records in smaller, non-overlapping batches. Monitor your DML operation performance metrics to determine the optimal batch size. If your logs show continuous locking errors, reduce the batch size or ensure the records in each batch do not share common parent records or lookup dependencies that trigger mass row locking.
Error logging should be handled via a custom framework that writes failures to a platform object, allowing for automated reconciliation scripts. Never rely on the standard API response to handle business logic errors at this scale.
