We recently migrated our data warehouse to Databricks on AWS, but our monthly costs are skyrocketing during large ETL runs. What are the best practices for configuring cluster autoscaling and using Spot Instances without risking job failures? We really need to optimize our DBU consumption immediately.
3 answers
To tackle rising costs, you must implement a multi-layered approach starting with Cluster Policies. By enforcing specific instance types and limiting maximum workers, you prevent developers from over-provisioning. I highly recommend switching your worker nodes to Spot Instances while keeping the Driver node on On-Demand to ensure stability. Additionally, enable Photon for your SQL warehouses; while the hourly rate is higher, the processing speed often reduces the total DBUs consumed per job significantly. Check your idle timeout settings as well to ensure clusters terminate promptly.
Have you looked into the specific DBU breakdown in your Cost Management console to see if the surge is from interactive clusters or automated jobs?
Always use the "Analyze" feature on your Delta tables before running joins. Proper data skipping and Z-Ordering can reduce the amount of data scanned, which directly lowers your DBU usage.
Great point, David. Z-Ordering on high-cardinality columns has saved our team nearly 30% on compute time during our daily morning refreshes.
Robert, we checked and it is definitely the interactive clusters left running over weekends. We are now looking for a way to automate a hard shutdown for any cluster that has been inactive for more than two hours. Is there a specific script or native Databricks feature you would recommend for this type of governance across multiple workspaces?