Our Airflow instance on EKS is getting quite expensive due to the high number of worker nodes being spun up for small tasks. Are there specific executor configurations or pod autoscaling strategies you recommend to keep the costs down without increasing latency?
3 answers
The biggest win for us was switching to the KubernetesExecutor with specific resource requests and limits for every task. By default, pods might request more CPU/RAM than needed, leading to underutilized nodes. You should also look into using 'Karpenter' instead of the standard Cluster Autoscaler on AWS. Karpenter is much faster at provisioning the right-sized instances (like Spot instances) for your workload. Also, ensure you are using 'Task Groups' and avoiding heavy logic in the DAG parsing code, which can bloat the Scheduler's memory usage.
Have you considered using the CeleryKubernetesExecutor to handle small, frequent tasks via Celery workers while offloading the heavy, long-running jobs to K8s pods?
Spot instances are your best friend for non-critical ETL. We cut our Airflow compute bill by nearly 60% just by moving the worker node groups to AWS Spot instances.
Just make sure you have solid retry logic in your DAGs. If a Spot instance is reclaimed, you don't want the whole pipeline to fail without a second attempt.
I hadn't thought about the hybrid approach. Currently, even a simple 'Hello World' python task spins up a whole pod, which seems like massive overkill for our daily reporting runs.