We are running a heavy-duty ML pipeline on a Kubernetes cluster. Frequently, we see tasks that are marked as "success" in the logs but remain in a "running" state in the Apache Airflow UI until they eventually timeout as zombies. This is causing our resource pods to hang. Is this a heartbeat issue between the KubernetesExecutor and the scheduler, or something else entirely?
3 answers
Zombie tasks usually occur when the Airflow scheduler loses track of a task that is still running on a worker, or if the process dies without updating the metadata DB. On Kubernetes, this often happens if your pods are being OOM-killed (Out of Memory) or if the scheduler_zombie_task_threshold is set too low for your cluster's network latency. Check your K8s events for "OOMKilled" status. If the pod disappears before it can send the final state to the DB, the scheduler will wait until the threshold is met before reaping it. Increasing the memory limits for your worker pods is the first step I'd take.
Are you using any specific resource requests and limits in your pod_override configuration? It sounds like the scheduler and the worker pod are falling out of sync during long-running training jobs.
Check your database connection pool. If the workers can't write the "success" state because the DB is full or slow, the task will just hang in the UI indefinitely.
Solid advice from Jason. I’ve seen this happen with Postgres when the connection limit is reached. It makes the task look like a zombie when it’s actually just a DB bottleneck
Douglas, yes, we are using pod_override to request GPUs. I noticed that when the model starts training, CPU usage on the pod spikes, which might be delaying the heartbeat signal back to the scheduler. Rebecca’s point about the zombie threshold also makes sense. We’ll try increasing the scheduler_health_check_threshold and ensuring the pod has a bit of CPU "headroom" so the heartbeat process doesn't get starved during the training phase.