Our background workers keep dying during peak data processing jobs. The logs show the terminates with exit code 137. What exactly does this numerical status code represent, and what structural changes do we need to implement to prevent our tasks from being abruptly killed?
3 answers
Exit code 137 explicitly states that your was terminated by an external operating system signal, specifically SIGKILL (signal 9), because it ran out of allocated RAM. The math behind the number is $128 + 9 = 137$, where 128 is the standard base for error signals. To fix this, you either need to optimize your application's memory footprints to avoid memory leaks, or increase the memory reservation thresholds inside your deployment configurations so the host allows the process to scale further under heavy load.
Adjusting the memory resource limit makes sense for scaling workloads, but what if the core application inside our contains a progressive memory leak? Won't increasing the limits simply delay the inevitable process termination instead of solving it?
It means the operating system stepped in and killed the process because the container violated its strict hardware memory allocation boundaries.
Exactly, Alice. It is a protective measure by the host operating system to make sure a single rogue container cannot consume all resources and crash the entire machine.
You are completely right; throwing more hardware at a memory leak is just a temporary patch. You must use application profiling tools like Eclipse Memory Analyzer or Node Clinic to track down the unreleased objects in your codebase.