We are migrating microservices to production. During load tests, these automated containerized applications crash constantly without memory caps, showing an exit code of 137. I want to understand how the OS decides to terminate these specific processes and how to safeguard our infrastructure.
3 answers
Exit code 137 explicitly indicates that your process was terminated by an external SIGKILL signal, which in this scenario is triggered by the kernel's Out-Of-Memory management system. When your automated containerized applications crash constantly without memory caps, they are exceeding safe operational thresholds of the node. The kernel uses an internal scoring algorithm based on memory usage percentages and process priority to choose a target. Uncapped containers running non-essential tasks are almost always selected first for immediate termination.
Could adjusting the OOM score adjust value for the container runtimes temporarily prevent these aggressive terminations in your testing cluster?
An exit code of 137 is a definitive sign of memory starvation. Defining strict resource boundaries is the only permanent solution available.
Well said, Deborah. Standardizing your deployment templates with uniform constraints removes the guesswork and keeps the environment stable.
Modifying the score might delay the termination, Gary, but it is highly risky. If the kernel cannot kill the runaway container, it will start terminating vital system daemons instead, which can lead to a complete drop of the entire physical node.