Our DevOps team is running into a recurring issue where our application pods get killed immediately after a rolling cloud server deployment finishes. The logs indicate an exit code 137, which points to memory issues, even though our Xmx parameter is configured well below the physical machine limits. Why would the container runtime kill the process if our internal Java heap settings seem correct?
3 answers
An exit code 137 means the operating system or container daemon sent an uncatchable SIGKILL because the container breached its absolute memory threshold. When you configure a cloud server deployment, you must remember that the total memory footprint of a Java process is the heap size specified by your parameter plus the off-heap memory. Off-heap allocations include Metaspace, thread stacks, garbage collection overhead, and direct byte buffers. If your container limit is set precisely to your heap limit, the extra off-heap memory usage will instantly cause a crash.
What diagnostic logging tools or APM agents can we attach to the startup sequence to catch the exact moment the off-heap utilization crosses the threshold line?
You simply need to increase the container memory limit in your resource definitions while keeping your heap parameter static to provide a sufficient buffer zone.
This is the most straightforward fix. Balancing the container limit with a 25% margin over the heap allocation completely eliminates the unexpected 137 termination errors.
You can add the native Java command line flag -XX:NativeMemoryTracking=summary to track this behavior. This allows you to output the exact internal breakdown of non-heap memory allocations to your standard output logs before the system gets targeted by the container runtime killer.