I keep encountering the "Error: socket hang up" message in my NodeJS application logs when making outbound API calls using the http module. It seems to happen randomly under high load, and it's crashing my background workers. Is this an issue with my local server configuration, a timeout on the remote server side, or something related to how Node handles connection pooling and keep-alive headers? I need to understand what is physically happening to the connection when this specific error is thrown.
3 answers
A "socket hang up" in NodeJS essentially means that the connection was terminated by one of the parties abruptly while a request was still in progress. Technically, it occurs when the client (your Node app) sends a request, but the remote server closes the connection before a response is fully sent back. This is often caused by the remote server's "Keep-Alive" timeout being shorter than your client's timeout. When the server decides the connection has been idle for too long, it sends a TCP FIN packet to close it. If your Node app tries to use that same socket right as it's closing, the "hang up" occurs. It’s a common headache when working with load balancers like Nginx or AWS ALBs that have strict idle timeout settings.
That explanation about the TCP FIN packet makes a lot of sense. However, if I am using a custom http.Agent with keepAlive: true, shouldn't NodeJS be smart enough to detect that a socket is being closed by the server before attempting to write a new request to it, or is there a specific race condition that happens within the internal event loop that prevents this?
In my experience, this usually happens when the remote server crashes or restarts. The socket is simply cut off, and Node throws this error because it was expecting more data bytes.
I agree with Linda. I saw this constantly during our migration to a microservices architecture. If the downstream service is unstable or under heavy GC pressure, it often drops sockets, leading to the exact "hang up" Joseph is seeing in his logs.
Daniel, you've pinpointed the exact issue. There is indeed a race condition. NodeJS might pick a socket from the pool that it thinks is "free," but in the few milliseconds it takes to start the write operation, the server's timeout hits and it closes its end. To mitigate this, many developers implement a retry logic specifically for the ECONNRESET or "socket hang up" errors, or they ensure the client-side keepAlive timeout is significantly shorter than the server-side timeout to ensure the client closes the connection first.