I'm running into an issue where my Load Balancer thinks a server is "Healthy" because the port is open, but the application is so slow that it's timing out for users. How should I configure my health check endpoints? Should I check the database connection or just a simple ping? I want to avoid the "zombie server" scenario where the LB keeps sending traffic to a struggling instance.
3 answers
You need to implement what's called a "Deep Health Check." Instead of just checking if the port is open (TCP check), the load balancer should hit a specific HTTP endpoint (e.g., /health) that actually performs a internal status check. This internal logic should verify that the application can reach its database, has enough free memory, and isn't under extreme CPU pressure. If any of these fail, the endpoint should return a 503 Service Unavailable. This tells the Load Balancer to stop sending new traffic immediately, allowing the server to either recover or be replaced by an auto-scaling group.
Won't a deep health check add unnecessary load to your database if you have hundreds of servers checking the connection every 5 seconds?
Avoid simple "pings" at all costs. A server can respond to a ping while its actual web service is completely locked up and useless to your customers.
Exactly, Linda. We learned that the hard way during a memory leak last month. The servers were "up" but every single request was timing out.
George, that is a valid concern. To solve this, we usually cache the result of the database check for a few seconds. That way, even if the load balancer pings the /health endpoint every 2 seconds, the application only runs the "real" database check every 10 seconds. It balances the need for accurate health monitoring with the need to keep our database from being flooded by internal monitoring requests.