Health Checks & Heartbeats

Overview

Health checks and heartbeats are the mechanisms by which a distributed system monitors the status of its components. They allow load balancers to stop sending traffic to broken servers and allow orchestrators to replace failed nodes automatically.

🧠 Mental model: A heartbeat is like a night watchman clicking a radio button every 60 seconds to say "I'm still awake." A health check is like a supervisor calling the watchman and asking, "Are you awake, and are all the doors locked?"

Key Concepts

Heartbeats (Push)

A node periodically sends a signal (a heartbeat) to a central monitoring service or to its peers (as in a Gossip protocol) to indicate it is alive. If the monitor misses a certain number of consecutive heartbeats, it declares the node dead.

Health Checks (Pull)

A central entity (like a Load Balancer or Kubernetes) periodically polls an endpoint on the node (e.g., HTTP GET /healthz). Health checks come in different depths:

Liveness Probe: "Are you running?" Just checks if the process is up. If this fails, the orchestrator usually restarts the container.
Readiness Probe: "Are you ready to serve traffic?" Checks if the app has finished booting, loaded caches, and connected to the database. If this fails, the load balancer stops routing traffic to it, but doesn't kill the process.
Deep Health Check: The endpoint actively pings its dependencies (DB, Redis, downstream APIs) before returning 200 OK.

Trade-offs

Deep health checks provide the most confidence but can cause cascading failures. If 100 app servers all run a deep health check every 5 seconds that pings the database, they might unintentionally DDoS the database. Furthermore, if a non-critical downstream service (like a third-party analytics API) goes down, a deep health check might mistakenly declare the entire app server unhealthy, taking down the whole system for a minor issue.

Interview Tips

Differentiate between Liveness (needs a restart) and Readiness (needs to be temporarily removed from the load balancer).
Advise caution with Deep Health Checks to avoid accidental DDoS or false-positive failures due to non-critical dependencies.
Mention that health check endpoints should be extremely fast and lightweight.

Summary

Heartbeats are signals pushed by a node to say 'I am alive.'
Health checks are pulled by a load balancer/orchestrator to verify a node's status.
Liveness probes determine if a process needs to be restarted.
Readiness probes determine if a node should receive user traffic.
Deep health checks verify dependencies but risk causing cascading failures.