Why Your Service Should Have Two Health Checks Not One

What you actually need to know

• One health check is doing two jobs and the jobs contradict each other.
• Liveness answers "should I kill this process," readiness answers "should I send it traffic."
• A combined check that depends on the database will trigger cascading restarts during a transient database issue.
• Readiness can call dependencies. Liveness cannot.
• Add a startup probe for any service that takes more than a few seconds to warm up.

The core argument

The single /health endpoint pattern is a habit that comes from an era before container orchestration. When a service ran on a fixed VM behind a load balancer, you only had to answer one question: is this thing okay. The platform did not restart processes. It just stopped sending traffic.

Now we have orchestrators that kill pods aggressively, and we have load balancers that route traffic conditionally, and these two systems need different answers. The orchestrator wants to know if it should give up on this process. The load balancer wants to know if this process can take traffic right now. They are different questions with different consequences if you get them wrong.

I have walked into several incidents where the root cause was a single health check that returned unhealthy because a database connection was briefly slow. The orchestrator killed the pod. The new pod started, hit the same slow database, returned unhealthy, got killed. The team had built itself a restart storm that took the whole service down during a five-second database blip that the database had already recovered from.

The fix is structural. Two endpoints. Different rules. Different consumers.

What each check should actually do

Liveness

Liveness is the simplest possible check that confirms the process is running and capable of responding. It does not touch the database. It does not call other services. It does not do anything that could fail because of something outside this process.

A liveness check that returns 200 means "I am here and answering." A liveness check that fails means "I am stuck or crashed, please restart me." It is the cheapest possible signal because the action it triggers, restarting the process, is destructive and you do not want false positives.

Readiness

Readiness is more expensive. It can check the database connection, the cache, the message queue, anything the service genuinely needs to handle a request. If readiness returns unhealthy, the load balancer stops sending new traffic. Existing connections drain. The pod is not killed. It just sits there waiting until it can serve again.

This is the right behavior when a dependency is slow. Stop sending traffic to instances that cannot handle it. Do not kill those instances, because killing them does not fix the dependency.

Startup

Startup is the trickiest because it only runs at boot. Use it for services that take significant time to initialize, like ones that load a large model or warm a sizable cache. Without it, you risk the orchestrator killing a slow-starting pod for failing liveness before it ever finished starting.

How long does it take to add

What each check must include

• Liveness: only checks internal to the process. No network calls. Fast and cheap.
• Readiness: checks dependencies the service actually needs to serve requests.
• Startup: tracks initialization progress.
• All three: return explicit status codes and minimal bodies. Detail goes in logs, not response bodies.
• Consistent paths: /healthz/live, /healthz/ready, /healthz/startup is a common convention.

Expert opinion

The first time I separated liveness from readiness in production, the team was skeptical. It felt like over-engineering for a problem we had not had. Three months later a Redis instance had a slow minute and our combined check would have restarted every pod in the fleet. Instead, readiness flipped to false for the duration, the load balancer stopped sending traffic, the pods stayed alive, and when Redis came back the fleet resumed without a single restart. That is the day the pattern earned its keep.

>

Yashveer Singh, founder of Yashveer Labs

How this played out on a real project

A SaaS platform I worked on had a single /health endpoint that called the database. During business hours the database had occasional 200ms spikes from a long-running report query. Each spike caused the health check to time out. The orchestrator restarted pods. The new pods hit the same database, timed out, got restarted. The error rate during these spikes was much worse than the database issue would have caused on its own.

We split the check into liveness and readiness. Liveness became a trivial in-process check. Readiness kept the database call but with a generous timeout and a small failure tolerance. The next time the database spiked, readiness briefly returned false on a few instances, the load balancer rerouted, and not a single pod was restarted. The platform's self-inflicted outage pattern disappeared. The discipline of separating concerns showed up again in the resilience patterns of circuit breakers and retries, which addresses related failure modes.

Common mistakes

1. Using a single /health endpoint for both consumers.
2. Calling the database from the liveness check.
3. Returning 500 instead of 503 for unhealthy, which mixes application errors with health signals.
4. Setting liveness probes too aggressively, causing healthy slow-starting pods to be killed.
5. Setting readiness probes too leniently, so traffic continues to a broken instance.
6. Not adding a startup probe for slow-starting services.
7. Putting expensive logic in a readiness check that fires every second. The check becomes a load source of its own.

A two week plan to fix it

1. Day one. Audit your current health check. What does it actually check? Who consumes it?
2. Days two and three. Split into three endpoints. Liveness, readiness, startup.
3. Days four and five. Update the orchestrator and load balancer configs to use the new endpoints with sane intervals and thresholds.
4. Week two. Roll out across one service first. Watch the behavior during the next minor dependency hiccup.
5. Week two ongoing. Roll out to the rest of the fleet. Document the pattern in your architecture decision records so the next service inherits it.
6. Long term. Make the three-endpoint pattern part of the new service template so this is never a question again.