Health Checks Are Lying to You (And Why That Matters)
Your /health endpoint says everything is fine. Your users disagree. Here's what I learned the hard way.
I built a health check endpoint last year. It was clean: checked database connection, cache layer, external APIs. Green light. Perfect.
Then production went sideways on a Friday at 9 PM.
The health endpoint said OK. The dashboard was green. But users in certain regions couldn't reach the service at all. Turns out our load balancer was misconfigured, and internal health checks were bypassing it entirely.
Lessson learned: internal metrics don't verify what matters — whether actual users can reach your service.
I now split health checks into two categories:
1. Internal diagnostics — these live in your dashboard. Database latency, queue depth, memory usage, cache hit rates. These tell you why something might break. They're for your on-call engineer at 3 AM.
2. External uptime verification — this is separate. I use a cheap external service (or just cron from another server) that actually hits your public endpoints and measures real response time. It catches what your internal checks miss: network issues, DNS problems, load balancer failures, regional outages.
With Seven Suite (my NestJS/Angular open source project), I added both. The internal endpoint is detailed — it shows me Redis state, database pool connections, background job queue status. But it only matters if my external monitor confirms the API is actually reachable.
This distinction saved us twice now. Once when a DDoS mitigation layer started filtering legitimate traffic. Once when a database replica failed and internal checks didn't catch it because they weren't reading from replicas.
Practical take: if you're the one on-call when things break, you need context and confirmation. The red line on your latency chart means nothing if the external uptime monitor is already paging you. They need to talk to each other.
Build your internal health endpoint for debugging. Build your external uptime check for reality. Make them disagree sometimes — that's when you actually learn something.
Green doesn't mean working. It just means your service thinks it's working.