web / health checks

updated November 05, 2026

I run health checks on web services during deploy and every minute to verify availability.

Servers serve GET /healthz. It returns 200 ok on success and 503 db unavailable on failure. The body is the same for every fault, so a public caller learns nothing about the internals. The server writes the real error to its journal. Cache-Control: no-store prevents stale responses. A two-second timeout prevents slow dependencies from hanging callers. The endpoint takes no credentials and reads minimal state.

Check the dependency, not the handle

SELECT 1 proves that the server runs SQL over the network. It touches no table, so a dropped table or a missing grant still returns 200.

I read one row from a core table instead:

SELECT 1 FROM games LIMIT 1

One row proves that the table exists and the app can read it. I use LIMIT 1 because count(*) scans the whole table and adds no signal.

An empty table is a pass, because the query still ran. An empty games table is a board with no games yet. In Go, that makes sql.ErrNoRows a pass.

Cache in memory to protect against floods

An external caller can flood an unauthenticated endpoint and exhaust database connections, causing health checks to time out.

I cache database check results in memory for two seconds. A double-checked sync.RWMutex serializes checks so the database receives at most one query every two seconds. If a client cancels its context, the handler reports an error without caching that cancellation:

type health struct {
	mu        sync.RWMutex
	checkedAt time.Time
	err       error
}

func (h *handler) serveHealth(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Cache-Control", "no-store")
	if err := h.checkDB(r.Context()); err != nil {
		log.Printf("healthz: %v", err)
		http.Error(w, "db unavailable", 503)
		return
	}
	_, _ = io.WriteString(w, "ok\n")
}

func (h *handler) checkDB(ctx context.Context) error {
	h.health.mu.RLock()
	if time.Since(h.health.checkedAt) < 2*time.Second {
		err := h.health.err
		h.health.mu.RUnlock()
		return err
	}
	h.health.mu.RUnlock()

	h.health.mu.Lock()
	defer h.health.mu.Unlock()
	if time.Since(h.health.checkedAt) < 2*time.Second {
		return h.health.err
	}

	queryCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
	defer cancel()

	var n int
	err := h.db.QueryRowContext(queryCtx, "SELECT 1 FROM games LIMIT 1").Scan(&n)
	if errors.Is(err, sql.ErrNoRows) {
		err = nil
	}
	if err == nil {
		h.health.checkedAt = time.Now()
		h.health.err = nil
		return nil
	}
	if ctx.Err() != nil {
		return err
	}
	h.health.checkedAt = time.Now()
	h.health.err = err
	return err
}

A static server checks liveness only

A server that serves files has no database. Its check is liveness: 200 ok proves that the process serves HTTP.

Ready is not healthy

A process that just started can answer /healthz before its first data refresh ends. My sports app puts that fact in a header on the same answer: X-Board-Fresh: 1 after the first refresh, and 0 before it. The status stays 200, so a monitor reads the process as healthy. Only the deploy reads the header.

What reads the answer

The sports app runs in a guest with two slots, and one slot takes the traffic. A deploy boots the new build in the idle slot.

The deploy checks from the same machine, because that is where the new process runs. It asks the idle slot at its own address, before the proxy sends it any traffic. The check proves that the new process is well. It does not prove that the internet can reach the service.

for i in $(seq 120); do
  if hdr=$(curl -sf -m 1 -D - -o /dev/null "$guest/healthz") &&
    page=$(curl -sf -m 2 "$guest/") && [[ $page == *Search* ]]; then
    [[ $hdr == *X-Board-Fresh:\ 1* ]] && return 0
  fi
  sleep 0.5
done

The deploy also reads the home page, because a 200 from /healthz does not prove that the page renders. After the first good answer, it waits up to 30 seconds more for the fresh header. A board that is still stale then does not stop the deploy, because the old slot reads the same feeds.

If the idle slot does not answer, the deploy stops it, points back at the previous build, and exits with an error. The old slot keeps the traffic. If the idle slot answers, the deploy points the proxy at it. It stops the old slot when the proxy shows no request in flight there.

A watchdog on the host probes the live slot of each guest every minute. It restarts a slot after three failed probes in a row. An exact 503 db unavailable is not a failed probe: the process answers and the database does not, so a restart would not help.

An outside monitor requests the public /healthz through the CDN. It proves what the local checks cannot: that DNS, the CDN, and the proxy reach the process. The generic body is what makes the public route safe to leave open.

← All articles