📌 Part of Blueprint: Production Self-Hosted Infrastructure → L3: Observability, Loops & Self-Healing

A Health Endpoint That Rendered the Dashboard: 1.5 s Idle, 4.2 s Under x265

· 8 min read · Engineering health-checksobservabilitysqlitelock-contentionsupervisorwatchdogperformance

Why a 1,500 ms Answer Reads as Absent

A health check is the cheapest possible contract: one request in, one honest boolean out, fast enough that the caller does not have to think about it. It is also the easiest endpoint in a system to let rot, because nothing it does looks wrong. It returns 200. The payload parses. The queue it reports on is genuinely healthy.

I run a media-processing control plane: a supervisor process holding a SQLite queue of roughly 3.8k rows, with per-lane worker threads, a scanner that reconciles the filesystem against that queue, and an HTTP surface used by three different kinds of poller. A desktop shell attaches to it, a tray watchdog watches it, and the dashboard itself polls its state endpoint every three seconds.

The failure that cost a morning: the shell reported “no supervisor on 8790 and none was started” while four encoding lanes were working through the queue. Nothing was down. curl against the same port answered 200 — in about a second and a half, with work running. The shell’s attach probe gives up at 0.5 s, so a live supervisor and an absent one produced the identical verdict: nothing arrived in time.

That is the shape of the bug worth writing down. The endpoint was not broken, it was answering a much larger question than the one it was asked, and a timeout is the one answer that cannot be distinguished from death.

The Endpoint Was Correct, and That Was the Bug

/healthz was calling sup.state() — the same method that renders the entire panel. That payload is not a status line. It assembles the ETA calculation, the advisory and upcoming views, the decision rows, infrastructure failures, job statistics, the licence read, a sysinfo snapshot and the guard states, then serialises all of it.

Measured on 2026-11-10 with one lane encoding:

Condition/healthz latencyShell budget
Idle1.5 s0.5 s
One lane running x2654.2 s0.5 s

Both numbers are already past the caller’s budget before any work arrives, so the endpoint had been failing its only job by design. The second-order damage is worse than the first: three pollers hit that path regularly, so a watchdog, an attach probe and a dashboard each dragged a full dashboard render onto a machine that was already encoding at every tick. A health check that costs what the health check measures is a feedback loop, not an observation.

The fix is not “cache the expensive call”. Caching a payload built for a different consumer just makes the wrong answer arrive sooner. The endpoint needed its own payload, deliberately smaller than the panel’s, with the fields a supervisor-liveness check actually needs.

Six Fields, Two Queries, One Side Connection

The replacement answers with six keys: ok, started, uptime_s, lanes_running, pending, done_today. Only two of them touch the database, and they are the same two counts state() opens with, so the panel and the probe cannot drift apart about the queue.

Three decisions are worth keeping:

Same expression, not same meaning. uptime_s is computed with the identical expression state() uses. Writing a “close enough” version is how two endpoints start disagreeing by a second, and a disagreement between a watchdog and a panel is the kind of discrepancy that burns a whole afternoon to chase.

A cache with its own lock, and a short TTL. Two counts measured 0.1 ms (pending by lane) and 5 ms (done today) on the live queue, which is cheap until three pollers ask at once. Two seconds of TTL is enough to absorb them and short enough that nobody can see a stale number for longer than a breath. uptime_s and lanes_running are computed per request, because both are free.

The count query runs on a read-only side connection, not the shared one. The shared connection is serialised behind a db_lock that the scanner holds in bursts. A probe that queues behind a scan still hands the shell a timeout — the exact failure this endpoint exists to prevent. Measured on 2026-11-10: the cold probe waited 0.72 s for that lock right after a restart, while the same two counts on a short-lived mode=ro URI connection cost 8-18 ms. Read-only is a second guarantee for free: this path can never write, migrate or checkpoint.

🔍 `Supervisor.health()` and the read-only count path (Python, 57 lines)
HEALTH_COUNT_TTL_S = 2.0

def health(self) -> dict:
    """The `/healthz` payload - deliberately NOT `state()`.

    `state()` renders the whole dashboard: insight views (ETA, advisory, upcoming, decisions,
    infra failures, job stats, decided rows, saved), the licence read, sysinfo, guards. Measured
    2026-11-10 with one lane encoding: 1.5 s idle, 4.2 s under x265.
    """
    now = time.time()
    with self._health_lock:
        if self._health_counts is None or (now - self._health_ts) > self.HEALTH_COUNT_TTL_S:
            self._health_counts = self._health_counts_uncached()
            self._health_ts = now
        pending, done_today = self._health_counts
    return {"ok": True, "started": self.started,
            # same expression state() uses, so the two endpoints can never disagree by a second
            "uptime_s": int(time.time() - time.mktime(time.strptime(self.started,
                                                                    "%Y-%m-%d %H:%M:%S"))),
            "lanes_running": sum(1 for w in self.lanes.values() if w.running),
            "pending": pending, "done_today": done_today}

def _health_counts_uncached(self):
    """`(cpu pending, done today)` on a READ-ONLY side connection.

    `self.conn` is serialised with `db_lock`, which the scanner holds in bursts, so a probe that
    queued behind a scan would still hand the shell a timeout. Measured 2026-11-10: the cold probe
    took 0.72 s right after a restart (lock wait), while the same two counts cost 8-18 ms on a
    side connection. `mode=ro` also means this path can never write, migrate or checkpoint.
    """
    count_sql = "SELECT lane, COUNT(*) n FROM files WHERE state='pending' GROUP BY lane"
    done_sql = "SELECT COUNT(*) FROM files WHERE state='done' AND verified_at LIKE ?"
    today = (time.strftime("%Y-%m-%d") + "%",)
    side = None
    try:
        import sqlite3
        side = sqlite3.connect("file:%s?mode=ro" % self.db_path.replace("\\", "/"),
                               uri=True, timeout=5)
        pending = 0
        for r in side.execute(count_sql):
            if r[0] == "cpu":
                pending = r[1]
        return pending, side.execute(done_sql, today).fetchone()[0]
    except Exception:                                     # noqa: BLE001 - fall back below
        pass
    finally:
        if side is not None:
            try:
                side.close()
            except Exception:                             # noqa: BLE001
                pass
    with self.db_lock:
        pending = 0
        for r in self.conn.execute(count_sql):
            if r["lane"] == "cpu":
                pending = r["n"]
        return pending, self.conn.execute(done_sql, today).fetchone()[0]

The fallback at the bottom is deliberate and worth naming: if the side connection cannot be opened, the probe degrades to the shared connection under the lock. That is slower and it is still bounded, which is the correct trade for an endpoint whose worst failure mode is silence. The two paths also index their rows differently — the side connection uses SQLite’s default tuple cursor, while the shared one returns sqlite3.Row — which is why the same two queries are unpacked positionally on one path and by column name on the other.

A Test That Pins the Dependency, Not the Duration

The regression to prevent is not “the endpoint got slow again”. It is “someone re-attached the dashboard renderer because it was already there and the payload looked richer”. Those are different properties, and only the second can be tested reliably — a duration assertion in a test suite is a coin flip on a loaded machine.

So the first test replaces sup.state with a function that raises, then calls health() and asserts the payload is sane. If anyone re-introduces the dependency, the test fails with a message that states the cost:

expected = sup.state()          # snapshot taken before state() is replaced

def boom():
    raise AssertionError("/healthz called state(): that is the 1.5-4.2 s the shell cannot "
                         "wait for")

sup.state = boom
got = sup.health()
assert got["pending"] == expected["queue"]["cpu"]["pending"]   # the panel and the probe agree
assert got["done_today"] == expected["done_today"]

The second property is the one a cache always threatens: it must not become a lie. A row added inside the TTL is allowed to be missing from the count; a row added before the TTL is not allowed to stay missing after it. The test adds a row, asserts the cached count, moves the cache timestamp back past the TTL, and asserts the new count arrives. A cache that never expires passes every timing test and reports a queue that emptied an hour ago.

The suite that holds this is now 1,596 tests, 15 skipped — and the deploy path was as boring as it should be: copy the module into the installed copy, restart through the boundary script with all lanes already stopped so no encode was interrupted, confirm autostart_lanes and every lane’s desired state came back unchanged.

Measured After Deploy

The Result

Carried forward, in three lines. The latency budget belongs next to the payload rather than in the caller’s configuration. Cheap-by-contract means a separate code path, not a faster one, because anything built on the panel’s payload inherits the panel’s future. And a shared connection is a queue: the 0.72 s was not slowness, it was waiting, and no profiler would have charged it to the health endpoint.

🛡️ Production Infrastructure & SRE

Dealing with flaky containers, cloud cost bloat, or unexplained outages?

Real production reliability comes from understanding the kernel and the protocol layers, not just restarting containers. I manage 162 containers across 3 hosts running 24/7 client workloads.

Lee Teong Hoe

Lee Teong Hoe · Mr Hoelee

Full-stack developer & DevOps engineer based in Malaysia. I run my own production infrastructure and build web systems for small businesses — this blog is the living record.

Hiring? Profile & stack · Malaysian SME? Website design & development ↗ · All posts