My eval is the health check

The thing that stops a deploy of mine is a health check. It runs about once a minute against the deployed system and gates the pipeline: health fails, the version doesn’t ship. In the field, eval means something narrower — DoorDash held a base-model swap when its offline harness scored the new one worse. Good practice, and never the thing that stops mine.

The load balancer runs my test suite left this out: the check validates every dependency the service can’t work without. It carries a rolling count of logged errors — enough inside a window and health flips. Both dials live in config, tuned per service. It returns the error text, and that instance’s memory, CPU, and I/O. A few extra lines of JSON, and the health response is the first page of the incident, already written.

engraving of a monaural trumpet-form stethoscope laid diagonally beside an open-cased pocket watch on a chain, the watch face showing Roman numerals and a subdial, the case open enough to hint at the movement

The orthodoxy says keep the check shallow: fail on a shared dependency and every instance fails at once, emptying the fleet. It has a scar behind it: Buildkite published a postmortem where exactly that happened, replacements included. I ask the other question: is that instance really healthy if its dependencies are broken? Keeping it in rotation only means the customer gets the error instead of the load balancer.

The one time this played out on me: Anywhere Real Estate, my mid-tier on Lambda caching whole response bodies, the master-data team’s ETL calling a webhook on every listing change. Their bug invalidated everything, all day. We effectively never cached. Every request went through to their APIs, and we blew up their Mongo. Our health checks called those APIs, and the calls were failing.

Nothing got pulled — on Lambda behind CloudFront there was nothing to pull. The check was the alarm, not the breaker. The degraded mode we designed in held: backend down, serve the last good page. Anyone with a bookmark was fine. Search and filters are the product, and they were dead.

Fig. 01 — Detection took minutes; everything after it waited on the team that owned the bug. Not to scale.

The two checks, side by side:

	Liveness ping	The deep check
Flips it	Not a broken dependency — that’s what staying shallow avoids	A dependency it can’t work without, or a rolling error count past its threshold
Behind a load balancer	Passes anyway — the instance stays in rotation and the customer gets the error instead of the load balancer	Fails and comes out of rotation — on a shared dependency, that can empty the whole fleet at once (Buildkite’s postmortem)

Datadog paged in minutes, and with nobody on call over there, finding a human took hours. We found their bug from our side of the fence: roughly twenty million invalidations a day didn’t pass a sniff test against only three million listings. Detection was never the bottleneck. Escalation was. I don’t ship a check that lies to keep a box in rotation, and each single point of failure it exposes is the next thing I harden.

More writing

I gate on what a test suite can check
The line for an autonomous system is verifiability, not consequence
Newer · Sep 2026

Autonomy expands with instrumentation, not capability
Why the most autonomous system I run got there through telemetry, lineage, and gates, not smarter models
Older · Aug 2026

Start

Tell me what’s stuck

I’ll tell you in about a day whether I’m the right person. The first conversation is fit, not a free architecture review.