Design the degraded mode before the shared store you don't own goes down.

What was on fire

The product was down. AWS was green.

A shared MongoDB was saturated. It belonged to the master-data team — one giant cluster holding everything about a listing, behind their Golang APIs on ECS/Fargate — and when it went, their APIs stopped answering and our health checks went red.

The cause was a bug in their ETL. It was calling our invalidation webhook about 20 million times a day against roughly 3 million listings, so we effectively never cached. Every page request went through to their APIs, and every one of those went to the one Mongo.

Then it got worse, through us. Each invalidation also asked our pre-render fleet — Chromium on Lambda, rendering whole pages to S3 — to re-render a page. That fleet scales on demand, so it scaled. Its renders hit our own site, our cache misses hit their APIs, their APIs hit the same database. Their bug, via us, brought the whole thing down.

Constraint set

The backend belonged to another team and stayed that way. We had REST endpoints and nothing else: no eventing to tell us a listing had changed, and APIs built for backend jobs rather than for a page, so composing one page meant several calls. A schema reshape or a cache on their side was not available to us. A redesign of that plane was never in the window. The product still had to survive it.

System diagram

Architecture decisions

All of these predate the incident. Nothing here was built under fire — what was already in place is what decided how the day went.

1. Cache in front of a store we didn’t own. Their APIs were slow. The mid-tier — Node.js on Lambda — gave the UI the response shapes it actually wanted and cached them. The pager does not care who holds the ticket.
2. The ladder, in the order we got to it. ElastiCache first, on a TTL of minutes, because a realtor who edits a listing expects to see it on the site immediately and we had no event to tell us when that happened. Holding whole response bodies in memory got expensive at that many listings, so the cache moved to S3 — the composed response stored as an object. Fast enough to serve, durable, and it scales past anything we would ask of it. CloudFront sits above both, storing whole rendered pages in S3.
3. A webhook instead of a TTL guess. We gave the master-data team’s ETL a webhook to call: changed listings land in a queue, the queue invalidates, the next request repopulates. It took the guesswork out. It also handed a team we did not control a direct line into our cache, which is the whole story.
4. Degraded mode designed in, not improvised. If the backend failed we still wanted to put something in front of the user, so serving the last good page from CloudFront was the default — not a call somebody made in the room that day.
5. Log every call. Logging every invalidation is what let us diagnose their bug from our side of the fence. That volume of logs became its own budget problem, which is a separate story.

Not again: don’t let a signal from a team you don’t control drive a fleet that scales itself. Rate-check it against the size of the catalog first.

Recovery / operate path

Datadog paged first. Our health checks run on the Lambdas and make real calls to their APIs. When those calls failed, the check went red — and nothing came out of rotation, because on Lambda behind CloudFront there is nothing for a red check to pull. It was the alarm, not the breaker.

The fallback held, and its limit is the honest part. Anyone with a bookmark got their page from CloudFront. Anyone actually using the site did not: search and filters are dynamic, have no cached page, and were dead. On a real estate site, that is the product.

Detection was never the bottleneck. Finding a human on the owning team took several hours — no on-call, so we escalated through their management chain. Another couple of hours for them to identify the overwhelmed Mongo, then support tickets to the vendor. After a delay Mongo released its resources on its own and the system started scaling back up. Their root-cause fix landed several days later. A very long outage, and hours of it went to finding someone to tell. No MTTR published. No game-day claim.

We knew what was wrong before they did. We log every call into the invalidation webhook, so the arithmetic was sitting there: about 20 million invalidations a day against roughly 3 million listings does not pass a sniff test. That is how we told them they had a bug.

What changed after

Not much needed to change on our side. The bug was theirs, the root-cause fix was theirs, and ownership of the store stayed where it was.

What I built afterward was proof: take an invalidation, rebuild the cache entry, diff it against what was there. “This listing changed” becomes checkable against what actually changed, which is how we showed which invalidations were spurious. The sniff test itself stayed arithmetic a human did. No standing invalidations-per-listing alarm claimed.

The two things that decided the day were never ours to fix. We had been told the store could handle the load. It could not, and when it broke there was no on-call to reach.

Anonymization notes

The client ships by name; the people do not. The backend team ships as the master-data team. No people, no org chart, no ticket IDs. The invalidation rate and the listing count are rounded as they were reported. Stack footnote, with the attribution that matters: our mid-tier was Node.js on Lambda behind CloudFront, with S3 and ElastiCache underneath; the master-data team’s APIs were Golang on ECS/Fargate over a single MongoDB. The timeouts cascaded on their plane, not ours.

Who this is for

• A product sitting on a shared store another team owns, reachable through REST endpoints and nothing else.
• An auto-scaling fleet driven by a signal from a team you don’t control.
• Users down, and the clock running on somebody else’s escalation path.

Related systems

Related writing

Related field notes

More systems

Start