A rendering decision made for users is a bill the crawler pays.

What was on fire

Nothing was paging. We had moved the Anywhere Real Estate site from Next.js to React, and on the thing we were hired for, it worked — the page appeared immediately and the data filled in as it arrived, which also hid how slow the master-data APIs behind it were. Googlebot got the same page and had to render it in memory, wait on the Ajax, and then work out what to index. There were roughly 3 million listings we wanted indexed.

The way I think about it: Google has a certain amount of compute that they’re willing to allocate to your site. Static pages let the bot knock through a whole bunch more of them inside that budget than pages it has to render itself. With 3 million listings, it’s really not great to ask Google to go and navigate each one on its own. It just doesn’t work well.

Google’s own documentation is looser than my framing. It defines crawl budget as crawl capacity plus crawl demand, and puts the practical concern at roughly a million or more unique pages — but it publishes no per-site compute allotment and no JavaScript cost multiplier. The threshold is the part that held.

None of this was planned. The site was built first, and the realization that a pre-render solution gated the push to prod came after.

Constraint set

We didn’t own the backend. The master-data team held the APIs and the change notifications, and we couldn’t make either faster, so rendering in the request path would have put every page view behind latency we didn’t control. Changes on that side, caching included, were not available to us.

The team was the other constraint, and it’s the honest one. The front-end capability I could staff was React, not Next.js, and nobody I had knew the stack that was already there. That chose the framework, and the framework chose the rendering problem.

System diagram

Fig. 01 · pinch or scroll to zoom · drag to pan

Close

+ − Fit

Architecture decisions

1. Build the fleet instead of buying it. We talked to Prerender.io. Their cost was just insane for the number of pages that we had, so building it ourselves was easy. Their published tiers still stop at 500,000 renders a month with anything past a million priced on request, and the billable unit is the render, not the cached hit. A weekly re-render of the whole catalog bills every page again.
2. Split bots from humans at CloudFront. User agent and request headers, matched against the published known-bot lists. Humans keep the React app, a bot gets the snapshot out of S3. The ceiling is honest: we probably didn’t get everything, but we got all the main players that we cared about. Google itself has called this pattern — dynamic rendering — a deprecated workaround since February 2024, and points at server-side rendering, static rendering, or hydration instead. Its cloaking rule turns on intent to manipulate rankings plus materially different content, not on detecting a user agent, so serving a crawler the same page it would have rendered itself stays on the right side of that line. Tolerated, not recommended.
3. A recursive crawl with one dial on it. Seed the homepage into ElastiCache, fire an SQS event, and Lambdas take micro-batches of about 10 URLs each: render, extract the internal links, push those back into ElastiCache, write the page to S3. The frontier and the dedup are the same object — a set of URLs added or completed, and a URL already in it doesn’t get rendered again. The governor is the concurrent-Lambda count, and it was just setting one configuration setting inside of AWS: up to 5,000 at full burst, about 300 as the everyday steady state, tuned all the time. Lambda’s default account limit is 1,000 concurrent executions per Region, raised on request. A per-function reserved-concurrency cap costs nothing.
4. Re-crawl on a schedule instead of trusting the change feed. The webhooks missed changes, and one edit in the internal portal could touch hundreds or thousands of listings at once. So: the whole site, weekly, plus on-demand runs scoped to a path — restrict a run to everything under /agent and the link extractor keeps only the matching URLs as it goes. Listing pages were the target. Search results and the sitemap pages were fine on their own.

Not again: don’t let the team you can staff choose the rendering architecture.

Recovery / operate path

What landed in the bucket was not the raw snapshot. The JavaScript came out first — no reason to ship React that will go re-download everything the moment it loads — and a comment block went into the footer recording what was pre-rendered, when, and how to find the logs for it. The page carried its own pointer back into the system that made it.

Full time to download and render went from 5 to 10 seconds to under 200 milliseconds a page — just enough time for CloudFront to hit the S3 bucket, pull the data back, and send it on. A full re-render of 3 million pages took about an hour and a half.

Built in the course of a few days, and it needed hand-holding from then on. What it cost was debugging. A lot of incremental issues that were really a pain to understand, debug, and decipher. The first answer was pushing everything into Datadog. The better one came later: every rendered page got a debug file beside it in S3, and appending /debug to any page URL returned the raw internals — what was pre-rendered, which links the crawl decided to follow, the state of the system. Chromium was the other cost. It is not memory-efficient and leaks all over the place. That collides directly with reusing a Lambda execution environment to dodge cold starts, so instance lifecycle and Chromium flags became things we managed rather than set. You fix one issue, and then some other things pop up. It’s one of those live-and-learn kind of things.

The SEO outcome I can’t hand you cleanly. Search Console’s indexed-page count fell off a cliff after the release. The same release changed the site’s URL structure — not a rendering decision — behind a redirect layer at CloudFront, and where that old-URL mapping was incomplete Googlebot got 404s instead. That cutover is its own story. The drop is real and I won’t pin it on client-side rendering alone. I also don’t have the landing. I left the project before things were finalized. Things were on a better trajectory, they got better over time, and they stabilized. No recovery number. No return-to-baseline claim.

What changed after

An invalidation from the master-data team meant one thing to us — pre-render that page. A normal day was somewhere between 100,000 and 2 million of them, and I never pinned that number down. Then a bug on their side made it 20 million in a day, and the fleet did exactly what it was built to do: it scaled up to meet the demand. Those renders hit our own site, our site hit their APIs, and their database saturated further. Their bug, via us, caused the whole thing to crumble.

The controls on my side were already there — the concurrency cap, and the URL set that skipped anything queued or already done. What was missing sat on the other side of the boundary. We had been told the store could handle the load. It could not. That is the part I carry now: a dependency’s claimed capacity is not its measured capacity, and a fleet that auto-scales against someone else’s change feed is a lever they can pull without knowing they pulled it.

And the larger call, on the record. In retrospect, would it have been better to just keep Next.js and not deal with all the pre-render stuff? Yeah, probably. We were hired for the user experience and we delivered it, with all accolades — so much better in React than in Next.js. We traded one for another, and that was probably in retrospect a problem or a failure. I’d probably do it differently if I had the opportunity.

Anonymization notes

The client ships by name; the people do not. The internal data team ships as “the master-data team” and nothing finer. No people, roles only. Numbers are as I gave them, rounded, and the invalidation baseline stays a range because I never pinned it down. Stack footnote: React, AWS Lambda running headless Chromium, CloudFront, S3, SQS, ElastiCache.

Who this is for

• A single-page app with millions of URLs that have to be in the index.
• A rendering architecture that got chosen by who you could hire.
• An auto-scaling pipeline sitting downstream of a change feed another team owns.
• Anyone about to defend serving crawlers pre-rendered HTML to a reviewer who has read Google’s current guidance.

Related systems

Related field notes

More systems

Start