A recursive pre-render fleet needs two controls before it needs speed: a ceiling on the workers, and a set that refuses to render the same URL twice.

A React front end and a catalog of millions of listing pages hand a crawler a bill it will not pay in full. My framing at the time: “Google has a certain amount of compute that they’re willing to allocate to your site.” Static pages let a bot “knock through a whole bunch more pages within that amount of compute” than pages it has to render in memory while waiting on Ajax calls. “It’s really not great to ask Google to try to go and navigate to each one of those pages on its own. It just doesn’t work well.”

So on a platform build at Anywhere Real Estate, the answer was a fleet.

Lambda starts Chromium, points it at our own site “to do essentially what Google bot was doing,” snapshots the rendered HTML, and writes it to S3 with a timestamp and the URL it came from. “If we saw a bot” at CloudFront, that snapshot is what got served.

That much is a rendering trick, and it took a few days. What made it a system I could leave running is smaller and much less interesting: a ceiling on how many workers may run at once, and a set that remembers which URLs are already done. This note is the pattern. None of the code below is from that build. It’s reference, written from the current docs.

The crawl budget

Google publishes the mechanism behind crawl budget, but not the number. Its crawl-budget guide defines the budget as “the set of URLs that Google can and wants to crawl,” built out of a crawl capacity limit — which “limits the total amount of time your server spends holding connections open for Google, factoring in both the number of parallel connections and their duration” — and crawl demand, which for Googlebot moves with “a site’s size, update frequency, page quality, and relevance, compared to other sites.” The sentence closest to my own: “There are limits to how much time and resources Google can devote to crawling any single site.” The same guide says who it is written for, in rough estimates rather than thresholds: large sites of 1 million or more unique pages changing moderately often, or sites of 10,000 or more pages changing daily.

What Google does not publish is a price for rendering JavaScript against parsing static HTML. The closest first-party statement is a Google Rendering engineer on the Search Off the Record podcast in July 2024, calling headless-browser rendering “very expensive” and “the exact amount of expensiveness … highly confidential” (Rendering JavaScript for Google Search, episode 77). Google’s JavaScript SEO documentation puts a render queue between crawling and indexing and dates the wait only as a few seconds that can run longer. The hard numbers in public are independent measurements, not Google’s: Vercel and MERJ instrumented nextjs.org through April 2024 and matched more than 37,000 Googlebot render-to-server pairs, landing on a median rendering delay of 10 seconds, a 90th percentile around three hours, and a 99th around eighteen. One site, not a guarantee. No ratio published anywhere — the “rendering costs twenty times more” line in circulation traces to a 2019 conference transcription whose recording is gone, so don’t cite it as a Google figure.

Is this still sanctioned

Serving a crawler a pre-rendered copy of a page is the technique Google named dynamic rendering, and Google deprecated it. The search documentation changelog records the demotion in two steps: the technique stopped being recommended in August 2022, and on 6 February 2024 the entry reads “We updated our documentation on dynamic rendering to clarify it’s a deprecated workaround.” The page itself is still live — titled “Dynamic Rendering as a workaround,” last updated 2025-12-10 — and it now points you at “server-side rendering, static rendering, or hydration as a solution” instead, while still naming two cases where the workaround applies: content that changes rapidly, and crawlers that don’t run JavaScript at all.

The cloaking question is separate and it has a documented answer. Google’s spam policies define cloaking as “presenting different content to users and search engines with the intent to manipulate search rankings and mislead users,” and list as an example “inserting text or keywords into a page only when the user agent that is requesting the page is a search engine, not a human visitor.” The dynamic-rendering page carries the other half: “Googlebot generally doesn’t consider dynamic rendering as cloaking” where the rendering “produces similar content.”

So the verdict is conditionally tolerated, not sanctioned, and you have to read it across two documents — there is no single Google sentence blessing a user-agent switch. There is no post-deprecation first-party statement addressing the pattern either way, and no reported penalty case against one. The rule that carries the weight is content equivalence: the copy the bot gets has to be the page, not a better page.

The front door

Bot detection at the edge started as a string match. “Seeing a bot was really just done by taking a look at the user agent and the request headers, and so there’s lists of user agents out there for all the known bots, and all you have to do is go off and compare it against that list.” The honest ceiling on that, in my own words at the time: “We probably didn’t get everything, but we got all the main players that we cared about.”

A user agent is a claim, not proof, and Google publishes two ways to check it. Its verification guidance gives a manual path — reverse DNS on the accessing IP, confirm the domain is googlebot.com, google.com, or googleusercontent.com, then a forward lookup back to the same address — and an automatic one, matching the address against Google’s own published IP ranges. Those ranges live in common-crawlers.json, which is where Googlebot’s addresses are published alongside its robots.txt token, Googlebot.

The caching half is the part that bites. Do not vary the cache on the header you are matching: CloudFront’s own guidance is that “the User-Agent header can have thousands of unique variations, so it’s generally not a good candidate for including in the cache key,” and the default cache key is the distribution’s domain name and the URL path, nothing else. So the split is a routing decision plus a rewritten path. The bot’s copy and the browser’s copy land on two different cache keys instead of fighting over one.

Where the check runs is a real choice with a documented cost either way. Where my own check ran is not something the record settles. “At CloudFront” is as far as it goes.

From the CloudFront Functions docs: a viewer-request function that routes a self-declared bot to the render cache and rewrites the path.

import cf from "cloudfront";

// Lowercased tokens, kept in the function because a CloudFront function
// has no network access. Google publishes its own token and address
// ranges; a real deployment verifies, it doesn't just match.
var BOTS = ["googlebot", "bingbot", "duckduckbot", "applebot", "yandexbot"];

function handler(event) {
  var request = event.request;
  var ua = request.headers["user-agent"];

  var claimsBot =
    !!ua &&
    BOTS.some(function (token) {
      return ua.value.toLowerCase().indexOf(token) !== -1;
    });

  if (claimsBot) {
    // Origin named in this distribution: the render-cache bucket.
    cf.selectRequestOriginById("render-cache");
    // The rewritten path is what lands in the cache key, so the bot copy
    // and the browser copy never share a cache entry.
    request.uri = "/pages" + request.uri.replace(/\/$/, "") + "/index.html";
  }

  return request;
}

The governor

The cap on a pre-render fleet is one setting, and that is the whole point. “The pre-render pipeline did have a governor. What we could do is control the quantity of concurrent Lambdas that were running at once, and we would do that all the time.” Steady state ran at a few hundred concurrent workers against a burst ceiling several times that — a dial we turned constantly, not a number picked once. “Everything worked; it was just setting one configuration setting inside of AWS to be able to make that happen.”

Two settings, actually, and they are independent. Lambda’s default account quota governs the Region. Reserved concurrency on a function “sets both the maximum and minimum number of concurrent instances allocated to your function” and “incurs no additional charges.” The second dial sits on the queue: maximum concurrency on the SQS event source mapping exists “to prevent one queue from using all of the function’s reserved concurrency or the rest of the account’s concurrency quota.” AWS’s own rule for the pair: “Don’t set maximum concurrency higher than the function’s reserved concurrency.”

The ceiling most people meet first isn’t the account quota at all. The same page: “By default, Lambda can scale to invoke up to 1,250 concurrent function instances for an Amazon SQS event source mapping. If this is insufficient for your use case, contact AWS support.” A raised account quota does not raise that on its own.

From the AWS CLI and Lambda docs — the numbers below are placeholders, and the point is that each dial is one call.

# Dial one: how much of the account's concurrency this worker may hold.
aws lambda put-function-concurrency \
  --function-name prerender-worker \
  --reserved-concurrent-executions 250

# Dial two: how much of that the crawl queue may claim, plus batching
# and per-message failure reporting.
aws lambda update-event-source-mapping \
  --uuid a1b2c3d4-5678-90ab-cdef-11111EXAMPLE \
  --scaling-config '{"MaximumConcurrency":200}' \
  --batch-size 10 \
  --maximum-batching-window-in-seconds 5 \
  --function-response-types ReportBatchItemFailures

Batch size defaults to 10 and goes to 10,000 on a standard queue, and the batching window defaults to 0 and buffers for up to five minutes. ReportBatchItemFailures lets a worker fail three URLs out of ten without dragging the other seven back onto the queue.

The set that remembers

Dedup is not a nice-to-have in a recursive crawl — it is the termination condition. “All of that was handled by putting in the URLs inside of ElastiCache. So we were able to manage inside of a list, right, or inside of a set every URL that we had added or completed already, and if the URL was in there, then we wouldn’t go off and pre-render that.”

A set rather than a lookup followed by a write, because the set command does both in one hop. Redis SADD returns “the number of elements that were added to the set, not including all the elements already present in the set” — one on a URL nobody had claimed, zero on one somebody already had. Read-then-write is a race under fan-out. SADD isn’t. ElastiCache runs Valkey, Memcached, and Redis OSS, and the set commands come with the first and the third.

The other reason the set has to be there is delivery semantics. AWS is blunt about it: “Lambda event source mappings process each event at least once, and duplicate processing of records can occur… we strongly recommend that you make your function code idempotent” (SQS event sources). A fleet that re-renders on every redelivery is a fleet that never finishes.

Path scoping lives in the same step. A run could be pinned to a URL prefix: “let’s say that on the website we just wanted to pre-render all the agent profiles, so we’d be able to say anything with a path that started with /agent,” and the link extractor “would go and only take the ones that it needed to be able to continue the pre-render process.” Filter before the set, not after. The set is what bounds the run, so anything that reaches it is something the run has committed to render.

From the AWS SDK and Redis docs: one worker invocation, from batch to enqueue.

import { PutObjectCommand, S3Client } from "@aws-sdk/client-s3";
import { SendMessageBatchCommand, SQSClient } from "@aws-sdk/client-sqs";
import type { SQSBatchResponse, SQSEvent } from "aws-lambda";
import { createClient } from "redis";

// renderPage drives a headless Chromium instance and returns the
// serialized DOM with its <script> elements removed, plus every href it
// found. footerProvenance returns an HTML comment naming the source URL,
// the render time, and where to go find the logs for it.
import { footerProvenance, renderPage } from "./render";

const s3 = new S3Client({});
const sqs = new SQSClient({});
const cache = createClient({ url: process.env.CACHE_URL });

const SEEN = "prerender:seen";
const PATH_PREFIX = process.env.PATH_PREFIX ?? "/";

export const handler = async (event: SQSEvent): Promise<SQSBatchResponse> => {
  if (!cache.isOpen) await cache.connect();
  const batchItemFailures: { itemIdentifier: string }[] = [];

  for (const record of event.Records) {
    const url = new URL(record.body);
    try {
      const { html, links } = await renderPage(url.href);

      await s3.send(
        new PutObjectCommand({
          Bucket: process.env.RENDER_BUCKET,
          Key: `pages${url.pathname}/index.html`.replace("//", "/"),
          Body: html + footerProvenance(url.href),
          ContentType: "text/html",
        }),
      );

      // Scope first: the set should only ever hold URLs this run will render.
      const candidates = links
        .map((href) => new URL(href, url))
        .filter((u) => u.origin === url.origin)
        .filter((u) => u.pathname.startsWith(PATH_PREFIX))
        .map((u) => u.origin + u.pathname);

      // SADD returns how many members were new. Zero means someone else
      // already claimed it — one round trip both asks and claims.
      const fresh: string[] = [];
      for (const candidate of candidates) {
        if ((await cache.sAdd(SEEN, candidate)) === 1) fresh.push(candidate);
      }

      // SendMessageBatch takes at most 10 entries per request.
      for (let i = 0; i < fresh.length; i += 10) {
        await sqs.send(
          new SendMessageBatchCommand({
            QueueUrl: process.env.QUEUE_URL,
            Entries: fresh.slice(i, i + 10).map((body, n) => ({
              Id: String(n),
              MessageBody: body,
            })),
          }),
        );
      }
    } catch {
      // Only this URL goes back on the queue, not the whole batch.
      batchItemFailures.push({ itemIdentifier: record.messageId });
    }
  }

  return { batchItemFailures };
};

The queue has a second producer in production, and it is the one that made the governor necessary. The master-data team’s webhook fired on every listing change — “this page changed, so let’s go re-render it” — straight into the same queue the recursive crawl feeds. A cap that only bounds the crawl bounds nothing.

Inside the snapshot

Two things happen to the HTML between Chromium and S3. The first is subtraction: “one of the things that we did for optimizing SEO was to strip out all of the JavaScript that was inside the page. There was no need to have JavaScript from React that’s going to go off and try to re-download everything automatically.” The second is provenance — comments injected at the footer recording what was pre-rendered, when, and “how to go find any logs based on it.” The artifact carries its own pointer back into the logging system, which is the same instinct as threading a request ID through everything and putting every log in one place.

That grew into a sidecar. Debug output went to S3 as a separate object beside each rendered page, reachable by appending /debug to any page URL, and it returned “all the raw internal details on what was pre-rendered, what were the decisions made on what links to do, the state of the system and everything else.” The path before that was “pumping a whole bunch of information into Datadog.” The sidecar came “as we started to get more sophisticated.” We were pushing “gigabytes per second at full load” of logs. Whether the sidecar is what fixed that bill is not something I’ve claimed, and this note doesn’t either.

The platform numbers are why the trade is sane in the general case. CloudWatch Logs throttles PutLogEvents at 5,000 transactions per second per account and Region — adjustable, and shared with every other thing in the account. S3 sustains “at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned prefix”, with “no limits to the number of prefixes in a bucket.” One debug object per render, keyed by the page’s own path, spreads across prefixes as the crawl spreads. One log line per render funnels into a single account-wide throttle.

Build or buy

The vendor call was made on price: “their cost was just insane for the number of pages that we had. So building it ourselves was easy.” The vendor was Prerender.io, and what it publishes today is checkable.

Its pricing page tops out, below Enterprise Plus’s custom pricing, at the highest tier carrying a public dollar figure — 500,000 renders a month. At a catalog of millions of pages there is no published number to compare a build against, which is its own answer.

The unit matters more than the tier. Prerender.io counts a render “every time Prerender.io’s rendering service starts up to process a page”: first-time caching, scheduled or on-demand recaching, an uncached bot request, and pages that come back 3xx, 4xx, or 5xx. Cache hits are free. And “by default, Prerender.io renders both a desktop and a mobile version of each page, counting two renders per page.” A weekly full re-crawl of a large catalog multiplies the page count by two, then by the number of runs a month — which is the arithmetic that made building it “easy,” and it’s the arithmetic worth doing before you build one.

No invoice published, mine included. The cost verdict above is mine, not a measurement.

What holds

The two controls in this note are the cheap ones. A concurrency ceiling is a configuration setting; a URL set is one command in the worker’s hot path. Everything else on the list arrives later and arrives at you: “we got it working, and then you try to optimize for speed, you try to optimize for memory usage, you try to optimize for logging, you try to optimize for cost… you fix one issue, and then some other things pop up.” Chromium alone is a category of that — “not super memory-efficient, and has memory leaks all over the place,” which collides directly with reusing a warm execution environment to dodge cold starts, and lands you in instance lifecycle management and Chromium flag tuning.

“I don’t think that you would have been able to figure out from the very get-go, even if you knew exactly what you wanted. It’s just one of those live-and-learn kind of things.” That is true of the optimizing. It is not true of the governor and the set. Those two you can put in on day one, and they are the reason there’s a system left to optimize.

Underneath

Docs checked 2026-09-02. AWS quotas, prices, and doc wording drift — Lambda’s concurrency scaling rate changed in November 2023 and CloudWatch’s log billing changed again in May 2025 — so re-verify any figure here against the live page before it goes in a design.

The quoted lines are mine, from an interview on 2026-08-31. The client is anonymized. The fleet’s own throughput and concurrency figures appear only in rounded form, because they’re recollection rather than anything I’ve gone back and measured — the exact numbers stay out until they’re worth standing behind.

Related system

More field notes

Start