Field notes

A canary on a Lambda alias rolls back at the traffic layer. The source layer only moves forward.

engraving of a small canary perched on a bar inside a domed brass birdcage hung by a ring from a chain, the ring carrying a faint illegible foundry-style scroll mark, seed scattered on the cage floor

I’ve done this in a variety of ways. The one I reach for on Lambda is a canary push: “as you deploy Lambda, you deploy a new version, and then after that you can route traffic incrementally from the old version to the new version. If, as you start to route traffic to the new version, you see the health go away, you would drop traffic going to the new version and put everything back to the old version.” It costs something up front — “it does take a little bit more orchestration time, it does take a little bit more time to be comfortable with, but in the world of AWS it works great, actually.”

None of the code on this page is my configuration — it’s reference, written from AWS’s current docs, the mechanism the platform documents and you can check yourself. The positions and the quotes are mine.

source layer — one way traffic layer — reversible code change pipeline new version alias weights health signal health goes away health holds all traffic on new version
Fig. 01 — The rollback is one weight change on the alias, not a trip back through the pipeline.

Fig. 01 · pinch or scroll to zoom · drag to pan

The health signal

The health check is what the canary watches, and it’s the same check that gates the pipeline: “it’s normally baked into the whole CI/CD pipeline if you’re checking the health and it’s failed… it shouldn’t be deployed.” That check is not a liveness ping. It validates dependencies, carries a rolling in-memory error tally that can flip it unhealthy, and returns the error text with it. It runs every minute or so. I’ve written up the whole doctrine separately — the load balancer runs my test suite.

Two things I won’t reconstruct here, because I’d be inventing them: what the health signal was wired to on my own Lambda canaries — CloudWatch alarms on the alias’s metrics, or a CodeDeploy hook calling the endpoint — and who moved the weights, CodeDeploy or orchestration I wrote. The exhibits below are what the platform documents, not what I ran.

Traffic, not source

An AWS deployment rollback strategy has two layers, and I only accept the reversal at one of them. Shifting traffic back to the old version is a rollback, and I’ll take the word for it: “I guess you are correct that it’s called a rollback… maybe change my naming convention to a rollback.” What I refuse sits a layer down: “definitely nothing as far as rolling back inside of GitHub or whatever. It’s just making changes to your code, running it through the pipeline, and getting it deployed again.”

The reason the cheap reversal exists at all is the deploy target. “If you didn’t have the kind of infrastructure to spin up a new Lambda version, or a new ECS container, and then route traffic to it, then what would have been done is, say, deploying static code to an EC2 instance” — rolling back the instance. Versioned targets are what turn a rollback into a weight change.

Published doctrine doesn’t draw the line where I do. Google’s SRE Workbook treats patching in production as the position you’re stuck in when rollback isn’t available: with no option to roll back to a known-good configuration, “our best option to fix the errors is to find defects in the production version, patch them, and deploy a new version during the outage. This course of action will almost certainly prolong the user impact of the bug.” PagerDuty’s incident-commander steps put it shorter — “Bad Deployment: Roll it back” — and sequence that ahead of knowing the cause. Both are arguing about a layer I’ve already automated. My source-layer rule is only safe because of the schema rules further down. Without those, published doctrine is right and I’m wrong.

Shifting the weight

A Lambda alias can point at two published versions at once, and the second one’s weight is the fraction of invocations it gets. AWS documents the constraints plainly: an alias points to a maximum of two versions, both must be published ($LATEST is not eligible), and both must carry the same execution role and the same dead-letter-queue configuration. The first version takes the residual weight — set the additional version to ten percent and the original is assigned ninety automatically. Routing is probabilistic, so at low traffic the actual split drifts from the configured one.

From the AWS CLI docs — the shift out, the shift back, and the finish:

# ten percent to version 2; version 1 keeps the rest
aws lambda update-alias \
  --function-name my-function \
  --name live \
  --function-version 1 \
  --routing-config AdditionalVersionWeights={"2"=0.1}

# health went away: every request back on version 1
aws lambda update-alias \
  --function-name my-function \
  --name live \
  --function-version 1 \
  --routing-config AdditionalVersionWeights={}

# health held: version 2 takes all of it
aws lambda update-alias \
  --function-name my-function \
  --name live \
  --function-version 2 \
  --routing-config AdditionalVersionWeights={}

Every invocation says which version ran it. Lambda writes the version into the START log line, exposes it as the ExecutedVersion dimension on alias metrics, and returns an x-amz-executed-version header on synchronous invocations.

Letting CodeDeploy hold the dial

CodeDeploy runs the same weight changes on a schedule, so nobody has to sit on the CLI. Its predefined Lambda deployment configurations come in three shapes, each name carrying a CodeDeployDefault. prefix. Four canaries — LambdaCanary10Percent5Minutes, and the same move at ten, fifteen, and thirty minutes — send ten percent first and the remaining ninety after the interval. Four linear schedules shift ten percent every one, two, three, or ten minutes. LambdaAllAtOnce shifts all of it immediately. The ECS equivalents sit on the same page, which is the other half of “a variety of ways.”

Attaching a CloudWatch alarm to a deployment group stops the deployment when that alarm fires. Rolling back is a second, separate setting — “Roll back when alarm thresholds are met,” which redeploys the last known good revision. Stopped is not the same as reverted, and the difference is one checkbox.

Configuration First shift Then Interval
Canary (LambdaCanary10Percent*) 10% Remaining 90% 5, 10, 15, or 30 minutes
Linear (LambdaLinear10Percent*) 10% +10% every interval, up to 100% Every 1, 2, 3, or 10 minutes
LambdaAllAtOnce 100% Immediate
Alarm attached, stop only Deployment stops when the alarm fires
Alarm attached, roll back Redeploys the last known good revision

From the AWS SAM docs, which wire both ends for you — the alias, the CodeDeploy application, the schedule, the alarms, and the rollback:

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      Runtime: nodejs24.x
      CodeUri: s3://bucket/code.zip
      # publishes a version on every code change and points "live" at it
      AutoPublishAlias: live
      DeploymentPreference:
        Type: Canary10Percent10Minutes
        Alarms:
          - !Ref AliasErrorMetricGreaterThanZeroAlarm
          - !Ref LatestVersionErrorMetricGreaterThanZeroAlarm
        Hooks:
          PreTraffic: !Ref PreTrafficHookFunction
          PostTraffic: !Ref PostTrafficHookFunction

The hooks are where a health check gets a vote. A Lambda deployment has exactly two, BeforeAllowTraffic and AfterAllowTraffic, and each is a Lambda function CodeDeploy invokes and then waits on. Neither one runs during the shift: the first runs before traffic moves at all, the second after all of it has moved. So for the ten minutes a tenth of production is on the new version, the alarms are the only thing watching.

From the CodeDeploy API and the AWS SDK for JavaScript v3 — a pre-traffic hook that calls the deployed function’s health endpoint and reports the verdict:

import {
  CodeDeployClient,
  PutLifecycleEventHookExecutionStatusCommand,
} from "@aws-sdk/client-codedeploy";

const codedeploy = new CodeDeployClient({});

interface HookEvent {
  DeploymentId: string;
  LifecycleEventHookExecutionId: string;
}

export const handler = async (event: HookEvent): Promise<void> => {
  let status: "Succeeded" | "Failed" = "Failed";

  try {
    // the deep check — dependencies, error tally, stats — runs behind this URL
    const res = await fetch(process.env.HEALTH_URL as string, {
      signal: AbortSignal.timeout(10_000),
    });
    status = res.ok ? "Succeeded" : "Failed";
  } catch {
    status = "Failed";
  }

  // report either way: silence for an hour is itself a failed deployment
  await codedeploy.send(
    new PutLifecycleEventHookExecutionStatusCommand({
      deploymentId: event.DeploymentId,
      lifecycleEventHookExecutionId: event.LifecycleEventHookExecutionId,
      status,
    }),
  );
};

That last comment is a documented rule, not a style preference: CodeDeploy treats the deployment as failed if the validation function doesn’t call back within one hour. The status takes only Succeeded or Failed.

The schema only moves forward

A schema change is the case with no traffic layer to hide in. “How do you roll back a database schema change without there being issues? I don’t know if I have clear perfect ways of doing it, but I can tell you what I’ve done in the past.” Three rules, and they’re what make refusing a source-layer rollback survivable.

  1. Nothing breaking, either direction. “Making sure that all your code is backwards compatible. Making sure that any new changes that you make are not breaking changes. That’s from an API perspective, that’s from a database perspective.”
  2. A code rollback never contracts the schema. “You’re not going to go back to the database and drop those tables or drop new columns that were created. You will just keep the database the way that it is, and then on the next push… your database migration scripts would facilitate.”
  3. Migrations ride in the service and run at startup. “When the service runs, the first time that it spins up, it will go and look at the database and see what version it’s at, and if it needs to be upgraded, it will upgrade on its own. Obviously you have a locking mechanism so that you don’t have multiple services trying to upgrade the database at the same time.”

Expand-only migrations are the whole trick. The schema moves one direction, the code moves either way, and the old version stays runnable the entire time the canary is out. Take that away and shifting the weight back stops being free — the version you’re shifting back to is now talking to a database it doesn’t recognize.

The rule

Rollback is a word about layers. At the traffic layer it’s a weight on an alias, it’s automatic, and I want it wired before the first deploy. At the source layer it’s a revert, a pipeline run, and a deploy of old code onto a schema that has already moved — so I fix forward instead, and I pay for that up front with non-breaking APIs and forward-only migrations. If your deploy target isn’t versioned, you don’t get to make this distinction at all. That’s the argument for versioning it.

Underneath

Docs checked 2026-09-02. Every technical claim above comes from AWS’s own documentation. Alias routing and its constraints: Implement Lambda canary deployments using a weighted alias. The predefined canary and linear schedules: Working with deployment configurations in CodeDeploy. The alarm-stops-versus-rollback distinction: Configure advanced options for a deployment group. The template shape: Deploying serverless applications gradually with AWS SAM. The hook list and the one-hour callback rule: AppSpec ‘hooks’ section. The hook’s parameters and its two usable status values: PutLifecycleEventHookExecutionStatus. nodejs24.x as a supported runtime: Lambda runtimes. The hook exhibit typechecks under TypeScript strict mode against @aws-sdk/client-codedeploy 3.1124.0.

The doctrine quotes are Google’s SRE Workbook on canarying releases and PagerDuty’s incident response documentation.

More field notes

Start

Tell me what’s stuck

I’ll tell you in about a day whether I’m the right person. The first conversation is fit, not a free architecture review.