Environment directories keep Terraform organized. Splitting by change velocity is what keeps state from trying to do too much.

At Cigna, Terraform state lived in S3, split across environments the way most shops do it. That part was never the interesting decision.

The split that mattered ran a different direction — not dev versus prod, but what changes constantly and what doesn’t change at all. Stable infrastructure gets Terraform. Anything tuned all the time does not.

Splitting Terraform state across environments

Splitting Terraform state across environments starts with directories, not workspaces: a dev directory, a test directory, a prod directory, each running its own terraform.tfvars. “That’s always been pretty standard” — every environment gets its own set of variables, and a modules/ directory underneath holds anything reusable across all three.

None of the code in this note is from a client repo. It’s reference, written from Terraform’s own module structure docs — the names are synthetic.

modules/
  ec2-worker/
  s3-bucket/
envs/
  dev/
    terraform.tfvars
  test/
    terraform.tfvars
  prod/
    terraform.tfvars

Workspaces or separate state files

Terraform workspaces vs separate state files — that choice really comes down to how much you trust one configuration to serve every environment. HashiCorp’s own docs are blunt about the limit: workspaces “are not appropriate for system decomposition or deployments requiring separate credentials and access controls,” and the CLI docs explain why: workspaces share a single backend, so real separation means “each subsystem should have its own separate configuration and backend.”

Directories are the version of that separation the interview describes — but directories by themselves don’t create separate state. A dev directory and a prod directory pointed at the same backend config would still write to the same S3 object. What actually separates them is the backend key underneath each directory: each environment’s own backend "s3" { key = ... }, one key per environment, so dev can’t touch prod’s state even by accident.

Modules: golden and cloned

The enterprise kept a set of golden modules in its own GitHub — a VPC module, an S3 module, one for pretty much every service worth standardizing — built to get a freshly provisioned account into shape: transit gateway, subnets, DNS, routing, organized the way the enterprise wanted it organized. Pick the ones you need, bring them into your project, wire them up. Lego blocks.

That works until a module doesn’t do what you need, or you need something faster than the module’s owners can turn around. “What a lot of teams end up doing is they clone the module… copy all the files into your local repo and then be able to modify and make the changes that you want there. There is no hard and fast rule that says you need to use these from GitHub.” The owning teams “aren’t necessarily quick to jump on it if you want to make changes or if you submit PRs to make updates.” Cloning and diverging locally is the accepted escape hatch. Nobody apologizes for it.

Inside a single project, the same shape repeats: a modules/ directory of submodules for anything reusable, and a main.tf that “should just be spinning up a bunch of individual modules. It shouldn’t really be creating resources on the fly.”

Secrets stay outside the resource graph

Terraform never creates secrets, full stop: “we don’t want to store any secrets or any kind of sensitive information in there.” Values get created directly in SSM Parameter Store or Secrets Manager, by hand, and Terraform’s role stops at reading them by reference with something like the aws_ssm_parameter data source.

That’s not the same as keeping the value out of state, though. The provider’s own docs warn: “the unencrypted value of a SecureString will be stored in the raw state as plain-text.” HashiCorp’s own state documentation is blunt about why that’s unavoidable — state stores values in plain text regardless of where they came from, so anything sensitive that ends up there, created by a resource or merely read back by a data source, is readable by anyone who can read the file.

What the boundary actually buys you sits upstream of state: the value is never created or managed by a Terraform run, so a leaked .tf file or a tfvars file checked into version control was never the thing holding it. On larger teams, that separation extends to who’s allowed to create the value in the first place. A product owner, not a developer, creates it via the console. Developers verify it’s right. Only then does the deployment proceed.

The lock table

Locking runs through a single DynamoDB table — one table, one entry per state file, so two deployments can’t step on each other. “That works pretty well.”

From HashiCorp’s own S3 backend docs:

terraform {
  backend "s3" {
    bucket         = "example-tfstate"
    key            = "orders-worker/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "example-tfstate-locks"
    encrypt        = true
  }
}

The same docs also cover use_lockfile, which locks through the state bucket itself instead of a separate DynamoDB table — and as of the current S3 backend docs, DynamoDB-based locking is deprecated and slated for removal in a future minor version. The table above is still the version I’ve actually run; use_lockfile is the direction new setups get pointed toward now. When a lock gets stuck instead of the state getting corrupted, that’s a different rung on a different ladder.

Split by change velocity

The split I’m actually proud of doesn’t run along environment lines at all. It runs along how often something changes. “Pure standalone infrastructure that isn’t going to change — you can have your own Terraform for that. And then afterwards, for things like your Lambdas… use something that’s either pure CDK or pure AWS CLI or APIs to push updates.” The thesis: “the biggest issue with Terraform and these other platforms is trying to make it do too much.”

A VPC doesn’t change week to week. A Lambda’s memory setting might change today because someone’s chasing a timeout. Running the full Terraform for that is real friction — “it’s painful to run the full Terraform” just to bump a number. So inside the resource, Terraform is told to stop watching that field. I never called the mechanism by name at the time. HashiCorp’s own lifecycle { ignore_changes } is where the name comes from — something else, a CLI call and not another Terraform run, manages the field from there.

From the AWS and Terraform docs — the resource, then the CLI calls that tune it afterward:

resource "aws_lambda_function" "orders_worker" {
  function_name = "orders-worker"
  role          = aws_iam_role.orders_worker.arn
  handler       = "index.handler"
  runtime       = "nodejs22.x"
  filename      = "lambda.zip"
  memory_size   = 512

  lifecycle {
    ignore_changes = [memory_size, reserved_concurrent_executions]
  }
}

aws lambda update-function-configuration \
  --function-name orders-worker \
  --memory-size 1024

aws lambda put-function-concurrency \
  --function-name orders-worker \
  --reserved-concurrent-executions 20

Both calls are documented in the AWS CLI reference: update-function-configuration for memory, put-function-concurrency for reserved concurrency.

One wrinkle: ignore_changes only applies to update operations. On a create — including a forced replacement, a runtime bump or filename change that recreates the Lambda — Terraform still considers those attributes, so the new function comes up with whatever’s written in the resource block, memory_size = 512 included, and the CLI-tuned settings are gone until that external process runs again.

Terraform state file too large

Terraform state file too large is a real problem for some teams — I just haven’t been one of them. “I can’t say that I’ve worked on any projects where that’s the case, but I’ve heard of that.” It isn’t the velocity split keeping state small, either: ignore_changes only tells Terraform to stop planning updates to memory_size and reserved_concurrent_executions — it doesn’t remove those attributes from state, and it doesn’t freeze them at whatever Terraform last applied. The Lambda above still carries both in state, and a normal terraform refresh syncs in whatever the CLI actually set, ignored fields included. What keeps state small here is scope, not behavior: this entry never put many resources in front of Terraform to begin with, and state size tracks resource count, not how often those resources change.

Terraform was the tool I actually knew going into that mandate. Deciding between Terraform and CDK on the next one came down to something else entirely — who held the mandate, not which tool was better. And none of the golden-module Lego trick works until an account is already organized enough to receive it. That part is landing-zone work, before Terraform ever runs.

More field notes

Start