Terraform state breaks. This is the ladder I climbed before nuking dev.

The biggest cause, in my experience: a local-machine deploy where our AWS session token dies mid-apply and we’re in Terraform limbo. After that, a flaky VPN killing an apply mid-flight, or a security-group rename — AWS won’t rename a security group in place, so you create a new one, reattach everything hanging off it, and drop the old one, which gets ugly fast on a group with a lot of resources tied to it. Destroying your Terraform state happens. It’s just a fact of life.

In order, the rungs are: a plan that throws the error, downloading the state, working out what’s actually wrong, a targeted import, nuke-and-reimport, and — only in dev — tearing the whole environment down and letting Terraform rebuild it.

Reading the plan output

Terraform state file corruption recovery starts on terraform plan — specifically a plan that errors because the state and what’s actually in AWS disagree, not a plain syntax or provider error. None of the commands below came off my terminal. They’re reference, written from the docs, and the addresses, IDs, and bucket names are placeholders.

terraform plan
terraform state pull > "state.backup.$(date +%s).json"
terraform state list
terraform state show 'aws_security_group.app'

That timestamped file is a local backup on its own, no matter what the bucket does — timestamped instead of a fixed name, so a second pull during a retry doesn’t overwrite the first one. Bucket versioning on the backend "s3" bucket is a separate, S3-side backup — turned on, a corrupted write still has an earlier object version to restore from directly. It isn’t the default, so it has to be set explicitly. state list and state show are just reading the state file’s own opinion of what exists, before touching anything.

Import, then what’s already gone

The next rung is a targeted terraform import: tedious, and the sequencing is usually what makes it painful — each resource needs a matching config block already written, one command per resource, and there’s no dependency graph doing the ordering for you.

terraform import 'aws_security_group.app' sg-0123456789abcdef0

Import needs the provider’s own ID format for that resource — a security group ID here, not a name. Sometimes the plan is arguing with reality because a resource is genuinely gone: you (or the console cleanup further down) deleted it by hand, and Terraform is still watching for it. That’s what terraform state rm is for — it forgets the resource, it does not delete it, and the next plan will try to create it again if it’s still needed.

terraform state rm 'aws_security_group.deprecated'

It’s the wrong tool for a resource that’s still there but stuck. Lambdas attached to a VPC leave behind network interfaces that don’t die when the Lambda does, and those Hyperplane ENIs stick around until no other function or version sharing that subnet-and-security-group combination is still around — long enough to block a dependent security group or subnet from deleting at all. Removing that security group from state wouldn’t release the ENI or delete the group. It would just leave Terraform trying to create a duplicate on the next plan. The actual fix there is waiting the ENI out, or chasing the dependency chain by hand once AWS lets go of it.

terraform state mv is the sibling command, for renaming an address in state without touching the real resource — same family, different job.

Restoring state from S3

Recover terraform state from backup when the state file itself, not just one resource, is the problem, and the local pull above isn’t recent enough on its own — S3 backend versioning holds every prior version of the state object, and you can roll the object back directly. Pick the version by timestamp from list-object-versions, the last one before the corrupting write, not just the newest older one. Take a fresh state pull backup before doing this, confirm nothing is mid-apply anywhere — no lock held, no CI run queued — and run terraform plan again afterward before trusting the restored version:

aws s3api list-object-versions --bucket my-tf-state-bucket --prefix env/dev/terraform.tfstate
aws s3api copy-object --bucket my-tf-state-bucket --key env/dev/terraform.tfstate \
  --copy-source 'my-tf-state-bucket/env/dev/terraform.tfstate?versionId=00000111122223333'

terraform force-unlock is a rung that shows up here too, for a lock left stuck instead of a state left corrupt — never run it against a lock a run still holds. That lock is almost always a single DynamoDB table, one entry per state file.

Nuke and reimport, dev only

Nuke-and-reimport is what’s left once targeted fixes and a restored state don’t get you there: tear down what’s actually broken and let Terraform rebuild it from a clean plan. In a dev environment that’s fine — you just recreate everything by running Terraform once it’s healthy again. In anything closer to production, you don’t do this. The account nuke I’ve done was never the account itself — it was going through the console and manually deleting resources one at a time until dev was clean enough to start fresh.

Several days, once

One of these took several days at Cigna, early on — banging my head against nuke-and-reimport and everything else I could think of before it broke loose. Some of that was inexperience with Terraform itself; some of it was that nobody wanted to talk to the people who’d just acquired us, so we fixed it ourselves instead of asking for help — teach a man to fish.

Assisted, not automatic

The real help I got: inside Cursor, I downloaded the state file from S3 — worth remembering that a state file can hold plaintext secrets, so it’s not something to feed into just any tool — and had it analyze what was going on, and it gave me a sequence of things to try. That was back when I didn’t trust AI the way I do now, so it was “help me debug and understand what’s in the state file, and what paths I should try” — assisted, not autonomous, and the risk is more mitigated now than it was before AI got involved. Same idea behind autonomy expands with what you can verify, not with what the model can do.

One rung gets easier to avoid if it’s built in ahead of time: the security-group rename that starts a lot of these. AWS won’t let you rename a security group after it’s created, and Terraform models that as a forced replacement. The create_before_destroy lifecycle is the standard mitigation — it stands the replacement security group up before tearing down the old one, instead of dropping the old one first:

resource "aws_security_group" "app" {
  name_prefix = "app-"
  description = "app tier security group"
  vpc_id      = aws_vpc.main.id

  lifecycle {
    create_before_destroy = true
  }
}

That’s the ladder, cheapest rung first: read the plan, pull and inspect the state, import or forget what’s actually gone, restore from S3 when the whole file’s bad, and reserve nuke-and-reimport and the teardown for dev.

More field notes

Start