The runbook everyone hated was the spec for the pipeline that replaced it.

What was on fire

Nothing was down. Everything was late. When I arrived at Express Scripts, there was no automated pipeline whatsoever. Every application was a manual deploy: put a ticket into Remedy, attach the script to deploy, the script to roll back, the script for everything else, and hand it to the people who ran them. Some of the big deployments took, no joke, up to 2 months. Releases came quarterly to every half year, specifically because it was so painful.

What that did to the business was latency. Something leadership wanted was maybe in place the next quarter, and there was a high level of stress, especially for people who weren’t used to that. The release team’s process was healthy by its own lights — the way we’ve always done it will always continue to work — and it was the only thing in the building that knew how code reached production.

Constraint set

The business wasn’t ready to deploy as often as we could go. They weren’t sure about the process. We came in with a mandate to make it happen at all costs and a promise of air cover, and the air cover had a ceiling: the platform lead didn’t have a direct line to the sponsor, and the competing leadership saying “no, slow down” did. There came a point where the sponsorship stopped absorbing the pushback, and we were on our own. It was a very political place, and we weren’t lifers, so we pushed forward as hard as we could and tried to get them on board.

System diagram

Architecture decisions

1. Codify the runbooks instead of waiting for the release team to hand over the secret sauce. We spent a lot of time reading the code and trying to understand how it worked, but the legacy process had already done the hard part for us. It made them formally write down every step of a deployment and every step of a rollback, and those documents were incredibly useful for understanding what their secret sauce actually was. Take that, look at the code and the infrastructure, and we could extrapolate exactly what was done and what was needed. The secret sauce ended up not so vital anymore, because we were able to take it and codify it.
2. One or two teams at a time, then normalize. Work with a couple of teams, really understand their process, bake it into Jenkins, then figure out how to normalize it so the same capability serves the next teams. The hard part was codifying it in a way that worked across dozens, if not hundreds, of teams without becoming a Jenkins pipeline that was just a bunch of spaghetti code.
3. A thin manifest over a fat library, with hooks for the rest. Each repo got a manifest declaring simple things — the application’s name, the application group it deployed into so several apps could share one firewalled-off subnet, the DNS name, environment variables, container size, autoscaling, the ports to open, the firewalls to open up. Nothing crazy; it was what different teams had already been doing, distilled. Beside it sat a Jenkinsfile that imported our custom library, initialized it, and let it go do the rest. For the edge cases a team could modify the Jenkinsfile or add into specific hooks. About 80% of teams used the manifest and got through; the other 20% needed light customizations, and we worked with them.
4. Keep the formal production approval, and put it inside the pipeline. There was still a process between a merge and production. You checked in, it ran all your tests, deployed into dev, deployed into QA, tests at each. To get into production, someone — a business sponsor or their equivalent — gave a formal approval in a platform we had tied in, and only then did the Jenkins flow continue and deploy. The gate that had lived in a ticket became a pause the pipeline waited on.
5. Couple it tightly, on purpose. The manifest and the library were welded to each other and to the targets. Years later that convention was the thing the AWS migration had to unwind — teams had never looked underneath it. The tight coupling was just fine back in 2016. Without it we probably would never have had the adoption we did, just by making it simple to change a manifest and add a Jenkinsfile inside each repo.

Not again: don’t make joining the platform harder than adding two files to a repo.

Recovery / operate path

It was roughly a year of building and getting things going. We went from 2 months for a deployment to deploying within a matter of hours, and it completely changed the game. The pipeline deployed everything from on-prem .NET stacks to on-prem Kubernetes to on-prem Pivotal Cloud Foundry, and then everybody decided to shift all of it to PCF. I argued against that one and lost, then built the on-ramp anyway — that decision is its own entry.

The politics didn’t go away because the pipeline worked. A leader I worked under used to say that when you try to do big change in an organization you end up with a third of the people gung-ho and on board, a third absolutely against it who will fight you tooth and nail, and a third in the middle who will do whatever they have to do to keep their jobs. We were fighting on the third that said absolutely no, and they had a lot of power, because they were the ones who brought code into production and held the secret sauce nobody else knew. Extracting that knowledge and automating it was scary for them. There was no single moment where it came to a head — a lot of fighting back and forth and never a lot of kumbaya. Ultimately the opposition lost its leadership, and a lot of our adoption problems went with it.

Did the platform ever put something bad into production? I’m sure it did. What’s the difference between the platform putting something bad into prod and a person doing it by hand? The whole point of QA was to catch that, and from a DevOps perspective it wasn’t our problem to sort out — that was the business process. From a technical perspective it did what it had to do. I don’t recall a deployment that failed and we couldn’t recover. After tens of thousands of deployments going like clockwork, that’s the testament.

No defect rate, no failure rate, no count of teams beyond “dozens, if not hundreds.” I don’t remember which approval platform we tied in, so it goes unnamed.

What changed after

Everybody switched over to our Jenkins platform and used it for many years. When Cigna bought the company, it had its own Jenkins pipeline, and it adopted a lot of the features and capabilities we had, and we migrated a lot of people onto that. GitHub Actions is the enterprise’s platform of choice now, and everybody has been migrating to it.

The convention outlived its authors and its targets, and the migration entry above is the bill for it. I’d make the same call.

Anonymization notes

The client ships by name; the people do not. My previous employer stays an alias — a payments company acquired by a major card network. Nobody is named: the platform lead is a former vice president who had built the same thing there; the Linux engineer, a third teammate whose name I’ve lost, and the sponsor appear by role only. The leadership on the other side of the fight is described as a position, not a person. The saying about thirds belongs to a leader I worked under, not to a book. The approval platform is unnamed because I don’t remember it. The numbers are my round phrases, not measurements: up to 2 months, quarterly to every half year, roughly a year, a matter of hours, about 80/20, dozens if not hundreds of teams, tens of thousands of deploys. Product names stay: Jenkins, Remedy, Pivotal Cloud Foundry, Kubernetes, GitHub Actions.

Who this is for

• A shop still deploying through tickets and hand-run scripts, where a big release is measured in months and the cadence is set by the pain.
• A platform team that is the newcomer in a building of long-timers, with air cover that has a ceiling.
• Anyone about to extract a release team’s tribal knowledge into automation and wondering where the spec is.
• Leadership weighing a tightly coupled convention that wins adoption now against the unwind it will cost at the next platform.

Related systems

Related writing

More systems

Start