A cluster mis-sized on the day it was built never trips a cost anomaly.

What was on fire

Nothing was down. Nothing was going to page, either — about 1,400 AWS accounts across the enterprise, security governed across all of them, cost governed on none.

The shape repeats: a team spins up something far larger than it needs. One had built an Amazon OpenSearch cluster at production size and left it running in lower environments, at roughly $30k a month. It had been burning for about six months when I found it.

It was healthy the whole time. That is what made it invisible: the cluster wasn’t drifting, it was wrong the day it was built, and a bill that starts wrong has no anomaly in it to find. Cost anomaly detection reads your own history, and this was the history.

The FinOps team had come online to read the billing data across every account and hand back suggestions. I was brought in from outside that team to go a step further — to find better ways of slicing the data than the tool they had bought could manage.

Constraint set

The raw export wasn’t mine to read. The FinOps team held the S3 bucket that the Cost and Usage Report lands in, and they held it for a reason I’d defend: it carries financials for the entire enterprise, not the slice I worked in. They fed it into Apptio Cloudability, which had a decent UI for per-account spend and a reporting layer being built on top of it.

So I had two things — the accounts I already worked in, and a licensed seat in the tool. Not the bucket.

The wholesale levers weren’t mine either. Reserved-capacity analysis across the estate belonged to the FinOps team, and that is the work that moves money by flipping a switch. I get brought in for the harder half: one team at a time, their codebase, their infrastructure. And anything a team agreed to still had to clear the same path to prod and architecture review as a feature would.

System diagram

The bucket is the whole shape of it. Everything above the line belonged to the FinOps team, everything below it is what a licensed seat and my own accounts could reach.

Architecture decisions

1. Build the departmental view beside the bought tool, not instead of it. Cloudability held the data and its UI wouldn’t slice it the way I needed, so I called the same endpoints its own web app called, for the accounts my department owned, and pulled the results into my own database. The raw export for the accounts I already had access to was the key — it told me exactly what the fields were, so I knew what to ask the tool for. Rejected: waiting on the bucket, or asking for another report inside a UI that had already refused the question.
2. Postgres and Grafana, green at the top, drill on red. One page for a leader or a team: everything green means nothing to do; anything red is a click into the account behind it. The facets were the ones that actually carry money — instance types and how hard they’re worked, orphaned EBS volumes, what’s sitting in S3, versioned buckets nobody reads, clones of buckets that already exist. Rejected: another spend-by-account table, which the tool already gave everyone.
3. Read their account, their implementation, and their Terraform before saying a word. Every one of these fixes lands on a team that had a reason for what they built. The OpenSearch cluster was overprovisioned because they were still learning the platform during bring-up and bought capacity instead of understanding — which you only get to say after you’ve read it. I wasn’t going to preach to a team without knowing the full scope of their problem.
4. Right-size by environment instead of re-platforming everything. Lower environments don’t get hit hard, so they went to OpenSearch Serverless. Production, when it came, went back to a provisioned cluster — my recollection, not something I went back and checked. The team’s objection was never architectural — they didn’t know the Terraform or how the data migration would work, so I built the proof of concept, validated the throughput, and wrote the migration through Kinesis Firehose myself. Once it was proved, nobody complained.

Not again: put the cost question inside the architecture review, at design time. The review that blocks the fix is the review that should have caught the design.

Recovery / operate path

The dashboard found it. The cluster came up as an outlier, I traced the account to the team that owned it, and then it was a conversation rather than a ticket. About two weeks from the outlier to the migration running for real, against about six months of it burning.

The save is on the order of $20–25k a month, on lower environments only. I never went back for the serverless run-rate afterward, so there’s no after-number here. The department ran to roughly 400 accounts, and across it the low-hanging fruit came to something like 15–20% — though that share is the FinOps team’s wholesale levers, not mine. My per-team rebuilds added maybe another 5–10%. These are my own estimates from the time, and no industry benchmark is being set beside them.

The honest ceiling is the interesting number. If every suggestion had been taken, the department could have cut about 50%. It wasn’t, and the reason isn’t analysis. Rightsizing a peak-sized cluster into an auto-scaling group of smaller instances is a real infrastructure change, and a real change costs a team a path-to-prod cycle and an architecture review. Against a roadmap, that loses to features every time.

Mostly two people opened those dashboards: me and my leadership. We pulled numbers out of it, made spreadsheets, and sent those. It was an analyst’s instrument, not a habit the organization picked up.

What changed after

The layer died. I squeezed what I could out of data and analytics, moved to the next bottleneck, and nobody else wanted to run it — the FinOps team was set on the third-party product they had bought.

What outlasted it is smaller and real: one team’s lower environments are right-sized, and the migration path they didn’t know how to build exists now. The wholesale levers stayed where they always were, on the FinOps team’s side of the fence. The half that was left was still sitting on the same backlogs when I moved on.

Anonymization notes

The client ships by name; the people do not. The FinOps team, the OpenSearch team, and my leadership appear as roles only, and no product owner is quoted — none was. The vendor is named because it’s load-bearing: Apptio Cloudability held the data, and I read it through the endpoints its own interface used, as a licensed user, for accounts my department owned.

Withheld: the department’s monthly spend. I remember it as a bill in the eight figures a month and I no longer remember whether that figure covered the department or the whole enterprise, so it prints with no owner attached and no dollar amount. The current account count is out too — 1,400 is the number at the time, which is the only one this story needs. The percentages above are my estimates, hedged as I gave them. Stack footnote: the Cost and Usage Report into Cloudability, Cloudability into PostgreSQL, Grafana over that; the cluster went Amazon OpenSearch Service to OpenSearch Serverless with Kinesis Firehose delivering into it, all of it defined in Terraform.

Who this is for

• An engineer who can see the bill in a vendor UI but not the raw export, and needs to slice it a way the tool won’t.
• A team that overprovisioned during bring-up to get something working, and never went back once it did.
• Anyone whose rightsizing fix has to buy a path-to-prod cycle before it can save a dollar.
• A leader whose cost analysis is running well ahead of anyone’s backlog.

Related systems

More systems

Start