Simplifying an over-built
Kubernetes platform onto Cloud Run
How I replaced an always-on Kubernetes cluster that reserved seventeen vCPUs to do about one core of work with a smaller, serverless platform on Cloud Run: fewer moving parts, one release path, and a lower bill by design.
A modest application
on an enterprise-sized platform.
The product was a learning platform: six web services sharing one database, plus an AI service for retrieval, content generation and voice.
All three environments lived in a single Google Kubernetes Engine cluster in a single cloud project, one namespace each. Every namespace was a complete stack with its own databases and cache, deployed by a GitOps controller from Helm values, with secrets copied into the cluster by a sync operator.
None of that was wrong. Each piece had been added for a reason. But together they made a platform whose operating surface was far larger than the application it carried.
The goal was not a new runtime for its own sake. It was to run the same product with less to operate and less to pay for.
Measure what the platform
was actually doing.
Before proposing anything, I took a read-only baseline of the live estate: what each workload reserved, what it used, and what it cost at list price.
- Workloads reserved about 17 vCPU and 48 GiB of memory.
- Across a seven-day window they used about one CPU core, in total.
- Roughly 30% of the reserved CPU, and 40% of the memory, belonged to platform tooling: the GitOps controller and the secret-sync operator.
- The main databases were 4 vCPU instances running at 3–11% CPU.
- The web services alone held at least fourteen replicas at minimum, around the clock.
The configuration told the same story. Around sixty Kubernetes manifests, two Helm charts, six GitOps applications, autoscalers and disruption budgets described a handful of HTTP services. Forty-seven CI workflow files across four repositories expressed three different ways to deploy.
The cost was not one oversized component. It was capacity reserved for a load that never arrived, plus the machinery needed to manage that capacity.
Delete layers
instead of optimizing them.
Tuning the cluster would have recovered some idle capacity, but every layer would still need upgrading, securing and debugging. I looked for the managed service that made each layer unnecessary.
Runtime
Node pools, autoscalers and minimum replicas became Cloud Run services that scale on requests and to zero outside production.
Deployment
The GitOps controller and Helm values became one pipeline that deploys an image digest directly, with its own checks.
Secrets
The sync operator became native secret references, each secret readable by exactly one runtime identity.
Database access
A connection pooler and shared password became a proxy sidecar that signs in with the service’s own cloud identity.
The network got simpler too. With no nodes there was nothing inside the private network to reach: no NAT gateway, no firewall rules to maintain, and data on private addresses only. Several separate public load balancers became one front door per environment, behind a web application firewall.
The cheapest component to run
is the one you no longer operate.
One project per environment,
built from the same code.
Instead of three namespaces sharing one project, each environment became its own cloud project, with a small admin project for what they share: Terraform state, pipeline identities, central logging and a budget.
Every environment is built from identical Terraform components, applied in a fixed order. They differ only in a variables file, and CI fails any change that lets the component code drift between them.
That gave the platform a property that mattered for cost: an environment can be fully defined and still switched off. Staging and production carry their network, registry and identities, while their databases, cache and front door stay disabled until a release is promoted into them. Until then they cost close to nothing.
Size each environment
for the job it does.
The baseline showed that the database tier, not compute, was the largest single cost. It was also the easiest to correct once each environment had its own project.
- Dev and staging: a small shared-core database and a basic cache instance.
- Production: a properly sized, highly available database and cache.
- All three: private IP only, encrypted connections only, and daily backups.
Some savings sat in the guardrails rather than the workloads. The baseline turned up organization-wide data-access audit logging that would bill every read and write, per gigabyte, for logs no one read; I flagged it to be narrowed before production traffic arrives. Every project carries cost-attribution labels, and a budget alerts at 50%, 90% and 100% of actual spend, and on forecast.
Most of the saving came from the database, not the runtime.
Make a bad release
unable to take traffic.
Three deployment models became one. The pipeline builds an image once, pushes it by digest, scans that exact digest, and uses only the digest from then on. Promotion to the next environment copies the same digest rather than rebuilding.
Each release then follows the same steps:
- Run database migrations as jobs, in order, stopping at the first failure.
- Deploy the new revision with zero traffic.
- Call it from inside the private network; it must answer healthy.
- Call it from the internet; it must be refused.
- Switch traffic, confirm the served digest, and run public checks through the firewall.
If any check fails, the previous revision is still serving everything, so there is nothing to roll back. I verified that by releasing a deliberately broken revision: the pipeline went red and the serving revision kept all of its traffic.
Pipelines sign in through workload identity federation. Across all four projects, no service account holds a long-lived key.
Turn hidden defaults
into explicit, reversible dials.
Serverless does not make cost go away. It makes cost follow decisions. I wrote each one down with its price and its reason, so it could be changed deliberately instead of inherited by accident.
Warm or cold
Scale-to-zero is free when idle, but the first request waits for the service to start. One warm instance per service removes the wait for a small fixed fee. It is set per service and per phase, not globally.
Background work
I priced always-on worker pools against queue-driven tasks that scale to zero. The team chose workers to keep the migration a lift-and-shift; the queue design is recorded as the next cost lever.
Hourly services
Components billed by the hour, such as a managed vector index, are switched on in Terraform when they are needed and off when they are not.
Durability
Point-in-time recovery and long log retention are production decisions. Development keeps daily backups and short retention.
A cost you chose is a decision.
A cost you inherited is a leak.
Less to operate,
and less to pay for.
≈38%
Lower modelled monthly run-rate
Midpoint · range 26–50% · list prices
57%
Lower database-tier cost
The largest single line
≈30%
Of reserved CPU was platform tooling
Removed with the cluster
0
Long-lived credentials
No service-account keys in any project
- Result 01
A smaller platform
The cluster, GitOps controller, secret-sync operator, connection pooler, NAT and node pools are gone. The legacy cluster has been retired.
- Result 02
Cost that follows use
Non-production can scale to zero, staging and production stay off until promoted, and the remaining fixed costs are labelled, budgeted and alerted.
- Result 03
One release path
Build once, promote by digest, and switch traffic only after checks pass. A failed release has been shown to leave production untouched.
- Result 04
A narrower attack surface
One firewalled entry point per environment, data on private addresses only, and no long-lived keys to leak.
The cost figures are a model at list prices, built from measured usage, and not yet a bill. While the old and new platforms overlap, spend goes up before it comes down. The model will be checked against real billing once the cutover is complete, and I would rather publish the range than a number I can’t yet defend.
Three principles
I would apply again.
- Lesson 01
Measure before you choose a runtime.
The baseline changed the plan. Most of the saving came from right-sizing the database and deleting tooling, not from the move to serverless.
- Lesson 02
Every layer you remove takes its failure modes with it.
A deleted controller needs no upgrades, no permissions and no on-call runbook. Simplification pays twice: in the bill and in the attention it frees.
- Lesson 03
Make every cost an explicit decision.
Warm instances, worker models, retention and recovery windows all cost money. Writing each down with its price makes it easy to change later.