Rebuilding CI runners
so they can’t overspend
How I traced a runaway cloud bill to self-hosted CI runners, stopped it within hours, and rebuilt the platform so the same failure cannot happen again.
The machinery behind
every software release.
Continuous integration, or CI, is the automated check every code change goes through before it reaches customers. When an engineer proposes a change, CI builds the software, runs the tests and packages the result, usually within minutes. Only changes that pass move on to release.
For a business, that makes CI part of how the product gets made, not a background tool. It is what lets a team ship small changes often without breaking what already works. When CI is slow, releases slow down. When it is down, nothing ships.
Four terms come up in the rest of this story:
Runner
The machine that actually runs a CI job. GitHub rents out its own runners, billed by the minute, or a team can run its own in its cloud account: more control over machine size, software and network, at the price of operating them.
Ephemeral runner
A runner created for one job and deleted afterwards. Every job starts clean, and nothing from one job can leak into the next. It also means every job downloads its tools from scratch.
Spot VM
Spare cloud capacity sold at a steep discount, which the provider can reclaim at short notice. A good fit for CI, where an interrupted job can simply run again.
NAT gateway
A shared exit to the internet for machines with no public address of their own. Google Cloud’s NAT charges for every gigabyte that passes through it, so heavy downloads behind NAT cost money on top of the machines.
CI touches every change, so its costs grow with how much the team ships.
One short-lived VM
for every CI job.
The company runs its CI on ephemeral self-hosted GitHub Actions runners. A small controller on Cloud Run checks GitHub every minute for queued jobs, starts one Spot VM per job, and deletes it when the job finishes.
The runners lived in a legacy sandbox project. A new platform project, built entirely in Terraform, was being prepared to take them over.
Within five days of going live, the runners had quietly become the most expensive line on the bill.
Networking became
the fastest-growing line.
During the week the runners went live, a new daily cost report and a surge alert I was building flagged Networking as the fastest-growing service. The billing export showed what was behind it.
In five days the runners pushed 9.2 TB through a Cloud NAT gateway, which bills per GB processed, and cost $422 in NAT charges alone. On the worst day, October 26, that was $232.19 for 5,133 GiB, against about a dollar a day the week before.
Proof,
not correlation.
Why NAT at all
The runner VMs had no external IP. Every download a CI job made, from Docker images and packages to tool caches and the runner itself, left through Cloud NAT. NAT charges per GB processed, so CI traffic was billed twice over.
The proof
The audit log showed the runner instance templates being recreated with external IPs on October 27 at 20:59 UTC. After that, runners kept working at 28–126 core-hours a day, and NAT traffic dropped to 0 GiB.
What remained
The NAT gateway still charges about $1.14 a day just for existing. It stays: twelve development services depend on it for outbound traffic, and keeping development in that project was a deliberate decision.
The invoice said how much.
The billing export said why.
Stop the leak first,
then fix it properly.
The emergency fix was to give runners their own external IPs, which bypass NAT, applied the same day. That stopped the leak within hours.
But it was made directly rather than through Terraform, on a platform with no ceiling on what it could spend: up to eight VMs and unmetered NAT. So the real work was rebuilding the platform properly.
An emergency fix outside Terraform is a stopgap, not a solution.
Cheap by design,
bounded by default.
The new platform keeps the same shape, one VM per job, but every part of it has a limit and every limit is in Terraform.
No NAT, by design
Runners get Standard Tier external IPs, and the network denies all inbound traffic. Outbound downloads cost nothing extra; nothing can connect in.
No cloud credentials
VMs have no service account, so a CI job cannot reach any Google Cloud API.
One job per VM
Each runner registers with a single-use (JIT) token, runs exactly one job, and is deleted.
Hard limits
At most two VMs, at most one of them non-Spot, which caps runner spend at about $92 a month. Runs older than 12 hours get no VM, and jobs stop at 175 minutes.
A breaker that acts
Every hour the controller reads the month’s runner spend from the billing export. At $90 of a $100 budget, pull request CI gets no runner until the month ends, while deploys keep theirs.
Visible in reporting
Runner resources are labelled, and the daily cost report gives runner spend its own line.
Every guard has a test
that proves it fails.
Configuration that looks right often does nothing, so every guard has a test proving it fails on bad input. The CI pipeline fails any change that:
- adds a NAT gateway to the runner component, or removes runners’ external IPs;
- opens any inbound firewall rule, or weakens the deny-all rule;
- gives a runner template a service account;
- raises the VM caps above the $100/month budget;
- turns off per-request billing on the controller;
- sets an organization policy anywhere except the intended project.
A guard you have never seen fail
is a guess, not a guard.
Six findings that
looked healthy and weren’t.
- Finding 01
The controller had never run.
Its scheduler called
/instead of/tickand got a 404 every minute since deployment. Green deployments, zero work done. - Finding 02
“Always on” by accident.
In the Terraform Google provider, a Cloud Run service with an explicit
resourcesblock defaultscpu_idleto false, which means instance-based billing. Pinged every minute, the controller never scaled to zero: about $49 a month for 3 seconds of work per call. One line,cpu_idle = true, fixed it, and a guard keeps it fixed. - Finding 03
The plan guard was blind to org policies.
It checked each change’s
projectfield, but organization policies carryparentinstead, so a policy change on the whole organization would have passed unchecked. I added tests that failed against the old guard, then fixed it. - Finding 04
Budget alerts couldn’t trigger automation.
Budget notifications to Pub/Sub need Google to grant its own service account access, which the organization’s domain-restricted-sharing policy blocks. Rather than weaken that policy, the breaker reads spend directly from the billing export.
- Finding 05
The new project blocked external IPs.
A canary run showed the org policy
compute.vmExternalIpAccessdenying every runner VM, and it failed safe: no VM, no fallback to NAT. The fix was a project-scoped exception, deployed with the guardrails above, after checking that the project’s only network denies all ingress. - Finding 06
API limits only show at apply time.
A 66-character budget name passed every plan, then failed against a 60-character API limit. Terraform preconditions now fail the plan instead.
A leak stopped,
and a ceiling put on top.
≈$3/day
Runner-related spend
From $270 on the worst day
0GiB
Runner data through NAT
From up to 5,133 GiB a day
≈$415/mo
Ongoing saving
Projection and estimate · see below
≈$92/mo
Hard ceiling on runner spend
Was unbounded · plus a breaker
- Result 01
Leak stopped
NAT data processing went from up to 5,133 GiB a day to 0, measured in the billing export. Runner-related spend went from $270 on the worst day to about $2–4 a day.
- Result 02
A controller billed per request
From about $49 a month, billed around the clock, to an estimated $3.50 a month, billed only while it works.
- Result 03
Bounded risk
Runner spend has a hard ceiling, an automatic breaker, budget emails, and a twice-daily cost surge alert. The October leak would have been flagged on its second day.
- Result 04
Guarded in CI
Reintroducing NAT, opening ingress, giving runners credentials or raising the caps past the budget now fails the pipeline, not the bill.
The ongoing saving of about $415 a month has two parts, and neither is a measured bill yet. About $370 is avoided NAT charges, a projection: 3.1 GiB per core-hour, times current CI volume, at $0.045/GiB. About $45 is the controller, an estimate from list prices, to be confirmed in the next day’s billing data.
The canary passed;
the cutover is next.
The new platform passed its first canary: a runner VM started with a public IP, ran the job, and was deleted. What remains:
- Make Spot runner startup more reliable, using a pre-built image instead of installing packages at boot.
- Move CI from the legacy runners to the new platform in an agreed change window.
- Retire the legacy runners, including the hand-made emergency fix.
Four principles
I would apply again.
- Lesson 01
Read the bill as data.
The billing export, joined with resource labels and usage, located the cause in minutes; the invoice total never would have.
- Lesson 02
Green is not evidence.
A scheduler hitting a 404, a guard that skipped a whole class of resources, and a CPU setting quietly billing 24/7 all looked healthy.
- Lesson 03
Cap the blast radius, then optimise.
Hard VM limits and a breaker matter more than any single saving: they turn an unbounded failure into a bounded one.
- Lesson 04
Fail closed.
When a policy blocked the new runners, they refused to start instead of silently falling back to the expensive path.