Right-Sizing, End to End: A Methodology That Survives Contact With Engineers

Every cost tool ships a right-sizing report. It is usually correct. It is also, in most organisations, ignored — a list of several hundred recommendations that no single person has the authority to apply, the context to validate, or the time to shepherd through review. The savings identified in that report have been identified every month for two years and captured approximately never.

The engineering problem was solved by percentile queries. The remaining problem is a change management problem: how do you make several hundred small, individually low-risk, collectively significant modifications to production workloads owned by twenty teams, without an incident and without a reorganisation. That is what this describes.

Step 1: Classify before you measure

Applying one sizing rule across a fleet produces bad recommendations for the workloads that matter most, and the credibility damage from one bad recommendation exceeds the value of fifty good ones. Sort every workload into four classes first — this takes an afternoon with a spreadsheet and the service catalogue.

Two things about this table. Memory always uses max, never a percentile — exceeding a memory limit kills the process, so the tail is not optional. And the automation column is the actually important one: non-production is where you build confidence and demonstrate that the process does not break things, so it goes first and it goes fully automatic.

Step 2: Get the observation window right

Fourteen days is the default and it is right for most services. It captures two weekly cycles, which covers the weekday/weekend pattern that dominates most business workloads, and it is short enough that you are not sizing against traffic from a previous architecture.

The exceptions are where this goes wrong:

• Monthly cycles. Billing runs, month-end close, payroll. A 14-day window will miss the peak entirely and recommend a request that OOM-kills the service on the 1st. Use 35 days, or exclude these and size them from the known peak.
• Seasonal businesses. Retail in Q4, tax software in Q1. Right-sizing in August against August traffic and leaving it alone is how you find out about your autoscaler's limits during peak season.
• Services deployed within the window. Anything with less than 14 days of history at its current version is not eligible. Exclude, revisit later.

The query, with the exclusions that matter:

# CPU: p95 over 14 days, per container, prod only
quantile_over_time(0.95,
  sum by (namespace, pod, container) (
    rate(container_cpu_usage_seconds_total{
      container!="", container!="POD",
      namespace=~"prod-.*"
    }[5m])
  )[14d:5m]
)

# Memory: true maximum working set
max_over_time(
  sum by (namespace, pod, container) (
    container_memory_working_set_bytes{
      container!="", container!="POD",
      namespace=~"prod-.*"
    }
  )[14d:5m]
)

Exclude container="POD" — that is the pause container and it will pollute your aggregates. And aggregate at the container level, not the pod level, because a pod with an over-requested sidecar and a correctly-sized main container needs a recommendation for the sidecar specifically. Sidecars are consistently the largest single source of fleet-wide waste, because they are set once in a template and copied into thousands of pods.

Step 3: Order the rollout by risk, not by savings

The instinct is to start with the biggest savings. That is the wrong order, because the biggest savings are usually attached to the largest and most critical services, and one incident in week one ends the programme.

Run it in four waves:

Wave 1 — non-production, everything, automatic. Typically 20 to 30 percent of total cluster cost and essentially zero risk. Apply it, let it run for two weeks, and report what happened: X dollars saved, zero incidents. You now have evidence for the conversation about wave 2.

Wave 2 — production over-provisioned only. Only workloads where the recommendation is lower than the current request. Reducing an over-request cannot cause the workload to run out of resources; the container was never using them. The genuine risk here is scheduling density — pods pack tighter, more land on each node, and node-level pressure changes. Real, but bounded and observable.

Wave 3 — production under-provisioned. Workloads that need more. This costs money and it is the wave people skip, which is a mistake — these are the workloads being throttled or OOM-killed, and fixing them is a reliability win that buys you the political capital for everything else. Ship these alongside wave 2 so the report shows both directions and reads as an accuracy programme rather than a cost-cutting one.

Wave 4 — tier-1, individually. One service at a time, with the owning team, deployed during their normal release process, watched for a full cycle. Slow by design.

Step 4: Make the change a pull request

Not a controller mutating live workloads. A PR against the manifest in git, with the evidence inline, reviewed by the owning team.

This matters more than any technical detail in this piece. A PR is reviewable, revertible, auditable, and it puts the change in the workflow the team already uses. A controller that silently resizes production workloads is a system that will eventually do something surprising during an incident, and the first time it does, it gets turned off permanently.

## Right-sizing: payments/checkout-api

| container | resource | current | proposed | basis          | 14d observed |
|-----------|----------|---------|----------|----------------|--------------|
| api       | cpu      | 2000m   | 850m     | p95 × 1.3      | p95 654m     |
| api       | memory   | 4Gi     | 2Gi      | max × 1.3      | max 1.54Gi   |
| envoy     | cpu      | 100m    | 40m      | p95 × 1.3      | p95 31m      |

Class: tier-1 user-facing
Window: 2026-05-19 → 2026-06-02 (14d, 0 restarts, 0 OOMKills)
Throttling: 0.00 (no CPU limit set)
Estimated monthly change: **−$1,840**

Rollback: `git revert` this commit; ArgoCD syncs within 3 minutes.

Include the observed numbers, not just the recommendation. An engineer who can see that max memory was 1.54 GiB over two weeks can sanity-check the proposal against what they know about the service, and that check is where the bad recommendations get caught.

Step 5: Verify, with a defined window and a defined signal

A right-sizing change is not done when it merges. It is done when it has survived a full traffic cycle. Watch four things for seven days:

• OOMKills. kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}. Any occurrence is an automatic revert, no discussion.
• CPU throttling. Only relevant if the workload has a CPU limit, which for most workloads it should not — the case against CPU limits is a prerequisite for this whole methodology.
• p99 latency against the pre-change baseline. Record the baseline before merging; comparing against memory afterwards does not work.
• Pending pods in the namespace. A wave of reductions that increases scheduling density can surface capacity problems elsewhere.

- alert: RightSizingRegression
  expr: |
    increase(kube_pod_container_status_restarts_total{
      namespace=~"prod-.*"
    }[1h]) > 2
    and on(namespace, pod)
    kube_pod_container_status_last_terminated_reason{
      reason="OOMKilled"
    } == 1
  for: 5m
  labels: { severity: page, source: rightsizing }
  annotations:
    summary: "OOMKill after right-sizing in {{ $labels.namespace }}"
    runbook: "git revert the right-sizing commit for this service"

Publish the revert rate. A programme with a 2 percent revert rate is well-calibrated. Zero reverts means you are being too conservative and leaving money on the table; 10 percent means your multipliers are wrong and you should fix them before continuing rather than pushing through.

Step 6: Close the loop, or it decays

Right-sizing is not a project. Workloads drift — a caching layer ships and halves CPU, a new feature doubles memory, traffic grows 40 percent — and a fleet sized correctly in June is 20 percent wrong by December.

Run the recommendation engine continuously and open PRs when drift exceeds a threshold. Forty percent in either direction is a reasonable trigger: tight enough to catch real change, loose enough that teams are not reviewing a resizing PR every fortnight. VPA in updateMode: "Off" is a perfectly good engine for this and it is already in most clusters; just be aware of its 24-hour decay half-life, which makes it unsuitable for the monthly-cycle workloads flagged earlier, and never let it run in Auto alongside a CPU-based HPA — that combination oscillates.

What this is worth, and what it is not

A cluster that has never been right-sized typically carries 40 to 60 percent over-allocation. Capturing most of that is a 25 to 35 percent reduction in cluster cost, which on a $200k monthly spend is $50k to $70k a month for roughly a quarter of one engineer's time on an ongoing basis. Few infrastructure projects have that ratio.

But be clear about the ceiling. Right-sizing improves the accuracy of allocation. It does not improve the relationship between allocation and what you provision. A cluster where every workload requests exactly what it needs can still run at 40 percent node utilisation, because the remaining gap is a packing problem — node shapes, topology constraints, fragmentation — with an entirely different set of levers. That is usually the larger number, and it is where to go next.

And none of it sticks without the organisational half. A right-sizing programme that runs as a platform-team initiative captures savings once and watches them erode. One that routes the signal to the owning team, in their workflow, with their approval, becomes part of how the organisation operates — which is the difference between a cost project and a functioning practice.