Home/Blog/How to Set Up Kubernetes Cost Allocation in Under a Week
Playbooks

How to Set Up Kubernetes Cost Allocation in Under a Week

Kubernetes is where cost allocation projects die. A single cluster looks like one bill to AWS or GCP. Inside it live 40 workloads owned by 8 teams, and none of them show up on the invoice. The billing export sees a $180K per month EKS charge; engineering sees nothing to attribute.

The fix is not exotic. It takes a week, a metrics agent, and a namespace mapping. This is the version that works.

Day 1: What metrics do you need in the cluster?

Two components, both open source, both idempotent to install.

  • kube-state-metrics. Exposes the current state of every Kubernetes object: pods, deployments, namespaces, resource requests and limits. This is the source of truth for what workloads exist and what they claim to need.
  • A per-pod usage source. cAdvisor or Prometheus node exporter, capturing per-pod CPU seconds, memory bytes, network bytes, and disk I/O. This is what workloads actually use, which is different from what they requested.

Together they let you compute two ratios per workload: request share of the node, and usage share of the node. The right cost allocation blends both, weighted by which resource is the binding constraint on that node.

Install both via Helm in the afternoon. They add well under 1% overhead in most clusters.

Day 2: How do you build the namespace-to-team mapping?

This is the day the project stalls at most companies, because it requires a real conversation instead of a technical fix.

  • Pull the namespace list. kubectl get namespaces gives you the raw set. Most clusters have 20 to 100.
  • Categorize each one. Team-owned, product-owned, shared platform, kube-system, or orphaned.
  • Talk to the platform team. For every namespace not clearly team-owned, ask who created it and what runs in it. This conversation is 60 to 90 minutes and it saves 10 hours of guessing later.
  • Publish the mapping. A single markdown file in the platform repo. Every namespace maps to exactly one team.

Then enforce the mapping. Any new namespace requires an entry in the file before it can be created. If your platform allows self-service namespace creation, this becomes a validating admission policy, not a manual process.

The mapping does not need to be perfect on day 2. It needs to exist. Corrections happen on day 5 when teams see their bill.

Day 3: What label convention should you enforce?

Two labels, and only two, are required at the workload level.

  • team. The owning team, matching the namespace mapping in most cases. Necessary because a single namespace can host workloads from multiple teams during a migration.
  • service. The specific product or service the workload delivers. checkout, recommendations, auth, search.

Two more are useful but not required.

  • environment. prod, staging, dev. Separates spend by environment so teams can see prod cost without dev noise.
  • component. For services with multiple deployments, api, worker, scheduler. Enables finer-grained attribution.

Enforce via a validating webhook that rejects workloads without the required labels. Kyverno and OPA both handle this in under 20 lines of policy. The enforcement is the point. Without it, labels drift, and drift means the unallocated bucket comes back within a quarter.

Day 4: How do you split shared cluster costs?

Every cluster has costs that do not belong to a single workload. Handle each category with a specific rule.

Shared cost Split by Rationale
Ingress controllers Request count per workload Workload that gets more traffic pays more
Log forwarders Log volume per namespace High-log services subsidize low-log ones otherwise
Cluster autoscaler CPU share of nodes it scales Scales workloads that requested more capacity
Node-level daemons Node CPU or memory Overhead scales with node size, not workload count
Idle capacity Route to platform Buffer is insurance, not per-workload cost
Cross-AZ traffic Source workload Attribute to the workload originating the transfer
Persistent volume unused Owning namespace Storage costs when nothing is using it are the team's problem

Every rule is one-time. Write it, ship it, and it holds until the cluster architecture changes. Do not try to make the rules dynamic. Static rules that are 90% right beat dynamic rules that are 100% right but nobody understands.

Day 5: What report do engineers actually look at?

Not the raw cost table. Engineers care about three things.

  • This week vs. last week. Their team's total cluster cost, week over week, with the delta highlighted.
  • Top 5 workloads by cost. Which of their services are eating the budget. Ranked, with month-over-month change.
  • Rightsizing opportunities. Workloads where request is more than 2x actual usage, with a projected saving from adjusting.

Everything else is data engineers will look at once and never again. The three above are what drive optimization behavior.

Deliver the report in Slack or email, weekly, on Monday morning. Not a dashboard link. A rendered summary that engineers can read in 30 seconds without clicking anywhere. Dashboards are for the FinOps team; summaries are for engineers.

What breaks in the first month of running Kubernetes allocation?

Three things, all predictable, all recoverable.

  • Labels drift. A team forgets to add team on a new workload. The admission policy catches it, but only if the policy is in enforce mode. Start in audit mode for a week, then flip to enforce.
  • Namespace mapping goes stale. Someone creates a namespace outside the process. The weekly report shows an unmapped namespace at the top of the platform team's queue. Groom weekly, not monthly.
  • Cost spikes look like allocation bugs. A workload doubles its cost overnight; the team's first assumption is that the allocation broke. Ship a "why did this change" drill-down that shows the change in usage vs. the change in unit price, so the team can distinguish signal from noise.

None of these are showstoppers. They are the reason allocation is a weekly practice, not a quarterly project.

How do you measure success after a month?

Three numbers on the platform team's dashboard.

  • Allocated percentage. Should be above 95% within two weeks. Any lower and the label enforcement is not working.
  • Unallocated dollars. The trend, not the absolute value. If it is dropping, the grooming ritual is working. If it is flat, something is being missed.
  • Number of teams reading the weekly report. Track Slack message opens or email opens. If engineering leaders are not looking at the report, the allocation exists on paper but not in the culture.

The third metric is the one that separates a technical allocation project from a functioning FinOps program.

The mistake to avoid

Most teams try to allocate Kubernetes cost by writing a giant SQL query over the billing export. That fails, because the billing export does not know what a namespace is. Ship kube-state-metrics and a cost agent on day 1, enforce two labels via admission policy by day 3, and treat the namespace mapping as a written artifact you groom weekly. The technical work is straightforward; the discipline of keeping the mapping current is what separates the teams who reach 95% allocation from the ones stuck at 40%.

kubernetes-costk8s-showbacknamespace-allocationfinopscontainer-cost

Frequently asked questions

Do we need a specialized agent or can we use the cloud billing exports alone?

You need something inside the cluster. Cloud billing shows the node cost, not the workload cost. Kube-state-metrics and cAdvisor provide per-workload CPU, memory, and network usage. That usage becomes the ratio you multiply the node cost by, to get per-workload cost. Without an in-cluster metrics source, allocation stops at the node level, which is not useful for chargeback.

What is the right label convention for cost allocation?

Two required labels: team and service. Optionally environment and component. Enforce them at admission via Kyverno, OPA, or a validating webhook. Reject workloads without the labels rather than accepting them and adding to unallocated. Retrofitting labels on existing workloads takes an ongoing effort; blocking new workloads without them is a one-time policy decision.

How do you allocate shared costs like ingress and log daemons?

Split by usage. Ingress controllers should be split by request count per workload. Log forwarders by log volume per namespace. Cluster autoscaler and node-level agents by CPU or memory share. Never split shared costs by headcount or by equal division across teams; that penalizes small teams running large workloads.

What about idle capacity? Who pays for the buffer?

Route idle to platform, not to individual workloads. Idle capacity exists to absorb spikes; it is a shared insurance policy, not a per-team overhead. Report it separately on the per-team dashboard as 'platform overhead' so teams see the cost of the buffer without being charged for it. Then track the buffer size as a platform KPI and tune it quarterly.

How accurate is Kubernetes cost allocation, really?

Within 5 to 10% of ground truth for CPU and memory, which is fine for chargeback and optimization decisions. Network and storage are noisier, typically 10 to 20% off, because per-pod attribution depends on labels and tags that are not always precise. The remaining uncertainty is smaller than the decisions you will make with the data, so do not wait for perfect accuracy before starting.

Every cloud dollar gets an owner

Pyrenis allocates 100% of AWS, GCP, Azure, and Kubernetes spend to the teams that create it, catches anomalies in hours, and ships savings with real numbers.

Request early access