7 Cloud Cost Anomalies Every FinOps Lead Should Alert On
Cloud bills do not grow smoothly. They mostly stay flat, and then one Tuesday afternoon someone deploys a change and the bill spikes 40% for the rest of the month. The FinOps team hears about it three weeks later on the invoice. By then the money is gone and the engineer who caused it is on to different work.
Anomaly detection exists to close that gap. It works when it targets the specific spike patterns that actually happen, not a generic "spend increased" alert. These are the seven that come up over and over.
Which anomaly costs the most per hour?
NAT gateway data transfer. It is the anomaly that shows up on every FinOps retrospective, and it is the most expensive per hour of any single misconfiguration.
The pattern: a workload that used to talk to a service through a VPC endpoint gets refactored, and traffic starts routing through the NAT gateway instead. Data transfer through NAT is charged per GB at rates that can hit $0.045 per GB, plus hourly NAT charges. A workload moving 1 TB per day suddenly costs $45 per day extra, and larger workloads can spike into the thousands.
Signal to alert on. NAT gateway data processed bytes, per NAT, deviates more than 3 standard deviations from the 14-day mean. Also alert on any NAT with sustained data transfer above 500 GB per day, regardless of history.
Typical impact. $500 to $5000 per day of surprise cost, per misconfigured workload.
What is the runaway logging pipeline pattern?
An application changes its log level to DEBUG, or a bug creates an error log per request, and the logging pipeline suddenly ingests 10x its normal volume. CloudWatch Logs bills at $0.50 per GB ingested. Datadog and similar tools bill by ingest as well.
The pattern is often invisible in dashboards because the logs still land, the alerts still fire, and the pipeline still works. The failure mode is financial, not functional.
Signal to alert on. CloudWatch Logs ingest bytes per log group deviates more than 3 standard deviations, or exceeds 100 GB per day on any single group. Cross-reference with per-service log volume from your logging tool.
Typical impact. $200 to $2000 per day per misconfigured service.
When do orphaned load balancers become a problem?
After a cluster shutdown or a Kubernetes upgrade, load balancers get provisioned by the ingress controller of the old cluster and never deleted. Each ALB or NLB costs $16 to $22 per month idle, and companies often accumulate 50 to 200 of them over 18 months.
Signal to alert on. New load balancer without any target group targets healthy for more than 48 hours. Also alert on load balancers with zero request count over the last 7 days.
Typical impact. $800 to $4000 per month at steady state, plus the compounding of adding more each quarter.
What does a GPU autoscaling misconfiguration look like?
The most expensive anomaly per node. A machine learning workload sets a min replica count of 4 on a GPU node group. Nobody uses it for two weeks. Each g5.xlarge costs $1 per hour, so a min of 4 costs $96 per day, or $2880 per month, doing nothing.
The pattern often comes from copy-pasting Helm values between environments, where the production min becomes the dev min, or from a "safe minimum" set during a launch that never gets reset.
Signal to alert on. Any GPU node with average CPU or GPU utilization under 5% over a 24-hour window. Second alert on min replicas above 2 for any workload in a non-production namespace.
Typical impact. $1000 to $10,000 per month per misconfigured workload.
How do you detect an S3 request storm?
A misbehaving client, often a Lambda or a Kubernetes CronJob, starts hitting S3 with millions of GET or LIST requests. S3 charges $0.0004 per 1000 GET requests, which sounds cheap until a stuck loop generates 10 million requests per hour.
Signal to alert on. S3 request count per bucket deviates 4 standard deviations from the 14-day mean, with a floor of 10 million requests per hour. Also alert on any bucket where LIST requests exceed GET requests by more than 10:1, which almost always indicates a misconfigured client.
Typical impact. $100 to $2000 per day.
What is the cross-region replication anomaly?
An engineer enables cross-region replication for disaster recovery, forgets to set a lifecycle policy, and the replicated bucket accumulates data at the same rate as the source. Data transfer between regions runs $0.02 per GB. On a 100 TB bucket replicating at 1 TB per day, that is $20 per day of transfer plus the storage duplication.
Signal to alert on. Data transfer bytes between regions, per source bucket, exceeds 500 GB per day, or grows by more than 25% week over week.
Typical impact. $500 to $5000 per month per misconfigured replication rule.
How do you find the forgotten proof-of-concept?
The slowest-burning but most common anomaly. A team spins up an ML training environment or a data pipeline for a specific experiment. The experiment ends. The infrastructure keeps running.
Unlike the other anomalies, this one does not spike. It plateaus at a moderately high number and stays there for months. The signal is not deviation from history; it is deviation from purpose.
Signal to alert on. Any resource created in the last 90 days that has been running for more than 30 days without CPU utilization above 20% or any external network traffic. Cross-reference with the git repo activity for the owning team.
Typical impact. $500 to $10,000 per month per forgotten workload, and companies accumulate 5 to 20 of these per year.
How do you route alerts so they get acted on?
Central FinOps channels are where alerts go to die. Route by ownership, from the moment of detection.
| Alert type | Route to | Escalation |
|---|---|---|
| Team-owned resource spike | Team's on-call or engineering channel | Team lead after 4 hours unacknowledged |
| Platform-owned shared spike | Platform team channel | Platform lead after 2 hours |
| Cross-cutting (NAT, transfer) | Team originating traffic, cc platform | Platform lead after 4 hours |
| Unallocated resource spike | Platform team, treated as grooming queue | FinOps lead if unactioned for 24 hours |
The routing table must be based on the same allocation mapping used for cost reports. If the mapping is stale, alerts go to the wrong team, and the team stops trusting the alerts. The mapping is the primary artifact; alerts are downstream of it.
What should the alert body actually say?
Not a chart. Not a link to a dashboard. Three sentences that answer three questions.
- What happened. "NAT gateway
nat-0abc123in us-east-1 processed 12 TB in the last 24 hours vs. a 14-day average of 1.5 TB." - What it costs. "Estimated impact: $540 per day if this continues, $16,000 per month."
- Who owns it. "Owner: platform-networking. Suggested action: check for a recently deployed workload with egress traffic."
Alerts that answer these three questions get resolved. Alerts that link to a dashboard get ignored. The difference in resolution time is measured in days.
The mistake to avoid
Most anomaly detection systems fire on statistical deviation and stop there. That produces alert fatigue within a month. The version that works targets specific known patterns, sets thresholds tuned to what actually breaks, routes to the team that owns the resource rather than a central channel, and writes alert bodies that let an engineer act without opening a dashboard. Get the seven patterns above wired correctly, and you will catch 80% of the surprise bills that would have hit your CFO's inbox at month end.
Frequently asked questions
What is a reasonable threshold for a cloud cost anomaly?
A statistical baseline is more useful than a fixed dollar or percentage threshold. Compute the rolling 14-day mean and standard deviation per service per team, then alert when today's daily spend exceeds mean plus 3 standard deviations. That catches real spikes without paging on normal weekly patterns. Add a dollar floor of $100 per day so tiny services do not generate noise.
How fast should anomaly alerts fire?
Within 6 hours of the spike starting. Cloud providers refresh billing data every 4 to 8 hours, so this is the realistic floor. Anything slower and the team hears about the problem after two full days of overspend. Anything faster requires usage-based inference, which is possible but noisy. Six hours is the sweet spot between signal and noise.
Who should anomaly alerts route to?
The team that owns the affected resource, not a central FinOps channel. Central channels create diffusion of responsibility; alerts sit unread for days because everyone assumes someone else is handling it. Route to team-specific channels via the allocation mapping, and cc the FinOps lead only for spikes above a dollar threshold, usually $500 per day of impact.
What causes false positive anomaly alerts?
Three main sources: (1) legitimate traffic spikes from marketing campaigns or product launches, (2) monthly batch jobs that only run once a period so the baseline treats them as anomalies, and (3) newly deployed services with no history for the baseline to compare against. Suppress with allowlist annotations that a team can add to their own workload, so the FinOps team is not the bottleneck.
Should we alert on cost or on usage?
Both, and they catch different failures. Cost alerts catch pricing changes, commitment expirations, and regional cost differences. Usage alerts catch runaway loops, misconfigurations, and traffic anomalies before they show up in billing. A mature program runs both, with usage alerts firing faster and cost alerts confirming the financial impact within hours.
Every cloud dollar gets an owner
Pyrenis allocates 100% of AWS, GCP, Azure, and Kubernetes spend to the teams that create it, catches anomalies in hours, and ships savings with real numbers.
Request early access