# GPU FinOps: Spot for Training, Reserved for Inference

* * *

# GPU FinOps: Spot for Training, Reserved for Inference

GPU spend is the fastest-growing line item in enterprise [cloud bills](https://zop.dev/resources/blogs/cloud-cost-anomaly-detection), per the [FinOps Foundation 2026 State of FinOps Report](https://www.finops.org/insights/the-state-of-finops/). A single 8-GPU training cluster costs $95,800 per month if left always-on. A team running four such clusters spends $4.6 million per year on training alone. Yet most of that spend is structurally avoidable, because GPU economics reward a discipline that CPU-era FinOps frameworks do not teach.

FinOps is the engineering practice of bringing financial accountability to variable cloud spend by aligning engineering, finance, and product on continuous cost decisions. Applied to GPUs, the discipline has one rule: spot for training, on-demand or reserved for inference. Get this wrong and you either burn cash on always-on training fleets or break latency SLOs by running production inference on instances that get evicted with two minutes notice. Get it right and the same workload costs 60% less without any change to model accuracy or serving latency.

## Why GPU Costs Break Traditional FinOps Playbooks

CPU FinOps has a settled playbook. Reserve the baseline, autoscale the burst, right-size every six months. That playbook fails on GPU workloads for three reasons.

| Property | CPU instance | GPU instance |
| --- | --- | --- |
| On-demand to spot discount | 70-80% | 60-70% |
| Eviction notice | 2 minutes | 2 minutes |
| Cold start (with model load) | 30-60s | 90-180s |
| Price gradient across SKUs | 2-3x | 5-10x |
| Saturation pattern | Variable load | Either bound or idle |

GPUs have a steeper SKU price gradient. An A10G costs $1.006 per hour. An A100 costs $3.67 per hour. An H100 costs over $12 per hour. A 5x gap between adjacent tiers means defaulting to the largest SKU is not a 30% mistake, it is a 500% mistake.

GPUs also fail differently under spot eviction. A CPU job that loses an instance restarts a request and recovers in seconds. A training job that loses a GPU instance restores a multi-gigabyte checkpoint, re-loads model weights, and resumes from the last save point. That recovery costs minutes to hours, depending on checkpoint frequency.

And inference workloads cannot tolerate the eviction at all. A real-time inference endpoint serving a sub-100ms latency SLO has no path to drain in-flight requests, warm a replacement instance, and re-route traffic in two minutes. The structural mismatch between spot eviction timing and inference SLOs is why the same pricing model that saves 65% on training breaks 99.9% availability on inference.

The fix is not to use one pricing model for all GPU workloads. It is to match each workload to the pricing model its tolerance profile fits.

## Spot for Training: When the 65% Discount Is Real

Training is fault-tolerant. It is checkpointed by design. The wall clock can extend by hours and the result is identical. That makes training the canonical spot workload, and the savings are large enough to fund a year of headcount.

Take a concrete example. Training a 7-billion-parameter LLM from scratch on one trillion tokens with 8x A100 GPUs takes approximately 21 days of wall-clock time. On a single `p4d.24xlarge` running on-demand at $32.77 per hour, that is $16,516 for a single training run.

The same job on spot capacity, at an average 65% discount, runs at $11.47 per hour. Even with two evictions costing 4 hours of recovery time each, the spot run completes in 521 hours at $5,975. That is a saving of $10,541 per training run.

| Scenario | Hourly rate | Wall hours | Total cost |
| --- | --- | --- | --- |
| On-demand baseline | $32.77 | 504 | $16,516 |
| Spot, 0 evictions | $11.47 | 504 | $5,781 |
| Spot, 2 evictions, hourly checkpoints | $11.47 | 521 | $5,975 |
| Spot, 5 evictions, hourly checkpoints | $11.47 | 547 | $6,274 |
| Spot, 5 evictions, daily checkpoints | $11.47 | 624 | $7,159 |

Notice the bottom row. With daily checkpoints instead of hourly, five evictions cost an extra 80 hours of wall-clock recovery time. The spot saving shrinks from $10,541 to $9,357. Five more evictions on daily checkpoints would erase the saving entirely.

![diagram](https://iili.io/BQosD0l.png align="center")

Three rules make spot training economics work. First, checkpoint at least every hour, ideally every 15 minutes, depending on training step time. Checkpoint storage costs cents per gigabyte-month on S3 or GCS. Recovery time without recent checkpoints costs hundreds of dollars per training run.

Second, structure jobs to be elastic to capacity. Use frameworks like [AWS SageMaker managed spot training](https://docs.aws.amazon.com/sagemaker/latest/dg/model-managed-spot-training.html) or Google Cloud TPU preemptible nodes that handle re-queue automatically. Hand-rolled spot recovery logic is a source of subtle bugs that cost more in engineer time than the spot discount saves.

Third, monitor the spot pool before launching long runs. The [AWS Spot Instance Advisor](https://aws.amazon.com/ec2/spot/instance-advisor/) reports historical eviction rates by SKU and region. The `p4d.24xlarge` in `us-east-1` averages 5-10% per month. The same SKU in `eu-west-1` can spike above 25% during scarcity. Region selection matters more for GPU spot than for CPU spot because the GPU pool is thinner.

This pattern works when training jobs are checkpointable, idempotent, and time-flexible. It breaks when training is on a fixed deadline (a customer demo, a paper submission), because the spot variability that costs you 4-8 extra wall-clock hours per eviction can push a launch by a day. For deadline-bound training runs, on-demand is the right choice despite the cost premium.

## Reserved for Inference: Why Spot Breaks Latency SLOs

Inference is the inverse workload. It is real-time. It serves user requests with sub-100ms latency targets. It cannot lose capacity without immediately impacting availability.

The two-minute spot eviction notice is the disqualifier. In two minutes, an inference fleet must drain in-flight requests, warm a replacement instance with model weights loaded, and route traffic away from the doomed instance. Model load alone takes 90-180 seconds on a `g5.xlarge`. There is no path to do all three within the eviction window.

The right architecture reserves the inference baseline and autoscales on top of it.

![diagram](https://iili.io/BQosy57.png align="center")

Inference traffic typically shows 3-5x peak-to-trough variation across a 24-hour cycle. The baseline that the fleet never drops below is the right size for a reservation. The peak burst is on-demand or scheduled scaling.

For an `a2-highgpu-1g` running A100 inference on Google Cloud, the on-demand rate is $3.67 per hour, or $2,673 per month per instance. A [3-year Committed Use Discount](https://cloud.google.com/compute/docs/instances/signing-up-committed-use-discounts) drops this to $1.58 per hour, or $1,151 per month. For a 10-instance baseline that runs 24/7 to handle p50 traffic, the saving is $15,220 per month, $182,640 per year. The same [reserved-capacity logic](https://zop.dev/resources/blogs/cosmos-db-reserved-capacity-63-percent-savings) that delivers 63% on Cosmos DB applies to GPU inference, with one difference: GPU reservations are tied to specific instance families, so a poorly forecasted commit traps you in a SKU you may want to migrate off.

The reservation strategy for GPU inference is inverted from CPU. With CPUs, you reserve the baseline and burst on-demand for the peak. With GPUs, the same logic applies, but the cost asymmetry is far larger. A reservation that covers your stable baseline, with on-demand reserved for the burst, captures the majority of the discount opportunity. Spot has no role in the latency-sensitive inference path.

| SKU | On-demand $/hour | 3-year reserved $/hour | Throughput (Llama-2-7B) | Cost per 1M tokens |
| --- | --- | --- | --- | --- |
| AWS g5.xlarge (A10G) | $1.006 | $0.42 | 90 tok/s | $1.30 |
| AWS inf2.xlarge | $0.76 | $0.31 | 130 tok/s | $0.66 |
| AWS p4d.24xlarge (8x A100) | $32.77 | $11.13 | 720 tok/s/GPU | $0.43 |
| GCP a2-highgpu-1g (A100) | $3.67 | $1.58 | 95 tok/s | $1.74 |

For latency-bound inference, the dollar-per-token math points to AWS Inferentia2 for supported model architectures, with A10G as the general-purpose default for models under 13B parameters. The A100 is justified for 70B-parameter models and above, where smaller GPUs cannot fit the model in memory.

## GPU SKU Selection: The 3-5x Gradient Most Teams Ignore

Most teams pick GPUs the way they pick CPUs: choose what is in the docs, scale up if it is slow. With CPUs, picking one tier too large costs 30%. With GPUs, it costs 300%.

The decision matrix is bounded by model size and latency target.

| Model size | Latency target | Recommended SKU | Hourly cost | Cost per 1M tokens |
| --- | --- | --- | --- | --- |
| Under 7B params | Batch (offline) | g5.xlarge spot | $0.30 | $0.40 |
| Under 7B params | Real-time (under 100ms) | inf2.xlarge | $0.76 | $0.66 |
| 7B to 13B params | Real-time | g5.2xlarge | $1.21 | $1.45 |
| 13B to 70B params | Real-time | g5.12xlarge | $5.67 | $2.10 |
| 70B+ params | Real-time | p4d.24xlarge | $32.77 | $0.43 |
| Mixed small models | Real-time | p4d with MIG | $32.77 / 56 workloads | varies |

The bottom row is the multi-instance GPU partitioning that most teams skip. [NVIDIA Multi-Instance GPU](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/) on the A100 partitions a single physical GPU into up to seven isolated instances. A `p4d.24xlarge` with 8 A100s and full MIG enabled serves up to 56 isolated inference workloads concurrently. For teams serving many small models, MIG cuts cost-per-model by up to 7x compared to one-model-per-GPU deployment.

This pattern works when you have a portfolio of small models with similar latency targets. It breaks when one model needs the full A100 memory bandwidth (such as long-context retrieval models), because MIG isolation also caps memory bandwidth per partition.

![diagram](https://iili.io/BQoLdzu.png align="center")

The default deployment of one model per GPU dedicates a $3.67 per hour A100 to a model that uses 8% of its memory and 12% of its compute. MIG fixes this without code changes. Most inference frameworks (Triton Inference Server, TorchServe, KServe) support MIG natively.

Inferentia2 is the second SKU lever. For supported model architectures (Llama, Mistral, GPT-style transformers under 13B parameters), Inferentia2 runs inference at 1.7x the throughput of an A10G at 75% the price. The combined effect is 1.7x more cost-efficient per million tokens served. For teams with predictable model architectures, the Inferentia2 migration is a one-week engineering investment with a 40% inference cost reduction.

## Non-Production GPU: The 25% Waste Hiding in Plain Sight

Production GPU costs get scrutinized. Non-production GPU costs do not. That is where 18-25% of total GPU spend disappears, an extension of the same [zombie-resource problem](https://zop.dev/resources/blogs/automate-dev-staging-environment-scheduling-aws) that drains CPU dev fleets.

The pattern is mundane. A data scientist spins up a `g5.2xlarge` Jupyter notebook on Friday afternoon to fine-tune a model. They go home. The notebook runs idle from Friday at 6 PM through Monday at 9 AM, a 63-hour weekend at $1.21 per hour. That is $76 per developer per weekend, before counting weekday idle time.

Multiply by a 50-person ML team and you have $3,800 per weekend, $15,200 per month, $182,400 per year, on idle development capacity nobody used. The same accountability gap that produces this waste in dev shows up in prod when [showback reporting](https://zop.dev/resources/blogs/chargeback-vs-showback-team-level-cloud-cost-accountability) does not break out GPU spend by team.

![diagram](https://iili.io/BQoLKqx.png align="center")

| 50-instance dev fleet (g5.2xlarge) | Hours per month | Monthly cost |
| --- | --- | --- |
| Always-on, no controls | 36,000 | $43,560 |
| Weekend shutdown only | 25,200 | $30,492 |
| Business hours + weekend shutdown | 9,900 | $11,979 |
| Idle detection + business hours + weekend shutdown | 5,940 | $7,187 |

A 50-instance GPU dev fleet with idle detection, business-hours scheduling, and weekend shutdown costs $7,187 per month. The same fleet always-on costs $43,560 per month. The saving is $36,373 per month, $436,476 per year, with zero impact on developer productivity. Idle detection wakes the instance back up when the developer returns. The kernel reconnects in under two minutes.

The three controls compound. Idle detection alone saves 30% because notebooks sit idle through meetings, lunches, and context switches even during the workday. Business-hours scheduling adds 30% on top. Weekend shutdown adds another 15%. Combined, they cut non-production GPU spend by 80%.

## Building a 90-Day GPU Cost Reduction Plan

GPU cost reduction is not a research project. It is a sequence of contained changes, each with a clear saving target. Run them in order.

| Phase | Weeks | Action | Effort | Expected monthly saving |
| --- | --- | --- | --- | --- |
| Baseline | 1-2 | Tag every GPU instance by workload type (training, inference, dev). Pull p95 utilization for 14 days. | 1 engineer-week | 0 (visibility only) |
| Non-prod controls | 3-4 | Deploy idle detection, business-hours scheduling, weekend shutdown for dev fleet | 2 engineer-weeks | 60-80% of dev GPU spend |
| Spot for training | 5-8 | Migrate training jobs to managed spot training. Set checkpoint interval to 30-60 minutes. | 3 engineer-weeks | 60-65% of training GPU spend |
| Reserved for inference | 9-12 | Identify inference baseline (lowest p5 over 30 days). Purchase 3-year Compute Savings Plans for that capacity. | 1 engineer-week + procurement | 40-65% of inference GPU spend |
| SKU right-sizing | 13 | Audit inference workloads. Migrate small-model inference to MIG or Inferentia2. | 4 engineer-weeks | 30-50% of inference GPU spend |

A team spending $200,000 per month on GPUs at the start of this plan typically lands at $80,000-$95,000 per month after 90 days. The path is mechanical. Each phase has a defined input (utilization data, workload tagging, SKU map) and a defined output (a configuration change, a reservation purchase, a fleet migration). The only judgment call is the inference baseline number, and that is determined by the data, not by intuition.

The discipline is not complicated. It is just different from CPU FinOps. Spot for training, reserved for inference, MIG for small models, idle detection for everything else. Apply each rule to the workload it fits, and the bill drops by 60% in a quarter without compromising training quality or inference latency.