FinOpsPractice
1 min read
◆ The PatternTracking cost per prediction and eliminating ML waste
ML workloads are expensive — GPUs, storage, compute for training and serving. Cost governance tracks cost per prediction, GPU utilisation, idle resources, and spot vs reserved savings. The goal: same model quality at lower cost, or better models at the same cost.
// Interactive — cost breakdown by category
| Strategy | Savings (vendor and practitioner ranges; varies widely) | Risk |
|---|---|---|
| Spot/preemptible instances | 60–90% | Interruption risk |
| Right-sizing instances | 20–50% | Under-provisioning |
| Model compression | 40–75% | Accuracy loss |
| Prediction caching | 50–80% | Stale results |
Pattern bridge: Cost governance applies the same risk-adjusted return thinking to infrastructure — maximise model value per dollar spent, just as transaction cost analysis measures trading efficiency.
Worked example
A 2.50-dollar-an-hour GPU that can serve 400 predictions a second. At 30% utilisation each 1,000 predictions cost $0.0058; at 80%, $0.0022 — 2.7× cheaper on the same hardware. Utilisation is usually the biggest lever, before any model change.
Where it misleads
- Cost per prediction ignores what a prediction is worth; the cheapest model is not the best value.
- Pushing utilisation to the limit raises latency sharply (see Latency & Throughput).
- Training, data and engineering time are often larger costs than serving.
In code
gpu_per_hour = 2.50 # dollars, on demand
capacity = 400 # predictions per second at full use
def cost_per_1k(utilisation):
return gpu_per_hour / (capacity * utilisation * 3600) * 1000
low, high = cost_per_1k(0.30), cost_per_1k(0.80)Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.
Sources
- FinOps Foundation, FinOps Framework, finops.org
- E. Strubell, A. Ganesh & A. McCallum, “Energy and Policy Considerations for Deep Learning in NLP”, ACL, 2019
- Designing Machine Learning Systems, C. Huyen, O’Reilly, 2022
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 1