Engineering, production, governance, cost, communication · insight-0284
Price inference per thousand predictions before choosing GPU serving
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A team proposes GPU instances to serve a gradient-boosting model that handles 50 requests per second.
Action
Benchmark latency and throughput on CPU and GPU with realistic batch sizes, compute cost per thousand predictions at expected utilisation, and choose GPU only when model type, batch size and volume keep it busy.
Reason
Accelerators pay off when there is enough parallel work to fill them; tree ensembles and small models often serve efficiently on CPU.
Boundary
Model-serving capacity planning for tabular, vision and language models.
Symptom or risk
GPUs sit mostly idle at low throughput, so cost per prediction rises sharply with no latency benefit for small tree models.
Exceptions
Large neural and LLM inference usually needs GPUs or specialised accelerators regardless of volume.
Trade-offs
CPU serving may need more replicas at peak; autoscaling changes the comparison.
Pitfalls
Comparing list prices without measured utilisation.
Example
Hypothetical prices: a GPU instance at 1.20 per hour serving 50 predictions per second costs about 0.0067 per thousand; a CPU instance at 0.20 per hour handling the same load costs about 0.0011.