LLMs, GenAI, RAG, agents, evaluation · insight-1650draft — in review
Compare self-hosting with an API on total cost at expected utilisation, including the people who run it
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Applied AI, Data scientist
- Industries
- healthcare, energy
Summary: situation, action, reason, boundary
Situation
A team sees the per-token price of a hosted model and a GPU rental price and proposes to self-host an open-weight model to save money.
Action
Compute break-even monthly tokens from GPU cost at planned utilisation plus a share of engineering and on-call time, against the hosted price at the same quality tier. Price privacy needs separately; recheck as volume grows.
Reason
A fixed monthly cost only beats a per-token price above a threshold volume, and that threshold rises with every cost you omit and with every percentage point of idle capacity.
Boundary
Build-versus-buy decisions for model serving, guardrails and evaluation tooling.
Symptom or risk
Idle capacity, on-call load and upgrades make self-hosting costlier and less reliable than the API at real volumes.
Exceptions
Data-residency or air-gap requirements that rule out the API regardless of price.
Trade-offs
The API choice adds vendor dependency and deprecation schedules; self-hosting adds operations burden.
Pitfalls
Comparing peak throughput with the API price, or assuming equal quality without the eval set.
Example
Hypothetical: a GPU node at 15,000-a-month engineer is 1 per million hosted tokens, break-even is 5.92 billion tokens a month, before idle capacity.