S23 · business case simulation

Self-hosting a quantized model against an API, with gates

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

5stages
40 minabout
0stages committed

The brief

A logistics company (simulated) extracts shipment id, issue type and requested action from customer emails. Volume is 5,000,000 emails a month; each request is about 700 input tokens and 60 output tokens.

All figures are hypothetical. Options and their measured results on a 1,000-email simulated golden set (which includes an 80-email slice of non-English messages):

Option	Overall correct	Non-English slice (n = 80)	Cost basis
Hosted large model	95.0%	90.0%	3.00 / 15.00 USD per million input / output tokens
Hosted small model	84.0%	70.0%	0.25 / 1.25 USD per million tokens
Self-hosted, 8-bit quantized open-weights	94.4%	86.3% (69 / 80)	2,100 USD a month per server; one server serves 2,000,000 emails a month at peak latency targets; 1,500 USD a month for operations
Self-hosted, 4-bit quantized	92.1%	78.8% (63 / 80)	2,100 USD a month per server; 2,800,000 emails a month per server; 1,500 USD a month for operations

Quality gates set by operations (simulated): at least 93% overall and at least 85% on the non-English slice.

Stage 1 of 5

First, the hosted large model, since it is the current default for comparison.

What is its monthly cost for 5,000,000 emails?

15,000 USD. Input is 3,500 million tokens at 3.00 (10,500) and output is 300 million tokens at 15.00 (4,500), and the two are added to give the monthly bill.10,500 USD, since the 60 output tokens per email are too few to affect the total and the monthly bill is driven by the long inputs.53,400 USD. Output is priced at 15.00 and input at 3.00, so 3,500 million input tokens are charged at the output price and the 300 million output tokens at the input price.1,250 USD, which is what this workload costs on the small model; the large model is priced the same per email because the emails are short.