S23 · decision value simulation

Batch pricing is cheap until the deadline arrives

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
35 minabout
0stages committed

The brief

A travel retailer (simulated) uses an LLM for three traffic classes. All prices and rates are hypothetical. Each request uses 1,500 input tokens and 300 output tokens on a model priced at 1.00 USD per million input tokens and 5.00 per million output tokens, so a real-time request costs 0.0030 USD (1,500 x 1.00 / 1e6 + 300 x 5.00 / 1e6).

Class	Requests a month	Latency requirement
A. Live chat suggestions	300,000	under 3 seconds
B. Email draft replies	500,000	within 4 hours
C. Nightly booking summaries	1,200,000 (about 40,000 a night)	ready by 07:00; the job starts at 22:00

A batch endpoint charges a hypothetical 50% less on both token prices (0.0015 USD per request) and works asynchronously. In the retailer's tests (simulated) 92% of requests finished within the 7 hours between 22:00 and 05:00 and the rest did not. Unfinished requests are not billed in this exercise, and the team plans to re-run them through the real-time endpoint between 05:00 and 07:00.

Stage 1 of 4

The team lists proposals for each class.

Which proposals are sound as stated? Select all that apply.

Move class C to the batch endpoint, with a fallback that re-runs unfinished requests in real time after the window closes.Move class A to batch as well, since halving the price of 300,000 requests is worth a slower first response.Move class B to batch with no fallback, because batches usually complete quickly and the 4-hour window is generous.Track the share of batch requests completed inside the window and the fallback volume as cost and reliability metrics.Submit the nightly job earlier than 22:00 when the day's bookings are final, to widen the window.