S23 · decision value simulation
How many chunks to retrieve, priced per correct answer
People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.
- 4stages
- 40 minabout
- 0stages committed
The brief
An insurer (simulated) runs an assistant that answers customers' questions about policy wording. The pipeline retrieves the top-k chunks of 400 tokens each and passes them with a 600-token instruction and the question (question tokens are small and ignored here); the answer is about 250 tokens. Volume is 400,000 questions a month.
All prices are hypothetical, USD per million tokens, for the single model in use: input 3.00, output 15.00. No caching or batching is assumed in this exercise.
A simulated golden set of 400 questions, each with a reviewed answer, scored the four settings:
| k | Input tokens per request | Correct answers (golden set, 400 questions) |
|---|---|---|
| 3 | 1,800 | 78.0% |
| 5 | 2,600 | 85.5% |
| 8 | 3,800 | 87.0% |
| 12 | 5,400 | 86.0% |
Stage 1 of 4
Per request cost is input tokens x 3.00 / 1,000,000 plus 250 output tokens x 15.00 / 1,000,000. Volume is 400,000 requests a month.
What is the monthly model cost at k = 8?