LLMs, GenAI, RAG, agents, evaluation · insight-0668
Budget GenAI features by cost per successful task, not price per token
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Data analyst, Applied AI
- Industries
- insurance, telecom
Summary: situation, action, reason, boundary
Situation
A business case multiplies expected requests by the list price per million tokens and calls that the cost.
Action
Model cost per resolved task: tokens per attempt times attempts per task, plus tool and infrastructure costs, plus human handling for failures, divided by successful outcomes; state every price as an assumption.
Reason
The business pays per outcome, and a cheaper model that needs more attempts or more human clean-up can cost more per resolved case.
Boundary
Business cases on days 54 and 55, vendor comparisons, pricing internal chargebacks.
Symptom or risk
Real cost lands several times higher because of retries, failed attempts, multi-step agents, evaluation runs and human review of escalations.
Exceptions
Early feasibility estimates, where an order-of-magnitude token cost is enough.
Trade-offs
A full model needs measured success and escalation rates, which only exist after a pilot.
Pitfalls
Ignoring input tokens, which usually dominate RAG and agent requests.
Example
Hypothetical inputs: model A costs 0.02 per attempt and resolves 90% of cases; model B costs 0.005 and resolves 60%; each failure costs 1.50 of agent time. A ≈ 0.02 + 0.10 × 1.50 = 0.17 per case; B ≈ 0.005 + 0.40 × 1.50 = 0.605.