LLMs, GenAI, RAG, agents, evaluation · insight-0671
Send offline workloads such as eval runs and nightly enrichment through a batch API
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- ML engineer, Applied AI, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Golden-set evaluations, document classification backfills or nightly summaries run through the real-time API with custom concurrency code.
Action
Move non-urgent workloads to the provider's batch interface, submit them with stable custom ids, poll for completion, and reconcile results by id.
Reason
Anthropic documents its Message Batches API as asynchronous, with each request processed independently, at lower cost and higher throughput than standard calls.
Boundary
Evaluation suites (days 50-51), backfills, periodic reports.
Symptom or risk
Offline jobs compete with user traffic for rate limits and pay interactive prices for work nobody waits on.
Exceptions
Anything a user or a downstream step waits on in real time.
Trade-offs
Results arrive later and individual requests can fail or expire, so jobs need retry handling for the failed subset.
Pitfalls
Matching results by order rather than by custom id.
Lessons and concepts
Claims and their sources
Anthropic's Message Batches API processes requests asynchronously and independently and is documented as reducing cost and increasing throughput for workloads that do not need immediate responses.