LLMs, GenAI, RAG, agents, evaluation · insight-0512
Check the stop reason before parsing, and treat truncated output as a failure
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Applied AI, ML engineer
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A batch job extracts structured itinerary changes and occasionally receives JSON that ends mid-object.
Action
Read the stop or finish reason on every response; treat token-limit stops as failed outputs, log them, and retry with a higher limit or a smaller input instead of repairing the text.
Reason
APIs report why generation ended, and a token-limit stop means the model did not finish, so any parse of the text is partial by construction.
Boundary
Structured extraction, long summaries, batch pipelines.
Symptom or risk
A lenient parser closes the brackets and stores records with missing fields as if they were complete.
Exceptions
Streaming interfaces may show partial text deliberately, labelled as incomplete.
Trade-offs
Higher limits raise cost per call; splitting inputs adds orchestration.
Pitfalls
Monitoring only parse errors, which miss truncations that happen to parse.
Lessons and concepts
Claims and their sources
Anthropic's documentation states that every successful response carries a stop reason and that a max_tokens stop means the output was cut off and is incomplete.