LLMs, GenAI, RAG, agents, evaluation · insight-0667
Handle errors that arrive mid-stream, after the response already started
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
The application streams model output to users and treats the initial HTTP 200 as success.
Action
Treat a streamed response as successful only when the end-of-message event arrives; handle error events explicitly, mark partial output as incomplete in the interface, and log it as a failure with the request id.
Reason
With server-sent events, the status code is sent before generation finishes; Anthropic's docs note that errors can occur after a 200 response and are delivered as stream events.
Boundary
Chat interfaces and any streamed generation consumed by code.
Symptom or risk
An overload or other error event arrives part-way through, the interface shows a truncated answer as if complete, and logs record a success.
Exceptions
Non-streaming calls, where the status code reflects the whole response.
Trade-offs
Retrying a half-streamed answer duplicates text on screen; decide whether to restart or show an error.
Pitfalls
Parsing streamed partial JSON into a downstream system before the stream completes.
Lessons and concepts
Claims and their sources
When receiving a streaming response over server-sent events, an error can occur after the API has returned a 200 status.