LLMs, GenAI, RAG, agents, evaluation · insight-0526
Use provider-enforced schemas instead of asking for JSON in the prompt
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Applied AI, ML engineer
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A pipeline extracts passenger name, flight and requested change from emails by asking the model to reply in JSON, then parses with a try/except.
Action
Use the provider's schema-enforced output mode (or constrained decoding when self-hosting), keep the schema within the documented supported subset, and still handle refusals and token-limit stops as explicit failures.
Reason
Vendor documentation describes schema-enforced modes that guarantee structural conformance, whereas plain JSON modes or prompt instructions only make valid JSON likely.
Boundary
Extraction, classification with fixed labels, tool arguments, any output parsed by code.
Symptom or risk
A small share of replies have missing keys, extra prose or invalid enum values, and the failures arrive as midnight parse errors or, worse, as silently dropped records.
Exceptions
Free-form prose outputs gain nothing from a schema.
Trade-offs
Supported schema features are limited, and very large schemas can hit complexity limits.
Pitfalls
Treating schema conformance as correctness; a valid object can hold a wrong flight number.
Lessons and concepts
Claims and their sources
OpenAI's documentation states that JSON mode only ensures valid JSON while Structured Outputs ensures adherence to the supplied schema, and that refusals and incomplete responses may not follow the schema.
Anthropic's documentation states that structured outputs use constrained decoding to follow the schema, lists unsupported JSON Schema features, and notes that refusals and max_tokens stops can produce non-conforming output.