LLMs, GenAI, RAG, agents, evaluation · insight-1590
Score extraction field by field with typed comparators instead of whole-record exact match
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Applied AI, Data scientist
- Industries
- aviation, logistics
Summary: situation, action, reason, boundary
Situation
An extraction pipeline is graded by exact string equality of the whole JSON output against gold records with eight fields.
Action
Normalise values with typed comparators (dates, amounts with currency, identifiers, enums), score each field for precision and recall including the explicit unknown value, and report field-level results with a record-level pass rate.
Reason
The metric should match the decision: downstream systems consume fields, and each field has its own error cost and fix.
Boundary
Structured extraction from documents, emails and tickets.
Symptom or risk
One wrong field zeroes the record, formatting differences ("1,200.00" versus "1200") count as errors, and nobody can see which fields drive failures.
Exceptions
When any wrong field makes the record unusable, the record-level metric is the decision metric; report both.
Trade-offs
Comparators encode judgments (is 10:05 equal to 10:00?); document them.
Pitfalls
Normalising so loosely that real errors pass.
Example
Hypothetical: seven of eight fields correct gives record exact match 0 and field accuracy 87.5%.