LLMs, GenAI, RAG, agents, evaluation · insight-0531

Extract from long documents section by section with provenance, then aggregate in code

Situation
An analyst asks a model to list every penalty clause from a long supplier contract in one call.
Action
Split the document into sections, extract per section into a schema that includes the page or section reference, then deduplicate, merge and count in code; spot-check a sample against the source.
Reason
Smaller inputs reduce long-context position effects, and provenance fields make every extracted item checkable.
Boundary
Contracts, regulatory filings, policy manuals, long reports.

Symptom or risk

Clauses in the middle are missed, duplicates appear, and nobody can trace an extracted clause back to its page.

Facts that span sections, such as a definition used later, need context passed between steps.

More calls and orchestration; cross-section reasoning becomes harder.

Letting the model do the final aggregation and counting, which code does exactly.

Liu et al. found models often used information in the middle of long inputs less reliably than information near the beginning or end.
Lost in the Middle: How Language Models Use Long Contexts