LLMs, GenAI, RAG, agents, evaluation · insight-0502

Test effective context at your own input lengths instead of trusting the advertised window

Situation
A team plans to paste a whole ground-handling contract (a hypothetical 300 pages) into one prompt because it fits inside the documented context window.
Action
Build a probe set with known answers placed at different depths and at production-like lengths, including questions that combine several facts, and measure accuracy by length and position before choosing whole-document prompting over retrieval.
Reason
Long-context studies report accuracy falling as inputs grow and depending on where the relevant text sits, so usable context is shorter than the maximum accepted input.
Boundary
Contract review, long-document Q&A, log analysis, multi-document summaries.

Symptom or risk

Answers miss clauses buried in the middle or mix up similar clauses, while everyone assumes that anything inside the window is used.

Inputs far below the window rarely need probing; a newer model may degrade less, which the probe will show.

A day of probing versus silent misses found later by users.

Testing only single-needle lookups, which are easier than multi-fact reasoning over long inputs.

RULER found that almost all tested long-context models degraded substantially as input length grew, even when they passed simple needle-in-a-haystack tests.
RULER: What's the Real Context Size of Your Long-Context Language Models?
Liu et al. found performance was often highest when relevant information was at the beginning or end of the input and degraded when it was in the middle.
Lost in the Middle: How Language Models Use Long Contexts