S32 · paper simulation

Does long context make retrieval obsolete? Read the claim

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
40 minabout
0stages committed

The brief

Your VP of engineering forwards a preprint with the note "Do we still need the RAG project?" The paper and its results are fictional, written for this exercise. Its abstract (paraphrased):

"We compare placing entire documents in the prompt with retrieval-augmented generation on three public question-answering sets. Long-context prompting answers 74% of questions correctly against 65% for retrieval, which we take as evidence that retrieval-augmented generation is no longer needed."

Facts from the methods section (fictional): 600 questions in total (200 per dataset); the largest source corpus is 120,000 tokens and the median 85,000; the retrieval baseline uses keyword search, three chunks of 128 tokens, no reranking and no query rewriting; correctness was judged by a language model, with no validation against human labels reported; the datasets are built from public encyclopedia pages; cost and latency are not reported.

Your own situation (simulated): a policy and procedure knowledge base of 52,000 pages, about 600 tokens per page, queried about 10,000 times a day.

Stage 1 of 4

The retrieval baseline is the other half of the comparison.

What is the most serious problem with how the two approaches were compared?

There are only three datasets, so no conclusion can be drawn about any method until at least ten benchmarks have been run with the same protocol and the same judge model.Long-context prompting was given more tokens than retrieval, which is unfair, so the comparison should be repeated with equal token budgets for both methods and with the same judge for each.The retrieval side is weak and untuned (three short keyword chunks, no reranking or query rewriting), so the 9-point gap may measure a weak retriever rather than retrieval as a method.The language-model judge was not the same model as the one answering, which biases scores in favour of whichever approach produces longer answers.