LLMs, GenAI, RAG, agents, evaluation · insight-1707
Seed facts into long documents to measure how many survive each level of a summary tree
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, Data scientist
- Industries
- finance, insurance
Summary: situation, action, reason, boundary
Situation
A 400-page report is summarised in chunks, the chunk summaries are summarised again, and the top-level summary is accepted because it reads well.
Action
Plant 20 to 30 known facts (a number, a date, a named exception) across a test document, run the pipeline, and count survival per level. Carry a structured list of entities and figures with source pointers up the tree, and compare with a single long-context pass if the document fits.
Reason
In Wu et al.'s recursive book summarisation, the authors discuss that errors at lower levels carry into higher-level summaries and that each summary sees only its own input. The tree structure is what makes loss multiply.
Boundary
Contracts, incident reports, earnings calls and any corpus summarised by chunk and merge.
Symptom or risk
A fact dropped or distorted at a lower level cannot return, so omissions and errors accumulate, and fluent output hides what the reader never sees.
Exceptions
Documents short enough for one pass, or tasks where only the gist matters.
Trade-offs
A carried fact list lengthens every merge prompt.
Pitfalls
Seeding facts the chunker always keeps together, or judging by a rater who has not read the source.
Example
Hypothetical: each summarising step keeps a given fact 85% of the time. In a three-level tree (chunk, section, whole) a fact survives with probability 0.85 x 0.85 x 0.85 = 61%; at 90% per step it is 73%.
Lessons and concepts
Claims and their sources
Wu et al. (2021, an arXiv preprint) describe recursively summarising book sections and then the summaries with a fine-tuned GPT-3, and, in their limitations discussion (read via an ar5iv rendering), note that lower-level errors compound at higher levels and that decomposition assumes parts can be summarised independently.