project-038 · LLM and GenAI · cross-industry
Chunking experiment on long technical papers: evidence recall at a fixed context budget
An R&D team wants to ask questions of long technical reports and feed retrieved chunks to a model with a fixed context budget. Which chunking policy - fixed-size token windows with overlap, paragraphs, section-aware paragraphs with headers, or sentence windows - puts the annotated evidence inside a 1,000-token context most often, and is the winner the same for different question types?
- Licence
- CC BY 4.0 (AllenAI data page and dataset card)
- Reference run
- none
At a glance
Data
QASPER train/dev v0.3 (NLP papers with question-level evidence paragraphs) (public) · Train 888 papers / 2,593 questions; validation 281 papers / 1,005 questions (dataset card); this project uses a subset
Details on dataBaseline
Paragraph chunks as given by the dataset, BM25 within the paper, greedily add the top-ranked chunks until the 1,000-token budget is reached (truncate the last one).
Details on baselineValidation
Develop and choose budgets and tie-breaking rules on 100 train papers; evaluate once on validation-split papers.
Details on validationMetric
Evidence recall at budget (any-annotator and strict variants), evidence precision (share of budget tokens that overlap evidence), mean number of chunks used, index size,…
Details on metric
Success criterion
Adopt a chunking policy only if its evidence recall at a 1,000-token budget beats the paragraph policy on the held-out papers with a paired-bootstrap interval (resampling papers) that excludes zero; otherwise keep paragraphs because they are simplest to cite. Comparisons are invalid unless every policy is held to the same token budget (not the same top-k).
Context
Chunking is often tuned by feel, and most comparisons are unfair: smaller chunks with the same top-k use less context, larger chunks with the same k use more. This project fixes the retriever (BM25, optionally one small embedding model held constant) and changes only the chunking, so the result isolates one variable. QASPER suits this because each question has paragraph-level evidence annotated by people who read the full paper. Retrieval here is within one paper at a time, which is a realistic "chat with this document" setting and differs from corpus-wide search.
Data
- Dataset
- QASPER train/dev v0.3 (NLP papers with question-level evidence paragraphs) (public)
- Licence
- CC BY 4.0 (AllenAI data page and dataset card)
- Size
- Train 888 papers / 2,593 questions; validation 281 papers / 1,005 questions (dataset card); this project uses a subset
- Verification
- identity verified
Access
Download qasper-train-dev-v0.3.tgz (about 10.8 MB) from the S3 URL listed in the AllenAI loading script; no login. The Hugging Face entry uses a loading script that recent datasets versions no longer run, so read the JSON directly. Cache the archive with its SHA-256. Fallback: if the bucket is unavailable, apply the same protocol to the academy lessons with builder-labelled evidence and say the external replication was not done.
Grain
One row per (paper, question, annotator answer). Evidence is a list of full paragraphs (or figure/table captions prefixed "FLOAT SELECTED"). Retrieval unit is a chunk produced by the policy under test; a chunk counts as a hit when it overlaps an evidence paragraph by a minimum number of characters (rule fixed in advance).
| Column | Meaning |
|---|---|
| paper_id / title / abstract | Paper identity and metadata |
| full_text.section_name / paragraphs | Section headings and their paragraphs, the material to chunk |
| question_id / question | Question written by someone who read only the title and abstract |
| answers.unanswerable | True when the annotator found no answer in the paper; excluded from evidence recall, reported separately |
| answers.evidence | Evidence paragraphs (or FLOAT SELECTED captions) used to answer |
| answers.extractive_spans / free_form_answer / yes_no | Answer forms; only used for slicing by answer type |
EDA and data-quality rules
Count questions by answer type, unanswerable flag and evidence kind; exclude FLOAT SELECTED evidence from the main metric (text chunking cannot retrieve a figure) and report its share.
Multiple annotators can give different evidence; define the rule (hit if any annotator's evidence is covered, plus a strict variant requiring the union) before running.
Paragraph length distribution per paper; flag papers with extraction artefacts (empty sections, very long merged paragraphs).
Check that evidence strings match a paragraph in full_text exactly; count and handle mismatches instead of silently dropping them.
Baseline
Paragraph chunks as given by the dataset, BM25 within the paper, greedily add the top-ranked chunks until the 1,000-token budget is reached (truncate the last one). Token counts come from one fixed tokenizer recorded in the config. A second reference point: the first 1,000 tokens of the paper (abstract and introduction), which tells you how much retrieval adds over position.
Features
Policies under test, each scored at budgets of 500, 1,000 and 2,000 tokens: fixed windows of 128/256/512 tokens with 0% and 25% overlap; paragraphs; paragraphs with the section heading prepended; sentence windows (a sentence plus one neighbour each side). Chunk ids map back to character offsets so overlap with evidence can be computed exactly. Deduplicate overlapping windows before filling the budget so overlap does not waste context.
Candidate methods
Section-aware paragraphs with headings prepended
Headings such as 'Experimental setup' carry the words questions use; cheap and keeps citations readable.
Fixed 256-token windows with 25% overlap
The common default; tests whether ignoring structure costs recall and how much context overlap wastes.
Sentence windows
Fine-grained chunks pack more distinct evidence into the budget but may cut the context a generator needs.
Validation design
Develop and choose budgets and tie-breaking rules on 100 train papers; evaluate once on validation-split papers. Bootstrap resamples papers (not questions), because questions on the same paper are correlated. Report per-budget curves, not one number.
Leakage checks
Questions were written from title and abstract only, so the abstract is a legitimate but strong source of lexical overlap; report results with and without the abstract indexed. No tuning on validation papers.
Bias and representativeness
All papers are NLP research in English with consistent structure; results may not transfer to contracts, manuals or scanned reports. Annotators were practitioners, not the authors (dataset card), so evidence labels are incomplete - a chunk can be useful yet unlabelled.
Robustness checks
Repeat the best two policies with a different tokenizer for budget counting and with one small embedding retriever (if a download is allowed) to check the ranking of policies is not an artefact of BM25.
Technical metrics
Evidence recall at budget (any-annotator and strict variants), evidence precision (share of budget tokens that overlap evidence), mean number of chunks used, index size, all by budget and policy.
Business interpretation
Probability that the context sent to the model contains what a human needed to answer, at a context cost the team can afford; a policy that needs twice the budget for the same recall doubles token spend (hypothetical price, labelled).
Error slices
Answer type: extractive vs yes/no vs free-form; yes/no questions often need evidence spread across sections.
Evidence count: single-paragraph vs multi-paragraph evidence.
Section of the evidence: method/results vs introduction; position effects show up here.
Implementation steps
Download and cache the archive; parse papers into paragraphs with character offsets; write tests for offset round-trips.
Implement each chunking policy as a pure function returning chunks with offsets; unit-test on a toy paper.
Implement budget filling with deduplication and a fixed tokenizer; test that every policy respects the budget exactly.
Run BM25 per paper for all policies and budgets on 100 train papers; inspect 20 misses per policy.
Freeze the protocol, run once on validation papers, bootstrap by paper, and write the per-budget curves and decision.
Reproduction
Learner-authored scripts; for example uv run python -m chunking.prepare --archive data/qasper-train-dev-v0.3.tgz then uv run python -m chunking.run --split train --papers 100 --seed 13 and uv run python -m chunking.run --split validation --final. CPU-only with BM25; record the archive hash, tokenizer name and version, and runtime. The optional embedding check requires a model download.
Limitations
Evidence recall is a retrieval proxy; it does not show that a generator answers correctly from the context.
Within-paper retrieval removes the hardest part of corpus search (finding the right document).
Token budgets depend on the tokenizer; results are reported for one tokenizer and may shift with another.
Production and monitoring
Store the chunking policy and tokenizer version with the index so a change triggers a re-index and a regression run. Keep chunk-to-source offsets for citations. Monitor context-budget saturation (how often the budget is full) and the share of answers that cite chunks cut mid-sentence.
Interview questions
Why is comparing chunk sizes at the same top-k misleading, and what would you hold constant instead?
Your sentence-window policy wins on evidence recall but answer quality drops. How could both be true?
Postmortem and alternatives
If the winner differs by budget, the real decision is the budget, not the policy; say which budget the product can afford. An alternative worth discussing is hierarchical retrieval (retrieve sections, then paragraphs), which trades one more retrieval step for better use of the budget.
Rubric3 criteria
| Criterion | Meets | Exceeds |
|---|---|---|
| Fair comparison | All policies evaluated at the same token budgets with one tokenizer and one retriever. | Shows how a same-top-k comparison would have produced a different (wrong) conclusion. |
| Evidence handling | Hit rule, annotator disagreement and FLOAT SELECTED evidence handled by rules fixed in advance. | Reports strict and lenient variants and explains when they disagree. |
| Decision | Policy and budget chosen with paper-level bootstrap intervals. | Error slices explain why the winner wins and where it would fail on other document types. |
References4
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
How questions and evidence were collected; defines what a hit means.
- The Probabilistic Relevance Framework: BM25 and Beyond
Fixed retriever used across policies.
- Lost in the Middle: How Language Models Use Long Contexts
Why more context is not automatically better once chunks reach a generator.
- RAGAs: Automated Evaluation of Retrieval Augmented Generation
Context-relevance and faithfulness as the next evaluation layer beyond evidence recall.