LLMs, GenAI, RAG, agents, evaluation · insight-0601

Chunk along the document's own structure before tuning chunk size

Situation
You are ingesting manuals, policies or contracts that have headings, numbered clauses or tables into a RAG index.
Action
Split first on structural boundaries (section, clause, table, list item), keep the heading path with each chunk, and only then cap length with a size limit and small overlap.
Reason
A chunk is the unit of evidence the model sees. If the unit is cut mid-rule, no retriever or prompt can restore the missing condition.
Boundary
Structured, authored documents: fare conditions, insurance wordings, runbooks, API docs.

Symptom or risk

Fixed-size token windows cut a fare rule from its exception, or a table from its header row, so retrieved chunks are fluent but incomplete.

Unstructured text such as chat logs or OCR output with no reliable headings; there, fixed windows with overlap are a reasonable default.

Structural chunks vary widely in length, so some exceed the embedding model's input limit and must be sub-split.

Trusting the parser: PDF extraction often flattens tables and drops headings, so inspect a sample of chunks before indexing everything.