LLMs, GenAI, RAG, agents, evaluation · insight-0613
Deduplicate near-identical chunks before they crowd out other evidence
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Applied AI, ML engineer, Data analyst
- Industries
- insurance, cross-industry
Summary: situation, action, reason, boundary
Situation
The corpus contains copies of the same policy in several places: the PDF, an intranet page and an email announcement.
Action
Hash normalised chunk text at ingestion to remove exact duplicates, and at query time collapse near-duplicates (high mutual similarity) to the single most authoritative or most recent copy.
Reason
Every context slot spent on a duplicate is a slot not spent on complementary evidence, and duplicates of different versions can contradict each other.
Boundary
Enterprise wikis, email archives, scraped web corpora.
Symptom or risk
The top five results are the same paragraph five times, so the context carries one piece of evidence and the answer misses the exception stated elsewhere.
Exceptions
When source multiplicity is itself a signal (for example, many customer complaints describing one issue), count duplicates instead of discarding them.
Trade-offs
Aggressive near-duplicate collapsing can merge two versions that differ only in a crucial number.
Pitfalls
Deduplicating on embeddings alone; check the text diff before collapsing policy versions.