Neural networks, deep learning, NLP · insight-1447
Build a contrast set before letting embedding similarity merge texts automatically
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Applied AI
- Industries
- aviation, saas
Summary: situation, action, reason, boundary
Situation
A duplicate-ticket merger links two tickets whenever the cosine similarity of their embeddings passes a threshold chosen on random pairs.
Action
Collect near pairs from your logs that differ only in polarity, quantity, date or entity, measure where they fall against the threshold, and add a verification step (extracted-field match or a pairwise model) before any automatic merge.
Reason
A bi-encoder embeds each text independently and compares by cosine, which is cheap but does not guarantee that opposite statements separate; a pairwise cross-encoder reads both together at much higher cost.
Boundary
Duplicate detection, FAQ matching, claim and complaint clustering.
Symptom or risk
Texts that differ in a negation, a number, a date or an airport share topic and wording and may still pass, so the merge destroys the one fact that separated them.
Exceptions
Human review before every merge makes the risk a cost question.
Trade-offs
Verification adds latency; a conservative threshold lowers recall.
Pitfalls
Tuning the threshold on random pairs only; building the contrast set from synthetic edits that never occur.
Lessons and concepts
Claims and their sources
Sentence-BERT encodes sentences independently and compares them by cosine similarity; the paper contrasts this with BERT pairwise scoring, which needs about 50 million inference passes for the most similar pair among 10,000 sentences.