Neural networks, deep learning, NLP · insight-1720
Exclude duplicate positives from a batch when training a retriever with in-batch negatives
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A bi-encoder is fine-tuned on question and answer pairs with in-batch negatives, and several questions share the same answer or the same question appears twice.
Action
Build batches with no repeated text or answer id, using a no-duplicates sampler, or group by a canonical answer id first when different wordings mean the same answer. Log how often a batch would have held a repeat.
Reason
Sentence Transformers' documentation says a duplicate of the anchor or positive in a batch acts as an incorrect negative; the no-duplicates sampler is its documented remedy.
Boundary
Contrastive and ranking losses that use other batch items as negatives, on data with skewed answer frequency.
Symptom or risk
The loss treats the correct answer for another question as a wrong one and pushes it away, and this happens most often for the most frequent answers.
Exceptions
Corpora where every positive is unique by construction.
Trade-offs
Batches become less random, and tiny datasets may not fill batches without repeats.
Pitfalls
Checking exact text equality only, while near-identical paraphrases still collide.
Example
Simulated (seed 0): 1,000 answers drawn with 1/rank frequency, batches of 64, 2,000 batches. About 44 percent of anchors share their positive with another pair in the batch, against about 6 percent under uniform draws.
Lessons and concepts
Claims and their sources
The Sentence Transformers losses reference describes MultipleNegativesRankingLoss as using the other batch items as negatives, warns that duplicates of the anchor or positive would act as incorrect negatives, and advises the NO_DUPLICATES batch sampler.
NoDuplicatesBatchSampler keeps any value from repeating within a batch, including across columns, because a duplicate would otherwise become a false negative for losses that use batch items as negatives.