project-047 · LLM and GenAI · saas
Similar-ticket search for a software support desk: BM25, TF-IDF, embeddings and hybrid on human-labelled duplicates
A software support desk wants to show agents "similar past tickets" when a new ticket arrives, so known issues are linked instead of re-solved. Does semantic search (sentence embeddings, or a hybrid with BM25) put a genuinely similar question in the top three more often than lexical search, and is the gain worth an embedding service?
- Licence
- Repository states no licence; underlying posts are Stack Exchange user contributions under CC BY-SA 2.5/3.0 (attribution, share-alike) - do not redistribute derived files without checking
- Reference run
- none
At a glance
Data
AskUbuntu question dataset (2014 dump) with human similar-question annotations (public) · About 167K questions; dev and test 200 queries each with 20 candidates (11 dev and 14 test queries have no similar candidate, download-verified)
Details on dataBaseline
Two lexical baselines: the provided Lucene BM25 order over the 20 candidates (setting 1, no computation needed), and your own BM25 over the full corpus (setting 2),…
Details on baselineValidation
Choose fields, fusion constant and any thresholds on the dev queries; evaluate once on test queries. Setting 1 (rerank 20 candidates) matches the benchmark protocol;
Details on validationMetric
MAP, MRR, precision@1 and @5 (setting 1, benchmark protocol); recall@10 and unlabelled-hit share (setting 2); query latency and index build time on CPU.
Details on metric
Success criterion
Adopt embeddings or hybrid only if precision@1 and MRR on the test queries improve over BM25 with paired-bootstrap intervals excluding zero (target set in advance) and the embedding index fits the team's latency budget (for example p95 under 100 ms per query on CPU at corpus scale, measured). Otherwise keep BM25 and invest in title quality and duplicate-linking workflow.
Context
Day 41 contrasts sparse and dense representations; Days 48 and 51 evaluate retrieval with ranking metrics. AskUbuntu is a real support forum with user-marked duplicates and human similarity annotations, which makes it a credible stand-in for support tickets. The benchmark's evaluation setting reranks 20 BM25 candidates per query; a production search must also find candidates in the whole corpus, so this project runs both settings.
Data
- Dataset
- AskUbuntu question dataset (2014 dump) with human similar-question annotations (public)
- Licence
- Repository states no licence; underlying posts are Stack Exchange user contributions under CC BY-SA 2.5/3.0 (attribution, share-alike) - do not redistribute derived files without checking
- Size
- About 167K questions; dev and test 200 queries each with 20 candidates (11 dev and 14 test queries have no similar candidate, download-verified)
- Verification
- download verified
Access
GitHub repository taolei87/askubuntu (no login): dev.txt and test.txt (200 queries each, 20 BM25 candidates, human labels), text_tokenized.txt.gz (about 46 MB, about 167K questions, bodies truncated to 100 words). Cache with hashes. The raw XML lives on Google Drive and is not needed.
Grain
One query question with 20 candidate questions, a subset labelled similar; corpus rows are (id, tokenised title, tokenised body).
| Column | Meaning |
|---|---|
| question_id | AskUbuntu question id (2014 dump) |
| title / body | Lower-cased, tokenised title and body; body truncated at 100 words |
| similar_ids | Human-annotated similar questions among the candidates (dev/test) |
| candidate_ids / bm25_scores | 20 candidates and their Lucene BM25 scores provided with the data |
| train_random | User-marked duplicates plus random negatives, for optional fine-tuning or threshold work |
EDA and data-quality rules
Exclude queries with no similar candidate from ranking metrics (11 dev, 14 test), report the counts, and keep them for a separate 'no good match' analysis.
Check that every candidate and query id exists in the corpus file; count missing ids.
Text is pre-tokenised and lower-cased; record that embeddings will see tokenised text, not the original formatting, which can reduce their quality.
Baseline
Two lexical baselines: the provided Lucene BM25 order over the 20 candidates (setting 1, no computation needed), and your own BM25 over the full corpus (setting 2), title-only and title+body.
Features
Representations: TF-IDF (fitted on the corpus excluding dev/test queries), a small open sentence-embedding model (model download; CPU feasible for 167K short texts with batching and caching), and a hybrid by reciprocal rank fusion. Field choice (title vs title+body) is an explicit experiment, because titles carry most of the intent in support questions.
Candidate methods
Sentence-embedding retrieval (bi-encoder)
Matches paraphrased problem descriptions ('wifi drops after suspend' vs 'network lost on resume').
Hybrid BM25 + embeddings via RRF
Keeps exact matches on package names and error codes while adding paraphrase recall.
Validation design
Choose fields, fusion constant and any thresholds on the dev queries; evaluate once on test queries. Setting 1 (rerank 20 candidates) matches the benchmark protocol; setting 2 (full corpus) measures the production task, using the labelled similar ids as relevant and treating unlabelled hits as unknown rather than wrong (report how many top-5 hits are unlabelled).
Leakage checks
Remove each query question from its own search results and from the TF-IDF fitting corpus. If you fine-tune on train_random, make sure no dev/test question appears in the training pairs.
Bias and representativeness
The data are from 2014, in English, about one product family; questions about newer releases or other products are absent. Popular topics have many duplicates and dominate averages; report results by topic cluster if possible.
Robustness checks
Truncate queries to the title only, add typos to ten queries, and check how each method degrades.
Technical metrics
MAP, MRR, precision@1 and @5 (setting 1, benchmark protocol); recall@10 and unlabelled-hit share (setting 2); query latency and index build time on CPU.
Business interpretation
Share of new tickets with a correct similar ticket in the top three (agent links it instead of re-solving), and the share of suggestions that waste agent attention.
Error slices
Queries whose similar questions share few words with them (lexical gap).
Title-only vs title+body representations.
Queries with many vs few labelled similar questions.
Implementation steps
Download, hash, parse; build the id-to-text map and verify all ids resolve.
Setting 1: score provided BM25 order, TF-IDF and embeddings over the 20 candidates; implement MAP/MRR/P@k with unit tests.
Setting 2: index the full corpus (BM25 and embeddings), search for each test query, compute recall@10 and unlabelled-hit share.
Add the hybrid; choose settings on dev; run test once; bootstrap the differences.
Write the recommendation with latency and the 'no good match' analysis.
Reproduction
Learner-authored; for example uv run python -m tickets.prepare, uv run python -m tickets.rerank --split test and uv run python -m tickets.fullsearch --split test. BM25 and TF-IDF are CPU-only; the embedding arm downloads a small model once and caches embeddings to disk with the model name and version.
Limitations
Forum questions are cleaner and more public than enterprise tickets, which contain logs, screenshots and private data.
Unlabelled hits in the full-corpus setting may be relevant; recall there is a lower bound.
Tokenised, lower-cased text removes formatting that real tickets have (stack traces, versions).
Production and monitoring
Index new tickets incrementally; log shown suggestions and agent clicks or links as weak labels (biased by position). Monitor the share of tickets with no suggestion above a similarity floor, and re-embed the corpus when the embedding model changes (store the model version with each vector).
Interview questions
Your embedding model wins in the 20-candidate benchmark setting but not in full-corpus search. Why can that happen?
How would you evaluate 'similar ticket' search when most relevant pairs were never labelled?
Postmortem and alternatives
Discuss whether a better duplicate-linking workflow (agents marking duplicates) creates more value than a better model, since it also produces the labels the next model needs.
Rubric3 criteria
| Criterion | Meets | Exceeds |
|---|---|---|
| Two evaluation settings | Benchmark rerank setting and full-corpus setting both run and reported separately. | Explains the gap between settings with examples and unlabelled-hit analysis. |
| Representation experiment | Fields and methods compared with paired intervals on test after choosing on dev. | Error slices show where embeddings help (lexical gap) and where they hurt (exact identifiers). |
| Operational judgement | Latency and index cost measured; recommendation follows the pre-set rule. | Feedback loop design for collecting labels without position-bias errors. |
References6
- Semi-supervised Question Retrieval with Gated Convolutions
Origin of the benchmark and its 20-candidate BM25 protocol.
- The Probabilistic Relevance Framework: BM25 and Beyond
Lexical baseline.
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Bi-encoder sentence embeddings for efficient similarity search.
- SentenceTransformers Documentation
Practical embedding and reranking models.
- Reciprocal rank fusion outperforms condorcet and individual rank learning methods
Rank fusion for the hybrid.
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
Out-of-domain behaviour of dense vs lexical retrieval; BM25 as a robust baseline to beat.