Skip to content
ML Academy

project-032 · Deep learning · e-commerce

Same product, different shop: TF-IDF vs sentence embeddings for product matching on Abt-Buy

A price-comparison team must link each product in retailer A's catalogue to the same product in retailer B's catalogue. Does a pretrained sentence-embedding model retrieve the correct match more often than a character n-gram TF-IDF baseline, and at what similarity threshold can a match be accepted without human review?

Licence
CC BY 4.0 (credit the website and the VLDB 2010 paper)
Reference run
none

At a glance

  • Data

    Abt-Buy (Leipzig entity-resolution benchmark) (public) · 1,081 Abt products, 1,092 Buy products, 1,097 gold matching pairs; about 135 KB zipped

    Details on data

  • Baseline

    Two baselines: (1) exact match on normalised names, which shows how much is trivially solvable;

    Details on baseline

  • Validation

    Split Abt products (queries) 50/50 at random into development and test halves, keeping all gold pairs of a query together;

    Details on validation

  • Metric

    recall@1, recall@5 and MRR with multiple relevant items per query; precision-coverage curve of top-1 matches over the similarity threshold; bootstrap intervals.

    Details on metric

Success criterion

On the held-out half of the Abt products:

  • recall@1, recall@5 and MRR for each method with bootstrap 95% intervals over queries;
  • an auto-accept threshold chosen on the development half to reach at least 95% precision for top-1 matches, and its coverage on the test half.

Recommend embeddings (or a hybrid) only if recall@1 or coverage at 95% precision improves outside the interval; otherwise keep TF-IDF.

Context

Day 38 (interpret an embedding similarity and its domain/evaluation caveat), with Days 39-40 for the encoder background. Product titles carry model codes where one character changes the product. Character n-grams match codes exactly; a general-purpose sentence encoder trained on sentence pairs may place two size variants of the same television line close together. Neither result is known in advance, which is the point of measuring. CPU only: the encoder is a 90.9 MB download (docs-only part if downloads are blocked) and encoding about 2,200 short titles takes seconds to a minute on a laptop.

Data

Dataset
Abt-Buy (Leipzig entity-resolution benchmark) (public)
Licence
CC BY 4.0 (credit the website and the VLDB 2010 paper)
Size
1,081 Abt products, 1,092 Buy products, 1,097 gold matching pairs; about 135 KB zipped
Verification
identity verified

Access

Abt-Buy.zip (134,745 bytes) linked from the Leipzig benchmark page; no authentication. The zip holds the two source files and a perfect-mapping file; inspect the CSV headers and encoding on first load and record them in the data card. Fallback if blocked: none that preserves the task; keep the project specified.

Grain

One row per product offer per shop; the mapping file has one row per matching pair. An Abt product can have more than one matching Buy record, so evaluation must allow several relevant items per query.

ColumnMeaning
idProduct identifier within its shop; join key to the mapping file.
nameProduct title, often containing brand and model code; the main matching field.
descriptionFree-text description; may be long or missing; truncated by the encoder beyond 256 word pieces.
manufacturer, priceAttributes listed by the provider for the e-commerce sets; check which shop actually populates each, and how price is formatted.
mapping (Abt id, Buy id)Gold matches; used only for evaluation and threshold tuning on the development half.

EDA and data-quality rules

  • Record row counts per file and per mapping; assert ids are unique and every mapped id exists.

  • Count Abt products with zero, one and several gold matches; queries with zero matches are excluded from recall@k and reported.

  • Missing-value rates for description, manufacturer and price by shop; parse price strings to numbers only after inspecting formats.

  • Normalise text the same way for every method (lowercase, unicode normalisation, collapse whitespace) and show five before/after examples.

Baseline

Two baselines: (1) exact match on normalised names, which shows how much is trivially solvable; (2) TF-IDF on character 3-5-grams within word boundaries (char_wb) over product names, cosine nearest neighbours among all Buy products. The vectoriser is fitted on the union of both catalogues' names; that is label-free and mirrors production, where the full catalogue is indexed.

Features

Text fields: name; name plus manufacturer; name plus the first 50 words of description. An extracted model-code token (alphanumeric tokens that contain digits, for example a code such as 'ABC-1234') supports an exact-code bonus in the hybrid. Embeddings are L2-normalised so cosine similarity equals a dot product.

Candidate methods

  • Sentence embeddings (all-MiniLM-L6-v2) with cosine nearest neighbours

    Captures paraphrase and word-order variation ('TV' against 'television'); the Day 38 object. Small enough for CPU.

  • Hybrid score: weighted TF-IDF cosine + embedding cosine + exact model-code bonus, weights tuned on the development half

    Combines exact-code precision with semantic recall; the usual practical answer when both signals carry information.

  • Embeddings on name plus description

    Tests whether extra text helps or dilutes the title signal; watch truncation at 256 word pieces.

Validation design

Split Abt products (queries) 50/50 at random into development and test halves, keeping all gold pairs of a query together; the Buy catalogue is the shared gallery for both. Blend weights and the auto-accept threshold are tuned on the development half only; the test half is read once. Bootstrap queries for intervals.

Leakage checks

No feature built from the gold mapping; thresholds and weights never tuned on the test half; if you fine-tune anything (optional), its training pairs come from the development half only. Note that the pretrained encoder may have seen public product pages; that is not label leakage, but it limits claims about unseen catalogues.

Bias and representativeness

Popular manufacturers with long, informative titles may be matched far better than the long tail. Report recall by manufacturer frequency band and by price band; a matcher that works only for big brands distorts price comparisons for smaller sellers.

Robustness checks

Perturb test titles: drop the manufacturer, reorder tokens, introduce one-character typos in the model code, change unit formats (for example inch written as a word against a symbol). Report recall@1 change per method. Also evaluate the tuned system without re-tuning on Amazon-GoogleProducts from the same page as a transfer task.

Technical metrics

recall@1, recall@5 and MRR with multiple relevant items per query; precision-coverage curve of top-1 matches over the similarity threshold; bootstrap intervals.

Business interpretation

Coverage at 95% precision (share of products auto-linked), review workload for the rest with a hypothetical seconds-per-pair, and the expected number of wrong links per 1,000 auto-accepted (each wrong link shows a customer a wrong price comparison).

Error slices

  • Titles with an extractable model code against titles without one.

  • Manufacturer frequency bands (top 10 brands against the long tail).

  • Size or capacity variants of the same product line (near-duplicates that differ by one number).

  • Queries with several gold matches; long descriptions that the encoder truncates.

Implementation steps

  • Studio 0-15 min: load, EDA rules, development/test split of Abt queries.

  • Studio 15-35 min: evaluation function for recall@k and MRR with multiple relevant items (unit-test it on a three-query toy example); exact-name and TF-IDF baselines.

  • Studio 35-60 min: encode both catalogues with the pre-downloaded model, compare; read 20 failures per method side by side.

  • Studio 60-90 min: hybrid weights and auto-accept threshold on the development half; decision-log entry.

  • Homework: test half read once; slices; robustness perturbations; the Amazon-GoogleProducts transfer task; one-page report.

Reproduction

Pin scikit-learn and sentence-transformers versions and the model revision hash in your own environment; cache embeddings as arrays with a checksum and the exact text template used; save the query split ids. One command recomputes all metrics from the cache. If model downloads are blocked, run and report the TF-IDF parts and mark the embedding parts as not run. No academy reference implementation.

Limitations

  • A small benchmark (about 2,200 offers) from a 2010 study; title conventions and catalogues have changed since, and two shops say little about a marketplace with thousands of sellers.

  • At this size an exhaustive similarity search is fine; at millions of items you need blocking or an approximate nearest-neighbour index, which adds its own recall loss.

  • Gold mappings in public benchmarks can contain errors; review a sample of 'false positives' before trusting precision.

  • A general-purpose encoder; a model fine-tuned on product pairs could behave very differently.

Production and monitoring

Version the encoder, text template and index together; a model change requires re-embedding both catalogues. Monitor the similarity-score distribution of accepted matches and audit a weekly random sample to estimate live precision, because there are no labels in production. Keep the exact-code rule as a guard: reject auto-matches whose extracted model codes disagree. New catalogue categories need a fresh threshold check.

Interview questions

  • Two products have cosine similarity 0.93 under your encoder but are different sizes of the same TV. What does that tell you about the embedding space, and how do you fix the matcher?

  • How do you compute recall@k when a query can have several correct matches, and why does it matter here?

  • How would you choose and monitor an auto-accept threshold when production has no labels?

Postmortem and alternatives

Record whether TF-IDF already solved most queries, where embeddings helped (paraphrased or reordered titles) and where they hurt (near-identical variants). If the hybrid wins, write down which component the gain came from by ablating each term. Alternatives: fine-tune the encoder on development-half pairs with a contrastive loss, add a cross-encoder re-ranker for the top 10 candidates, or use structured attribute extraction (brand, model code, size) before any similarity.

Rubric4 criteria

CriterionMeetsExceeds
Evaluation correctnessrecall@k and MRR handle several relevant items, unit-tested on a toy case; development/test split by query.Bootstrap intervals and a precision-coverage curve that drives the threshold choice.
Baseline strengthExact-name and character n-gram TF-IDF baselines reported before any embedding.Ablation of the hybrid showing each term's contribution.
Error analysisSide-by-side failures for both methods with the model-code and manufacturer slices.Robustness perturbations and the Amazon-GoogleProducts transfer result, explained.
DecisionRecommendation with coverage at 95% precision and the review workload under stated assumptions.Production design with label-free monitoring and a guard rule, plus the trigger for revisiting the threshold.

References5