Neural networks, deep learning, NLP · insight-0466
Do not treat a t-SNE or UMAP picture of embeddings as evidence of separation
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, Data analyst, Applied AI
- Industries
- retail
Summary: situation, action, reason, boundary
Situation
A slide shows customer-message embeddings in 2D with neat colored clusters and claims the model separates segments.
Action
Use the plot for exploration only. Back any claim with a measurement in the original space: a linear probe's accuracy, nearest-neighbour label agreement or a clustering score, with a random-label baseline.
Reason
Wattenberg et al. show t-SNE cluster sizes mean nothing, distances between clusters might not mean anything, and results change with perplexity.
Boundary
Any 2D projection of embeddings shown to stakeholders.
Symptom or risk
Projection settings change cluster sizes, gaps and even apparent structure, so decisions rest on a picture rather than on a measurement.
Exceptions
Purely illustrative figures, labelled as such and not used for decisions.
Trade-offs
Quantitative checks are less persuasive visually; pair the plot with the number.
Pitfalls
Tuning perplexity until the picture matches the story.
Lessons and concepts
Claims and their sources
Wattenberg, Viegas and Johnson show that t-SNE cluster sizes mean nothing, distances between clusters might not mean anything, and plots vary with perplexity.