project-049 · LLM and GenAI · cross-industry

Embedding drift monitoring for a virtual assistant, evaluated on simulated traffic with known injected shifts

A virtual-assistant platform wants an early warning when incoming user requests change - a new topic surging, more out-of-scope requests - before intent accuracy quietly degrades. Which embedding-based drift detector raises alarms soon after a real shift without firing every week on normal variation, and does an alarm say anything about whether the shift is harmful?

Dataset: CLINC150 (150 intents in 10 domains plus out-of-scope queries) as the source for seeded simulated weekly traffic
Licence: CC BY 3.0 (repository LICENSE file and dataset card)
Reference run: none
Lessons: day 41 day 52 day 55

Data
CLINC150 (150 intents in 10 domains plus out-of-scope queries) as the source for seeded simulated weekly traffic (public) · data_full: 15,000 train, 3,000 val, 4,500 test in-scope queries plus 100 / 100 / 1,000 out-of-scope (download-verified); 10 domains x 15 intents
Details on data
Baseline
Non-embedding monitors: weekly share of each predicted intent compared with the reference using a chi-square test (or population stability index), and the weekly…
Details on baseline
Validation
Run 50 seeded simulations per scenario, each 16 weeks: weeks 1-6 no drift, then an injected shift (domain mix moves toward travel by 10, 20 or 40 percentage points;
Details on validation
Metric
Detection delay (weeks after shift onset), detection rate within four weeks, false alarms per no-drift week, all per scenario and magnitude;
Details on metric

Success criterion

Choose the detector with the shortest median detection delay on injected shifts subject to a false-alarm rate of at most one alarm per 20 no-drift weeks across seeded simulations (target set in advance). The report must also show, for each injected shift, whether intent accuracy actually dropped - detection and harm are separate claims.

Context

Day 52 covers observability; Day 55 asks for monitoring in a production design; Day 41 supplies text representations. Real traffic never comes with drift labels, so detectors are hard to evaluate. Here the traffic is simulated from CLINC150: weekly batches are sampled with a schedule that injects known shifts (domain mix change, out-of-scope surge) at known weeks, which gives ground truth for delay and false alarms. The simulated traffic is labelled as simulated everywhere.

Data

Dataset: CLINC150 (150 intents in 10 domains plus out-of-scope queries) as the source for seeded simulated weekly traffic (public)
Licence: CC BY 3.0 (repository LICENSE file and dataset card)
Size: data_full: 15,000 train, 3,000 val, 4,500 test in-scope queries plus 100 / 100 / 1,000 out-of-scope (download-verified); 10 domains x 15 intents
Verification: download verified

Access

GitHub clinc/oos-eval data/data_full.json and data/domains.json (no login), or the Hugging Face parquet mirror clinc/clinc_oos. The weekly traffic is generated by a seeded learner script from these files; the schedule and seed are saved with every run.

Grain

Source rows are single queries with an intent (or oos). Simulated unit is a week: a batch of, say, 400 queries sampled according to the week's scheduled mix.

Column	Meaning
text	User query
intent	One of 150 intents or oos
domain	Derived from domains.json (banking, credit_cards, travel, utility, work, small_talk, meta, auto_and_commute, home, kitchen_and_dining)
week / scenario	Derived: simulated week index and the injected shift active that week (none, domain_shift, oos_surge, gradual)
drift_true	Derived ground truth: whether the week's sampling distribution differs from the reference

EDA and data-quality rules

Reference window built only from train queries; monitored weeks sample from val/test so the same sentence never appears in reference and monitored data.
Verify domain mapping covers all 150 intents exactly once.
Record the sampling schedule per scenario and check realised mixes against it (sampling noise at 400 queries per week is part of the test).

Baseline

Non-embedding monitors: weekly share of each predicted intent compared with the reference using a chi-square test (or population stability index), and the weekly out-of-scope prediction rate from the deployed classifier. These are what most teams already have.

Features

Embeddings: LSA (TF-IDF + truncated SVD fitted on the reference window; CPU, no download) and optionally a small sentence-embedding model (download). Detectors on embeddings: MMD two-sample test with a permutation p-value, a domain-classifier test (train a classifier to separate reference from current week; AUC with a permutation threshold), and mean-embedding cosine shift with a threshold set on no-drift simulations. Multiple-testing control across weekly tests is part of the design.

Candidate methods

MMD permutation test on LSA embeddings
Distribution-level test with an interpretable p-value; cost grows with batch size.
Classifier two-sample test
Also tells you which queries look new (highest classifier scores), useful for triage.
Predicted-label distribution test (baseline, no embeddings)
Cheap and already explainable; misses shifts that stay within the predicted-label mix.

Validation design

Run 50 seeded simulations per scenario, each 16 weeks: weeks 1-6 no drift, then an injected shift (domain mix moves toward travel by 10, 20 or 40 percentage points; out-of-scope share rises from 2% to 8% or 15%; or a gradual ramp). Thresholds are set on a separate set of no-drift simulations. Report detection delay distributions and false alarms per no-drift week.

Leakage checks

Thresholds and SVD components must not be tuned on the evaluation simulations; keep calibration and evaluation seeds separate. The intent classifier used for harm analysis is trained on train queries only.

Bias and representativeness

CLINC150 queries were crowdsourced paraphrases and are cleaner and more uniform than real traffic; real weekly variation (holidays, campaigns) will raise false alarms. Report this as a lower bound on real false-alarm rates.

Robustness checks

Vary batch size (100, 400, 1,600 queries per week) and embedding choice; a detector whose ranking flips with batch size needs a stated operating range.

Technical metrics

Detection delay (weeks after shift onset), detection rate within four weeks, false alarms per no-drift week, all per scenario and magnitude; compute time per weekly check.

Business interpretation

Weeks of degraded service avoided versus analyst time spent on false alarms; for each shift, the measured drop (or not) in intent accuracy and out-of-scope recall.

Error slices

Shift type and magnitude: small domain shifts vs out-of-scope surges.
Batch size.
Harmful vs harmless shifts: alarms on shifts that did not reduce accuracy.

Implementation steps

Load data and domain map; write the seeded traffic simulator with scenario schedules and unit tests that realised mixes match schedules within sampling error.
Train the intent classifier on train queries (TF-IDF + logistic regression) for the baseline monitor and harm analysis.
Implement LSA embeddings and the three detectors; calibrate thresholds on no-drift simulations.
Run evaluation simulations; compute delay, detection and false-alarm metrics; run the harm analysis.
Write the monitoring design: which detector, batch size, alert routing and the triage view.

Reproduction

Learner-authored; for example uv run python -m drift.simulate --scenarios all --sims 50 --seed 100 and uv run python -m drift.evaluate. CPU-only with LSA; the optional sentence-embedding variant downloads a model once. All outputs are labelled as simulated traffic.

Limitations

Simulated shifts are clean mixture changes; real drift also includes new phrasing, new products and seasonality.
Detection says the distribution changed, not that the system got worse; harm needs labels or proxies.
Results depend on batch size; small weekly volumes make every detector slow.

Production and monitoring

Compute the chosen detector weekly per product line, store the reference window version, and show the most 'novel' queries (classifier scores) to an analyst with every alarm. Refresh the reference window on a documented schedule, not after every alarm, or you will absorb drift silently. Pair drift alarms with a small weekly labelled sample to measure harm.

Interview questions

Your drift detector fires; accuracy on a labelled sample is unchanged. What do you do?
How would you evaluate a drift detector when production traffic has no drift labels?
Why is resetting the reference window after each alarm dangerous?

Postmortem and alternatives

Expect the out-of-scope surge to be caught fastest by the simple out-of-scope rate monitor; discuss whether the embedding detectors earn their complexity by catching shifts the label-based monitors miss.

Rubric3 criteria

Criterion	Meets	Exceeds
Ground-truth evaluation design	Seeded simulator with known shift onsets; separate calibration and evaluation seeds.	Shows the effect of batch size and shift magnitude on delay and false alarms.
Detection vs harm	Reports accuracy change for each injected shift alongside alarms.	Proposes an alert policy that combines drift and harm signals.
Monitoring design	Chosen detector, cadence and reference-window policy documented.	Triage view and runbook for an alarm, with ownership.

References5

Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift
Two-sample-test framing of shift detection and comparison of detectors.
An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction
Dataset and the out-of-scope problem.
Indexing by latent semantic analysis
LSA embeddings used as the CPU-only representation.
Evidently documentation
Example tooling for drift reports in production.
Designing Machine Learning Systems
Data distribution shifts and monitoring in ML systems.

At a glance