Skip to content
ML Academy

project-013 · Classical ML · manufacturing

Yield-failure screening from 590 sensor signals: feature selection without fooling yourself

A semiconductor fab can send about 10% of production units to extra inspection before final line test. Can in-line sensor signals choose those units so that more failures are caught than by random inspection, and which signals should process engineers investigate first?

Licence
CC BY 4.0 (as stated on the UCI page)
Reference run
none

At a glance

  • Data

    SECOM (UCI 179) (public) · 1,567 units; 591 features per the page (confirm the column count on load); 104 fails

    Details on data

  • Baseline

    Random inspection (expected recall equals the inspection rate). Second baseline: the single sensor with the best training-period separation, thresholded at the 10%…

    Details on baseline

  • Validation

    Chronological split: first 75% of units (by timestamp) for training, last 25% for testing. Inside training, forward-chaining folds for tuning.

    Details on validation

  • Metric

    Recall and precision at the inspection rate with bootstrap intervals, PR-AUC, balanced error rate, selection frequency per signal cluster, optimism of the leaky variant.

    Details on metric

Success criterion

  1. On the chronologically last 25% of units: recall of failures at a 10% inspection rate, with a bootstrap interval; success if the interval's lower bound is above the 10% expected from random inspection.
  2. Balanced error rate reported for comparability with the provider's published baselines (about 33-40% for 40-feature selections), treated as a soft reference because their protocol details are not given.
  3. Signals reported to engineers are those selected in at least 60% of resampled training fits, grouped by correlation clusters.

Context

SECOM is semiconductor process data donated to UCI: 1,567 production units, several hundred sensor features with varying missingness, a pass/fail label from in-house line testing (104 fails), and a timestamp per unit. The provider frames the task as feature selection to find the signals that matter. With far more features than failures, selecting features on all data before cross-validating is the classic way to report an excellent model that does not work.

Data

Dataset
SECOM (UCI 179) (public)
Licence
CC BY 4.0 (as stated on the UCI page)
Size
1,567 units; 591 features per the page (confirm the column count on load); 104 fails
Verification
identity verified

Access

Direct download of the zip (secom.data 5.1 MB, secom_labels.data 39.7 KB, secom.names) from the UCI page; no account. Space-separated text with NaN for missing values. Fallback: none needed.

Grain

One row per production unit; features are unnamed sensor or measurement values; labels file aligned by row.

ColumnMeaning
feature_1 ... feature_N (secom.data)Anonymous sensor / process measurements; NaN = missing
label (secom_labels.data)-1 = pass, 1 = fail at in-house line test
timestamp (secom_labels.data)Date-time stamp of the test point for that unit

EDA and data-quality rules

  • Reconcile the page's feature count with the parsed column count; align labels by row and assert equal lengths.

  • Missingness per feature and per unit; plot missingness against time (sensor added or removed mid-period?).

  • Constant and near-constant features and exact duplicate columns are counted (removal happens inside the pipeline).

  • Fail rate over time and gaps in the timestamps; decide the time split from this plot.

Baseline

Random inspection (expected recall equals the inspection rate). Second baseline: the single sensor with the best training-period separation, thresholded at the 10% inspection rate, chosen on training data only.

Features

All sensors, with median imputation plus missing-indicator columns (a missing reading can be informative), a variance filter and standardization, all inside the pipeline. Optional: correlation clustering to report signal groups instead of individual columns.

Candidate methods

  • L1-regularized logistic regression with class weights

    Embedded feature selection that yields a short signal list; stable selection can be measured by resampling.

  • Random forest or extra-trees with balanced class weights

    Handles non-linear thresholds; importance computed by permutation on held-out data, not impurity.

  • Gradient boosting with native missing-value handling (HistGradientBoosting)

    Uses missingness patterns directly and often performs well on wide tabular data; checked for overfitting with so few fails.

Validation design

Chronological split: first 75% of units (by timestamp) for training, last 25% for testing. Inside training, forward-chaining folds for tuning. Every step that looks at labels or feature statistics (imputation, filtering, selection, scaling) runs inside each fold. A deliberately leaky variant (selection on all data, then random CV) is run once to show the optimism.

Leakage checks

Feature selection or imputation outside CV; random splits that ignore process drift; using test-period statistics for scaling. The leaky-variant comparison quantifies the effect.

Bias and representativeness

Process drift (maintenance, recipe changes) shifts sensor baselines; label noise in pass/fail; only 104 fails, so every metric is uncertain.

Robustness checks

Missingness thresholds, imputation choices, resampled selection stability, alternate inspection rates (5%, 10%, 20%).

Technical metrics

Recall and precision at the inspection rate with bootstrap intervals, PR-AUC, balanced error rate, selection frequency per signal cluster, optimism of the leaky variant.

Business interpretation

Fails caught per 100 inspections and expected cost = missed-fail cost x missed fails + inspection cost x inspections (costs hypothetical).

Error slices

  • Test-period months: does performance decay with time since training?

  • Units with high missingness.

  • Missed fails: which signal clusters, if any, separate them.

Implementation steps

  • Download, checksum, parse; align labels; run the EDA rules.

  • Define the chronological split; build pipelines with in-fold preprocessing.

  • Fit baselines and candidates; tune with forward-chaining folds.

  • Evaluate on the test period at the inspection rate with intervals; run the leaky variant for contrast.

  • Run resampled selection stability; group signals by correlation; write the engineer-facing signal list with the caveat that selection is not causation.

Reproduction

Learner-built. One command; cached raw files with checksums; seeds and split date in config; pinned environment.

Limitations

  • Anonymous features: engineers must map signal indices to process steps themselves.

  • Few failures and a single period; results may not transfer after process changes.

  • Selected signals are predictive markers, not proven causes of yield loss.

Production and monitoring

Score units as measurements arrive; signals from later process steps are not available for earlier decisions, so models may be needed per decision point. Monitor sensor drift and missingness, refresh after recipe changes, and feed inspection outcomes back as labels.

Interview questions

  • Your CV balanced error was 20% but the test period gives 35%. What likely happened?

  • Should a missing sensor reading be imputed away or used as a signal?

  • How do you present a signal list to engineers without implying causation?

Postmortem and alternatives

Statistical process control on a few stable signals may deliver most of the value with less maintenance. With process-step metadata, step-wise models aligned to decision points would be more useful than one model on all signals.

Rubric3 criteria

CriterionMeetsExceeds
Leakage controlAll preprocessing and selection inside folds with a chronological split.Quantifies the optimism of the leaky variant and explains it.
Evaluation under scarcityRecall at the inspection rate with bootstrap intervals.Discusses what 25-30 test failures can and cannot show and proposes how to collect more evidence.
Signal reportingSelection-frequency list grouped by correlation.Engineer-facing note that separates prediction from causation and proposes verification steps.

References6