Classical ML, validation, metrics, interpretation, debugging · insight-0154

Tune the threshold on cross-validated or fresh validation predictions, never on training predictions

Situation
You search thresholds to maximize a business metric such as profit or F-beta.
Action
Use TunedThresholdClassifierCV with a business scorer, or tune on a separate validation set, and report the final threshold's performance on an untouched test set.
Reason
The threshold is a fitted parameter; tuning it on the same data used to fit the model overfits like any other hyperparameter.
Boundary
Binary classifiers whose operating point is chosen from data.

Symptom or risk

Thresholds tuned on training-set scores fit overconfident in-sample predictions and fail on new data.

A threshold fixed by policy (regulation or a contractual service level) is not tuned and needs only verification.

Internal cross-validation costs refits; prefit mode is cheaper but needs a genuinely new validation set.

Using a scorer with the wrong positive label; built-in scorers assume the default positive class.

TunedThresholdClassifierCV tunes the threshold with internal cross-validation against a chosen scorer; the docs warn never to tune it on the training data.
Tuning the decision threshold for class prediction (scikit-learn user guide 3.3)