Skip to content
ML Academy

project-010 · Classical ML · banking

Card-fraud alerts under a fixed review budget: ranking on day 1, tested on day 2

A card issuer's fraud operations team can review a fixed number of alerts per day. Which transactions should be alerted so that the most fraud value is stopped within that budget, and how much fraud should the team expect to catch?

Licence
Public (as stated by OpenML)
Reference run
none

At a glance

  • Data

    creditcard (OpenML data id 1597; ULB / Worldline credit card fraud data) (public) · 284,807 rows x 31 columns; 492 positives

    Details on data

  • Baseline

    Business rule: alert the highest-amount transactions up to the budget. Second baseline: logistic regression on Amount and hour of day.

    Details on baseline

  • Validation

    Time split: day 1 (Time < 86,400 s) for training, with its last six hours held out for early stopping and threshold choice; day 2 for testing.

    Details on validation

  • Metric

    PR-AUC (average precision), recall and precision at the budget, recall of fraud amount at the budget, with bootstrap intervals;

    Details on metric

Success criterion

  1. On the second day of data (test), recall of fraud transactions and of fraud amount at the alert budget (hypothetical: 0.2% of transactions) with 95% bootstrap intervals.
  2. The selected model beats the amount-ranking rule baseline by more than the half-width of the bootstrap interval of the difference; otherwise the rule stays.
  3. PR-AUC reported as the threshold-free summary; accuracy is not reported as a success metric.
  4. A written explanation of how undersampling (if used) changes predicted probabilities and how they were corrected.

Context

The OpenML copy of the ULB/Worldline dataset contains card transactions by European cardholders over two days in September 2013: 284,807 transactions, 492 frauds (0.172%). For confidentiality, 28 features are PCA components; only Time (seconds since the first transaction) and Amount are raw. There is no card or customer identifier, so the behavioural features that usually dominate fraud models (velocity, merchant novelty) cannot be built. The provider does not say which rows the PCA was fitted on; if it used both days, the components carry information from the test period.

Data

Dataset
creditcard (OpenML data id 1597; ULB / Worldline credit card fraud data) (public)
Licence
Public (as stated by OpenML)
Size
284,807 rows x 31 columns; 492 positives
Verification
identity verified

Access

sklearn.datasets.fetch_openml(data_id=1597, as_frame=True) or download from openml.org; no account. Cache the file and record its OpenML version and checksum. A Kaggle copy exists but needs a Kaggle login; it is not required.

Grain

One row per card transaction; no card, customer or merchant key.

ColumnMeaning
TimeSeconds elapsed since the first transaction in the dataset
V1-V28Principal components of undisclosed original features
AmountTransaction amount (not PCA-transformed)
Class1 = fraud, 0 = genuine

EDA and data-quality rules

  • Confirm Time spans two days and plot transaction and fraud counts per hour; the split uses Time, so daily cycles must be visible.

  • Count exact duplicate rows and check whether duplicates cross the day boundary.

  • Correlations among V1-V28 within the training day: near-zero correlations on the full data but non-zero on one day would be consistent with a PCA fitted on both days.

  • Amount distribution by class, including zero-amount transactions.

Baseline

Business rule: alert the highest-amount transactions up to the budget. Second baseline: logistic regression on Amount and hour of day. Random alerting gives recall equal to the budget share and anchors the lift.

Features

V1-V28 as given, log(1 + Amount), hour of day derived from Time (cyclic encoding). No resampling by default; class weights are a tuning option. Card-level aggregates are impossible here and are listed as the main missing signal.

Candidate methods

  • L2 logistic regression with class weights

    Strong, fast, interpretable on PCA inputs; a calibrated baseline for expected-loss thresholds.

  • Gradient boosting (LightGBM or HistGradientBoosting)

    Captures non-linear patterns in the components and amount; usually stronger on tabular fraud data, to be confirmed under a time split.

  • Isolation Forest (unsupervised)

    Represents the situation where labels arrive late or are scarce; shows how far an anomaly score gets without labels.

Validation design

Time split: day 1 (Time < 86,400 s) for training, with its last six hours held out for early stopping and threshold choice; day 2 for testing. Random K-fold would mix days and is reported only as a contrast. Bootstrap the test day (stratified by class) for intervals; only a share of the 492 frauds (a few hundred at most) fall in the test day, so intervals are wide; count them first.

Leakage checks

Possible look-ahead through PCA fitted on both days (cannot be fixed, must be stated); random splits mixing days; duplicates crossing the split; offline evaluation assumes labels are known instantly, while real chargebacks arrive weeks later.

Bias and representativeness

No customer attributes are available to audit; amount-weighted objectives favour catching large frauds and may under-protect customers with small, frequent fraud. Alerts also burden genuine customers (declines, calls).

Robustness checks

Alert budgets of 0.1%, 0.2% and 0.5%; class-weight settings; seeds; Isolation Forest contamination parameter.

Technical metrics

PR-AUC (average precision), recall and precision at the budget, recall of fraud amount at the budget, with bootstrap intervals; calibration of probabilities if an expected-loss threshold is used.

Business interpretation

Fraud value caught per day at the budget, false alerts per caught fraud, and net value = caught amount x recovery rate - review cost x alerts (recovery rate and review cost hypothetical).

Error slices

  • Amount bands: small-amount frauds missed by the amount rule and possibly by the model.

  • Hour of day (night hours have fewer transactions and a different fraud share).

  • Missed frauds at the budget (false negatives): inspect their scores and amounts.

  • Zero-amount transactions.

Implementation steps

  • Fetch from OpenML, cache, checksum; run EDA rules.

  • Create the day-1/day-2 split and the validation window; compute baselines.

  • Train candidates; choose the operating point on the validation window at the budget.

  • Evaluate on day 2 with bootstrap intervals; compare with the amount rule.

  • If undersampling is tried, show the probability shift and apply the prior-correction before any expected-loss calculation.

  • Write the report with an alert-budget curve (recall versus budget) and a missed-fraud review.

Reproduction

Learner-built. One command fetches (or reads the cache), trains and writes the report; seeds and budget in config; pinned environment. The OpenML data version is recorded with the results.

Limitations

  • Two days of data: no seasonality, drift or label delay can be studied.

  • Anonymised PCA features prevent feature engineering and explanation in business terms.

  • Possible look-ahead in the PCA cannot be removed.

Production and monitoring

Real-time scoring with a latency budget, a rules layer for known patterns, champion-challenger evaluation, and monitoring that accounts for label delay (chargebacks) and for feedback: only alerted transactions get reviewed labels quickly, which biases retraining data. Keep a small random review sample to estimate true recall.

Interview questions

  • The model has 99.9% accuracy. Why is that meaningless here, and what do you report instead?

  • You undersampled to 1:1 and the predicted fraud probabilities look far too high. Why, and how do you fix them?

  • How do label delays and alert-only feedback change your validation and retraining design?

Postmortem and alternatives

With card identifiers, velocity and history features would matter more than any model choice. An example-dependent cost-sensitive objective (weighting by amount) is the alternative when the business values fraud amount rather than counts.

Rubric3 criteria

CriterionMeetsExceeds
Validation realismTime-based day-1/day-2 split with a separate validation window.Contrasts with random K-fold and explains the PCA and label-delay caveats.
Metric choiceRecall and precision at a budget and PR-AUC with intervals.Adds amount-weighted recall and an alert-budget curve tied to operations capacity.
Imbalance handlingUses class weights or no resampling and explains why.Demonstrates and corrects the probability shift from undersampling.

References5