Skip to content
ML Academy

project-035 · Deep learning · telecom

Debugging a neural training script that runs but does not learn

A (simulated) churn-model training script in a telecom analytics team runs without errors, the loss curve looks plausible, and validation AUC sits near chance. Can you find and fix every planted bug with systematic checks (shape assertions, loss at initialisation, overfitting one batch, a numerical gradient check, an input-independent baseline) instead of tuning hyperparameters, and leave a regression test for each?

Dataset
Synthetic telecom churn snapshots (seeded generator)
Licence
academy-generated synthetic data; no third-party licence
Reference run
executed

At a glance

  • Data

    Synthetic telecom churn snapshots (seeded generator) (synthetic) · 20,000 rows × 10 features plus true probability and label (configurable); under 2 MB in memory

    Details on data

  • Baseline

    Three anchors: the prevalence model (log loss of predicting the base rate), logistic regression with one-hot contract type (should be near the oracle except for the…

    Details on baseline

  • Validation

    Stratified 70/15/15 train/validation/test split with a fixed seed. Debugging uses train and validation only; the test split confirms the final fixed model once.

    Details on validation

  • Metric

    Validation log loss and AUC against the oracle and logistic regression; loss at initialisation against the expected value (ln 2 ≈ 0.693 with a zero output bias, about…

    Details on metric

Success criterion

  • All six planted bugs found. For each: the check that exposed it, a minimal failing test, the fix, and validation AUC and log loss before and after.
  • The fixed network reaches validation log loss within 0.01 of logistic regression on the same split, and you report the gap to the oracle (Bayes-optimal) log loss, which is computable because the data are synthetic with known probabilities.
  • The test suite runs in under 60 seconds on CPU.

Context

Week 7 (Days 31, 33, 34, 35). Silent failures are the expensive ones: a script that crashes gets fixed, a script that trains to a mediocre score gets shipped. This project trains the habits from the practitioner literature: check the loss at initialisation, overfit a single batch, compare against an input-independent baseline, verify gradients numerically, and add complexity one step at a time (Karpathy, 2019; CS231n notes). The data are synthetic, generated from a known logistic model with interactions, so you know the best achievable score and can tell a bug from a hard problem. The instructor fixture is a buggy numpy (or PyTorch) script; the learner never sees a list of the bugs, only the symptom.

Data

Dataset
Synthetic telecom churn snapshots (seeded generator) (synthetic)
Licence
academy-generated synthetic data; no third-party licence
Size
20,000 rows × 10 features plus true probability and label (configurable); under 2 MB in memory
Verification
unverified

Access

Generated by a seeded function in the project repository (make_churn(n=20000, seed=7)), no network. Synthetic: simulated customers, not real subscribers. Features are drawn from stated distributions and the label from a known logistic model with two interaction terms, so the true probability of each row is stored for oracle metrics. Optional transfer check: run the fixed script on scikit-learn digits as a 10-class task.

Grain

One row = one simulated customer at a monthly snapshot; label = churn in the following month.

ColumnMeaning
tenure_months, contract_typeMonths as a customer (0-72) and contract type {monthly, one-year, two-year}; long contracts lower churn.
monthly_charge, data_usage_gb, roaming_shareMonthly bill, data use and share of usage abroad; the bill interacts with contract type in the generator.
support_calls_90d, dropped_call_rateSupport contacts and network quality; both raise churn probability, with an interaction between them.
p_true, churnGenerator probability (for oracle metrics only, never a feature) and the sampled 0/1 label (about 20% positive by design).

EDA and data-quality rules

  • Assert the generator's stated properties: row count, positive rate within 1 point of the design, no NaN, contract categories present.

  • Assert shapes at every boundary: X is (n, d) float64, y is (n,) or (n, 1) consistently, and predictions match y's shape before any loss is computed.

  • Check that row order is shared by X and y after every shuffle (a hash of the first rows of X and y before and after).

  • Compute the oracle log loss and AUC from p_true on the validation split; it is the ceiling, and logistic regression should sit close to it.

Baseline

Three anchors: the prevalence model (log loss of predicting the base rate), logistic regression with one-hot contract type (should be near the oracle except for the interactions), and the oracle itself from p_true. A network that lands near the prevalence model is broken, not unlucky.

Features

Standardised numeric features (scaler fitted on the training split only) and one-hot contract type; the same matrix for the network and logistic regression. p_true is excluded and a test asserts its absence.

Candidate methods

  • The provided two-layer MLP (10→32→1, ReLU, sigmoid output, binary cross-entropy), debugged

    The object of the exercise; small enough that every check runs in seconds.

  • Logistic regression on the same matrix

    Reference that cannot share the network's bugs; any large gap is evidence of a bug, not of model choice.

Validation design

Stratified 70/15/15 train/validation/test split with a fixed seed. Debugging uses train and validation only; the test split confirms the final fixed model once. Each bug fix is re-measured on validation with 3 seeds so improvements are not seed noise.

Leakage checks

One planted bug is a leak: the scaler is fitted on all rows before the split. It does not stop learning; it makes validation slightly optimistic and is caught by a test that the scaler sees training indices only. Also assert p_true never enters the features.

Bias and representativeness

Synthetic customers, so no real-group fairness claim is possible. Use the known generator to check calibration by contract type: the fixed model should be calibrated within each contract segment, and a persistent segment gap after fixing usually means a remaining bug or a missing interaction.

Robustness checks

After fixing: shuffle the row order and re-run (results should be unchanged up to seed noise); run with float32 and float64; run the fixed script on digits as a 10-class transfer check to make sure fixes were general and not tuned to the churn file.

Technical metrics

Validation log loss and AUC against the oracle and logistic regression; loss at initialisation against the expected value (ln 2 ≈ 0.693 with a zero output bias, about 0.50 with the bias set to the logit of a 20% base rate); maximum relative error of the gradient check per parameter group; number of bugs found and tests written.

Business interpretation

Time to diagnosis and the number of checks that would have caught each bug automatically in CI. In a real team, a silent bug that ships costs every decision made on the model's scores; here you estimate that cost as the difference between the buggy and fixed model's expected retention-campaign value under a stated hypothetical value per correctly targeted churner.

Error slices

  • Per bug: which check exposed it (shape assertion, loss at init, overfit one batch, gradient check, input-independent baseline, code review).

  • Per contract type: calibration of the fixed model against the generator.

  • High-support-call customers, where the generator's interaction lives and a missing-nonlinearity bug shows first.

Implementation steps

  • Studio 0-15 min: reproduce the symptom; compute the three anchors (prevalence, logistic regression, oracle). Write down hypotheses before touching the code.

  • Studio 15-40 min: cheap checks first: shape assertions at every boundary, loss at initialisation, the input-independent baseline (train on zeroed inputs; if it matches the real model, the features and labels are disconnected).

  • Studio 40-65 min: overfit 32 rows with regularisation off; run a centred-difference gradient check in float64 on 16 rows, with dropout off, comparing each parameter group's relative error with the CS231n thresholds.

  • Studio 65-90 min: fix, add one failing-then-passing test per bug, re-measure with 3 seeds; record each bug in a bug log (symptom, check, cause, fix, test).

  • Homework: remaining bugs, the robustness runs, the digits transfer check, one test-split confirmation, and a one-page postmortem.

Reproduction

The instructor fixture (buggy script plus generator) lives in the learner's project repository; the fixed version must pass pytest -q in under 60 seconds and reproduce its metrics with one seeded command (for example uv run python train.py --seed 0). Planted bugs in the fixture, for instructors only: labels permuted independently of features; a (n,) versus (n, 1) broadcast that turns the loss into an n × n mean; sigmoid applied twice (probabilities fed to a loss that expects logits); a missing ReLU derivative mask in the backward pass; the scaler fitted before the split; dropout left active at evaluation. A builder-authored academy reference implementation (not learner evidence) generates the data and runs the checks.

Limitations

  • Planted bugs are cleaner than real ones; real failures often combine two small bugs with interacting symptoms.

  • A tiny network and tabular synthetic data; data-pipeline and distributed-training bugs are out of scope.

  • The oracle ceiling exists only because the data are synthetic; on real data you must use strong baselines instead.

Production and monitoring

Turn the checks into permanent tests that run in CI on a small fixture: shape contracts at module boundaries, loss-at-init within tolerance, overfit-a-batch within N steps, gradient check on the custom layers, scaler-fit-on-train-only, and an evaluation-mode assertion. Track them as part of an ML test rubric, and block a model release when any fails.

Interview questions

  • A model trains without errors but validation AUC is 0.52. Walk me through the first five things you check, in order, and why that order.

  • What does a successful overfit-one-batch test prove, and what does it not prove?

  • How do you run a gradient check on a network with ReLU, and how do you interpret a relative error of 1e-3?

Postmortem and alternatives

For each bug, write which check caught it, how long it took, and which earlier check could have caught it faster. Typical lesson to test against your own log: the label-permutation and broadcast bugs are found in minutes by an input-independent baseline and a shape assertion, while hyperparameter tuning would never find them. Alternative approaches: a framework's automatic differentiation removes the backward-pass bug class but not the others; data contracts catch the leakage bug earlier.

Reference implementation

Status: executed — outputs below are from the recorded run

cd python && uv run python -m reference.r15_debug_silent_training_bugs.run

r15 results - eight planted bugs, ten checks, fixed one at a time

Recorded run of cd python && uv run python -m reference.r15_debug_silent_training_bugs.run (full mode). Numbers are copied from outputs/results.json, bug_metrics.csv, detection_matrix.csv and fix_path.csv, written by that run. Data: simulated telecom churn customers (seed 7, 20,000 rows, positive rate 19.95%; train 14,000 / validation 3,000 / test 3,000, stratified, split seed 0). Network: numpy MLP 10-32-1, SGD with momentum, 30 epochs.

ItemValue
Date2026-10-12
Seedsrun seeds 0, 1, 2 (initialisation, dropout, batch order); generator seed 7; split seed 0
Python / numpy / scikit-learn3.12.13 / 2.5.3 / 1.9.1
Runtime of the recorded run34.5 s wall, 34.5 s CPU (single thread); smoke run about 15 s; test file about 13 s

Anchors (validation split)

AnchorLog lossAUC
Prevalence model (predict the training base rate)0.49990.5000
Logistic regression, same matrix0.37710.8271
Oracle (p_true from the generator)0.35940.8407

Each bug alone (mean of 3 seeds, sd in brackets; validation)

ConfigurationVal AUCVal log lossWhat the number looks like
correct script0.8344 (0.0004)0.3658 (0.0008)reference
broadcast_loss0.7798 (0.0042)0.4993 (0.0003)loss equals the prevalence model's
label_misalignment0.5626 (0.0247)0.4976 (0.0010)no signal learned
sigmoid_twice0.7801 (0.0068)1.2823 (0.0022)ranks fairly, probabilities useless
no_shuffle0.7029 (0.0433)1.3511 (0.0100)worse than prevalence; unstable across seeds
missing_relu_mask0.8146 (0.0046)0.3976 (0.0053)looks like a slightly weak model
regulariser_scale0.8194 (0.0016)0.3998 (0.0017)looks like a slightly weak model
scaler_before_split0.8345 (0.0005)0.3658 (0.0006)indistinguishable from correct
dropout_at_eval0.8305 (0.0015)0.3703 (0.0018)small loss, non-repeatable predictions
all eight together0.5000 (0.0000)1.4064 (0.0000)constant output

Which check exposed which bug (X = alarm; checklist order left to right, seed 0)

Configurationshapealigninit lossbatch ordergradcheckoverfit 1 batchzero inputref. gapscalereval determ.first alarm
correct script..........none
broadcast_lossX....XXX..shape
label_misalignment.X....XX..align
sigmoid_twice..X..X.X..init loss
no_shuffle...X..XX..batch order
missing_relu_mask....XX.X..gradcheck
regulariser_scale.....X.X..overfit 1 batch
scaler_before_split........X.scaler
dropout_at_eval.........Xeval determ.

Evidence behind the alarms (values printed by the checks): shape - loss elements (32, 32) against predictions (32, 1); align - 0.683 of training rows carry their own label; init loss - 0.759 against an expected 0.500 for sigmoid_twice (the correct script: 0.501); batch order - batch positive-rate sd 11.3 times the random-batch value; gradcheck - 313 of 385 coordinates fail (max relative error 2.0) for the missing mask, 0 of 385 (4e-07) for the correct script; overfit one batch - regulariser_scale ends at 0.267 as configured and 0.002 with L2 off (the correct script: 0.004 and 0.002); zero input - real and zeroed-input validation loss 0.501 vs 0.500 (labels) and 0.500 vs 0.500 (broadcast); reference gap - label_misalignment leaves logistic regression 0.141 above the oracle on the same prepared matrix (correct matrix: 0.018) while the net-minus-LR gap is -0.002, which says the fault is upstream of the model; scaler - standardiser fitted on 20,000 rows against 14,000 training rows; eval determinism - two predictions of the same rows differ by up to 0.361.

Fixing one at a time (checklist order, all eight present at step 0; 3 seeds)

StepFixedVal AUCVal log loss
0none0.50001.4064
1broadcast_loss0.48341.3853
2label_misalignment0.54131.3852
3sigmoid_twice0.50001.4064
4no_shuffle0.50000.5000
5missing_relu_mask0.81630.4009
6regulariser_scale0.83090.3698
7scaler_before_split0.83050.3703
8dropout_at_eval0.83440.3658

Findings

  1. A checklist of ten checks found all eight bugs and stayed silent on the correct script. Every planted bug raised at least one alarm; the correct script raised none. The eight bugs had eight different first alarms (shape, alignment, init loss, batch order, gradcheck, overfit-one-batch, scaler contract, eval determinism); the zero-input and reference-gap checks were generic symptoms that fired on several bugs without naming a cause.
  2. No single check is enough. The gradient check caught 1 of 8 bugs (the missing ReLU mask) and is blind to a consistent-but-wrong loss: broadcast_loss, sigmoid_twice, regulariser_scale and no_shuffle all have gradients that match their own loss. The loss-at-init check caught 1 of 8 (sigmoid_twice); the broadcast bug starts at exactly the right value (0.500) because predicting the base rate is its fixed point. Overfit-one-batch passed for label_misalignment (the net memorised 64 permuted labels) and for no_shuffle (a random 64-row subset was used).
  3. Metric-looking-plausible bugs cost real quality while "looking fine". missing_relu_mask and regulariser_scale lose 0.015-0.020 AUC and 0.032-0.034 log loss against the correct script, with smooth, believable curves; tuning would have been tempting. scaler_before_split moved neither metric (0.8345 vs 0.8344) at 14,000 training rows because 30% of rows barely shifts a mean; only the contract test sees it. dropout_at_eval costs 0.0045 log loss and makes every prediction non-repeatable.
  4. Bugs interact and the validation metric does not improve monotonically while fixing them. Fixing broadcast_loss first changed little (log loss 1.4064 to 1.3853; AUC 0.483, which is chance level at this sd), fixing sigmoid_twice pushed log loss up by 0.021 (1.3852 to 1.4064), and fixing no_shuffle moved the model to the base rate (0.5000), which is a better log loss but not a model that learned. Learning started only at step 5, when the missing ReLU mask was fixed: the missing mask and L2 5e-2 together collapsed the network to a constant, although each alone still learns (AUC 0.8146 and 0.8194). The log-loss path rose by more than 0.005 at one step (step 3), so the "fix path never gets worse" criterion is not met; progress must be judged by the checks' alarms going quiet, not by the validation metric. (In the all-bugs configuration the broadcast branch of the loss shadows sigmoid_twice; it becomes active only after step 1.)
  5. The fixed script is within 0.0064 log loss of the oracle and better than logistic regression. Validation log loss 0.3658 (sd 0.0008, 3 seeds) against oracle 0.3594 and logistic regression 0.3771; AUC 0.8344 against 0.8407 and 0.8271. The network is 0.0113 better than logistic regression (it can use the two interactions), so "not worse than logistic regression by more than 0.01" is met, but "within 0.01 in absolute value" is not met (|gap| = 0.0113).
  6. Test split, read once: AUC 0.8242, log loss 0.3758 (3 seeds), against oracle 0.8308 / 0.3701 and logistic regression 0.8118 / 0.3886 on the same test rows. The test split is harder than validation for every model (oracle AUC 0.8308 vs 0.8407), so the 0.010 AUC drop from validation is mostly the split, not the fix.
  7. Calibration by contract type is good. Seed-averaged mean prediction vs mean label on validation: monthly 0.275 vs 0.274 (generator 0.282), one-year 0.133 vs 0.135 (0.135), two-year 0.095 vs 0.083 (0.087); the two-year segment is over-predicted by 0.012 (626 rows, within sampling noise of a 8.3% rate, sd about 0.011).
  8. Robustness re-runs agree. Same split, training rows stored in a different file order: AUC 0.8339 / log loss 0.3668 (sd 0.0018 / 0.0003) against 0.8344 / 0.3658. float32: AUC 0.8344, log loss 0.3658 (differences below 1e-7).
  9. Business view (hypothetical economics: 120 value per correctly targeted churner, 4 cost per contact, 10% of customers contacted; validation split, 1,000-customer scale; ties in the constant-output model broken by row order). Fixed model reaches 73.0 churners and 8,360 value; all eight bugs together reach 18.3 and 1,800; logistic regression 69.0 and 7,880; the oracle 74.3 and 8,520. The values are hypothetical and only the ordering is informative.

Success criteria of the project record

CriterionStatus
All planted bugs found; per bug the exposing check, a failing test, the fix, before/after validation AUC and log lossMet for 8 planted bugs (tests in test_r15_*.py; before/after in the tables above). Two bugs have no measurable metric effect (scaler_before_split) or a very small one (dropout_at_eval).
Fixed net within 0.01 log loss of logistic regression; gap to oracle reportedPartly: not worse than LR by more than 0.01 (it is 0.0113 better); not within 0.01 in absolute value. Oracle gap 0.0064 reported.
Test suite under 60 s on CPUMet: 14 tests in about 13 s.
Not doneDigits 10-class transfer check; written bug-log postmortem timings (there is no human diagnosis time to measure).

What this does not prove

  • It does not show that this checklist order is optimal or complete; it is the order of these eight bugs in one script.
  • Planted bugs are independent switches; real bugs are not. Detection results for "one at a time" do not transfer to two small bugs with interacting symptoms (finding 4 shows an example).
  • One generator seed and one split; three run seeds. Differences of 0.001 are inside noise; the seed sd of the correct script (0.0008 log loss) is small relative to the bug effects but the 2-4 sd sizes of the mild bugs are not a precise estimate.
  • Check thresholds (init loss 0.03, overfit 0.05, batch order 2x, reference gap 0.01 and 0.05) were set by the builder before the run and are not calibrated for other models; the missing-mask overfit value (0.065 vs threshold 0.05) is close to the line.
  • The scaler-leak effect is unmeasurable here because a 14,000-row training set barely moves a mean; on a small data set or with a time-ordered split it would be larger. No claim is made about that.
  • Synthetic customers: no real-group fairness or business conclusion.

Reproduce

cd python && uv run python -m reference.r15_debug_silent_training_bugs.run
cd python && uv run pytest tests/test_r15_debug_silent_training_bugs.py

Builder-authored reference. It is not learner evidence and is never credited to your portfolio.

Rubric4 criteria

CriterionMeetsExceeds
Systematic diagnosisHypotheses written before code changes; cheap checks before expensive ones; each bug linked to the check that exposed it.Shows which check would have caught each bug earliest and reorders the team checklist accordingly.
TestsOne failing-then-passing test per bug; suite under 60 seconds.Tests generalise (shape contracts and evaluation-mode assertions usable in other projects) and run in CI.
Evidence of the fixValidation metrics before and after each fix with 3 seeds; final model near logistic regression and the oracle.Calibration by contract type against the generator and the digits transfer check.
PostmortemBug log with symptom, check, cause, fix and test.Draws a reusable team practice and estimates the business cost of the silent failure under stated assumptions.

References6