Skip to content
ML Academy

project-028 · Deep learning · aviation

Will this flight fill? A from-scratch numpy MLP vs gradient boosting on airline booking snapshots

Twenty-one days before departure, which flight-days will finish at or above 90% load factor, so that the revenue manager can close the lowest fare bucket earlier on them? And does a small neural network add anything over a pickup rule, logistic regression and gradient boosting for this decision, on the same split and the same metric?

Dataset
Synthetic airline sample database (flight_day joined to the 21-days-out booking_curve snapshot)
Licence
academy-generated synthetic data; no third-party licence
Reference run
executed

At a glance

  • Data

    Synthetic airline sample database (flight_day joined to the 21-days-out booking_curve snapshot) (synthetic) · 20,824 flight-days (19 flights, 11 routes, 2022-01-01 to 2024-12-31); 270,712 booking-curve snapshots; about 19 MB SQLite

    Details on data

  • Baseline

    Three references, all on the same split and metric. (1) Prevalence: predict the training base rate for every flight (the log-loss floor any model must beat).

    Details on baseline

  • Validation

    Time-based split with a label-availability embargo: Validation flights depart 2023-07-01 to 2023-12-31.

    Details on validation

  • Metric

    Primary: log loss (probabilities feed a cost-based threshold). Secondary: Brier score, PR-AUC (positives are about a quarter of flights), ROC-AUC, reliability curve.

    Details on metric

Success criterion

The project succeeds when you can defend one recommendation with evidence, whichever model wins:

  • The selected model beats the pickup-rule baseline on validation log loss and PR-AUC.
  • Its predicted probabilities are usable for a threshold decision (reliability curve and Brier score reported).
  • You recommend the MLP over gradient boosting only if its mean validation log loss over 5 seeds is lower by more than 2 seed standard deviations and it is not worse on the 2024 shifted-route slice. Otherwise you recommend gradient boosting (or logistic regression) and say why.

A write-up that correctly shows the MLP losing or tying meets the bar. Success is a sound comparison, not a neural win.

Context

This is the Week 7 neural lab (Day 35) and a candidate for the Day 56 neural project. You train one small network and three classical references on the same time-based split, then prove the network is correct (shape assertions, a numerical gradient check, overfitting a tiny batch) before you trust its score. Expect gradient boosting to be hard to beat: in a 45-dataset benchmark, tree-based models remained the strongest family on medium-sized tabular data of around 10K samples (Grinsztajn et al., 2022). This table has about 20.8K flight-days and a dozen usable features, so that finding applies roughly, not exactly. The data are synthetic: they come from the academy's airline simulator, so any conclusion is about this generator, not about real airlines. The reference implementation r06 covers the core comparison; your learner work adds the seeds, the cost analysis, error slices and the written decision.

Data

Dataset
Synthetic airline sample database (flight_day joined to the 21-days-out booking_curve snapshot) (synthetic)
Licence
academy-generated synthetic data; no third-party licence
Size
20,824 flight-days (19 flights, 11 routes, 2022-01-01 to 2024-12-31); 270,712 booking-curve snapshots; about 19 MB SQLite
Verification
executed

Access

Built locally by python/capstone/airline_sim.py (build_sample_db, seed 42) into python/artifacts/airline_sample.sqlite. No network and no authentication. If the file is missing, rebuild it with the simulator. Synthetic data: generated by the academy simulator, not real airline records.

Grain

Modelling grain: one row = one flight number on one departure date, with the cumulative bookings observed 21 days before departure joined from booking_curve (13 snapshots per flight-day: 90, 60, 45, 30, 21, 14, 10, 7, 5, 3, 2, 1, 0 days out). Label: load_factor >= 0.90 at departure.

ColumnMeaning
date, flight_noDeparture date (ISO text) and flight number; together the primary key of flight_day.
route, market, capacityOrigin-destination pair, market in {domestic, intl-short, intl-long}, seats on the aircraft (160, 180 or 280).
booking_curve.cumulative_bookings at days_to_departure = 21 (and 30, 45 for pickup)Bookings on hand at each snapshot; the main predictor available at the decision time.
passengers, load_factorOutcome. load_factor = passengers / capacity; passengers are capped by capacity, so the label says 'finished full', not 'demand exceeded 90%'.
fare_level, avg_fare, revenue, bookings_final, cancellationsOutcome or outcome-derived (fare_level is a per-route tercile of realised price). Excluded as features.
weekday, holiday, seasonality, eventCalendar fields; treated as known in advance (assumption: the event calendar is published before D-21).
competitor_capacity, competitor_priceCompetitor schedule (assumed known in advance) and the departure-day competitor price (not known at D-21; excluded).

EDA and data-quality rules

  • Grain check: (date, flight_no) is unique in flight_day, and every flight-day has exactly one 21-day snapshot. Count and report missing snapshots instead of silently inner-joining them away.

  • Feature-availability audit before any model: a table with every column, the time it becomes known, and keep/drop. fare_level, avg_fare, revenue, passengers, bookings_final, cancellations and competitor_price are dropped because they are only known at or after departure.

  • Label sanity: positive rate by market and by year (about 27% overall in the seed-42 build); confirm that load_factor never exceeds 1 in the clean table and that the 280-seat flights are not mislabelled.

  • Plot cumulative bookings at D-21 divided by capacity against final load factor, by market. This relationship is the pickup baseline; if a model cannot beat it, the extra features add nothing.

  • Use flight_day, not flight_day_raw. If you choose the raw table for practice, deduplicate and repair it first and report how many rows each rule changed.

Baseline

Three references, all on the same split and metric. (1) Prevalence: predict the training base rate for every flight (the log-loss floor any model must beat). (2) Pickup rule: a one-feature logistic fit on cumulative_bookings_d21 / capacity, which is what an analyst would do in a spreadsheet. (3) L2-regularised logistic regression on the full allowed feature set. The neural network has to beat (2) to be useful and (3) to be worth its complexity.

Features

One shared feature matrix for every model so that the comparison isolates the learner: D-21 load ratio (bookings / capacity), pickup from D-45 to D-21 and from D-30 to D-21, capacity, one-hot route, market, weekday and month, holiday and event flags, seasonality, competitor capacity. Numeric columns are standardised with statistics fitted on the training period only (logistic regression and MLP); the gradient-boosting model gets the unscaled matrix. No target encoding, no features computed across the split boundary.

Candidate methods

  • From-scratch numpy MLP (2 hidden layers, 32 and 16 ReLU units, sigmoid output, binary cross-entropy)

    The learning artefact. Writing the forward and backward pass yourself forces the shape and gradient checks of Days 31-33. He initialisation, mini-batch SGD or Adam, L2 penalty and early stopping on validation log loss. Expected to be competitive with logistic regression, not necessarily with boosting.

  • HistGradientBoostingClassifier (or LightGBM) with early stopping

    The strongest default for medium-sized tabular data with mixed feature types and threshold-like effects (for example, a holiday or a nearly full cabin). Few hyperparameters; the benchmark to beat.

  • L2 logistic regression on the full feature set

    A calibrated, explainable linear reference. If the MLP cannot beat it, its hidden layers are not finding useful interactions in this table.

Validation design

Time-based split with a label-availability embargo:

  • Validation flights depart 2023-07-01 to 2023-12-31. A forecast for a 2023-07-01 departure is made on 2023-06-10, so the training set may only contain departures up to 2023-06-09 (21-day embargo; later labels were not yet known).
  • Test flights depart in 2024 and are scored once at the end, with a model refitted on departures up to 2023-12-10.
  • Hyperparameters, early-stopping epochs and the decision threshold are chosen on validation only.
  • The MLP runs with 5 seeds (initialisation and batch order); gradient boosting with 5 seeds if you enable subsampling.
  • For the test comparison, use a paired bootstrap over departure weeks (blocks) for the log-loss difference between the two best models.

Leakage checks

Feature-availability audit (above); 21-day label embargo between training and validation; scaler and any imputation fitted on the training period only; fare_level is derived from realised prices and must never enter; the test year is read once. Add a unit test that fails if any excluded column appears in the feature matrix.

Bias and representativeness

No personal data, so the fairness question here is segment performance: compare calibration and recall by market and by aircraft size (the 280-seat flights fill at a different rate). Two structural biases to name: passengers are capped by capacity (censored demand), so the model learns 'fills', not 'would have exceeded'; and the training data were produced under the existing fare policy, so a model that closes fares will change the bookings it later learns from.

Robustness checks

The simulator steps competitor capacity up 25% on IST-ADB, IST-AYT and IST-FRA from 2024-03-01: report test metrics on those routes before and after the break. Perturb D-21 bookings by plus or minus 5% to mimic late booking corrections and report the change in flagged flights. Report seed-to-seed spread for every neural result.

Technical metrics

Primary: log loss (probabilities feed a cost-based threshold). Secondary: Brier score, PR-AUC (positives are about a quarter of flights), ROC-AUC, reliability curve. Neural results as mean ± sd over 5 seeds; the test-year difference between the two best models with a block-bootstrap 95% interval.

Business interpretation

A hypothetical cost model, labelled as such: closing the discount bucket on a flight that would not have filled spills discount passengers (cost per wrong close), not closing on a flight that fills dilutes revenue (cost per missed close). Choose the threshold that minimises expected cost on validation for dilution-to-spill ratios of 1, 2 and 4, and report flagged flights per week and expected cost against the pickup rule. The ratio is an assumption to vary, not a fact about airline economics.

Error slices

  • Market (domestic, intl-short, intl-long) and aircraft size: calibration and recall per slice.

  • IST-ADB, IST-AYT, IST-FRA before and after 2024-03-01 (competitor capacity break).

  • Borderline flights with final load factor between 0.85 and 0.95, where the label is most fragile.

  • Holiday and event departures, which are rare in training and drive the largest booking surges.

Implementation steps

  • Studio 0-15 min: build the feature table with one SQL join of flight_day to the 21-day snapshot (plus 30 and 45 for pickup), write the availability audit, create the embargoed time split.

  • Studio 15-35 min: fit prevalence, pickup rule and logistic regression; record validation log loss and PR-AUC in an experiment log (one row per run: model, seed, features, metric).

  • Studio 35-60 min: numpy MLP. Write the shape table by hand, then assert every intermediate shape in the forward pass; check the loss at initialisation (about 0.693 with a zero output bias, about 0.58 when the bias is set to the logit of a 27% base rate); run a float64 gradient check on 16 rows; overfit 32 rows to near-zero loss.

  • Studio 60-80 min: train the MLP with early stopping and the gradient-boosting model on the same split; plot epoch learning curves (train and validation loss) and a training-size curve (10, 25, 50, 100% of the training period) for all three learners.

  • Studio 80-90 min: one decision-log entry: current best model, evidence, what would change your mind.

  • Homework: 5 seeds for the MLP; one learning-rate or L2 experiment (3 values, 3 seeds each); cost-based threshold and sensitivity; error slices; read the test year once; a one-page report with the recommendation.

Reproduction

Reference implementation r06 (status: implemented; recorded results in python/reference/r06_numpy_mlp_vs_baseline/RESULTS.md):

cd python && uv run python -m reference.r06_numpy_mlp_vs_baseline.run

Behaviour: rebuilds artifacts/airline_sample.sqlite with seed 42 if missing; numpy, pandas and scikit-learn only; no downloads; about 80 seconds on a laptop CPU for the recorded full run (5 seeds). Writes to python/reference/r06_numpy_mlp_vs_baseline/outputs/: metrics.json (per model and seed), checks.json, gradcheck.json (gradient-check errors), learning_curves.json and learning_curves.png, and test_slices.csv. Unit tests cover the shape assertions, the gradient check and a short fit. Not covered by r06 (left to your learner version): the embargo-ignored comparison, recall by segment and the +/-5% bookings perturbation. Your learner version must also run with one command, pinned dependencies and a printed seed.

Limitations

  • Synthetic data: the MLP-vs-boosting result describes the academy simulator, whose functional form you did not choose; do not generalise it to real airlines.

  • The label is censored at capacity, so the model predicts 'finishes full', not unconstrained demand; a regression on final bookings would keep more information.

  • A single decision time (D-21). Real revenue management re-forecasts at every snapshot, and errors at different horizons have different costs.

  • Excluding the departure-day competitor price removes a real signal; an as-of competitor price feed would be needed to use it legitimately.

Production and monitoring

Score each night for departures reaching D-21; fall back to the pickup rule if the snapshot is missing. Labels mature 21 days after scoring, so monitor input drift (distribution of the D-21 load ratio by market) daily and calibration by market monthly. Run in shadow mode next to the analysts' current rule before any fare action, and log overrides: once the model closes fares, future bookings are policy-dependent and naive retraining learns the model's own effect. Version the feature list with the availability audit as a contract test.

Interview questions

  • Your MLP beats gradient boosting by 0.004 validation log loss. What evidence do you need before you ship it, and what would you report if the gap is inside the seed noise?

  • Give two reasons a neural network can underperform gradient boosting on a table like this, and one change you would try before concluding.

  • Walk me through why fare_level must be excluded even though it is in the same table and improves validation AUC.

  • How does acting on this model change the data you will train the next version on?

Postmortem and alternatives

Expected pattern to examine, not a result: the pickup ratio carries most of the signal, gradient boosting captures the holiday and capacity thresholds, and the MLP lands at or behind logistic regression. If so, the honest finding is that hidden layers do not pay for themselves here. Alternatives worth one paragraph: monotonic constraints on the booking ratio in the boosting model; predicting final bookings and thresholding later; treating capacity-capped flights as censored observations. Record which assumption (feature availability, the cost ratio, or the 2024 break) most changed the decision.

Reference implementation

Status: executed — outputs below are from the recorded run

cd python && uv run python -m reference.r06_numpy_mlp_vs_baseline.run

r06 results - numpy MLP vs logistic regression and gradient boosting

Recorded run of cd python && uv run python -m reference.r06_numpy_mlp_vs_baseline.run (full mode, 5 seeds). Every number below is copied from outputs/metrics.json, outputs/checks.json, outputs/gradcheck.json and outputs/test_slices.csv, written by that run. The data are synthetic (academy airline simulator, seed 42).

ItemValue
Python / numpy / scikit-learn3.12.13 / 2.5.3 / 1.9.1
Runtime of the recorded run79.0 s wall, 69.6 s CPU (single thread; the laptop was shared with other jobs, so wall time is inflated)
Rows20,824 flight-days; 43 features at D-21; 0 missing D-21 snapshots
Splittrain 9,975 (departures to 2023-06-09, positive rate 22.0%); validation 3,496 (2023-07-01 to 2023-12-31, 32.6%); refit for test 13,471 (to 2023-12-10, 25.4%); test 6,954 (2024, 31.0%)

Findings

  1. The network is correct. Every intermediate shape is asserted; the parameter count is 1,953, equal to the formula (43x32+32 + 32x16+16 + 16+1). Loss at initialisation is 0.686 against ln 2 = 0.693 with a zero output bias, and 0.5286 against the base-rate entropy 0.5275 when the output bias is set to the logit of the training base rate. 32 rows were overfit to a loss of 0.001 (accuracy 1.0) in 2,260 full-batch Adam steps. The float64 centred- difference gradient check over all 1,953 coordinates passes: the largest absolute error is 2.0e-11 and the largest relative error 1.3e-6 (parameter group W2); no coordinate fails the 1e-8 + 1e-5 x |numeric| criterion. A second unit test shows the check does flag a deliberately mis-scaled first-layer gradient.
  2. Every learner beats the pickup rule by a wide margin, and the pickup rule alone is already far better than the base rate. Validation log loss: prevalence 0.661, pickup rule 0.239, logistic regression 0.0857, gradient boosting 0.0887 +/- 0.0004 (5 seeds), MLP 0.1096 +/- 0.0007 (5 seeds). PR-AUC: 0.326, 0.881, 0.989, 0.988, 0.986.
  3. The MLP loses. It is worse than gradient boosting by 0.021 validation log loss (about 37 pooled seed sd, so this is not seed noise) and worse than logistic regression by 0.024. On the 2024 test year (read once) log loss is 0.0881 for logistic regression, 0.0946 +/- 0.0003 for boosting and 0.1109 +/- 0.0025 for the MLP. The paired block-bootstrap interval (53 departure weeks, 1,000 resamples) for MLP minus boosting is +0.0162 [0.0122, 0.0200], and for MLP minus logistic regression +0.0228 [0.0191, 0.0263]; both exclude zero.
  4. The pre-written decision rule recommends against the MLP, and in favour of the simplest model. The MLP is not better than boosting by 2 sd and is also worse on the shifted-route slice (test log loss 0.081 vs 0.069 on IST-ADB, IST-AYT and IST-FRA from 2024-03-01). The lowest validation log loss belongs to L2 logistic regression, which is also the lowest on the test year. Recommendation: logistic regression, with gradient boosting as the second choice.
  5. Training-size curve. Validation log loss at the most recent 10 / 25 / 50 / 100% of the training window (998 / 2,494 / 4,988 / 9,975 rows): logistic regression 0.2155 / 0.0893 / 0.0870 / 0.0857; boosting 0.1212 / 0.0968 / 0.0893 / 0.0887; MLP 0.1221 / 0.1097 / 0.1104 / 0.1097. The MLP is flat from 25% of the data and never closes the gap, so more data alone would not have rescued it here. At 10% of the data the unregularised logistic regression (C fixed at its full-data value of 10,000, not re-tuned per size) overfits and boosting is better.
  6. Weak regularisation of the linear model was best. The validation log loss of logistic regression is flat for C >= 10 (0.0859 to 0.0857) and clearly worse for C <= 1 (0.0889 at C = 1, 0.1879 at C = 0.01). The chosen C = 10,000 is the largest value in the grid, but the curve is flat there, so the grid edge does not matter.
  7. The MLP's probabilities are pulled toward the middle (under-confident at both ends). On the test year its top bin (predicted 0.9 to 1.0, mean 0.962) has an observed rate of 0.995 and its 0.8 to 0.9 bin (0.858) observed 0.930, while its 0.1 to 0.2 bin (0.145) observed 0.070; boosting is closer at the top (0.982 vs 0.983 and 0.850 vs 0.826). Brier score on test: 0.0269 logistic regression, 0.0285 boosting, 0.0322 MLP.
  8. Slices. The test log-loss ordering logistic regression < boosting < MLP holds in every market, every aircraft size, before and after the 2024-03-01 competitor-capacity step, and on borderline flights (final load factor 0.85 to 0.95: 0.355 / 0.376 / 0.390). The MLP is last in every slice. The only change in the first two places is on the 266 holiday departures (boosting 0.0665 vs logistic regression 0.0672, a tie in practice) and the 114 event days (boosting 0.091 vs 0.096), both small slices. The pickup rule is bad on the 732 long-haul flights of the 280-seat aircraft (test log loss 1.04): a single booking-ratio curve does not transfer across aircraft sizes.
  9. Business reading (hypothetical cost ratios; not airline economics). With thresholds chosen on validation, the expected test cost per flight-day for dilution-to-spill ratios 1 / 2 / 4 is 0.0378 / 0.0498 / 0.0640 for logistic regression, 0.0383 / 0.0533 / 0.0684 for boosting, 0.0429 / 0.0591 / 0.0734 for the MLP and 0.0735 / 0.0892 / 0.1087 for the pickup rule, at about 41 to 47 flagged flights per week for the learned models and 46 to 53 for the pickup rule. The learned models cost roughly one third to one half less than the pickup rule at every ratio (logistic regression 41 to 49% less, the MLP 32 to 42% less).

What this does not prove

  • It does not show that neural networks are worse than boosting or linear models for booking data. It shows that this small MLP, with these fixed settings and a small tuning effort, lost on this synthetic generator. The generator was not designed by the learner and may be close to linear in the features; the run does not test that explanation.
  • The tuning effort was unequal (logistic regression: 7-value C grid; MLP: 4-value L2 grid, early stopping, fixed learning rate, batch and architecture; boosting: fixed settings plus iteration count). A tuned MLP (other widths, schedules, weight averaging, a linear skip path) might close part of the gap; no such search was run.
  • It does not show anything about real airlines, about other decision horizons than D-21, or about unconstrained demand (the label is "finished at or above 90%", and passengers are capped by capacity).
  • The positive rate moves from 22% in the training window to 33% in validation and 31% in the test year, so thresholds fitted on validation need not be optimal on test; the cost figures in finding 9 carry that shift.
  • Seeds vary initialisation, batch order and boosting feature subsampling only; split and data are fixed. The sd over 5 seeds is a rough estimate (the sd of an sd at n = 5 is large). Differences between boosting and logistic regression (0.006 on test) are small and were not given their own bootstrap interval.
  • The cost analysis uses invented cost ratios; the flagged-per-week numbers are counts on the 2024 synthetic schedule.
  • The record's embargo-ignored comparison, error-slice recall by segment and perturbation test (D-21 bookings +/- 5%) were not run; the 2024 break routes are covered only through log loss by slice.

Reproduce

cd python && uv run python -m reference.r06_numpy_mlp_vs_baseline.run
cd python && uv run pytest tests/test_r06_numpy_mlp_vs_baseline.py

Builder-authored reference. It is not learner evidence and is never credited to your portfolio.

Rubric4 criteria

CriterionMeetsExceeds
Fair comparisonAll models use the same embargoed split, features and metric; neural results are mean ± sd over seeds; the test year is read once.Adds a block-bootstrap interval on the test difference and a training-size learning curve that explains which model is more data-efficient.
Correctness evidence for the networkShape assertions in the forward pass, a float64 gradient check with reported relative errors, loss at init matches the expected value, a tiny batch is overfit.Gradient check covers every parameter group and reports coordinates near ReLU kinks; the checks run as automated tests.
Leakage and availabilityAvailability audit table; outcome-derived columns excluded with reasons; scaler fitted on training data only.Explains the 21-day label embargo and shows the metric change when it is ignored.
Decision and communicationRecommends one model with the cost-based threshold, states the hypothetical cost ratio and the synthetic-data limitation.Shows how the recommendation changes across cost ratios and on the 2024 shifted routes, and names the trigger for revisiting it.

References7