Core day 1 of 60 · Week 1 · Part I · application depth

Senior Data Scientist Mindset

A revenue manager asks you to "build an AI model that predicts no-shows". Before writing any code, how do you decide whether the answer is a rule, an analysis, or a model - and which baseline would prove the choice?

By the end: For three short business requests you choose rule, analysis or model, name the decision each one changes, and state the baseline a model would have to beat.

At a glance

Classify the request by the decision it serves, choose the cheapest method that changes it, and make any model beat a strong baseline.

Five questions
Describe, predict, estimate, what-if, decide: same table, different evidence.
Decision first
Name the decision, its timing and the cost of being wrong before any method.
Rule or analysis first
Known logic, few inputs or a why-question rarely need a model.
Baseline
The strongest simple method, computed only from data known at decision time.
Net value
Accuracy gain times volume times cost of error, minus the cost of running the model.

Recall · 5 min

Today's recall block is a short diagnostic (about five minutes). Answer honestly and rate your confidence: it sets your starting review plan, not a grade. Self-reported experience is not evidence; your answers are.

Open the diagnostic

Concept · 10 min

One table of records (orders, accounts, machines, flights) can answer five different questions: what happened, what will happen, how big is it, what if we act, and what should we do. Each needs a different method and fails in a different way. Seniority starts with noticing which one you were asked: the request says "predict" when the decision often needs a causal answer or a simple analysis.

The same flight table answers five different questions - and each needs different evidence

Predict first

"If we lower fares 5% next month, how many more seats will we sell?" - which question type is this?

Text description of the figure

A set of request cards (for example "why did IST-AYT revenue fall in March", "how many no-shows on ZX104 Friday", "what if we overbook two more seats") that you sort into describe/diagnose, predict, estimate, causal effect and decision. After sorting, each card reveals the method family, the validation it needs and its typical failure mode.

Five questions, five sectors, one data warehouse Overview

Describe and diagnose
Retail: why did returns of the new sneaker line jump 20% last month?
Predict
Banking: which overdue accounts will miss their next payment?
Estimate
Manufacturing: average repair time per machine stoppage, with an interval?
Causal effect
E-commerce: if we raise the free-shipping threshold, what happens to order size?
Decide
Logistics: how many couriers should we roster for Saturday?

Six questions before any method (here: a collections desk) Sequence

Which decision changes?
Who acts and how often: which overdue accounts get a call today. Owner: collections, daily.
What is known at decision time?
Balance, days overdue, past payments; not this afternoon's incoming transfer.
What does an error cost?
Often asymmetric: a missed defaulter costs far more than one needless call.
What is the simplest option?
Call everyone over 30 days overdue: a fixed rule, a lookup table, an analysis, or a model.
What must a model beat?
The strongest simple baseline, on unseen data, used the way it will be used.
What would change your mind?
Name the evidence before you look.

Rule, analysis, experiment or model? Diagram

About the past?
- no: to A what-if?
yes
Analysisdecompose and compare
A what-if?
- no: to Logic known?
yes
Experimentor causal assumptions
Logic known?
- no: to Repeated at scale?
yes
Ruleauditable and cheap
Repeated at scale?
- yes: to Model vs baseline
no
Lookup tabletoo few outcomes to learn
Model vs baselineonly if the gain pays

The verb in the request ("predict", "explain", "optimize") is a weak signal. Classify by the decision and by what would count as evidence.

When each answer wins Comparison

Criterion	Rule	Analysis	Model
Wins when	Logic known and stable, few inputs, audit matters, data scarce	The question is what happened or why	Many interacting signals, a repeated decision at scale, outcomes observed
Evidence it needs	Agreement on the logic; a check that it is applied	A decomposition with clean data and a fair comparison	Beats the baseline on a time-ordered split, net of running cost
Typical failure	Goes stale silently when the world changes	Reads a correlation as a cause	A gain that exists only in a leaky or random evaluation

A causal question ("what if we raise the free-shipping threshold?", "what if we overbook more?") needs an experiment or causal assumptions; no predictive model answers it alone (Day 5).

The baseline ladder: each rung adds structure, the floor is noise Illustration

Synthetic: 364 days x 3 routes, Poisson no-shows (route means 6 / 9 / 4, Fri-Sun x1.3), seeded. Fit on Jan-Sep, scored on Oct-Dec. Same process as the Python block, different random draws, so the numbers differ from the Python run.

Each rung adds one piece of structure. The lookup is within 3% of the noise floor: that sliver is all any model could still win on this data.

The gap between the best simple baseline and the noise floor is the most any model can win. Measure it before building anything.

Work decision-first: decision, decision time, cost of error, simplest option, Baseline, evidence that would change your mind.
Key terms: Prediction, Inference (statistical), Estimation, causal effect and prescriptive analytics.
Details are in the pictures above and in the optional technical depth below.

Assumptions

The decision and its cost of error can be stated; if nobody can say what changes, no method is justified yet.
Past data are representative of the period the decision applies to; regime changes (a new competitor, a pandemic-like shock) break baselines and models alike.
The outcome is observable afterwards, so error can be measured; without feedback a model cannot be evaluated or monitored.

Application and workplace decision · 10 min

Three requests arrive in one morning. For each, write (a) the decision it changes, (b) rule, analysis or model, (c) the baseline or comparison you would show.

Revenue management: "Flag flights likely to have more than 10 no-shows so we can overbook them."
Commercial director: "Why did IST-AYT revenue fall 8% in March versus last March?"
Payments: "Block bookings paid with cards that are on our confirmed-fraud list."

Then run the code:

A year of synthetic no-shows on three routes, split at a cutoff (the past fits, the future scores).
It compares a global average with an auditable route-by-weekday lookup.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
import pandas as pd

rng = np.random.default_rng(1)
days = pd.date_range("2024-01-01", periods=364, freq="D")
base = {"IST-ESB": 6.0, "IST-AYT": 9.0, "IST-LHR": 4.0}  # mean no-shows per flight (synthetic)
df = pd.DataFrame([(d, r) for d in days for r in base], columns=["date", "route"])
df["weekday"] = df["date"].dt.weekday
lam = df["route"].map(base) * np.where(df["weekday"] >= 4, 1.3, 1.0)  # Fri-Sun higher
df["no_shows"] = rng.poisson(lam)

cutoff = pd.Timestamp("2024-10-01")  # fit on the past, evaluate on the future
train, test = df[df["date"] < cutoff], df[df["date"] >= cutoff].copy()

global_mean = train["no_shows"].mean()
lookup = train.groupby(["route", "weekday"])["no_shows"].mean().rename("pred")
test = test.join(lookup, on=["route", "weekday"])

mae_global = (test["no_shows"] - global_mean).abs().mean()
mae_lookup = (test["no_shows"] - test["pred"]).abs().mean()
cost_per_seat_error = 50  # EUR; assumed blended cost of a denied boarding or an empty seat
flights_per_year = len(base) * 365
gain = (mae_global - mae_lookup) * cost_per_seat_error * flights_per_year
print(f"global-mean baseline MAE: {mae_global:.2f} no-shows/flight")
print(f"route x weekday lookup MAE: {mae_lookup:.2f} no-shows/flight")
print(f"lookup vs global: EUR {gain:,.0f} per year (linear-cost assumption)")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 27 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

Decision analysis: expected value and the action threshold5 minThe expected-value rule derived from two costs, worked by hand on an inspect-or-skip decision, and why it turns models into threshold machines.

The expected value of an action, and the threshold it implies Formula

\begin{aligned} \mathbb{E}[V(a)] &= \textstyle\sum_{o} P(o)\,V(o, a) \\ \text{act} &\iff p\,L_{\text{miss}} > c_{\text{act}} \iff p > c_{\text{act}} / L_{\text{miss}} \end{aligned}

$V(o, a)$: value of outcome o if you take action a; the sum weights it by how likely o is
$p$: probability the bad event happens if you do nothing (a defect, a churn, a missed payment)
$L_{\text{miss}}$: loss when you skip the action and the event occurs
$c_{\text{act}}$: cost of acting (an inspection, a retention call, a manual review)
$c_{\text{act}} / L_{\text{miss}}$: the break-even probability: with a EUR 4 inspection against a EUR 60 defect, 4/60 = 0.067

Act when -c_act > -p x L_miss, i.e. p > c_act / L_miss: every repeated act-or-skip decision reduces to a probability threshold set by the two costs, never by a model's default 0.5. The same rule prices a retention call (telecom), a claim review (insurance) or a follow-up slot (healthcare); only the two costs change.

Worked by hand: inspect a unit (EUR 4) or risk a EUR 60 defect Table

Defect probability p	EV(skip) = -p x 60	EV(inspect) = -4	Better action
0.02	-1.20	-4.00	skip
0.0667 (= 4/60)	-4.00	-4.00	indifferent: the threshold
0.10	-6.00	-4.00	inspect
0.50	-30.00	-4.00	inspect

Hypothetical manufacturing line, verified by computation. The switch happens at p = 4/60, so the line needs a calibrated p per unit from a model, not a yes/no. Expected value assumes the bet repeats many times; when one loss is catastrophic, risk attitude matters too (Day 16 adds asymmetric costs, Day 26 quantiles).

The act-or-skip threshold moves fast when the loss estimate is off Chart

Inspection cost fixed at EUR 4. If the defect really costs EUR 40, not 60, the threshold moves from 0.067 to 0.10 and every unit with p between the two flips from inspect to skip. Before trusting a threshold, check how wrong a cost estimate must be to change the decision.

Pricing an accuracy gain and the break-even volume5 minThe net-value formula, the break-even algebra solved for volume with a worked e-commerce example, and how uncertainty in the gain moves the call.

The net value of switching to a model Formula

V = \Delta \text{MAE} \times c \times N - C_{\text{run}}

$\Delta \text{MAE}$: error reduction against the baseline on a time-ordered evaluation
$c$: cost of one unit of error (assumed linear here; asymmetric from Day 16)
$N$: number of decisions per year
$C_{\text{run}}$: yearly cost to build, serve, monitor and maintain the model

The gain scales with volume; the running cost does not, so below some volume a better model loses money. Linear cost is a simplification: with asymmetric costs the cost-optimal decision becomes a quantile (Days 16 and 26).

Break-even volume: set the net value to zero and solve Formula

V = \Delta e \times c \times N - C_{\text{run}} = 0 \;\Rightarrow\; N^{*} = \frac{C_{\text{run}}}{\Delta e \times c}

$\Delta e \times c$: gain per decision: error reduction times the cost of one unit of error
$N^{*}$: decisions per year at which the model exactly pays for itself
$N > N^{*}$: every decision above break-even is profit; below it the model loses money

The same algebra answers "how much accuracy do we need" (fix N, solve for delta-e) and "how cheap must it run" (fix the gain, solve for C).

Worked example: an e-commerce returns model (hypothetical numbers) Sequence

Gain per decision
The model prevents 0.04 mishandled returns per order; each costs EUR 15: gain = 0.04 x 15 = EUR 0.60 per order.
Yearly gain
V(N) = 0.60 x N - 90,000, with EUR 90,000 per year to build, serve and monitor.
Solve V = 0
N* = 90,000 / 0.60 = 150,000 orders per year.
Translate to volume
150,000 / 365 is about 411 orders per day: below that daily volume the model loses money.
Stress the inputs
Halve the per-return cost to EUR 7.50 and N* doubles: the decision often hinges on the cost estimate.

Redo it on paper, then in Python. The arithmetic is trivial; writing it before building is what a senior adds.

When a better model pays, and how sure you can be Illustration

One knob: the accuracy gain. Value grows with volume; the running cost does not, so every line starts below zero.

Break-even volume is inversely proportional to the gain: halve the gain and you need twice the flights.

Where it breaks: the gain is itself an estimate. With 0.30 +/- 0.10, at the break-even volume (20.5/day) the switch is a coin flip; it takes far more volume to be confident.

Computed from V = gain x EUR 40 x 365 x flights/day - EUR 90,000 (assumed costs, linear in error). Right panel assumes the measured gain is normal with standard error 0.10 across time-ordered windows.

Near break-even the point estimate of the gain does not decide; its uncertainty across several time-ordered windows does (Day 25).

Baselines, headroom and the noise floor4 minThe rungs of a baseline ladder, the floor no model can pass, and the library defaults that hide a weak baseline.

Headroom decides whether a model is worth pricing at all Diagram

Best baseline errorroute x weekday lookup, time-ordered
- to Headroom
Noise floortrue median per case; else long-history lookup
subtract
Headroombaseline error minus floor
Could it pay?vs running cost
- no: to Report the finding
yes
Price a modelnet value and break-even volume
Report the findinga model is the wrong investment

With Poisson no-shows the floor is set by the counts themselves, not by data volume. When the floor is unknown, estimate it: the error of the best lookup on a long history, or the spread of outcomes for identical cases. Small headroom is a result, delivered before anyone builds a model.

Why the floor uses the median Formula

\begin{aligned} \arg\min_{c}\; \mathbb{E}\,\lvert Y - c \rvert &= \operatorname{median}(Y) \\ \arg\min_{c}\; \mathbb{E}\,(Y - c)^{2} &= \mathbb{E}[Y] \end{aligned}

$Y$: no-shows on one flight, a random count even when its rate is known exactly
$c$: a single forecast for that flight
$\lvert Y - c \rvert$: absolute error: the best constant is the median, so a fair floor uses the true median
$(Y - c)^{2}$: squared error: the best constant is the mean (Day 16 builds on this)

Even a perfect model keeps the Poisson noise of each flight. The metric decides which summary of that noise is the target.

Library defaults that shape a baseline comparison (scikit-learn 1.9) Table

Tool	Default	What it means for you
DummyRegressor	strategy='mean'	a global constant: a strawman rung, not the bar a model must beat
DummyRegressor	'median' or 'quantile' available	match the strategy to the metric: median for MAE
DummyClassifier	strategy='prior'	predicts class shares; accuracy then equals the majority share
KFold	shuffle=False	contiguous blocks, yet later days still train models scored on earlier ones
TimeSeriesSplit	n_splits=5, gap=0	set gap to the decision horizon so training stops before the cutoff

Checked against the installed scikit-learn 1.9.1. A strong baseline is usually one you write yourself, like the route x weekday lookup.

Analytics families, and explain against predict4 minHow descriptive, diagnostic, predictive and prescriptive work stack, and why inference and prediction choose different variables and validation.

Four analytics families, each building on the one below Stack

PrescriptiveWhat should we do? A prediction plus a decision rule and costs: a reorder quantity per store.
PredictiveWhat will happen? Tomorrow's grid load by hour, for the energy trading desk.
DiagnosticWhy did it happen? A decomposition of last month's telecom churn spike.
DescriptiveWhat happened? Insurance claims by region and policy type last quarter.

Where each method fits Map

Rule or checklist

Write it down and audit it.

Automated rule

Encode it; monitor that inputs still match.

Analysis and judgment

Too few cases to learn from; decompose and decide.

Model against a baseline

Many outcomes to learn from; prove the gain.

Explaining and predicting are different goals Comparison

Criterion	Explain (inference)	Predict
Main question	How does the outcome depend on each input?	What will the outcome be for new cases?
Judged by	Fit, assumptions, interpretable parameters	Error on data the model has not seen
Variables chosen for	Their causal or theoretical role	Their predictive value at decision time
Failure when mixed	Good parameters, poor forecasts	Good forecasts read as causes

After Breiman (2001) and Shmueli (2010). Neither is better; mixing their goals is the mistake: a model can predict well with confounded coefficients, and a good causal model can predict poorly (Day 5 shows both). Say which goal a request serves before choosing variables or a validation design.

Why the split decides the gain4 minHow a random split inflates a model's advantage over a lookup, shown on 30 simulated years.

A random split rewards interpolation; deployment needs extrapolation Illustration

Random CV (top strip) scatters test days between training days, so the local model can read the level on both sides. The time-ordered split (bottom) must forecast Oct-Dec from September.

Across 30 simulated years the local model looks 8% better than the lookup under random CV (median) and 6% worse on the future quarter. The gap is created by the evaluation, not the model.

Synthetic: daily Poisson no-shows (mean 6, Fri-Sun x1.3) times a wandering AR(1) level; 30 seeded years. Local model = weekday lookup + mean residual of the 6 nearest training days.

Random split or time-ordered split? Comparison

Criterion	Random cross-validation	Time-ordered split
What the model sees	Days on both sides of each test day	Only days before the cutoff
What the score measures	Interpolation between known days	Forecasting the future, as in use
In the decision case	7% better than the lookup	1% better than the lookup

Under random folds, test days sit between training days, so any feature that tracks the recent level reads the answer's neighbours. The gap between the two scores is the part of the gain that exists only in the evaluation design (Days 15 and 25).

Score the same model twice: the gap is the evaluation's share Diagram

Score it twicerandom CV and time-ordered folds
Large gap?random minus ordered
- yes: to Gain is interpolation
no
Gain survivesprice it net of running cost
Gain is interpolationdeployment never offers it
re-evaluate
Time-ordered foldsgap = decision horizon, several windows

The diagnosis costs one extra evaluation run. Report both numbers side by side; only the time-ordered one predicts what deployment will see.

Where first projects fail2 minFour failure modes to check before anything is built.

Four ways a first project goes wrong Overview

Wrong question
Solving the stated request instead of the decision behind it.
Weak baseline
Beating zero or a global mean proves nothing.
Leaky baseline
A baseline computed on the evaluation period looks unbeatable.
Gross value
Pricing an accuracy gain without the cost of running the model.

What a staff-level answer addsadvanced3 minDecision cadence, headroom, feedback loops, kill criteria and ownership, plus how each role reads the same request.

What a staff-level answer adds Overview

Decision cadence
Start from who acts, how often and at what grain; accuracy questions come after.
Headroom first
Measure lookup minus noise floor before estimating any model's gain.
Feedback loops
Acting on predictions changes the data the next model learns from.
Kill criteria
Agree in advance what result retires the model or the baseline.
Ownership
Name who answers when the number is wrong at 06:00 on a Friday.

Acting on a model rewrites the data the next model learns from Loop

Model scores casesdefault risk, expected no-showssets ↓
Policy actscredit limits, overbooking levelsshapes ↓
Outcomes observeddefaults at chosen limits onlylogged as ↓
Next training setold policy's outcomes unrecordedreviewed against ↓
Kill criteriaagreed before launchkeep and retrain ↓
↺ back to 1 · Model scores cases

Once credit limits follow a model, the defaults you observe are those of the limits the model chose; once overbooking follows a model, so do denied boardings. Next year's history can no longer show the old policy's outcomes, which is why the kill criteria must be agreed before launch.

Four lenses on one request (a telecom churn spike) Overview

Data analyst
Is the churn spike real, in which plans and regions, and are the numbers clean?
Data scientist
Which decision, baseline, metric and validation design?
ML engineer
How do scores reach the retention desk on time, and what if the pipeline fails?
Senior
All three, plus: what does a wrong answer cost, and who is accountable?

In interviews, the decision you chose not to automate, with its measured headroom, is a stronger story than the model you built.

Memory anchor

Decision first, baseline second, model last - and price the gain net of its cost.

Useful for

Triage of incoming requests - deciding what to build before anyone builds it.
Writing a one-page problem brief that names the decision, decision time, cost of error and baseline.
Explaining to stakeholders why a rule or an analysis is the better first deliverable.

Not suitable for

Settling a causal question ("what if we change price?") - classification of the question tells you that you need an experiment or causal assumptions, but does not supply them.
Replacing domain knowledge - the decision-first chain only works if someone can state the decision and its costs.

Traps

Solving the stated request instead of the decision

"Predict readmissions" gets a predictive model, but the real decision is which discharged patients get one of 30 weekly follow-up calls, with asymmetric costs. A good forecast with the wrong decision rule still wastes the slots.

How to avoid it

Write down the decision, who makes it and the cost of each error type before choosing a method.

A weak or leaky baseline

Comparing against "zero" or against a baseline computed on the evaluation period makes any model look good or makes the baseline look unbeatable.

How to avoid it

Use the strongest simple method, computed only from data available at decision time.

Senior reasoning

A senior reframes before solving. They ask what decision changes, what is known at that moment, and what an error costs, then propose the cheapest method that changes the decision, with a named baseline and a stated piece of evidence that would change their mind. They treat "we should use AI" as a hypothesis to test, not a requirement, and they price accuracy gains net of running cost and against the uncertainty of the gain itself. The chain is the same in every sector - credit approvals, claim triage, maintenance scheduling, retention offers - only the costs change.

Interview question

"Tell me about a time you decided NOT to use machine learning." How would you structure the answer?

Outline your answer first.

Recall cards scheduled after completion

Name the five kinds of question one dataset can answer, with an airline example of each.
When does a rule or an analysis beat a model, and what must a model beat to be adopted?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Rules of Machine Learning: Best Practices for ML Engineering — Martin Zinkevich (Google)core
Rule 1 (launching without ML and starting from heuristics) and Rule 4 (keep the first model simple) support the rules-before-ML stance. (Rules 1 and 4)
Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author) — Leo Breiman, 2001optional
The contrast between data-modeling (inference) and algorithmic-modeling (prediction) cultures behind today's question types. (Sections 1-3)
To Explain or to Predict? — Galit Shmueli, 2010optional
Explaining vs predicting as different goals with different modeling and validation choices.

Optional sections

Deep dive · 10 min

Analytics families. Descriptive (what happened), diagnostic (why), predictive (what will happen), prescriptive (what should we do). Prescriptive work combines a prediction with a decision rule and costs - for example, an reorder quantity chosen from a demand forecast and the cost of a stockout versus a unit left on the shelf.

Two cultures. Statistics traditionally builds a model of how the data were generated and reads its parameters (inference); much of machine learning treats the generating process as unknown and judges models by predictive accuracy on new data (Breiman, 2001). Shmueli (2010) shows that "explain" and "predict" lead to different variable choices, validation and success criteria - a model can predict well with confounded coefficients, and a good causal model can predict poorly. Neither culture is "better"; mixing their goals is the mistake.

Role lenses on the same request. A data analyst checks the numbers behind the request (is the churn spike real, which plans and regions, data quality). A data scientist frames the decision, baseline, metric and validation. An ML engineer asks how predictions reach the decision at the right time, how they are monitored and what happens when the pipeline fails. A senior holds all three and adds: what is the cost of being wrong, and who is accountable for it?

Interview practice · 10 min

A stakeholder insists on "an AI model" for prioritizing collections calls at a bank. You believe a days-overdue-and-balance rule is enough. How do you handle it?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-11 · depends on pandas, numpy

1 Recall · 5 min

2 Concept · 10 min

Assumptions

3 Application and workplace decision · 10 min

4 Mini-assessment · 8 min

Technical depth · optional · 27 min in total

Useful for

Not suitable for

Traps

Senior reasoning

Interview question

Recall cards scheduled after completion

Prerequisites and related concepts

References

Optional sections

Recall · 5 min

Concept · 10 min

Application and workplace decision · 10 min

Mini-assessment · 8 min