Core day 59 of 60 · Week 12 · Part II · evaluation depth

Senior Interview Day

You get one senior question and eight minutes: "A gradient-boosting model cuts forecast error from 7.4 to 7.1 seats per departure in a rolling-origin backtest, and the commercial director wants it live next week. What do you say?" How do you answer, take a challenge on your weakest claim, and revise without simply defending or simply folding?

By the end: You answer one timed senior question, respond to a Socratic follow-up that targets the weakest claim in your answer, produce a revised answer next to the first, and score both against a five-line rubric.

At a glance

Frame, structure, answer with units, then find your own weakest claim under challenge and revise it on evidence, keeping the first answer visible.

Frame first
At most three questions that change the answer: decision, cutoff and baseline, cost of error.
Recommendation first
Then evidence in decision units, the main risk and the next piece of evidence.
Read the spread
Median gain and the share from the best period, not only the mean.
Concede and narrow
Restate the challenge, test it, change only what the evidence moves.
Keep both answers
The revision is the evidence of judgment only if the first answer stays visible.

Recall · 5 min

Without notes, say aloud in one minute: the decision, grain, cutoff, baseline and metric unit of the forecasting framing card from Day 2, and what an empty seat costs in the course assumption. Then clear your due review cards.

Open due reviews

Concept · 10 min

An interview grades a process you can show in eight minutes: framing before answering, a structure the listener can follow, numbers with units and uncertainty, and what you do when challenged. The last separates levels most. A senior finds the weakest claim before the interviewer does, tests it and revises aloud, without defending or folding.

An error reduction is worth money only after volume and unit cost are applied, and only if it exceeds what the model costs to run

Predict first

One of six rolling-origin cutoffs supplies most of a model's total gain. What is the most important next question?

Text description of the figure

A decision cost calculator in which the error level, decision volume, the two unit costs and a stocking quantile are set. It shows the MAE, the rule-of-thumb value MAE x volume x mean unit cost, and the yearly cost of the median versus the cost-optimal quantile. It has no running-cost or per-cutoff input; those are in the code block.

The timed answer protocol Sequence

Clarify (about 90 s)
Decision and consumer, cutoff and baseline, metric unit and cost of error. No reply: state assumptions.
Structure (15 s)
Name the parts: what the number means, is it robust, what is it worth, recommendation.
Answer (up to 5 min)
Recommendation first, evidence with units, the main risk, the next evidence.
Take the challenge
Restate it, say what it tests, find the evidence, then concede and narrow or hold.
Revise (1 min)
Two or three sentences, and name the evidence you lacked.

The follow-up, computed: where does the gain come from? Illustration

Five of six cutoffs are positive, but cutoff 6 supplies 83% of the total.

Dropping any ordinary cutoff barely moves the mean. Dropping the best one takes it below the break-even gain (dashed, 0.075 seats at an assumed EUR 6,000 running cost per 1,000 departures).

all six cutoffs
without the best cutoff

Break-even running cost: EUR 24,000 with all cutoffs (blue), EUR 4,800 without the best (red).

The application's toy rolling-origin table (synthetic). EUR 80 per seat is the course assumption; running costs are hypothetical.

Per-cutoff gains, the mean with each cutoff left out, and the running cost at which the decision changes sign.

Taking a challenge without defending or folding Diagram

Restate itone sentence
name the claim
Say what it testswhich claim is thin
compute
Find the evidencecompute, do not argue
check
Evidence moves answer?
- yes: to Concede and narrow
no
Holdshow the evidence that answers it
Concede and narrowkeep what survives

Deleting the inconvenient cutoff as an outlier and deferring to "more data" are both ways of not answering.

What the same eight minutes look like at two levels Comparison

Criterion	Mid-level	Senior
First move	Explains gradient boosting	Asks what is decided, at which cutoff, against which baseline
Numbers	Mean error, no units, no spread	Seats and euros, per cutoff, net of running cost
Under challenge	Defends the first answer, or retracts it without evidence	Finds the weakest claim first, tests it, narrows
Ending	Yes or no	A scoped recommendation and the rule that would change it

Key terms: Socratic follow-up, Timed answer protocol, Baseline, Cost of error.
Five steps, about eleven minutes: clarify (at most three questions), structure (name the parts), answer (recommendation first), take the challenge (restate, test, concede or hold), revise (two or three sentences).
A follow-up is usually a question, not an objection: "which cutoff drives this?", "what would change your mind?", "what if errors cost different amounts?".
Score both answers on five lines: framing, evidence with units, uncertainty, decision with a trigger, quality of revision.
In the app a timed attempt is saved before model answers appear; the rubric self-assessment counts as interview evidence and is worth only your strictness. The interview gym under Practice is optional.

Assumptions

You answer unaided and against a timer; open-book attempts are labelled practice and do not count as evidence of this skill.
The interviewer is a partner, the app or a prepared card; simulated opinions are not those of real people.
The first answer is kept next to the revision, because a revision only shows learning if the original is visible.

Application and workplace decision · 10 min

Part A, timed (eight minutes, before running anything). "At D-14 the pickup baseline has an MAE of 7.4 seats per departure. A gradient-boosting model reaches 7.1 over six monthly rolling-origin cutoffs. The decision is closing fare class L at D-14; an empty seat costs about EUR 80 (course assumption). The director wants it live next week." Write or record your answer.
Part B. Run the code, which holds the follow-up information, and read the follow-up: "which cutoff drives the improvement, and does it change your recommendation?"
Part C. Write the revised answer in three sentences, keeping the first beside it.
Part D. Score both on the five lines. Optionally repeat with a question from your own project or the gym.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np

# Toy rolling-origin results (synthetic): MAE in seats per departure at six monthly cutoffs, D-14 snapshot.
base = np.array([7.2, 7.5, 7.3, 7.6, 7.4, 7.4])  # pickup baseline, mean 7.4
model = np.array([7.4, 7.3, 7.2, 7.5, 7.3, 5.9])  # gradient boosting, mean 7.1
EUR_PER_SEAT = 80  # assumed cost of one seat of absolute error (course assumption)
RUN_COST = 6000  # assumed model running cost per 1,000 departures (hypothetical)

gain = base - model  # seats of error removed per departure at each cutoff
rest = np.delete(gain, gain.argmax())  # the same gains without the best cutoff


def net(g):
    return g.mean() * EUR_PER_SEAT * 1000 - RUN_COST


print(f"mean gain {gain.mean():.2f} seats; wins {(gain > 0).sum()} of {len(gain)}; median {np.median(gain):.2f}")
print(f"best cutoff supplies {gain.max() / gain.sum():.0%} of the total gain")
print(f"net per 1,000 departures: all cutoffs EUR {net(gain):,.0f}; without best cutoff EUR {net(rest):,.0f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 35 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The method: from error costs to the forecast you act onadvanced6 minThe expected cost of acting on a forecast, the critical-ratio optimum derived from it, and why the pinball loss scores exactly that decision.

Every forecasting question hides a quantity committed before demand is known Diagram

Forecast distribution Fdescribe demand, not one number
quantile
Commit qstock, roster, buy energy, protect seats
demand
Demand D above q?
- yes: to Short by D − q
no
Over by q − Deach unit costs c_over: waste, idle staff
Short by D − qeach unit costs c_under: lost sale, overtime

Two errors, two unit costs. A store's stock, a call centre's roster, energy bought day-ahead and seats protected from a fare class are the same problem.

The decision behind a forecast: expected cost of acting on q Formula

C(q) = c_{over}\,\mathbb{E}\left[(q - D)^{+}\right] + c_{under}\,\mathbb{E}\left[(D - q)^{+}\right], \qquad C'(q) = c_{over}\,F(q) - c_{under}\,\bigl(1 - F(q)\bigr)

$q$: the quantity you commit to: units stocked, staff rostered, energy bought day-ahead, seats protected
$D$: the uncertain demand, with distribution F; the forecast's job is to describe F, not one number
$(q - D)^{+}$: units over: waste, idle staff, energy sold back below cost; each costs c_over
$(D - q)^{+}$: units short: lost sales, overtime, imbalance penalties; each costs c_under
$C'(q)$: one more unit is over with probability F(q) and saves a shortage with probability 1 - F(q)

Setting C'(q) = 0 gives F(q*) = c_under / (c_under + c_over); C''(q) = (c_under + c_over) f(q) >= 0, so it is a minimum. This is the single-period newsvendor problem.

The marginal argument, without calculus Sequence

Add one unit to q
Ask what the extra unit does on a random day.
Surplus with probability F(q)
Demand stays at or below q: the unit is wasted and costs c_over.
Saves a shortage with 1 - F(q)
Demand exceeds q: the unit is sold and saves c_under.
Stop where they balance
Keep adding while c_under (1 - F(q)) > c_over F(q), i.e. while F(q) < c_under / (c_under + c_over).

The result: the critical ratio picks the quantile Formula

q^{*} = \frac{c_{under}}{c_{under} + c_{over}}

$c_{under}$: cost of one unit of under-forecast
$c_{over}$: cost of one unit of over-forecast
$q^{*}$: the forecast quantile that minimises expected cost; 0.5 only when the costs are equal

MAE scores the median forecast, which is optimal only when c_under = c_over. Any other cost pair moves the decision to another quantile of the same forecast distribution (Day 26).

Why the pinball loss scores exactly this decision Formula

c_{under}(D - q)^{+} + c_{over}(q - D)^{+} = (c_{under} + c_{over})\,\rho_{\tau}(D - q), \qquad \rho_{\tau}(u) = u\,(\tau - \mathbf{1}[u < 0]), \quad \tau = \frac{c_{under}}{c_{under} + c_{over}}

$\rho_{\tau}$: pinball (quantile) loss: slope tau for under-forecasts, 1 - tau for over-forecasts
$u > 0$: short by u: (c_under + c_over) x tau x u = c_under x u
$u < 0$: over by -u: (c_under + c_over) x (1 - tau) x (-u) = c_over x (-u)

Expected decision cost is a constant times the expected pinball loss at tau, so ranking models by pinball loss at tau ranks them by money. MAE is the special case tau = 0.5 (Gneiting, 2011).

Three consequences an interviewer probes Overview

The median is a special case
Optimal only when both errors cost the same.
Ratio
Doubling both costs leaves the optimal quantile where it was.
Judge models at tau
On pinball loss at tau, or on simulated cost, never on MAE alone.

Worked by hand: ten days of demand4 minThe expected cost of every stock level computed from ten observed days, the marginal rule that stops at the optimum, and the cost of using the median instead.

Worked by hand: ten store-days of a perishable item (retail, hypothetical) Table

Stock q	F(q)	Expected units over	Expected units short	Expected cost (1 x over + 3 x short)	Cost change to q + 1
4	0.3	0.1	1.4	0.1 + 4.2 = 4.30	1 x 0.3 - 3 x 0.7 = -1.8
5	0.6	0.4	0.7	0.4 + 2.1 = 2.50	0.6 - 1.2 = -0.6: add one
6	0.8	1.0	0.3	1.0 + 0.9 = 1.90 (minimum)	0.8 - 0.6 = +0.2: stop
7	0.9	1.8	0.1	1.8 + 0.3 = 2.10	0.9 - 0.3 = +0.6
8	1.0	2.7	0.0	2.70	+1.0

Demand on the ten days: 3, 4, 4, 5, 5, 5, 6, 6, 7, 8. Waste costs 1 per unit, a lost sale 3, so tau = 3 / 4 = 0.75; the smallest q with F(q) >= 0.75 is 6, where the marginal cost turns positive. The median (5) costs 2.50. Recomputed in Python.

The ten days, drawn at the optimum and swept across stock levels Illustration

At a stock of 6: 10 units wasted and 3 sales lost over ten days, so the expected cost is 1 x 1.0 + 3 x 0.3 = 1.90 per day.

Cost falls while F(q) is below 0.75 and rises after: the minimum is 1.90 at q = 6. Stocking the median (5) costs 2.50, 32% more on the same data.

Hypothetical retail demand from the worked table: waste costs 1 per unit, a lost sale 3. In Python, np.quantile(d, 0.75, method='inverted_cdf') returns the same stock.

With costs 1 and 1 the answer would be the median: the forecast did not change, the decision did.

In Python: q = np.arange(3, 9); cost = [(np.maximum(k - d, 0) + 3 * np.maximum(d - k, 0)).mean() for k in q] reproduces the cost column.

Properties and where the answer breaksadvanced5 minWhat has to hold for the critical ratio to be right, a diagnostic for each assumption, and a computed case where the better MAE costs more.

Five assumptions under the critical ratio: how each breaks, the check and the fix Hierarchy

Critical ratio q*right only while five assumptions hold
- demand censoredQuantiles too lowcheck sold-out days; unconstrain demand
- costs not linearWrong optimumcheck cost per unit at volumes; simulate
- quantile miscalibratedCoverage below τcheck backtest coverage; recalibrate
- stock carries overNot one periodcarry-over: use an inventory policy
- costs are guessesFragile optimumrecompute q* over cost ranges; report it

Censoring is the commonest silent failure: a sold-out shelf or full queue records what was served, not wanted. Calibration matters most: an overconfident 0.75-quantile covers demand on far fewer than 75% of days; recalibrate or widen with conformal residuals (Day 26). Nonlinear costs: capacity, goodwill after stock-outs, step penalties.

The better MAE can be the worse decision Illustration

MAE: A 6.43, B 6.92 seats, so A wins at the median. Above q* = 0.725 B is cheaper: the ranking flips once errors cost different amounts.

Why: A's quantiles are overconfident. Its high quantiles cover fewer departures than they claim, so it under-stocks exactly when shortages are expensive.

Seeded synthetic demand (1,200 departures, true noise sd 8 seats). c_over = EUR 80 (course assumption); c_under is set by q*. Expected cost = (c_under + c_over) x pinball loss at q*.

Two synthetic forecasters scored at the cost-optimal quantile. The median ranking holds only when both errors cost the same.

A worked end-to-end mini-case5 minA retail ordering model taken from framing to a recommendation with a trigger, every number computed from an eight-cutoff table.

End-to-end mini-case: pinball loss at tau = 0.75 over eight weekly cutoffs (hypothetical) Table

Cutoff	Baseline pinball	Model pinball	Gain
1	2.31	2.09	0.22
2	2.28	2.02	0.26
3	2.35	2.17	0.18
4	2.26	2.12	0.14
5	2.30	1.99	0.31
6	2.33	2.15	0.18
7	2.27	2.07	0.20
8	2.30	1.95	0.35

A grocery chain's fresh-produce order model, units per store-day; waste 1 EUR, lost sale 3 EUR, so tau = 0.75. Mean gain 0.23, median 0.21, best cutoff 19% of the total, mean without it 0.213; eight wins of eight. Recomputed in Python.

The mini-case answered with the five-step protocol Sequence

Clarify
Decision: order quantity per store-day for 200 stores. Costs: 1 EUR waste, 3 EUR lost sale, so tau = 0.75 and MAE is the wrong yardstick.
Structure
What the number means, is it robust, what is it worth, recommendation.
Answer with units
Expected cost = 4 x pinball: 4 x 0.23 = 0.92 EUR per store-day, 67,160 EUR a year for 200 stores; 47,160 net of 20,000 running cost.
Robustness
Median 0.21; no cutoff supplies over 19%; net without the best one about 42,150. Check backtest coverage of the 0.75-quantile.
Recommend and trigger
Roll out region by region; reopen if coverage falls below 70% or the gain over the baseline drops under 0.10 for four weeks.

Same statistics, opposite recommendation: the seat case and the grocery case Comparison

Criterion	Seat case (lesson question)	Grocery mini-case
Cutoffs and metric	6 monthly cutoffs, MAE in seats	8 weekly cutoffs, pinball at tau = 0.75
Mean gain	0.30	0.23
Median gain	0.10	0.21
Wins	5 of 6	8 of 8
Share from the best cutoff	83%; net -EUR 1,200 without it	19%; net about EUR 42,150 without it
Sign-test chance of that many wins	7 / 64 = 0.11	1 / 256 = 0.004
Recommendation	No global switch; test whether the window recurs	Roll out region by region with a trigger

The paired t on the eight grocery gains is 9.1 (scipy ttest_rel). Six cutoffs could never reach the grocery case's level of evidence, whatever the data.

The rubric, line by line2 minWhat strong and weak look like on each of the five lines.

The five-line rubric, weak against strong Comparison

Criterion	Weak	Strong
Framing	Starts with the method	Decision, cutoff, baseline and cost of error fixed in the first 90 s
Evidence with units	Unitless error, one mean	Seats per departure and euros per 1,000 departures
Uncertainty	A single average as the result	Per-cutoff spread, median, share from the best period
Decision with a trigger	Ship or do not ship	Scope, test and the rule that would switch
Quality of revision	Holds unchanged or folds	Changes only what the evidence moved; names missing evidence

Clarifying in ninety seconds, inside a 45-minute slot3 minThree questions that change the answer and where the case sits in a longer interview.

Three clarifying questions that change the answer Overview

Decision and consumer
Who acts on the forecast, and what do they change? Here: closing fare class L at D-14.
Cutoff and baseline
At which snapshot, against which simple rule? Here: D-14, pickup baseline 7.4 seats.
Unit and cost of error
What is one seat of error worth, and are both directions equal?

If nobody answers, state your assumptions aloud and carry on.

A 45-minute slot, two pacings Illustration

A rough plan for one interview format, not a standard: interviewers allot time differently. The case block uses this lesson's protocol (clarify 1.5, structure 0.25, answer 5, challenge 3, revise 1 minutes).

Senior pacing: the recommendation is said at about minute 5.3 and the case closes at minute 13.75, inside one question's budget. Method first: the recommendation arrives at minute 14, and the challenge eats the time a revision needed.

The same case question inside a longer interview. Where the recommendation lands decides whether there is time to revise.

Reading where a gain comes from3 minMean, median, wins and the share from one period, before any recommendation.

Follow-up data: gain over the baseline per cutoff (seats, toy numbers) Chart

Cutoff 1-0.2 seats
Cutoff 20.2 seats
Cutoff 30.1 seats
Cutoff 40.1 seats
Cutoff 50.1 seats
Cutoff 683% of the total gain: what happened in this window?1.5 seats

Mean 0.30, median 0.10, five wins out of six. Wins counted per cutoff hide that four of them are tiny.

Reading a rolling-origin gain before recommending Table

Statistic	Value in the case	What it tells you
Mean gain	0.30 seats	The headline; sensitive to one window
Median gain	0.10 seats	The typical cutoff
Wins	5 of 6	Counts tiny wins as wins
Share from best cutoff	83%	Whether the gain is one event
Mean without best cutoff	0.06 seats	Below the running cost on the assumptions

Worth per 1,000 departures (assumed EUR 80 a seat, EUR 6,000 to run) Chart

Gross value, all cutoffs24,000 EUR
Running cost-6,000 EUR
Net, all cutoffs18,000 EUR
Net without the best cutoff0.06 seats x 80 x 1,000 - 6,000-1,200 EUR

Computed by the application code from toy gains. EUR 80 a seat is the course assumption; the EUR 6,000 running cost is hypothetical. The sign of the decision depends on one window.

First answer and revision, side by side Comparison

Criterion	First answer	Revised answer
Recommendation	Ship next week	No global switch yet
Evidence	0.3 seats = EUR 24,000 per 1,000 departures	Median 0.1; one cutoff gives 83%; net -1,200 without it
Next step	None	Shadow run across the next comparable window
Trigger	None	Switch if at least 0.2 seats outside that window

How little "five of six wins" provesadvanced2 minA sign test on six cutoffs, and why the per-cutoff table beats any single test statistic.

How much do "wins at five of six cutoffs" prove? Formula

P(W \ge 5 \mid n = 6,\ p = 0.5) = \frac{\binom{6}{5} + \binom{6}{6}}{2^{6}} = \frac{7}{64} \approx 0.11

$W$: number of cutoffs where the model beats the baseline
$p = 0.5$: no real difference: each cutoff is a coin flip
$0.11$: a tie would produce five or more wins about one time in nine

A sign test ignores the size of each win, which is its point here. Six cutoffs are too few to make any test decisive; the per-cutoff table is the evidence.

The follow-up bank3 minFive follow-ups that recur, what each tests, and a first sentence for each.

Five follow-ups that recur, what each tests and the evidence to bring Overview

Which cutoff drives this?
Tests robustness. Bring per-cutoff gains, the median and the share from the best cutoff.
What would change your mind?
Tests falsifiability. Bring a written threshold and the window it applies to.
What if errors cost different amounts?
Tests whether MAE is the decision metric. Bring both unit costs, the critical ratio, calibrated quantiles.
How would you know it works live?
Tests monitoring and rollback. Bring a shadow-run plan, a weekly metric and the rollback path.
What is the simplest thing that works?
Tests baseline discipline. Bring the baseline row on the same cutoffs, net of running cost.

Restate the follow-up, name what it tests, then bring the evidence. Practise each aloud.

Practising without an interviewer2 minHow to make a written or recorded attempt count.

Making a solo attempt count Sequence

Unaided and timed
Open-book attempts are practice, not evidence.
Record the first answer
Before reading any follow-up or model answer.
Get interrupted
A partner asks one question about your weakest number at minute three.
Score both
On the rubric, the same day, strictly.

Memory anchor

Frame, structure, answer with units, take the challenge, revise aloud - keep the first answer beside the second.

Useful for

Preparing for senior interviews, design reviews and promotion panels, where process is visible in minutes.
Practising the habit of locating your own answer's weakest claim before someone else does.
Turning any project result into a two-minute answer with units, uncertainty and a decision.

Not suitable for

Measuring technical depth across many topics; one timed question samples a process, not breadth.
Replacing live practice with a human who interrupts; a written or recorded attempt cannot reproduce pressure.

Traps

Answering the method instead of the decision

The answer explains boosting and cross-validation in detail and never says whether to ship, so the listener cannot tell whether you can make a call.

How to avoid it

Lead with the recommendation and the decision unit, then support it.

Folding under a challenge

The interviewer asks a question and you retract the recommendation without evidence, which reads as having no position at all.

How to avoid it

Concede only what the evidence shows, quantify it and keep the parts that survive.

Senior reasoning

A senior treats the first answer as a draft that will meet the strongest challenge, so they volunteer its weakest claim and say what would change it. They convert error to decision units, check how stable a gain is across periods, and offer a revision that preserves value (a shadow run, a restricted rollout) rather than a yes-or-no. They keep the first answer visible, because the revision is the evidence of judgment.

Interview question

"Your new model is better on average, but a colleague says the improvement is not real. How do you respond?" How would you structure the answer?

Outline your answer first.

Recall cards scheduled after completion

What are the five steps of the timed answer protocol?
A model's mean gain is positive and it wins at five of six cutoffs. What do you check before recommending it?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Quantiles as optimal point forecasts — Tilmann Gneiting, 2011optional
Quantiles are the optimal point forecasts under asymmetric piecewise linear loss, the result behind the critical-ratio chapter.
Out-of-sample tests of forecasting accuracy: an analysis and review — Leonard J. Tashman, 2000core
Reviews rolling-origin evaluation and why accuracy should be examined across origins and horizons, which is the follow-up the case turns on.
Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations — Evan Miller, 2024optional
Compare models with paired differences and report uncertainty rather than a single mean, supporting the per-cutoff view of the gain.
Rules of Machine Learning: Best Practices for ML Engineering — Martin Zinkevich (Google)optional
Practitioner guidance to start from a simple baseline and keep the first model simple, a prior the interview answer starts from. (Rules 1 and 4)

Optional sections

Interview practice · 15 min

A grocery chain's demand model improved MAE from 12.0 to 11.4 over the last three months. The head of commercial says it is great and wants a press release. The same evaluation shows the error rose in the two largest regions. What do you say?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-18 · depends on numpy

1 Recall · 5 min

2 Concept · 10 min

Assumptions

3 Application and workplace decision · 10 min

4 Mini-assessment · 8 min

Technical depth · optional · 35 min in total

Useful for

Not suitable for

Traps

Senior reasoning

Interview question

Recall cards scheduled after completion

Prerequisites and related concepts

References

Optional sections

Recall · 5 min

Concept · 10 min

Application and workplace decision · 10 min

Mini-assessment · 8 min