Core day 60 of 60 · Week 12 · Part II · evaluation depth

Final Assessment and Capstone

You have reached the end of the core curriculum. Without claiming seniority from completion, what can you defend with evidence across framing, data quality, baselines, modelling, evaluation, production, risk and business value, and which of those would a hiring committee find missing?

By the end: You answer a mixed cumulative probe, defend one of your three projects end to end with every claim tied to an artefact or marked not shown, read your scorecard for sufficiency, recency and source, and name the gaps your 30/60/90-day plan must close.

At a glance

Defend one project claim by claim with artefacts, read the scorecard for sufficiency, recency and source, and turn every gap into dated, assessed evidence.

Evidence, not attendance
First attempts, scenario stages, interview rubrics and artefacts count; completion never does.
Ten-element defense
Decision to revisit trigger, each claim tied to a named artefact or marked not shown.
Sufficiency
At least three distinct items and total weight 2, or the dimension says insufficient evidence.
Recency and source
Weight halves every 45 days; repeats count a quarter; reference work counts nothing.
Gaps become a plan
Each 30/60/90-day period closes one reported gap with named, assessed evidence.

Recall · 5 min

Closed notes, one line each: (1) The Part I chain from grain to decision. (2) What your neural comparison proved and did not prove. (3) Recall@5 versus MRR@10 and why the held-out part stays closed. (4) The elements of a decision record. Then clear your due review cards; the probe below is separate.

Open due reviews

Concept · 10 min

Day 60 measures evidence, not attendance. Opening sixty lessons proves you opened them. What can be defended is what you produced and what was assessed. A hiring committee reasons the same way: which claims have an artefact, which have an assessed answer, which have neither. No completion count and no score certifies seniority.

Every project defense starts by matching problem shape to a baseline and a model family

Predict first

A project's report shows a strong result but no baseline row. What does the defense have to add first?

Text description of the figure

A matrix of problem shapes against baseline, classical, machine-learning and deep options. Selecting a cell shows which option fits that problem and why, which is the framing step a defense must justify before any result.

The final assessment in four parts Sequence

The probe
Twelve cumulative items mixing concepts across Parts I and II; first attempts only.
The defense
One project, ten elements, every claim tied to an artefact or marked not shown.
The scorecard
Read each dimension for sufficiency, recency and source on /progress.
Committee and plan
Scenario 027 and the Week 12 Sunday studio; a 30/60/90-day plan from the gaps.

Base weight by source of evidence (published scoring policy) Chart

First attempt: core or probe1
Scenario stage, interview rubric1
Teach-back0.75
Recall review0.5
Repeat of a seen item0.25
Reference implementationnever your evidence0
Completion or streaknever an input0

Published policy version 2026-10-11.1 (docs/SCORING_POLICY.md; teach-back weight from the scoring engine). Self-rated items are worth only the strictness you applied.

An estimate can appear, vanish and return without a single new mistake Illustration

Synthetic ledger of seven attempts on one dimension. Weights, minimums, the estimate and its approximate standard error come from the scoring engine itself (policy version in docs/SCORING_POLICY.md).

Shaded: days on which the dimension shows an estimate (at least 3 distinct items and weight 2). It appears on day 16, disappears on day 43 with no new answer at all, and returns only after new first-attempt items.

One dimension of a synthetic learner, scored every day by the academy's own scoring engine.

How one dimension, then one role, gets a number Diagram

3+ distinct items?
- no: to Insufficient evidence
yes
Total weight at least 2?
- yes: to Estimate
no
Insufficient evidencenever 0
Estimateweighted mean x 100
per role
Half of role covered?
- yes: to Role report shown
no
Report withheldinsufficient evidence
Role report shown

Insufficient evidence is never shown as 0 and is left out of averages.

Toy ledger: total recency-weighted evidence per dimension (minimum 2) Chart

validation-metrics4 items: estimate 792.38
rag-retrieval2 recent items: insufficient1.8
forecasting2 items: insufficient1.32
system-design3 items, 140-150 days old: insufficient0.31

Output of the application code. Role coverage is 1.0 / 3.9 = 0.26 of the illustrative role weights, so the role report is withheld.

The probe: twelve cumulative items, each mixing two or more concepts. First attempts are evidence; repeats are practice.
The defense: one project, ten elements, each claim tied to an artefact or marked not shown.
The scorecard on /progress, read for sufficiency (at least three distinct items and weight 2, otherwise Insufficient evidence), recency (Evidence weight halves every 45 days) and source (first attempts 1, reviews 0.5, repeats 0.25, reference implementations and AI-generated text 0).
A role report appears only when at least half of the role's weighted dimensions have estimates.
The committee and the plan: scenario 027 reviews your own evidence; the 30/60/90-day plan ties each period to a reported gap and the evidence that closes it.

Assumptions

Evidence counts only if it is yours, recorded and a first attempt; viewing solutions, opening lessons and receiving emails are not evidence.
Self-rated items (interview rubric, teach-back, written scenario stages) are as strict as you were; the report cannot correct that.
A role report summarises this programme's recorded attempts. It is not a qualification, and a gap in it may mean missing evidence rather than missing skill.

Application and workplace decision · 10 min

Final defense prompt. Choose one project and defend it in ten minutes, written or recorded: "Show me that this solution is sound, from the decision it supports to the evidence that would change it."
Use the ten elements, name an artefact for each claim, and end with the one claim you are least able to support.
Then run the code: it scores a toy ledger with the published policy so you can see how sufficiency, recency and source change an estimate.
Open your own /progress page and write one sentence per dimension: the estimate or "insufficient", the reason, the next piece of evidence.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

# Toy ledger of a synthetic learner, scored with the academy's published policy (weights, 45-day half-life, minimums).
KIND_W = {"core": 1.0, "probe": 1.0, "review": 0.5, "practice": 0.25}
ledger = [  # (dimension, item, kind, first attempt, score 0-1, age in days)
    ("validation-metrics", "q-015-2", "core", True, 1.0, 48), ("validation-metrics", "q-016-2", "core", True, 0.0, 44),
    ("validation-metrics", "q-020-p3", "probe", True, 1.0, 38), ("validation-metrics", "q-025-1", "core", True, 1.0, 28),
    ("validation-metrics", "q-016-2", "core", False, 1.0, 20),  # repeat of a missed item: counts as practice
    ("forecasting", "q-023-1", "core", True, 1.0, 32), ("forecasting", "q-026-3", "core", True, 0.0, 22),
    ("rag-retrieval", "q-051-2", "core", True, 1.0, 6), ("rag-retrieval", "q-050-p4", "probe", True, 1.0, 8),
    ("system-design", "q-019-1", "core", True, 1.0, 150), ("system-design", "q-020-p5", "probe", True, 1.0, 150),
    ("system-design", "q-052-1", "core", True, 1.0, 140),
]
estimate = {}
for dim in sorted({r[0] for r in ledger}):
    rs = [r for r in ledger if r[0] == dim]
    w = [KIND_W[k if first or k == "review" else "practice"] * 0.5 ** (age / 45) for _, _, k, first, _, age in rs]
    items, total = len({r[1] for r in rs}), sum(w)
    ok = items >= 3 and total >= 2  # policy minimums: 3 distinct items and weight 2
    estimate[dim] = round(100 * sum(x * r[4] for x, r in zip(w, rs)) / total) if ok else None
    shown = f"estimate {estimate[dim]}" if ok else "insufficient evidence"
    print(f"{dim}: {shown} ({items} items, weight {total:.2f})")

role = {"validation-metrics": 1.0, "forecasting": 0.7, "system-design": 1.0, "rag-retrieval": 0.4, "mlops-llmops": 0.8}  # illustrative
coverage = sum(w for d, w in role.items() if estimate.get(d) is not None) / sum(role.values())
print(f"role coverage {coverage:.2f}: report {'shown' if coverage >= 0.5 else 'withheld (insufficient evidence)'}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 32 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The method: an estimate is a weighted mean with a standard erroradvanced6 minThe estimator behind every number on /progress, its standard error derived from the variance of a weighted sum, and the effective sample size.

The estimate and its effective sample size Formula

\hat p = \frac{\sum_i w_i s_i}{\sum_i w_i}, \qquad w_i = b_i \cdot 0.5^{a_i/45}, \qquad n_{\text{eff}} = \frac{(\sum_i w_i)^2}{\sum_i w_i^2}

$s_i$: score of item i, from 0 to 1
$b_i$: base weight of its source (1; 0.75 teach-back; 0.5 review; 0.25 repeat)
$a_i$: age in days
$n_{\text{eff}}$: effective sample size; below the item count when weights differ. SE = sqrt(p(1-p)/n_eff)

The estimand p is your probability of answering an unseen item of this dimension correctly on a first attempt, now. Each item is one noisy observation of it; older and weaker sources are trusted less. Three items at weight 1 give n_eff = 3, and one wrong answer moves the estimate by about a third.

Deriving the standard error of a weighted mean Formula

\operatorname{Var}(\hat p) = \frac{\sum_i w_i^{2}\,\operatorname{Var}(s_i)}{\left(\sum_i w_i\right)^{2}} = \frac{p(1-p)}{n_{\text{eff}}}, \qquad n_{\text{eff}} = \frac{\left(\sum_i w_i\right)^{2}}{\sum_i w_i^{2}}

$\operatorname{Var}(s_i)$: a right-or-wrong item with success rate p has variance p(1 - p); items are treated as independent
$w_i^{2}$: weights are constants, so they come out of the variance squared: Var(w s) = w^2 Var(s)
$n_{\text{eff}}$: equals n when all weights are equal and shrinks as they spread; it never exceeds the item count
$p(1-p)$: estimated by the estimate itself; the engine floors it at 0.01 so a perfect record still shows some uncertainty

Two lines of algebra from Var(aX + bY) = a^2 Var(X) + b^2 Var(Y) for independent X and Y. It assumes a stable skill over the weighted period, which is exactly what recency weighting is there to approximate.

Down-weighting old evidence is a choice to know less, more recently Chart

Four items, all answered todayweights 1, 1, 1, 14 n_eff
Four items aged 0, 30, 60, 90 daysweights 1, 0.63, 0.40, 0.25: 2.28² / 1.623.2 n_eff

Effective sample size n_eff = (sum of weights)² / sum of squared weights, computed with the policy's 45-day half-life.

How precise is an estimate? (computed with the policy formula) Table

Evidence	Total weight	Estimate	n_eff	Approx. SE
3 first attempts, 2 right, all recent	3.00	67	3.0	27 points
6 first attempts, 5 right, all recent	6.00	83	6.0	15 points
12 first attempts, 9 right, all recent	12.00	75	12.0	12.5 points
5 first attempts aged 0, 0, 45, 45, 90 days; one of the newest wrong	3.25	69	4.1	23 points

SE = sqrt(p(1 - p) / n_eff), with p(1 - p) floored at 0.01. Read two dimensions as different only when their bands barely overlap: 74 plus or minus 23 and 61 plus or minus 20 are the same evidence.

The same estimator outside this app Overview

E-commerce
A recency-weighted product rating: recent reviews count more than old ones.
Manufacturing
An EWMA control chart of a production line's defect rate.
Collections
A team's recent cure rate, weighted toward the latest weeks.

In each, report the number with its effective sample size, not with the raw count of records.

Exponential half-life, by hand5 minThe decay written three ways, a four-item ledger pushed through the policy formula by hand, and the day an estimate disappears with no new mistake.

An exponential half-life, three ways to read it Formula

w(a) = b \cdot 2^{-a/h} = b\,e^{-\lambda a}, \qquad \lambda = \frac{\ln 2}{h}, \qquad a_{x} = h \log_{2}\frac{1}{x}, \qquad W(t + \Delta) = W(t)\,2^{-\Delta/h}

$h$: half-life: 45 days in the policy; every 45 days of age halve an item's weight
$\lambda$: the same decay as a rate: ln 2 / 45 = 0.0154 per day
$a_x$: age at which weight falls to a share x: a quarter after 90 days, a tenth after 45 x log2(10) = 149.5 days
$W(t + \Delta)$: with no new items, the total weight of a dimension decays by the same factor, so it can drop below the minimum of 2

By hand: 2^(-15/45) = 2^(-1/3) = 0.794; 2^(-30/45) = 0.630; 2^(-90/45) = 0.25. Under a steady flow of items, the mean age of the weight is about h / ln 2 = 65 days.

Recency: weight = base x 0.5^(age / 45 days) Chart

Old evidence fades, so a dimension can move from an estimate back to insufficient evidence without any new mistake.

Worked by hand: four items in one dimension (synthetic) Table

Item	Source (base weight)	Age	Weight b x 0.5^(a/45)	Score	Weight x score
P	core, first attempt (1)	0 days	1.000	1	1.000
Q	probe, first attempt (1)	15 days	0.794	0	0
R	core, first attempt (1)	45 days	0.500	1	0.500
S	teach-back (0.75)	90 days	0.75 x 0.25 = 0.188	1	0.188
Total	4 distinct items	—	2.481 (minimum 2: met)	—	1.688

Estimate 1.688 / 2.481 = 0.680, so 68. n_eff = 2.481^2 / (1 + 0.630 + 0.25 + 0.035) = 3.2, SE = sqrt(0.68 x 0.32 / 3.2) = 0.26: 68 plus or minus 26 points. Recomputed in Python with the engine's formula.

Four items reach a total weight of 2.48, just past the minimum of 2 Bridge

P: core, answered today1
Q: probe, 15 days old0.5^(15/45)+0.794
R: core, 45 days oldone half-life+0.5
S: teach-back, 90 days old0.75 x 0.25+0.188
Total weight (minimum 2)2.482

The worked ledger as a bridge. Each item adds base weight x 0.5^(age/45); the oldest, weakest source adds least.

The worked estimate expires after 14 days with no new item Chart

Total weight
Minimum weight 2

On day 15 the dimension shows insufficient evidence, although nothing was answered wrongly. One item also swings a record this small: had Q been answered correctly, the estimate would be 2.481 / 2.481 = 100 instead of 68, which is what SE = 26 points means.

In Python: w = np.array([1, 1, 1, .75]) * 0.5 ** (np.array([0, 15, 45, 90]) / 45), then (w * s).sum() / w.sum() and w.sum()**2 / (w**2).sum().

Properties: the half-life trades responsiveness for precisionadvanced4 minSteady-state weight, effective sample size and evidence age for three half-lives, and a simulation of a learner whose skill changes.

The half-life as a bias-variance knob (one item every 3 days, steady state) Table

Half-life	Total weight	n_eff	SE at p = 0.70	Mean age of the evidence
15 days	7.7	14.5	12 points	20 days
45 days (policy)	22.1	43.3	7 points	63 days
90 days	43.8	86.6	5 points	128 days

With r = 2^(-3/h): total weight 1 / (1 - r), n_eff = (1 + r) / (1 - r), mean age 3r / (1 - r), from geometric series. A shorter half-life reacts sooner to real change and pays in precision; a longer one is precise about the past. Computed in Python.

Three half-lives, one learner whose skill jumps on day 120 Illustration

Reaches the midpoint 24 days after the jump; average band ±12 points.

Reaches the midpoint 36 days after the jump; average band ±8 points.

Reaches the midpoint 84 days after the jump; average band ±7 points.

Seeded synthetic learner: one first-attempt item every 3 days, true success rate 0.55 until day 120 and 0.85 after. Estimate and ±1 SE band from the policy formula with three half-lives; minimum-evidence rules are left out so the curves stay continuous.

The estimate and its one-SE band under the policy formula with three half-lives. Short half-lives follow the jump and wobble; long ones are smooth and late.

Three ways the weighted estimate misleads, and the check for each Hierarchy

Estimate p̂assumes independent, comparable items
- items not independentOverstated n_effvariants of one item: one idea
- difficulty differsBiased p̂easy runs inflate it: read item difficulty
- self-rated itemsMeasures strictnessask an outside reader to rescore one

Diagnostics in order of cost - count distinct concepts behind the items, read the item difficulties, have one rubric rescored by someone else.

The defense, element by element4 minTen elements and the artefact each claim needs.

The ten-element defense, with the artefact each claim needs Table

Element	Artefact that supports it
Decision and consumer	Framing card (Day 2)
Data and provenance	Data contract, corpus manifest, quality report
Baseline	Baseline row on the same split and metric
Model or change	Run-log rows, ablation table
Evaluation and uncertainty	Frozen split, seed spread, interval, slice table
Production	Serving note, monitoring plan, rollback path
Risk	Leakage check, injection tests, privacy note
Value net of cost	Decision-unit calculation with labelled assumptions
One diagnosed failure, one limitation	Incident or error-analysis note
What would change the decision	Decision record with revisit triggers

Any row without an artefact is said aloud as "not shown". End with the claim you are least able to support.

Role lensesadvanced3 minHow one record turns into four role reports, and the engine rules behind coverage, score and the missing list.

One record, four role lenses Illustration

dimension with an estimate (width = its role weight)
insufficient evidence or none

Synthetic learner (first-attempt core items only). Role weights, minimums and the 0.5 coverage rule are the academy's published policy, computed by its scoring engine.

One record: 16 dimensions have some evidence, 7 have enough for an estimate. The role lens decides whether that is half of what the role weighs.

Each bar is a role's total weight from the published role weights. The same estimates cover half of one role and not of another.

How a role report is assembled (scoring engine rules) Table

Rule	Value in the engine	What it means when reading
Coverage needed	0.5 of the role's total weight	Below it the role shows insufficient evidence, never a score
Role score	Weighted mean over covered dimensions only	At 0.55 coverage it says nothing about the other 45%
Missing list	Uncovered dimensions with role weight 0.7 or more	The first places a committee will probe
Strengths, concerns	Estimate 75 or above; below 60	Read with the SE: 74 and 76 are the same evidence

Values read from src/lib/engine/evidence.ts (policy 2026-10-11.1).

Reading your scorecard like an auditor3 minFive tempting readings of your own record and what an auditor says instead.

Five tempting readings of your own scorecard, and the auditor's reading Comparison

Criterion	Tempting reading	Auditor's reading
An estimate of 79	So I am strong here	On an effective sample of four: a wide band, and only first attempts count
A burst of repeats	Will lift the dimension	Repeats weigh 0.25 and add no distinct item; new items do
A high role score	So the role fits	Check coverage first: the score ignores every uncovered dimension
Self-rated rubrics	Are strong evidence	Worth the strictness applied; ask an outside reader to rescore one
No system-design estimate	So I skip the topic	Name the artefacts the ledger cannot see and plan assessed items

Facing the committee3 minDemonstrated, plausible and unknown claims, and a strong answer when a committee calls a dimension thin.

Three kinds of claim in front of a committee Overview

Demonstrated
An artefact or an assessed first-attempt answer exists. Say it plainly.
Plausible
Adjacent evidence suggests it. Say so, and say what would show it.
Unknown
Nothing yet. Volunteer it with a dated plan.

"Your system-design evidence is thin." Two answers Comparison

Criterion	Weak	Strong
Opening	My forecasting work shows I can design systems	Agrees: the design items are 140-150 days old
Evidence offered	Experience no artefact shows	The Day 58 ADR and assumption register, and what they omit
The gap	Argues it away	Names what the artefacts do not cover
Commitment	None	A scenario stage and a revised ADR, dated, in the 30-day plan

Turning gaps into a plan2 minWhat goes into each period of the 30/60/90-day plan.

A 30/60/90-day plan built from the gaps Sequence

Days 1-30
Close the highest-weight insufficient dimension with first-attempt items and one scenario stage.
Days 31-60
One portfolio improvement tied to a not-shown rung, such as a seed table or a generation-mode review.
Days 61-90
A transfer case on new data; re-read the scorecard and update the decision log.
Every period
One practice goal, one portfolio improvement, the evidence that would show each is done.

Defending without the app2 minWhat the scorecard cannot see, and a one-page evidence index any reviewer can check in seconds.

Memory anchor

Claims need artefacts; scores need three recent first-attempt items; completion proves nothing.

Useful for

Reviewing your own record before an interview, a promotion discussion or a portfolio export.
Turning gaps in a scorecard into a dated plan with the evidence that closes each gap.
Preparing the Week 12 Sunday hiring-committee simulation and the transfer case.

Not suitable for

Certifying seniority, hiring readiness or job performance; the report summarises recorded attempts in this programme only.
Replacing human review; self-rated items and one author's artefacts need an outside reader to be fully credible.

Traps

Reading completion as competence

Sixty finished days become "senior-level", and the first probing question about a missing baseline or a reused held-out set shows the gap.

How to avoid it

Lead with named artefacts and assessed answers, and state the gap you know about.

Reading insufficient evidence as failure, or as a licence to claim

The learner either panics because the report shows no score, or asserts the skill anyway because the report cannot contradict it.

How to avoid it

Treat it as a to-do for assessed, recent items and cite artefacts only for what they show.

Senior reasoning

A senior reads their own scorecard like an auditor. They separate three claims: demonstrated (an artefact or assessed answer exists), plausible (adjacent evidence suggests it) and unknown (nothing yet). They volunteer the weakest area with a dated plan, they do not quote completion, and they keep the defense to what each artefact actually shows.

Interview question

"Which of your three projects are you least sure about, and what evidence would settle it?" How would you structure the answer?

Outline your answer first.

Recall cards scheduled after completion

What are the ten elements of an end-to-end project defense?
How do you read a dimension that shows insufficient evidence?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Model Cards for Model Reporting — Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru, 2019optional
Short structured reports of intended use, evaluation conditions and limitations, a model for the limitations and provenance part of a defense.
Leakage and the reproducibility crisis in machine-learning-based science — Sayash Kapoor, Arvind Narayanan, 2023optional
A survey of leakage across published ML-based science and its model info sheets, supporting the leakage and reproducibility parts of the defense.
Artificial Intelligence Risk Management Framework (AI RMF 1.0) — National Institute of Standards and Technology (NIST), 2023optional
Governance vocabulary for documenting risk, measurement and ownership when the defense covers a GenAI system.

Optional sections

Deep dive · 10 min

What the weights mean. A dimension with $n$ records has weights $w_i = b_i \cdot 0.5^{a_i/45}$ , where $b_i$ is the base weight of the evidence kind (1 for first-attempt core, probe, scenario, interview and studio items; 0.75 for teach-back; 0.5 for reviews; 0.25 for repeats) and $a_i$ is the age in days. The estimate is $\sum w_i s_i / \sum w_i$ . The approximate standard error uses an effective sample size $(\sum w_i)^2 / \sum w_i^2$ , which is smaller than $n$ when weights are unequal.

Why a small sample estimate deserves caution. Three items at weight 1 give an effective sample of three; one wrong answer moves the estimate by about a third. The report prints the item count so that you can see this.

Defense without the app. If you work outside the app, keep a one-page evidence index per project: artefact path, claim it supports, date, and a line saying whether it was produced under timed, unaided conditions. A reviewer can then check any claim in seconds.

Interview practice · 15 min

A committee member says your forecasting project looks solid but your system-design evidence is thin. What do you say, and what do you do next?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-18

1 Recall · 5 min

2 Concept · 10 min

Assumptions

3 Application and workplace decision · 10 min

4 Mini-assessment · 8 min

Technical depth · optional · 32 min in total

Useful for

Not suitable for

Traps

Senior reasoning

Interview question

Recall cards scheduled after completion

Prerequisites and related concepts

References

Optional sections

Recall · 5 min

Concept · 10 min

Application and workplace decision · 10 min

Mini-assessment · 8 min