Core day 30 of 60 · Week 6 · Part I · evaluation depth

Part I Master Assessment

You now hold 29 days of evidence: answers, checkpoint probes, scenario stages and studio outputs. A cumulative 12-item probe is about to sample all of Part I. How do you read its result so that it adds information, and how do you turn the evidence on your Progress page into at most three Part II review priorities you can defend?

By the end: You complete a 12-item cumulative probe closed-book, classify each miss by concept pair and confidence, and write at most three Part II review priorities, each tied to a measured item count, a Part II day that will stress it and a retest.

At a glance

Read a small cumulative probe for patterns, not a grade, and pick at most three Part II review priorities that your evidence can defend.

Small sample
12 items at 70% carry a standard error of about 13 points.
Patterns
Misses that share a concept pair or a confident answer say more than the score.
Unknown is not zero
Too few items means collect evidence, not review.
Four filters
Enough evidence, weak after uncertainty, a pattern, a Part II day that needs it.
Three at most
Each with a retest in Week 7; revise at the Day 40 checkpoint.

Recall · 5 min

Closed-book warm-up, two minutes each: (1) name the three questions that triage a what-if input to a forecasting model; (2) what must be true of the errors that calibrate an interval at the cutoff; (3) why does a pooled association between fare and bookings mislead? Then clear your due review cards.

Open due reviews

Concept · 10 min

A 12-item probe is a small sample: at a true 70%, its standard error is about 13 points, so it cannot rank your dimensions alone. Its value is the pattern: every item mixes concepts, so misses show which pairs of ideas fail together and whether you were confident when they did. Nothing here is a pass mark.

Which model family earns a slot depends on the problem shape - the same evidence-first logic that picks review priorities

Predict first

With 3,000 labelled rows and 25 numeric features, which belongs in the first evaluation?

Text description of the figure

A matrix of problem shapes by model families. Selecting a cell shows what to try first, how that family fails on that shape, and what evidence would move you toward or away from it, without naming a winner.

What today's evidence can and cannot tell you Overview

A small probe
12 items: too few to rank dimensions on their own.
Mixed items
Each item mixes at least two earlier concepts, so misses point at pairs.
The Progress page
Estimates with standard errors and item counts, confidence table, recurring mistakes.
Four filters
Turn that evidence into at most three priorities.

How precise is a 12-item score? Formula

\text{SE} = \sqrt{\frac{p\,(1-p)}{n}} = \sqrt{\frac{0.7 \times 0.3}{12}} \approx 0.13

$p$: your true accuracy, here 70%
$n$: number of scored items
$\pm 2\,\text{SE}$: about 26 points either side: a single probe can land anywhere from mid-40s to mid-90s

The binomial approximation; the platform's own scoring policy also weights items by recency.

What a 12-item score can and cannot tell you Illustration

Exact binomial probabilities of each score. The same learner can land anywhere in the blue range on a 12-item probe: the probe alone cannot tell 60% from 80%.

Four times the items halve the spread. This is what the stored evidence on the Progress page adds to a single probe.

One learner, six dimensions with known true accuracy (in brackets) and item counts. Bars: how often each dimension shows the lowest raw score. The truly weakest is Dimension A, but thin dimensions take the label far more often than their accuracy deserves.

Exact binomial probabilities (left, middle). Right: 3,000 simulated attempts at the stated true accuracies, items treated as independent; ties share the label. Illustrative learner, not your data.

Exact score distributions for one learner at 12 and 48 items, and how often raw scores point at the wrong weakest dimension: ranking noisy estimates is a max-of-k problem (Day 15), and the lowest of several equal scores is biased low.

Four filters, in order Diagram

Enough evidence?3+ distinct items
- yes: to Weak after its SE?
no
Collect evidencescenario stage, retest
Weak after its SE?
- yes: to A pattern?
no
Not a prioritykeep practising
A pattern?repeated miss
+ urgency
Part II needs it soon?
+ urgency
Prioritywith a retest

The first two filters gate (yes moves on); a pattern or an early Part II need only raises urgency when the answer is yes. Keep at most three priorities, and send thin evidence to collection, never to a study block.

Score and item count decide the action Map

Unknown

A good score on few items: collect evidence before trusting it.

Strength

Maintain with spaced recall; no review slot needed.

Unknown

A low score on few items: collect evidence, do not review yet.

Priority candidate

Weak even after its standard error: review, then retest.

The Progress page holds what a probe cannot: an estimate per dimension with its standard error and item count, accuracy by stated confidence, and recurring mistakes.
Insufficient evidence means unknown, never zero: collect evidence before reviewing.
Four filters, in order: enough evidence; weak once uncertainty is included; a pattern (a recurring misconception, a confident miss, the same error type such as leakage, grain or Baseline in several costumes); a Part II day that needs it soon.
The Part II days that need each Part I concept directly are listed in the depth chapter "What Part II will lean on".
Keep at most three priorities, each with a retest; revise at the Day 40 checkpoint.

Assumptions

Your evidence is first attempts without notes; viewing answers or reference solutions is not credited, and older evidence counts less (the policy halves weight every 45 days).
The numeric weights in the sketch are illustrative choices for a teaching rule, not the platform's scoring policy.
Thirty days of measurements are a sample of what you know; a priority list is a plan to be re-measured, not a verdict.

Application and workplace decision · 10 min

Take the 12 probe items below closed-book, writing a confidence of 1-5 for each before you check.
Open /progress and copy its dimension table, the insufficient-evidence list, the confidence table and "My mistakes".
Run the sketch, which applies the four filters to an illustrative learner, then rerun it with your numbers.
Write at most three priorities in decision-log.md: dimension or concept id, evidence, first Part II day and retest.

The Week 6 Sunday studio (studio-w06-sun) is the extended assessment for these steps.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

# Illustrative Progress-page export for a synthetic learner; replace the numbers with your own.
evidence = {  # dimension: (score, stderr, items, repeated misconception misses, misses at confidence 4-5)
    "validation-metrics": (62, 8, 16, 3, 2),
    "classical-ml": (58, 7, 12, 1, 0),
    "business-judgment": (64, 8, 9, 0, 1),
    "forecasting": (78, 6, 14, 0, 0),
    "statistics": (55, 18, 4, 0, 0),
    "data-engineering": (40, 25, 2, 0, 0),
}
first_need = {  # first Part II day whose concepts list a Part I concept of that dimension as a direct prerequisite
    "validation-metrics": "Day 34 (learning curves), Day 43 (eval set)",
    "classical-ml": "Day 31-32 (regression, logistic), Day 35 (baseline)",
    "business-judgment": "Day 35 (baseline), Day 59 (defense)",
    "statistics": "Day 38 (variance, covariance)",
    "data-engineering": "Day 53 (data contracts)",
}
bar = 70
ranked, collect = [], []
for dim, (score, se, n, repeats, overconf) in evidence.items():
    if n < 3 or se > 12:                       # too little evidence to call it weak or strong
        collect.append(f"{dim} ({n} items, +/-{se})")
        continue
    gap = max(0, bar - (score + se))           # weak only if even the optimistic end is below the bar
    ranked.append((gap + 4 * repeats + 6 * overconf, dim, score, se, n))
print("priority  dimension            score  items  first Part II need")
for urgency, dim, score, se, n in sorted(ranked, reverse=True)[:3]:
    print(f"{urgency:8.0f}  {dim:20s} {score:3d}+/-{se:<2d} {n:5d}  {first_need[dim]}")
print("collect evidence before reviewing:", "; ".join(collect))

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 29 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The standard error of a score, derived and workedadvanced9 minThe binomial SE derived from one item up, a 60% score worked at five sample sizes, and the Wilson interval computed by hand for 3 of 5.

The standard error of a score, derived in four steps Sequence

One item
Score it X_i = 1 (right) or 0 (wrong) with P(X_i = 1) = p. Then E[X_i] = p and Var(X_i) = E[X_i^2] - p^2 = p - p^2 = p(1 - p).
n independent items
Variances of independent terms add: Var(X_1 + ... + X_n) = n p(1 - p).
The score
p-hat = (X_1 + ... + X_n) / n, so Var(p-hat) = n p(1 - p) / n^2 = p(1 - p) / n.
Plug in
SE = sqrt(p-hat (1 - p-hat) / n). It shrinks with sqrt(n): halving it takes four times the items.

p-hat estimates your accuracy on items like these, unbiased and consistent; its SE is its spread over repeated probes of the same size, not a property of you. Independence is the load-bearing step: items that share a misconception are positively correlated, so the true SE is larger.

Worked: 60% correct on 5, 10, 20, 40 and 80 items Table

Items (right)	SE = sqrt(0.6 x 0.4 / n)	Wald 95%: 0.6 +/- 1.96 SE	Wilson 95%
5 (3)	0.219	0.17 to 1.03: past 100%	0.23 to 0.88
10 (6)	0.155	0.30 to 0.90	0.31 to 0.83
20 (12)	0.110	0.39 to 0.81	0.39 to 0.78
40 (24)	0.077	0.45 to 0.75	0.45 to 0.74
80 (48)	0.055	0.49 to 0.71	0.49 to 0.70

The same 60% means "somewhere between a quarter and nine tenths" on five items and "about half to seven tenths" on eighty. The Wald band breaks on few items (textbook rule: trust it only when n p-hat and n (1 - p-hat) are both above about 5); Wilson stays inside 0-1. Computed in Python (statsmodels Wilson).

The Wilson interval, worked for 3 right out of 5 Formula

\frac{\hat p + \frac{z^2}{2n}}{1 + \frac{z^2}{n}} \pm \frac{z}{1 + \frac{z^2}{n}} \sqrt{\frac{\hat p(1-\hat p)}{n} + \frac{z^2}{4n^2}}

$\text{centre}$: (0.6 + 3.8416/10) / (1 + 3.8416/5) = 0.984 / 1.768 = 0.557: pulled towards 0.5
$\text{half-width}$: 1.108 x sqrt(0.048 + 0.0384) = 1.108 x 0.294 = 0.326
$\text{band}$: 0.557 +/- 0.326 = 0.23 to 0.88, inside 0-1 where the Wald band was not

Wilson inverts the score test instead of plugging p-hat into the SE, which is why it behaves on small n and near 0 or 1. Recomputed in Python.

Standard error of a score by number of items (true accuracy 70%) Chart

Halving the error needs four times the items. That is why a dimension with four items is "unknown", and one with sixteen is a measurement.

Mean scores, sample sizes and comparing two estimates5 minPartial-credit scores, the items a given precision costs, the SE of a gap between two dimensions, and the same arithmetic in four sectors.

Scores with partial credit, and comparing two dimensions Formula

\begin{aligned} \mathrm{SE}(\bar x) &= \frac{s}{\sqrt n} \\ \mathrm{SE}(\bar x_A - \bar x_B) &= \sqrt{\mathrm{SE}_A^2 + \mathrm{SE}_B^2} \end{aligned}

$\text{partial credit}$: six studio sections scored 1, 0.5, 1, 0, 0.5, 1: mean 0.67, s = 0.41, SE = 0.41 / sqrt(6) = 0.17
$\text{two dimensions}$: 58 +/- 9 against 74 +/- 7 (illustrative): gap 16, SE sqrt(81 + 49) = 11.4, only 1.4 SE
$s$: the sample SD of item scores; for 0/1 items s^2 is about p(1 - p), the binomial case

A 16-point gap between two thin estimates is not yet a ranking. Both lines assume independent items and independent dimensions. Computed in Python; the next figure draws the comparison.

Overlapping bands are not the test: the gap has its own standard error Illustration

Gap 16 with SE 11.4: only 1.4 SE from zero, so the 95% band of the difference includes zero. Two thin estimates cannot be ranked yet.

The two bands overlap, yet the gap of 12 is 2.1 SE from zero, because the SE of a difference is sqrt(SE_A² + SE_B²), smaller than SE_A + SE_B. Overlap is too strict a test.

Illustrative scores and standard errors (the first pair is the worked line above); independent dimensions assumed; bands are estimate ± 1.96 SE, computed here.

Two pairs of dimension estimates with 95% bands, and the band of their difference. Judge the gap against the SE of the difference, not by whether the two bands touch.

How many items buy a given precision? Formula

n = \frac{z^2\, p(1-p)}{h^2} = \frac{1.96^2 \times 0.7 \times 0.3}{0.10^2} \approx 81

$h$: the 95% half-width you want, here +/- 10 points
$h = 0.05$: about 323 items: halving the band costs four times the items

Solve the Wald half-width for n. No single probe gets near this; precision comes from accumulating first attempts over weeks.

The same arithmetic outside a learning platform Overview

Insurance claims QA
An auditor samples 12 claims and finds 2 errors: the error rate could be anywhere from about 5% to 45%.
Lab or factory QC
10 samples pass out of 10: Wilson still allows a defect rate of up to about 28%.
Chatbot evaluation
20 graded conversations at 85% cannot rank two prompts that differ by 5 points.
SaaS A/B test
Conversion on a few hundred users per arm: the SE of the difference decides, not the gap.

Wilson bands for 2 of 12 and 10 of 10 computed in Python. Same rule everywhere - quote n and the SE before the rate.

How the platform weighs your evidenceadvanced3 minRecency weights, the minimum-evidence rule and the effective number of items, computed with the scoring code.

Weights, effective items and the standard error Formula

\begin{gathered} w_i = b_i\,0.5^{a_i/45}, \qquad n_{\text{eff}} = \frac{(\sum_i w_i)^2}{\sum_i w_i^2} \\ \mathrm{SE} = \sqrt{\frac{p(1-p)}{n_{\text{eff}}}} \end{gathered}

$b_i$: base weight of the evidence kind: 1 for core, probe, scenario and studio; 0.5 review; 0.25 repeat
$a_i$: age of the answer in days
$n_{\text{eff}}$: equals the item count when all weights are equal; smaller when they differ

From docs/SCORING_POLICY.md. A dimension needs at least 3 distinct items and a total weight of 2, otherwise it shows insufficient evidence.

How the platform weighs old evidence Illustration

Weight of one answer by its age: it halves every 45 days, starting from the base weight of its kind.

Four first-attempt items answered on one day, nothing since. The total weight sinks below the minimum of 2 after about 46 days, and the dimension returns to "insufficient evidence", not to a low score.

Twelve items: six answered today, six answered earlier. The older half counts for less, so the estimate behaves like fewer items and its standard error grows, although nothing was answered wrongly.

Computed with the platform's own estimateDimensions on synthetic records (policy in docs/SCORING_POLICY.md); scores of the twelve items are illustrative.

Computed with the scoring code itself; weights halve every 45 days, thin old evidence falls back to unknown, and mixed ages count as fewer items.

How the sketch scores urgency2 minThe rule behind the ranking, with one worked line.

The sketch's urgency rule (illustrative weights) Formula

\begin{aligned} \text{urgency} &= \max\bigl(0,\ \text{bar} - (\text{score} + \text{SE})\bigr) \\ &\quad + 4\,r + 6\,c \end{aligned}

$\text{bar}$: your target, 70 in the sketch
$r$: misses of a recurring misconception
$c$: misses at confidence 4 or 5

validation-metrics: max(0, 70 - 70) + 4 x 3 + 6 x 2 = 24. The weights are a teaching choice, not the platform's scoring policy.

Urgency points from the sketch (illustrative learner, today's run) Chart

validation-metrics62 +/- 8 on 16 items; 3 repeats, 2 confident misses24
classical-ml58 +/- 7 on 12 items9
business-judgment64 +/- 8 on 9 items6

Points as computed by today's run of the application sketch for an illustrative learner; the formula, not this ranking, is what to reuse.

Reading the confidence table and each miss3 minWhich combination of confidence and correctness deserves review time first, and three lines to write for every miss.

Confidence and correctness Map

Danger zone

Confident and wrong: a belief to unlearn; weigh it most.

Solid

Confident and right: maintain with spaced recall.

Known gap

Unsure and wrong: you already know to review it.

Fragile

Unsure but right: retest before relying on it.

A confident miss is the strongest single signal: the belief will not correct itself. If your accuracy at confidence 4-5 is not higher than at 1-2, the self-assessment itself is the weak instrument.

Reading one miss, in three lines Sequence

The concept pair
Which two earlier ideas the item mixed, for example grain and baseline.
The belief
Why the wrong option looked right: the misconception named in its explanation.
The confidence
What you wrote before checking; 4 or 5 makes it a belief to unlearn.

Two misses with the same belief are one miss with a recurrence of two.

What Part II will lean on2 minThe Part I concepts that Part II days use directly, and when.

Part I concepts that Part II days need directly Table

Part I concept	First Part II day
Linear and logistic regression	Days 31-32
Learning curves	Day 34
Baseline-first	Day 35
Variance and covariance	Day 38
Encoding	Day 41
Train, validation and test	Day 43
Serving patterns	Days 52 and 58
Data contracts	Day 53
Drift monitoring	Day 58
Case clarification	Day 59

From the concept graph's direct prerequisites.

Writing a priority you can defend2 minFive fields for each of at most three priorities.

One priority, written down Sequence

Dimension or concept id
Name the concept pair if the misses cluster, not the day.
Evidence
Score, SE, item count, repeats, confident misses; first attempts without notes, recent ones weighted more.
First Part II day that needs it
From the table above.
Retest
Three fresh items in the first week of Part II.
Revisit
Re-measure at the Day 40 checkpoint and change the list if the numbers moved.

Evidence versus preference1 minWhat each way of choosing review topics risks, and what checks it.

Choosing review topics by preference or by evidence Comparison

Criterion	By preference	By evidence
Picks	Topics you enjoy or expect to need	Weak after uncertainty, patterned, needed soon
Risk	Repeats the pattern that produced the repeated misses	Over-weighting a noisy estimate
Checked by	Nothing	A retest in Week 7 and the Day 40 checkpoint

What a staff-level self-assessment addsadvanced2 minFive habits that make a self-assessment credible in a senior interview.

What a staff-level self-assessment adds Overview

Uncertainty first
Quote the item count and standard error before the score.
Unknown is not weak
Thin evidence earns a measurement, not a study block.
Error types over topics
One belief failing in three costumes is one skill.
A date to be wrong
Every priority carries a retest and the Day 40 re-measurement.
A practice that prevents it
Tie each fix to a work habit, such as registering the split before any fit.

Memory anchor

Unknown is not weak: pick review topics where enough items, a pattern and a near-term need agree, and set a date to re-measure.

Useful for

Turning a month of scored attempts into a short, defensible review plan before Part II starts.
Reading any small-sample assessment with its uncertainty and patterns: a probe, a claims-QA audit, a lab QC batch, a chatbot evaluation set.
Rehearsing the "how did you decide what to learn next" question in senior interviews.

Not suitable for

Certifying competence: the estimates are summaries of first attempts on this platform, not a qualification.
Choosing a career role; the role reports show coverage and concerns, not suitability.

Traps

Ranking by the raw score

The dimension with the lowest score on three or four items gets the review time, while a dimension with a measured weakness and repeated high-confidence misses is left alone.

How to avoid it

Read items and standard error beside the score; weak means the optimistic end is below the bar.

Reviewing by topic instead of by error type

Misses in SQL, Python and forecasting items are reviewed as three topics, although one skill (point-in-time availability, say) failed in all three.

How to avoid it

Name the concept pair or error type behind each miss before choosing what to study.

Senior reasoning

A senior treats a self-assessment like any other measurement: sample size, uncertainty, patterns, and a decision rule fixed before looking. They separate unknown from weak, they weight a high-confidence miss above a low-confidence one, and they leave a re-measurement date on every plan so that it can be wrong cheaply.

Interview question

"Tell me about a weakness you found in your own skills and what you did about it." How would you structure the answer?

Outline your answer first.

Recall cards scheduled after completion

What are the four filters for choosing a review priority from your evidence?
Why can a 12-item probe not rank your dimensions?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Align Assessments, Objectives, Instructional Strategies (Eberly Center, Carnegie Mellon University) — Eberly Center for Teaching Excellence & Educational Innovation, Carnegie Mellon Universityoptional
Why assessments should be aligned with objectives and teaching activities, the basis for matching probe items to evaluation-level objectives.
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations — Sander Greenland, Stephen J. Senn, Kenneth J. Rothman, John B. Carlin, Charles Poole, Steven N. Goodman, Douglas G. Altman, 2016optional
Common misreadings of confidence intervals and p-values that apply when reading a small-sample score.
On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation — Gavin C. Cawley, Nicola L. C. Talbot, 2010optional
Selection bias when the same data are used to choose and to report, relevant to probes p3 and p10.

Optional sections

Deep dive · 10 min

Reading the standard error. For a proportion $p$ over $n$ independent items, the standard error is $\sqrt{p(1-p)/n}$ . At $p=0.7$ and $n=12$ it is 0.13, and at $n=48$ it is 0.07. Items from one lesson are not independent (they share a misconception), so the real uncertainty is larger. Combining a probe with stored first-attempt evidence is the cheap way to gain items.

Recency and weights. The scoring policy weights core, probe, diagnostic and scenario evidence at 1.0, reviews at 0.5 and repeat attempts at 0.25, and halves any weight every 45 days. A dimension can therefore fall back to insufficient evidence if it was last measured long ago. This is intentional: old evidence is weaker evidence.

Confidence calibration. Accuracy at confidence 4-5 should be higher than at 1-2. If it is not, your self-assessment is the weak instrument, and the priority is to review the topics where you were certain and wrong, because those are the ones you would not have found by feel.

Reading a miss. For each wrong probe item write three things: the concept pair, the belief that made the wrong option attractive (the misconception tag in the explanation), and whether you were confident. Two misses with the same belief are one miss with a recurrence of two.

Part II demand, from the concept graph. Direct dependencies are listed in the mechanism. The indirect ones follow the same pattern: a weakness in validation shows up again in Day 35's neural-versus-baseline comparison, Day 43's evaluation set and Day 45's evaluation lessons, which is why it is worth one early fix.

The Sunday studio. The Week 6 Sunday studio contains the four assessment sections: closed-book recall, the finished forecasting report with an unfamiliar transfer case, a Part I defense and the Part II review. Complete the review section after the probe, so that it uses its result.

Interview practice · 10 min

A hiring manager says: "Your test results show a weak spot in validation. Convince me you are not a risk." How do you respond?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-18

1 Recall · 5 min

2 Concept · 10 min