Core day 43 of 60 · Week 9 · Part II · evaluation depth

Pretraining and Fine-Tuning

A colleague proposes fine-tuning a model on 4,000 past agent replies "so the support bot knows our policies". What evidence decides between a better prompt, retrieval and fine-tuning, and what must exist first?

By the end: You freeze a small versioned evaluation set with a recorded baseline, tag each failure by the capability that is missing, choose prompting, retrieval or fine-tuning from those counts, and check candidate training data for overlap with the evaluation items.

At a glance

Build a frozen evaluation set and baseline first, tag what is missing, and pick the cheapest lever that supplies it; fine-tuning is for behaviour, not facts.

Three levers
Prompt and retrieval change the input; fine-tuning changes the weights.
Eval set first
Frozen, versioned, hashed; baseline with an interval before any adaptation.
Missing capability
Knowledge -> retrieval; format -> schema; behaviour -> prompt, then tuning.
Facts go stale
Weights absorb new facts poorly and cannot cite or update them.
Contamination
Training text that overlaps eval items inflates the score without any gain.

Recall · 5 min

No reviews due? Answer from memory. (1) Why does lowering temperature not fix an answer the model was never given the facts for (Day 42)? (2) What are the three roles of train, validation and test data (Day 15)? (3) What is the baseline a new method has to beat (Day 1)?

Open due reviews

Concept · 10 min

You can change a model's behaviour with three levers: a better prompt, retrieval of the right documents, or fine-tuning the weights. A model after Pretraining already writes fluent text, so you adapt only for a Missing capability. Fine-tuning to "teach facts" is the common mistake: weights absorb format and behaviour well, new facts poorly, and baked-in facts go stale.

The missing capability, not the novelty of the method, picks prompting, retrieval or fine-tuning

Predict first

Policy facts change monthly and answers must cite the current policy. Which lever comes first?

Text description of the figure

Toggles for the gaps found in error analysis (knowledge, format, behaviour, vocabulary, latency or cost) and controls for how many labelled examples exist, the status of the evaluation set and whether a searchable corpus exists. The panel always opens with the evaluation-set step, then lists prompting, retrieval and PEFT fine-tuning, each with a verdict from "try first" to "not suited" and its reasons, followed by the evaluation to run first and an ordered plan. The rules are practitioner heuristics, not measurements.

Four ideas to hold on to Overview

Measure first
A frozen evaluation set and a baseline come before any lever.
Name the gap
Tag each failure with the capability that is missing.
Cheapest lever
Prompt, retrieval, then fine-tuning; they add up.
Keep it clean
Training text that copies eval items fakes a gain.

How a chat model is built, and where your adaptation sits Diagram

Very large text corpusunlabelled
Pretrainingnext-token loss (Day 42)
- to Supervised fine-tuning
Your fine-tuneusually PEFT (Day 44)
Supervised fine-tuninginstruction-response pairs
- to Preference tuning
Retrievalcurrent documents (Day 48)
- back to Your fine-tune
Preference tuningRLHF or DPO (Day 45)
- to Assistant model
Promptinstructions, examples, schema
- back to Retrieval
Assistant modelwhat you call via an API
Your eval set + baselinefrozen, tagged
- back to Prompt

The top row is done by the model builder. Your work starts with the evaluation set and moves left only while failures remain.

From failure tags to the lever Diagram

Failures on frozen settagged
Missing capability?
- format: to Schema + validator
- behaviour: to Survives a good prompt?
- cost: to Smaller adapted model
knowledge
Retrieval with citationsknowledge that changes
Schema + validatorformat (Day 47)
Survives a good prompt?
- no gap left: to Keep the prompt
- gap remains: to Fine-tune
Smaller adapted modelcost or latency
Keep the prompt
Fine-tunecurated examples; SFT, usually PEFT

A practitioner decision rule consistent with the OpenAI accuracy guide (as read 2026-10-11) and the retrieval-versus-fine-tuning evidence cited below.

The three levers Comparison

Criterion	Prompt	Retrieval	Fine-tuning
What changes	The input: instructions, examples, schema	The input: documents added at request time	The weights
Best for	Format and simple behaviour	Knowledge that is private or changes	Consistent behaviour, style, cost at scale
When a fact changes	Edit the text	Update the document; answers can cite it	Retrain; old fact stays in weights
What you own afterwards	A versioned prompt (Day 47)	An index and a freshness pipeline	A model artefact to retest, including safety

Eval set v1 baseline: passes, and failures by missing capability (app output) Chart

Knowledgeallowances, windows, terminals4 items
FormatJSON wrapped in prose2 items
Behaviourmissed an escalation1 items
Passed5 items

Knowledge dominates, so retrieval of the current policy is the first lever, not a fine-tune.

Write a frozen, versioned Evaluation set and record the best-prompt baseline with an interval before any adaptation.
Tag each failure: knowledge, format, behaviour or cost/latency. The counts pick the cheapest lever; levers combine.
Fine-tune (Supervised fine-tuning (SFT), Instruction tuning) only for behaviour a good prompt cannot hold.
Remove training examples that overlap evaluation items (Benchmark contamination) and rerun safety tests after tuning.

Assumptions

The tagged failures represent production traffic; a sample of easy chats under-states knowledge and behaviour gaps.
Pass or fail can be judged reliably, by exact match, a validator or a rubric; otherwise the set measures the judge.
Facts that change live in a source of truth retrieval can read; if none exists, build it first.

Application and workplace decision · 10 min

12 synthetic airline-support items with the recorded answers of the best prompt (the baseline); each item carries the capability that is missing if it fails.
The script freezes the set as version 1 with a hash, reports the pass rate with a 95% Wilson interval, tallies failures by capability and checks two candidate fine-tuning examples for 4-gram overlap.
Predict first: which capability causes most failures, and would a one-item gain on 12 be visible?

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import hashlib, json, math
from collections import Counter

# eval set v1: synthetic airline-support items written BEFORE any adaptation
# (id, capability missing if it fails, question, expected answer, recorded baseline answer)
items = [
    ("e01", "knowledge", "Free cabin bag on ZX Basic?", "8 kg", "10 kg"),
    ("e02", "knowledge", "Online check-in closes before departure?", "2 hours", "3 hours"),
    ("e03", "knowledge", "Terminal for ZX flights to Frankfurt?", "Terminal 1", "Terminal 2"),
    ("e04", "knowledge", "Free cancellation window on Flex?", "24 hours", "48 hours"),
    ("e05", "knowledge", "Checked bag on Basic?", "1 x 20 kg", "1 x 20 kg"),
    ("e06", "format", "Intent as JSON: money back for cancelled flight", '{"intent":"refund"}', "The intent is refund."),
    ("e07", "format", "Intent as JSON: my bag is missing", '{"intent":"baggage"}', 'Sure: {"intent":"baggage"}'),
    ("e08", "format", "Intent as JSON: change my seat", '{"intent":"seat"}', '{"intent":"seat"}'),
    ("e09", "format", "Intent as JSON: flight is late", '{"intent":"delay"}', '{"intent":"delay"}'),
    ("e10", "behavior", "Customer threatens a lawsuit over denied boarding", "ESCALATE", "I am sorry to hear that."),
    ("e11", "behavior", "Customer asks for the pilot's home address", "REFUSE", "REFUSE"),
    ("e12", "behavior", "Customer thanks the agent", "ACK", "ACK"),
]
version = hashlib.sha256(json.dumps([i[:4] for i in items]).encode()).hexdigest()[:8]
passed = [exp == got for *_, exp, got in items]
n, k = len(items), sum(passed)
z, p = 1.96, k / n  # Wilson 95% interval for a pass rate
centre = (p + z * z / (2 * n)) / (1 + z * z / n)
half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
print(f"eval set v1 sha256={version}  baseline pass {k}/{n} = {p:.2f}  95% CI [{centre - half:.2f}, {centre + half:.2f}]")
print("failures by missing capability:", dict(Counter(tag for (_, tag, *_), ok in zip(items, passed) if not ok)))

def grams(s, n=4):
    w = s.lower().split()
    return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}
candidates = ["Q: Intent as JSON: money back for cancelled flight A: refund", "Q: What time do gates close A: 20 minutes before"]
eval_grams = set().union(*(grams(q) for _, _, q, *_ in items))
print("training candidates overlapping the eval set:", sum(bool(grams(c) & eval_grams) for c in candidates), "of", len(candidates))

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 27 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The training objective, computed by hand4 minToken cross-entropy and perplexity on four tokens, and why every stage, fine-tuning included, is continued maximum likelihood.

The loss every stage minimises, token cross-entropy Formula

H = -\frac{1}{N} \sum_{t=1}^{N} \log_2 p_\theta(x_t \mid x_{<t}) \qquad \mathrm{PPL} = 2^{H}

$p_\theta(x_t \mid x_{<t})$: probability the model gave the token that actually came next
$N$: number of scored tokens (pretraining scores all; SFT usually only response tokens)
$\mathrm{PPL}$: perplexity, the effective number of choices per token; in nats, PPL = e^H

Minimising H is maximum likelihood: it is exactly maximising the probability of the training text. Fine-tuning runs the same minimisation on new pairs, which is why it is continued MLE, not a different kind of learning.

Perplexity by hand: a 4-token maintenance log line (IoT toy, synthetic) Table

Token (after the previous ones)	Model p	log2 p	Contribution -log2 p
pump	0.25	-2	2
pressure	0.5	-1	1
too	0.125	-3	3
high	0.0625	-4	4
mean H = (2+1+3+4)/4	-	-	2.5 bits

Hand-set probabilities, checked in numpy. PPL = 2^2.5 = 5.66: the model is as uncertain as a uniform choice among 5.66 tokens; guessing uniformly over 8 tokens would give PPL 8. Lower is better, and it measures corpus fit, not task success.

Loss and perplexity watch the run; the pass rate makes the decision Comparison

Criterion	Loss / perplexity	Task pass rate
Measures	Fit to a token corpus: how surprised the model is per token	Success on your frozen, tagged items
Comparable across	Only models with the same tokenizer on the same corpus	Any systems scored on the same set version
Use it for	Monitoring training: divergence, overfitting on the validation split	Adaptation decisions, with an interval

A model with a different vocabulary scores a different token sequence, so its perplexity is not on the same scale.

Fine-tuning as continued maximum likelihood, by handadvanced4 minThe masked SFT loss written out, computed on one support pair, and what follows from starting at the pretrained weights.

Supervised fine-tuning: the same loss, new data, a mask Formula

\mathcal{L}_{SFT}(\theta) = -\frac{1}{|R|} \sum_{t \in R} \log p_\theta(y_t \mid x, y_{<t}), \qquad \theta_0 = \theta_{\mathrm{pre}}

$x$: the prompt (instruction, customer message); read by the model but not scored
$R$: positions of the response tokens y; only these enter the loss (the mask)
$\theta_0 = \theta_{\mathrm{pre}}$: training starts from the pretrained weights, so SFT is continued maximum likelihood, not a new kind of learning
$\partial \mathcal{L} / \partial z_t = p_t - e_{y_t}$: at each scored position the logit gradient is predicted probabilities minus the one-hot target (Day 32)

Pretraining is the same expression with R = every position and x empty. What changes between stages is the data and the mask, never the estimator.

SFT loss by hand on one telecom support pair (synthetic, hand-set probabilities) Table

Token	Part	Model p	Scored?	-log2 p
Cancel my roaming add-on	prompt x (4 tokens)	-	no: mask 0	-
Roaming	response	0.5	yes	1
add-on	response	0.25	yes	2
cancelled	response	0.5	yes	1
mean over R	-	-	3 tokens	4 / 3 = 1.33 bits
one gradient step	-	-	-	raises those three p a little; the prompt words are not pushed

Checked in numpy. Every one of the 4,000 replies in the application's proposal adds terms like these: what is frequent across them (tone, format, the usual policy) moves fastest; a fact stated once moves a few terms.

Three properties that follow from continued maximum likelihood Overview

Small data overfits fast
A few hundred pairs are memorised within epochs: watch validation loss after each one.
High learning rate forgets
Large steps erase pretrained behaviour (catastrophic forgetting): start low, re-test base skills.
Label noise is learned
The estimator maximises the likelihood of whatever the replies say, wrong policy included.

Pretraining, SFT and preference tuning3 minWhat each training stage uses as data and objective, and what it changes.

Training stages compared Table

Stage	Data	Objective	What it changes
Pretraining	Very large unlabelled corpus	Next-token cross-entropy	Language, broad knowledge to a cutoff
Continued pretraining	Large in-domain unlabelled text	Same objective	Domain vocabulary and style (Gururangan 2020)
Supervised fine-tuning	Prompt-response pairs	Next-token loss on responses	Follows instructions, formats
Preference tuning	Ranked or paired responses	RLHF or DPO (Day 45)	Which responses are favoured

What the evidence says about fine-tuning for facts Overview

Retrieval won
Retrieval beat unsupervised fine-tuning for injecting knowledge in the settings studied (Ovadia 2024).
New facts are slow
Examples with new knowledge were learned slowly and raised hallucination (Gekhman 2024).
Staleness
A fact in the weights cannot be updated or cited without retraining.

Contamination as train-test dependence, three layersadvanced4 minThe bias overlap adds to a pass rate, derived; where overlap comes from, which layer you can measure and how to screen for it.

Contamination as train-test dependence: the bias it adds Formula

\mathbb{E}[\hat p] = (1-c)\,p_{\mathrm{clean}} + c\,p_{\mathrm{seen}}, \qquad \mathrm{bias} = c\,(p_{\mathrm{seen}} - p_{\mathrm{clean}})

$c$: share of evaluation items the training data contains or paraphrases
$p_{\mathrm{clean}}$: pass rate on items independent of training: the capability you want to estimate
$p_{\mathrm{seen}}$: pass rate on seen items, pushed toward 1 by memorisation
$\mathrm{bias}$: hypothetical: c = 0.25, clean 0.60, seen 0.95 gives 0.69 observed, 9 points of gain that is not capability

The Day 9 boundary restated: a pass counts as capability only if the item is independent of everything that shaped the model, so the estimator is unbiased only at c = 0. More items shrink the interval but leave the bias untouched.

Three layers of contamination Stack

Training-to-eval overlapYour fine-tuning data copies eval items. Measurable: n-gram or embedding screen.
Few-shot copied from evalPrompt examples are eval items; also inflates the score.
Pretraining contaminationPublic benchmarks may sit in the base corpus; often not inspectable (Sainz 2023).

Overlap manufactures a gain the model does not have Illustration

Seeded simulation (200 items, 200 draws per point; shaded: 5th to 95th percentile of the headline). Pass probabilities are assumptions stated in the code, not measurements of any model.

The true gain is 5 points. With 20% of items overlapping training the headline averages 11.2 points; scoring only clean items brings it back to 4.5. The headline grows with every overlapping item, while the model improved by the same amount throughout.

The same true improvement, scored on sets with more and more items copied into the fine-tuning data.

Screening for overlap Diagram

Candidate examples
4-gram or embedding hit?
- no: to Keep
yes
Drop or retirethe example, or the eval item
Keep

The n in an n-gram screen: false alarms against misses Illustration

verbatim copies (12)
light edits (12)
paraphrases (12)
unrelated questions (12)

Small n flags unrelated questions too (100% at n = 1, 42% at n = 2): shared phrases such as "intent as json" are not contamination. Large n misses light edits (0% caught at n = 5) and even verbatim copies of short questions (8% at n = 8). At the application's n = 4 the screen catches 100% of copies but only 33% of light edits and 8% of paraphrases: short items need a lower n, character n-grams or an embedding screen.

Evaluation questions from the application code; candidates generated by seeded rules (copy, edit, synonym swap) plus 12 hand-written unrelated support questions. The shares are those of this small set; the shape is the point.

Four kinds of candidate training example screened against the application's 12 evaluation questions at every n.

Building and versioning the evaluation set3 minThe fields every item needs and how versions keep scores comparable.

Anatomy of an evaluation item: six fields, each with a job Overview

id
A stable reference, so systems can be compared item by item across versions.
input
The real request, taken from traffic, not invented to suit a method.
expected answer or rubric
Makes pass or fail checkable by exact match, a validator or a written rubric.
capability tag
Knowledge, format, behaviour or cost: the counts pick the lever.
source, author, date
Provenance, so stale items can be found and retired.
split
Development, or the locked test portion used only for decisions (Day 45).

Versioning the set Sequence

Freeze
Publish version 1 with a content hash of ids, inputs and expected answers.
Score
Every system is scored on the same version; store outputs.
Fix by release
A wrong expected answer becomes version 2, never an in-place edit.
Rescore
Re-run baseline and candidates on version 2 before comparing again.

Freeze the set before you see an adapted model: a set written after reading a candidate's outputs drifts toward what that candidate does well. The hash makes any silent edit visible.

How many items are enough?3 minInterval width against set size, and why 12 items can only show large effects.

How wide is a pass-rate interval? (95%, pass rate 0.5) Chart

Half-width = 1.96 x sqrt(p(1-p)/n). 400 items still leave +/- 5. Start with 12 to 20 real cases to learn the failure tags; grow past 100 before deciding on small gains. Paired, item-level comparisons help (Miller 2024).

The Wilson interval the application prints Formula

\frac{\hat p + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat p(1-\hat p)}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}

$\hat p$: observed pass rate (5 of 12 = 0.42 in the application)
$n$: number of items
$z$: 1.96 for 95%
$z^2 / 2n$: pulls the centre toward 0.5, so the interval never leaves [0, 1] on small sets

For 5 of 12 it gives 0.19 to 0.68 (application output). Prefer it to the normal approximation below about 30 items or near 0 and 1.

How often a real 10-point gain is visible, by set size Illustration

paired: only items where the systems disagree count
unpaired: two separate pass rates

Same simulated systems (pass rates 0.60 and 0.70), 300 evaluation sets per size. With 12 items a real 10-point gain is detected 9% of the time even when paired; with 100 items paired testing finds it 54% of the time against 26% unpaired.

Seeded simulation with a shared item difficulty (logistic model); the gain, the difficulty spread and the tests are stated in the code. Real gains, correlations and therefore the curves differ by task; compute your own before sizing a set.

Simulated evaluation sets in which hard items are hard for both systems. Pairing uses that correlation; two separate pass rates waste it. Ask what gain the set can detect before the run: small NLP comparisons are often underpowered (Card et al. 2020).

A paired comparison counts only the items that changed Formula

z = \frac{b - c}{\sqrt{b + c}}

$b$: items the candidate passes and the baseline fails
$c$: items the baseline passes and the candidate fails
$b + c$: discordant items; items both pass or both fail carry no information about the difference

McNemar's test, normal approximation (hypothetical: b = 12, c = 4 gives z = 2.0). With fewer than about 20 discordant items use the exact sign test.

Curating fine-tuning data2 minFive steps that matter more than the number of examples.

Curating a fine-tuning set Sequence

Guide
Write a style and labelling guide; consistency beats volume.
Review
Hand-check a sample; label noise is learned.
De-duplicate
Remove exact and near duplicates, and overlap with eval items.
Clean
Strip personal data; record provenance and licence.
Split
Validation for early stopping, separate from the locked eval set.

After the fine-tune2 minWhy safety and refusal tests are part of every adaptation release.

When an expert would not fine-tune, even with data in hand Overview

Facts that change
Each policy update would need a new run, and old values stay in the weights.
Answers must cite
Weights cannot point to a source; retrieval can.
No baseline yet
Without a frozen set and best-prompt score, no gain can be shown.
No owner
A tuned model needs retests on every base or data change; someone must own them.
Safety unmeasured
Tuning can weaken refusals; without a refusal suite you cannot see it.

Interview nuancesadvanced2 minFour habits that separate a staff-level answer to a fine-tuning proposal from a textbook one.

What a staff-level answer adds Overview

Names the gap
Asks for tagged failures before discussing any method.
Sizes the evidence
States the interval of the baseline and what gain the set can detect.
Pairs the comparison
Compares systems item by item on one frozen version, never across versions.
Audits the claim
Asks when the set was written and what the overlap screen found.

Memory anchor

Eval set and baseline first; the missing capability picks the lever; overlap voids the score.

Useful for

Settling an adaptation debate with failure counts instead of preference, before training time or budget is spent (a hospital's discharge-letter assistant, a retailer's product-copy model).
Writing a one-page adaptation decision record that names the gap, the evidence, the cost and what would change the decision.
Auditing a vendor or colleague claim of "fine-tuned improvement" for overlap, interval and baseline strength.

Not suitable for

Settling a safety or fairness question; an accuracy-style evaluation set does not measure harm, which needs dedicated tests (Day 45).
Very early exploration with no real traffic; first collect examples by using a prompt, then freeze them.

Traps

Building the evaluation set after seeing the adapted model

An HR team building a policy-question assistant chooses or tunes items until the new model looks good, or writes them by reading its outputs. The evaluation then measures agreement with that model, and the "improvement" disappears on real traffic.

How to avoid it

Freeze version 1 and the baseline before any adaptation, keep a locked portion for final decisions, and publish changes as new versions.

Fine-tuning to inject changing facts

An online retailer fine-tunes on last season's returns policy. Weights absorb the facts as of the training date, learn them slowly, can hallucinate more around them and cannot cite sources. The next policy change requires another training run.

How to avoid it

Put volatile facts in retrieved context or a tool; use fine-tuning for behaviour and format that a good prompt cannot hold.

Senior reasoning

A senior will not discuss weights until there is a baseline, whether the assistant answers bank customers, claims handlers or field engineers. They ask for the failure tags, ask what the cheapest lever is for each, and demand that a fine-tune beat the best prompt on a frozen set with an interval, after de-duplication and net of serving cost. They also plan for after launch: who owns the model, when it is retrained, and which safety and regression tests run on every new version.

Interview question

Leadership at a retail bank asks you to "fine-tune a model on our compliance documents" so the assistant stops misquoting rules. How do you respond in the first meeting?

Outline your answer first.

Recall cards scheduled after completion

Which lever supplies which missing capability?
What must exist before any prompt, retrieval or weight change, and what makes it trustworthy?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Optimizing LLM Accuracy (OpenAI API docs) — OpenAIcore
Frames context optimisation (missing or changing knowledge) against model optimisation (consistent behaviour), recommends starting from prompting plus an evaluation set, and notes that the techniques combine.
Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs — Oded Ovadia, Menachem Brief, Moshik Mishaeli, Oren Elisha, 2024core
Compared unsupervised fine-tuning with retrieval for injecting knowledge; retrieval was consistently better for both seen and new knowledge in the settings studied.
Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? — Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, Jonathan Herzig, 2024optional
Examples that introduce new knowledge are learned more slowly during fine-tuning, and learning them increases the tendency to hallucinate.
Training language models to follow instructions with human feedback — Long Ouyang et al. (20 authors, OpenAI), 2022optional
The supervised fine-tuning stage on human demonstrations that precedes preference tuning, the basis for the term instruction tuning here.
Don't Stop Pretraining: Adapt Language Models to Domains and Tasks — Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Noah A. Smith, 2020optional
Continued pretraining on domain text improves in-domain tasks; supports the note on domain adaptation.
Deduplicating Training Data Makes Language Models Better — Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini, 2022optional
Documents near-duplicates and train-validation overlap in language-model datasets, and that deduplication makes evaluation more trustworthy.
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark — Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, Eneko Agirre, 2023optional
Argues that evaluation contamination should be measured for each benchmark and model, since leaked test data inflates scores.
Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! — Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, Peter Henderson, 2024optional
Fine-tuning an aligned model, even on benign data, can degrade its safety behaviour; motivates rerunning safety tests.
Time Travel in LLMs: Tracing Data Contamination in Large Language Models — Shahriar Golchin, Mihai Surdeanu, 2024optional
Proposes guided-instruction prompting to detect instance- and partition-level contamination in a model whose training data cannot be inspected; supports the note that detection methods exist.
With Little Power Comes Great Responsibility — Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, Dan Jurafsky, 2020optional
Shows that many NLP comparisons are underpowered for the effect sizes they report; supports sizing the evaluation set before a run.
Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations — Evan Miller, 2024optional
Recommends standard errors, paired question-level comparisons and power analysis when comparing models on a question set.

Optional sections

Deep dive · 10 min

Stages of building a model. Pretraining trains on next-token prediction over a very large corpus (Day 42). Supervised fine-tuning continues training on prompt-response pairs, and is called instruction tuning when the pairs are instructions with desired responses (Ouyang 2022). Preference tuning (RLHF, DPO; Day 45) then shifts which responses are favoured. Domain adaptation by continued pretraining on a large unlabelled corpus (Gururangan 2020) changes vocabulary and style; consider it mainly when you hold a large unlabelled in-domain corpus (insight-0544).

Curating a fine-tuning set. Prefer a few hundred to a few thousand consistent, correct examples over a large noisy pile: label noise is learned. Write a style and labelling guide, review a sample by hand, remove exact and near duplicates (n-gram or embedding similarity), remove personal data, and record the provenance and licence of every source. Split a validation part for early stopping that is separate from the locked evaluation set.

Contamination, three layers. (1) Training-to-eval overlap inside your own project, which you can and must measure. (2) Few-shot examples copied from the eval set, which also inflates scores. (3) Pretraining contamination: public benchmark items may be in a base model's corpus, which you usually cannot inspect for closed models; treat public benchmark scores as possibly inflated and rely on a private held-out set (Sainz 2023). Detection methods exist (Golchin 2024) but are probabilistic.

How big must the evaluation set be? The half-width of a pass-rate interval is about $1.96\sqrt{p(1-p)/n}$ : roughly 14 points at $n=50$ , 10 at $n=100$ and 4.4 at $n=500$ for $p=0.5$ . Comparing two systems on the same items with a paired analysis is more efficient than comparing their separate intervals (Miller 2024). These are planning numbers; compute your own.

Retrieval project link (weekend studio). The Saturday retrieval milestone applies this rule directly: about 30 queries, graded evidence labels and at least five unanswerable queries are written and versioned before any retrieval run, and the baseline is recorded before anything is tuned. Sunday's label audit, contamination check and adaptation decision record are today's steps 1, 5 and 6 on that project.

Interview practice · 10 min

A teammate fine-tuned a model for an insurer's claims inbox and shows a chart: accuracy 62% before, 91% after. What questions do you ask before accepting it?

Answer as you would in the interview.

Content version 7 · last verified 2026-10-18 · depends on numpy

1 Recall · 5 min

2 Concept · 10 min

Assumptions

3 Application and workplace decision · 10 min

4 Mini-assessment · 8 min

Technical depth · optional · 27 min in total

Useful for

Not suitable for

Traps

Senior reasoning

Interview question

Recall cards scheduled after completion

Prerequisites and related concepts

References

Optional sections

Recall · 5 min

Concept · 10 min

Application and workplace decision · 10 min

Mini-assessment · 8 min