LLM Evaluation and Alignment
By the end: You design a ten-case task-specific evaluation with checkable criteria, run a pairwise judge in both orders and read position, verbosity and contamination effects from recorded outputs, and state which claims about RLHF and DPO are established and which are still contested.
At a glance
Evaluate on your own checkable cases, treat a model judge as a biased instrument to calibrate, and gate releases on regressions and safety, not on a leaderboard.
Your cases
Benchmarks answer someone else's question and may be contaminated.
Code before judges
Format, ids, citations and arithmetic are checked in code; judge only the rest.
Judge both orders
Flips are ties; control length; use another family; validate on humans.
Regression gate
Every production failure becomes a case; safety slices block release.
RLHF and DPO
Preference tuning shapes answers toward what raters and judges like, often longer.
Recall
No reviews due? Answer from memory. (1) What must exist before any adaptation run, and what does a hash on it protect (Day 43)? (2) Why is a 10-point gain on 40 items not yet evidence (Day 43)? (3) What does a grouped split protect against in text data (Day 41)?
Concept
A public benchmark says how a model does on someone else's questions, possibly ones it saw in training. Your decision needs your own cases with checkable criteria. When a second model grades, it is an instrument with habits: it favours the first slot, the longer answer and its own family. Measure the instrument before trusting its numbers.
Predict first
A judge agrees with humans on 81% of pairs in each order. Can you trust a candidate win rate computed from one order?
Text description of the figure
Controls for one of three synthetic judges, the order used (candidate shown first, baseline shown first, or both orders keeping only consistent verdicts) and a toggle that removes items flagged as contaminated. Readouts show the first-position win rate, the share of verdicts that flip when order is swapped, agreement with the reference labels, and the candidate's win rate by the judge and by the labels. A table splits all, clean and contaminated items. The fixture is simulated; no real judge is involved.
Your cases
Write cases from your traffic with a check for each.
Code first
Anything a program can verify needs no judge.
Calibrate the judge
Both orders, length control, other family, human agreement.
Gate releases
Regression and safety slices run on every change.
- Golden setcases, checks, slices
- System outputsbaseline and candidate
- Checkable by code?
- yes: to Code checks
no - Judge human-validated?
- yes: to Pairwise judge
not yet - Human labelssample of pairs
- Code checksschema, ids, citations, sums
- to Rates with intervals
- Pairwise judgeboth orders, flips = ties
- Rates with intervalsper slice, paired
- Release gateregression + safety
A judge that has not been checked against human labels sends you back to label a human sample first. Everything a program can verify skips the judge entirely.
Equal agreement with humans in each order (0.81, 0.82) hid errors that lean toward the first slot: one order can swing the win rate by tens of points.
- judge, candidate shown first
- judge, baseline shown first
- both orders, flips = half
With no pull both orders sit near the human rate (0.52). At pull 0.7 the candidate-first order reports 0.68 and the other 0.35; judging both orders and splitting flips stays at 0.51.
Agreement with humans in one order falls slowly (0.83 to 0.80 at pull 0.7) while the flip rate climbs to 0.37. A per-order agreement number cannot reveal the bias; the swap test can.
Simulated judge with the application’s scoring rule (not a real model), 600 seeded pairs per point; only the position pull varies. A TypeScript re-implementation, so its rates differ slightly from the Python output.
The application's simulated judge with only the first-slot bonus varied. Per-order win rates split apart; the both-order rate does not move.
- Judge verdictmeant to track quality alone
- tracks the slotPosition biasverdict flips when A and B swap
- controlJudge both ordersflips count as ties
- tracks lengthVerbosity biaslonger wins more than humans say
- controlLength controllength-controlled comparison or a cap
- tracks the authorSelf-preferenceown family's outputs rated higher
- controlJudge from another family
- tracks its habitsUnvalidated rubricmeasures the judge, not the task
- controlAgreement with human labelskappa on a labelled sample
- tracks the slotPosition biasverdict flips when A and B swap
Sources: position (Zheng 2023, Wang 2024), verbosity (Dubois 2024), self-preference (Panickssery 2024). Swapping order cancels only the first branch; each other branch needs its own control.
- RLHF: SFT model
- to Human pairwise rankings
- DPO: SFT model
- to Chosen vs rejected pairs
- Human pairwise rankings
- to Reward model
- Chosen vs rejected pairs
- direct: to Classification loss
- Reward modelfits the preferences
- RL (PPO) + KL penaltystay near the reference
- to Tuned model
- Classification losschosen vs rejected, relative to the reference
- to Tuned model
- Tuned model
- Tuned model
The difference is the missing middle: DPO trains directly on chosen and rejected pairs, with no reward model and no RL loop. Christiano 2017 and Ouyang 2022 (RLHF); Rafailov 2023 (DPO). Which works better is setup-dependent (Xu 2024).
- Build a Golden set of your cases, check with code first, and judge the rest as a Pairwise comparison in both orders (Position bias).
- An LLM-as-judge needs a length control, another model family and agreement with human labels.
- A Regression suite with safety slices gates every prompt, model or data change.
- RLHF and DPO tune toward preferred answers, which also shapes what judges like.
Assumptions
The task cases represent real traffic and slices; a set written by the builders under-states hard inputs.
Judge criteria are written down and a human sample defines "agreement"; otherwise the judge is an opinion generator.
The fixtures are simulated or hand-written; real judges are noisier and differ by model and task.
Application and workplace decision
- 200 answer pairs from a baseline and a candidate assistant; humans labelled which is better.
- A simulated judge (not a real model) with a built-in pull toward the first slot and the longer answer ran once with each system first.
- Predict first: if the judge agrees with humans about 80% of the time in each order, what win rate does it report in each order?
Python
import numpy as np
rng = np.random.default_rng(45)
n = 200
better = rng.integers(0, 2, n) # human-preferred system per pair: 0 = baseline, 1 = candidate
longer = rng.integers(0, 2, n) # which system wrote the longer answer
def judge(first):
"""Simulated judge (not a real model): quality + pull towards the first slot and the longer answer + noise."""
score = np.zeros((n, 2))
for s in (0, 1):
score[:, s] = (better == s) * 1.0 + 0.5 * (longer == s) + 0.7 * (first == s) + rng.normal(0, 0.6, n)
return score.argmax(1) # id of the system picked
v_base_first = judge(np.zeros(n, int)) # baseline shown first
v_cand_first = judge(np.ones(n, int)) # candidate shown first
agree = lambda v: (v == better).mean()
print(f"agreement with humans: baseline-first {agree(v_base_first):.2f}, candidate-first {agree(v_cand_first):.2f}")
print(f"candidate win rate: baseline-first {v_base_first.mean():.2f}, candidate-first {v_cand_first.mean():.2f}, human {better.mean():.2f}")
both = (v_base_first == 1) & (v_cand_first == 1) # candidate wins in both orders
neither = (v_base_first == 0) & (v_cand_first == 0) # baseline wins in both orders
print(f"both orders: candidate wins {both.mean():.2f}, baseline wins {neither.mean():.2f}, flips counted as ties {1 - both.mean() - neither.mean():.2f}")
decided = both | neither
print(f"agreement with humans on decided pairs: {(both[decided] == (better[decided] == 1)).mean():.2f} ({decided.sum()} of {n} pairs)")
picks = np.r_[v_base_first, v_cand_first] == np.r_[longer, longer]
print(f"longer answer picked: judge {picks.mean():.2f}, human preference {(better == longer).mean():.2f}")
Mini-assessment
Technical depth
A pass rate as a binomial estimate, by handWhat a pass rate estimates, its standard error derived, and a 95% interval computed on 17 of 20 cases two ways.
- the true pass rate on the population of cases your set is drawn from: the quantity you want
- cases passed and cases run; each case is one independent pass-or-fail trial
- unbiased for p; its variance p(1-p)/n follows from the variance of a sum of independent trials
- largest at p = 0.5; quadrupling n halves it
Wald interval: p-hat plus or minus 1.96 SE. It breaks near 0 and 1 and on small n, where the Wilson interval (Day 43) is the default. A pairwise win rate is the same estimator on pairs, with each flip scored as half a win.
| Step | Arithmetic | Result |
|---|---|---|
| Estimate | 17 / 20 | p-hat = 0.85 |
| Standard error | sqrt(0.85 x 0.15 / 20) = sqrt(0.006375) | 0.080 |
| Wald 95% | 0.85 +/- 1.96 x 0.080 = 0.85 +/- 0.156 | 0.69 to 1.01: past 1, impossible |
| Wilson centre | (0.85 + 1.96^2 / 40) / (1 + 1.96^2 / 20) = 0.946 / 1.192 | 0.794 |
| Wilson half-width | 1.96 x sqrt(0.006375 + 1.96^2 / 1600) / 1.192 | 0.154 |
| Wilson 95% | 0.794 +/- 0.154 | 0.64 to 0.95 |
Checked in Python. Twenty cases cannot tell an 85% assistant from a 65% one. Independence is assumed: ten cases from one customer, template or document are closer to one trial than ten, so cluster by source or the interval is too narrow.
Agreement beyond chance: Cohen's kappa, derived and computedadvancedWhy raw agreement flatters a judge, kappa derived from chance agreement, and two worked 2 x 2 tables that show the base-rate effect.
- share of items on which judge and human give the same label
- agreement expected if the two labelled independently, each at its own base rate per class c
- agreement above chance
- the most agreement above chance that was possible, so kappa is the share of it achieved
- 1 is perfect agreement, 0 is chance level, below 0 worse than chance
Raw agreement rewards a grader for sharing the other grader's base rate; kappa removes that part. Skewed classes make high kappa hard to reach (Artstein and Poesio 2008).
| Judge label ↓ | A better | B better |
|---|---|---|
| A better (judge 30) | 20 | 10 |
| B better (judge 20) | 5 | 15 |
Diagonal = agreement: p_o = (20 + 15) / 50 = 0.70. Judge says A on 0.60, the nurse on 0.50, so chance agreement p_e = 0.60 x 0.50 + 0.40 x 0.50 = 0.50 and kappa = (0.70 - 0.50) / (1 - 0.50) = 0.40. Checked with scikit-learn's cohen_kappa_score: 70% raw agreement is only 40% of the way from chance to perfect.
Counts
50 claim letters: both say acceptable 43, judge only 3, human only 2, both say not acceptable 2.
Observed
p_o = (43 + 2) / 50 = 0.90.
Base rates
Judge says acceptable on 46 of 50 (0.92), the human on 45 (0.90).
Chance
p_e = 0.92 x 0.90 + 0.08 x 0.10 = 0.836: two graders who almost always say yes agree by default.
Kappa
(0.90 - 0.836) / (1 - 0.836) = 0.064 / 0.164 = 0.39.
Read it
90% raw agreement here is no better than 70% on the balanced task. Report kappa and agreement on the rare class.
Checked with cohen_kappa_score. Rare failures are exactly what a release gate is for, so a judge validated on raw agreement can miss most of them.
- Two humans label50-100 pairs, one rubric
- to Human-human kappa
- Judge labels themsame pairs, both orders, per slice
- to Judge-human kappa
- Human-human kappathe ceiling the task allows
- to Near the ceiling?
- Judge-human kappabootstrap interval over items
- Near the ceiling?
- no: to Change the judge
yes - Use it on this taskre-validate after a judge update
- Change the judgebetter prompt or another model
Kappa from 50 items is itself an estimate, so report its interval (a bootstrap over items is the simple way) before calling a judge validated. A judge validated on one task does not transfer to another.
How many pairs, and how sureThe spread of a win rate at 50, 200 and 800 pairs, and the planning formula for a margin, inverted from the standard error.
Seeded binomial draws; red dots are runs in which the truly better candidate does not look better. Pairs are treated as independent; ties and flips would widen the spread further.
Same candidate, same true win rate (0.55), 60 evaluations per size. With 50 pairs, 10 of 60 runs show the candidate losing or tied (observed rates 0.36 to 0.72); with 800 pairs, 1. The interval halves each time the set grows four-fold.
Every dot is one honest evaluation of a candidate with a true win rate of 0.55. To compare two systems, score the same items and report the paired difference with an interval (Miller 2024).
- the margin you want either side (0.05 for plus or minus 5 points)
- expected win rate; 0.5 is the worst case
- 1.96 for 95%
The standard error solved for n: set z x sqrt(p(1-p)/n) equal to the margin m. Worst case: about 385 pairs for plus or minus 5 points, about 97 for plus or minus 10 (computed). Flips counted as ties and clustered items widen it.
Writing task casesThe fields of a golden-set case and three test types that find real failures.
| Field | Example |
|---|---|
| input | A policyholder asks whether water damage from a burst pipe is covered |
| check | Exact intent, schema, cited chunk id, or a written rubric |
| slice | intent, language, refusal, unanswerable |
| source and date | Production log, author, date written |
| split | development or locked held-out |
Minimum functionality
Simple cases the system must pass, such as a plain refund request.
Invariance
A change that must not alter the answer: a name, a typo.
Directional
A change that must move the answer one way: add a legal threat, expect escalation.
Choosing the graderCode, human or model judge, and what each one misses.
| Criterion | Code | Human | Model judge |
|---|---|---|---|
| Best for | Schema, allowed values, ids, citations, arithmetic | Reference labels, tone, policy nuance | Open-ended quality at scale, after validation |
| Cost per item | Near zero | High | Moderate; doubled for both orders |
| Main risk | Passes valid but wrong content | Inconsistent raters; write criteria | Position, length and self-preference bias |
Judging pairs without fooling yourselfConsistent verdicts, flips as ties and the length pull that survives swapping.
Swapping order does not remove a length pull; a length control does (Dubois 2024).
The candidate is no better than the baseline but writes the longer answer in 74% of pairs. Swapping removes the slot effect and still leaves a length bonus; comparing like with like (candidate longer: 0.65, candidate shorter: 0.36, weighted equally) removes most of it. Humans: 0.53 and 0.50 in the same groups.
Seeded simulation with the application’s judge rule (not a real model). The length control here is a simple stratification; Dubois et al. 2024 use a regression-based control. Real length effects differ by judge and task.
A candidate that is no better but usually longer, scored with one control added at a time.
Contamination screens and release gatesReport clean and flagged cases separately and let a safety slice block a release.
Report both. Prefer private cases or cases written after the model's data cutoff (Sainz 2023).
| Slice | Old | New | Decision |
|---|---|---|---|
| Fare rules | 9/10 | 10/10 | ok |
| Baggage | 8/10 | 9/10 | ok |
| Refusal / safety | 10/10 | 8/10 | BLOCK release |
Two task slices improved and one safety slice regressed; a gate blocks on the safety slice whatever the average.
RLHF and DPO, the objectiveadvancedWhat DPO minimises and why optimising a proxy reward eventually hurts.
- chosen and rejected answers to prompt x
- frozen reference model (usually the SFT model)
- strength of the implicit KL penalty; larger keeps the policy closer to the reference
- logistic function; a zero margin gives loss log 2 = 0.693
In the optional lab a margin of 0.150 gives loss 0.621. Rafailov et al. 2023.
Margin
beta times how much more the policy favours the chosen answer than the reference does.
Zero margin
Policy equals reference: loss log 2, the steepest learning signal.
Weighting
Each pair's gradient is scaled by sigma(-margin): pairs already ranked right count less.
Beta
Small beta lets the policy drift further from the reference before the loss saturates.
Watch
Track length, diversity and a held-out human metric, not the loss alone.
Rafailov et al. 2023; the weighting follows from differentiating the loss in the equation above. Re-measure diversity after tuning: RLHF generalised better than SFT out of distribution but reduced output diversity (Kirk 2024).
Rises with every step of optimisation, to 3.89 at best-of-256. Read alone, it says: keep going.
Peaks at best-of-8 (-0.07, KL 1.20 nats) and falls to -0.57 at best-of-256: the most optimised answer is not the best one.
The proxy pays for length, so selection buys longer answers (+1.62 sd). Tracking length and a held-out gold metric next to the reward exposes the drift.
Best-of-n computed exactly over a seeded pool of 20,000 sampled answers (n = 1 to 256, KL = ln n - (n - 1) / n). The proxy and gold formulas are constructed to show the mechanism Gao et al. 2023 report (proxy up, gold up then down) and the length drift Singhal et al. 2023 describe; the numbers are not from either paper.
Best-of-n selection against a proxy reward that partly pays for length. The mechanism, not the paper's numbers.
Established versus evolvingWhich claims about judges and preference tuning are settled and which are still argued.
- Judge position biasZheng 2023, Wang 2024
- Contamination inflates scoresSainz 2023
- Length drift under RLHFSinghal 2023
- Reward over-optimisationGao 2023
- Judges replacing humanstask-dependent
- DPO versus tuned PPOXu 2024: setup-dependent
High agreement
Agreement in each order can stay high while win rates swing with the slot.
Swap fixes every bias
It cancels position, not length or self-preference.
Higher reward
A proxy rises past the point where held-out human scores fall.
Leaderboard rank
Public sets may be contaminated and answer someone else's question.
Judges replace humans
Only on a task where agreement with human labels was measured.
Memory anchor
Judge both orders, control length, validate against humans, screen for overlap, gate on a regression suite.
Useful for
Turning a vague "is it better" into a task-specific, versioned evaluation with checks, slices and intervals (an insurer's claim-letter drafts, a retailer's product Q&A).
Auditing a judge-based win rate or a leaderboard claim for order, length, family and overlap effects.
Building a regression suite that gates prompt, model and data changes, including safety and refusal slices (a clinic's patient-message assistant must never give dosing advice).
Not suitable for
Replacing human judgement on high-stakes or novel criteria; a judge validated on one task does not transfer to another.
Settling safety by an accuracy-style pass rate; harm testing needs dedicated adversarial and policy test sets.
Traps
Reporting a judge win rate from one order
An e-commerce team scores its product-question assistant with one judge order. The judge's pull toward the first slot or the longer answer decides the result, the ranking reverses when the order is swapped, and the team ships on an artefact.
How to avoid it
Judge in both orders, count consistent verdicts, treat flips as ties, control length and validate against a human sample from the same task.
Tuning prompts on the evaluation set
An HR policy assistant's prompt is iterated forty times against the same 30 cases, which it ends up fitting. The score rises without a real gain and the held-out behaviour is worse; the set no longer measures anything independent.
How to avoid it
Keep a development set for iteration and a locked held-out set for decisions, record versions, and refresh with new production failures.
Senior reasoning
A senior distrusts a number until the instrument is checked, whether it scores a bank's mortgage assistant or a telecom's troubleshooting bot. They ask what the cases are, how "better" was defined, which judge, in which order, with what length control, validated against which human labels, and whether any case could have been seen in training, and they read agreement as kappa, not raw percent. They separate claims that are established from claims still debated, and they turn every failure into a case that blocks the next regression.
Interview question
How would you evaluate whether a new prompt for a bank's mortgage-questions assistant is better than the current one?
Recall cards scheduled after completion
- Name three biases of an LLM judge and one control for each.
- How do RLHF and DPO differ, and what is still contested?
Prerequisites and related concepts
References
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacore
Documents position, verbosity and self-enhancement biases of LLM judges and their agreement with human preferences on MT-Bench and Chatbot Arena.
- Large Language Models are not Fair Evaluatorscore
Shows pairwise verdicts can flip when answer order is swapped and proposes order-balanced calibration.
- LLM Evaluators Recognize and Favor Their Own Generationsoptional
LLM evaluators rated their own outputs higher than others' outputs that humans judged equal; supports using a judge from another family.
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorsoptional
Automatic evaluators prefer longer outputs; a regression-based length control made the metric harder to game.
- Holistic Evaluation of Language Modelsoptional
Multi-metric, multi-scenario evaluation (accuracy, calibration, robustness, fairness, efficiency) as an argument against a single leaderboard number.
- NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcore
Benchmark test data leaking into pretraining corpora inflates scores; argues for measuring contamination per benchmark and model.
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationsoptional
Recommends standard errors, paired question-level comparisons and power analysis for model evaluations.
- sklearn.metrics.cohen_kappa_score (scikit-learn documentation)core
Defines Cohen's kappa as (p_o - p_e) / (1 - p_e) with chance agreement p_e; used to check both worked agreement tables.
- Survey Article: Inter-Coder Agreement for Computational Linguisticsoptional
Survey of chance-corrected agreement coefficients, including how skewed category distributions make high kappa hard to reach; supports the skewed-task example.
- Beyond Accuracy: Behavioral Testing of NLP Models with CheckListoptional
Capability-by-test-type design (minimum functionality, invariance, directional expectation) for building task tests.
- Training language models to follow instructions with human feedbackcore
Supervised fine-tuning, reward model and PPO pipeline from human feedback, the practical template for RLHF.
- Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcore
Optimises preferences directly with a classification-style loss, without an explicit reward model or RL loop.
- Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studyoptional
In the dialogue and code tasks studied, carefully tuned PPO beat DPO and DPO was sensitive to distribution shift; evidence that the comparison is setup-dependent.
- Scaling Laws for Reward Model Overoptimizationoptional
Optimising a proxy reward model eventually lowers the gold reward; supports tracking a held-out metric during preference tuning.
- A Long Way to Go: Investigating Length Correlations in RLHFoptional
Much of the reward improvement in RLHF came from longer outputs; supports checking whether a preference gain reflects length alone.
- Deep Reinforcement Learning from Human Preferencesoptional
Learns a reward model from pairwise human comparisons, the conceptual origin of the RLHF step described in the mechanism.
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!optional
Fine-tuning an aligned model, even on benign data, can degrade its safety behaviour; supports gating releases on safety slices.
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackoptional
Survey of open problems and limitations of RLHF across human feedback, the reward model and the policy; supports the failure sites named in the deep dive.
- Understanding the Effects of RLHF on LLM Generalisation and Diversityoptional
RLHF generalised better out of distribution than supervised fine-tuning but substantially reduced output diversity; supports re-measuring diversity after preference tuning.
Optional sections
Deep dive
Benchmarks versus task evaluations. A public benchmark is useful for screening and for tracking a model family; it answers a question about its own distribution. HELM argues for many metrics across many scenarios rather than one number (Liang 2023). For selection you need your own cases, because ranking can differ on your inputs and because benchmark text may be in a model's training data.
Judge validation, step by step. Write the rubric, label 50-100 pairs by humans (two raters on a sample to estimate rater agreement), run the judge in both orders, and report agreement per slice, with an interval. A judge whose agreement with humans is no higher than humans' agreement with each other is at the ceiling; a judge below that needs a better prompt or a different model. Re-validate after a judge model update.
RLHF in more detail. Step 1: supervised fine-tuning on demonstrations. Step 2: sample several responses per prompt, have people rank them, and fit a reward model to the rankings. Step 3: optimise the policy to maximise reward minus a penalty on divergence from the reference policy. Failure sites are the human feedback (noisy, biased, inconsistent), the reward model (a proxy that can be exploited) and the policy optimisation (Casper 2023). Over-optimising the proxy lowers the true reward after a point (Gao 2023).
DPO loss. For a prompt with chosen and rejected , with reference policy and scale : . The loss is when the policy equals the reference and falls as the policy raises the relative likelihood of the chosen answer. The lab computes one example.
Side effects to measure after preference tuning. Output length (Singhal 2023), diversity (Kirk 2024) and core capabilities (an "alignment tax") should be re-measured on your task suite, not assumed.
Retrieval project link (weekend studio). Sunday's contamination check computes the longest shared word sequence between each query and its labelled chunks and flags queries that copy source wording or were written after seeing retrieval output; that is the overlap screen of step 5 applied to your own queries. The same session asks for two ways a model judge can be biased, which are position and length effects from this lesson.
Hands-on lab: Build a 10-case eval, screen it for overlap, gate a release and compute one DPO loss
Run the code. Part 1 evaluates ten hand-written fare-rule cases with a programmatic check and a crude overlap screen against a public text. Part 2 is a release gate on slices. Part 3 computes the DPO loss for one preference pair with made-up log-probabilities. Then add two cases of your own, one that should be refused and one with no answer in the knowledge base; rewrite c01 so the overlap screen no longer flags it and say whether the case got easier or harder; and change the margin in Part 3 so the loss falls below 0.5.
import math
# Part 1: a 10-case task-specific eval with a programmatic check and a contamination screen
public_text = ("the free cancellation window is 24 hours after booking on flex fares. "
"passengers may carry one cabin bag of up to 8 kg on basic fares. "
"online check in closes 2 hours before departure for all domestic flights.")
cases = [ # (id, prompt, required substring, recorded answer from a hypothetical system)
("c01", "What is the free cancellation window on flex fares?", "24 hours", "24 hours after booking"),
("c02", "How heavy may a cabin bag be on basic fares?", "8 kg", "up to 8 kg"),
("c03", "When does online check in close for domestic flights?", "2 hours", "2 hours before departure"),
("c04", "Can I change the name on a flex ticket after booking?", "not", "Yes, any time."),
("c05", "Is a refund due if the airline cancels my flight?", "refund", "You are due a refund."),
("c06", "Does the basic fare include a checked bag?", "no", "No, only a cabin bag."),
("c07", "What happens to my return leg if I skip the outbound?", "cancel", "It may be cancelled."),
("c08", "Can a minor travel alone on a domestic flight?", "yes", "Yes, with a guardian form."),
("c09", "Is lounge access included on flex fares?", "no", "No."),
("c10", "How early should I arrive for an international flight?", "3 hours", "About 2 hours."),
]
pub_words = set(public_text.replace(".", "").split())
def overlap(q): # share of the question's words that also occur in the public text
w = q.lower().replace("?", "").split()
return sum(x in pub_words for x in w) / len(w)
leaky = {cid for cid, q, *_ in cases if overlap(q) >= 0.6}
ok = {cid: req.lower() in ans.lower() for cid, _, req, ans in cases}
clean = [c for c in ok if c not in leaky]
print(f"pass rate all {sum(ok.values())}/10; flagged as public-text overlap {sorted(leaky)}; clean-only {sum(ok[c] for c in clean)}/{len(clean)}")
# Part 2: regression gate on slices (old vs new version, hypothetical recorded pass counts)
old = {"fare rules": (9, 10), "baggage": (8, 10), "refusal / safety": (10, 10)}
new = {"fare rules": (10, 10), "baggage": (9, 10), "refusal / safety": (8, 10)}
for slice_, (k, n) in new.items():
drop = old[slice_][0] - k
print(f"{slice_:<17} {old[slice_][0]}/{n} -> {k}/{n} {'BLOCK release' if drop > 0 and 'safety' in slice_ else 'ok' if drop <= 0 else 'review'}")
# Part 3: the DPO loss for one preference pair, with made-up log-probabilities (illustration only)
beta = 0.1
lp = {"chosen": (-12.0, -13.0), "rejected": (-15.0, -14.5)} # (policy, frozen reference) sequence log-probs
margin = beta * ((lp["chosen"][0] - lp["chosen"][1]) - (lp["rejected"][0] - lp["rejected"][1]))
print(f"DPO margin {margin:.3f}, loss {-math.log(1 / (1 + math.exp(-margin))):.3f} (log 2 = {math.log(2):.3f} at margin 0)")
Interview practice
A vendor pitching to a hospital's patient-messaging team claims their model "beats" yours by 8 points on a leaderboard and by 70% win rate under a GPT-style judge. How do you decide whether to switch?