Core day 50 of 60 · Week 10 · Part II · evaluation depth

GenAI Application Lab

A colleague hands you last night's evaluation of an insurer's policy-question assistant prototype: retrieval looks strong, the answers read well and one tool task is marked passed. What do you inspect before anyone calls it ready, and which single experiment do you propose next?

By the end: From a recorded evaluation result you compute recall@k with an interval, a retrieval-by-answer table, slice results, cost per supported answer, repeated-run consistency and attack success, say which claims the data does and does not support, and propose one pre-specified experiment.

At a glance

Read an LLM application's evaluation layer by layer, with sample sizes and intervals, then pick one pre-specified experiment on the largest fixable failure.

Cases first
Who wrote the queries, how many, which slices, which versions, what is held out.
Locate the failure
Recall by slice, then the retrieval-by-answer table.
Consistency and attacks
All-runs-pass beside per-attempt; injection obeyed beside utility.
Cost per supported answer
Divide spend by answers that were correct and supported, not by requests.
One experiment
Largest fixable bucket, one change, expected movement written first.

Recall · 5 min

If no cards are due, answer from memory before today's block: name the stages of a RAG pipeline in order with one failure symptom each, three stopping conditions for an agent loop, and why a schema-valid output may still not be allowed to act.

Open due reviews

Concept · 10 min

By Day 50 the parts exist: retriever, context assembler, validated model output, a loop with limits. One headline accuracy hides which layer fails, and reading an evaluation is a separate skill from running one. Ask, in order: what are the numbers based on, where do failures sit, what could be wrong with the evaluation, and which cheap experiment would change the decision?

Recall@k moves with method, k and chunking, and a small query set cannot separate close methods

Predict first

Two configurations differ on three of fourteen queries, all in favour of B. What does an exact sign test say about "B is better"?

Text description of the figure

Controls for retrieval method (BM25, TF-IDF, LSA, hybrid RRF), chunk size, metric, query style and k over 40 labelled queries from the academy's own text. A grid shows the mean metric for every method and chunk size, a list shows the worst queries with a diagnosis (for example the rank at which the best relevant document appears), and a table of candidate next experiments gives measured changes with paired bootstrap intervals, clearly marked upper bounds and the count of unreachable queries. Retrieval only; no answers are generated.

The seven-point inspection order Sequence

Cases
Golden set size, query authors, slices, versions, dev and held-out, unanswerable and injection cases.
Retrieval
Recall@k at the production k, by slice, with an interval.
Answers
Retrieval-by-answer table: which cell holds the failures?
Abstention
Of unanswerable questions, how many were declined?
Tool tasks
End state over repeated runs: all-runs-pass beside per-attempt.
Security
Injected instructions obeyed, beside task utility under attack.
Cost and latency
Median and tail latency; cost per supported answer, prices labelled.

Recall@3 by slice, with the uncertainty drawn Illustration

Recorded synthetic fixture from the application block (gold ranks per query); the n = 50 bars keep the observed rate and only change the sample size, an illustration of interval width, not a prediction.

Thin bars: the recorded run (n = 4 or 5 per slice). Thick bars: the same observed rates if each slice had 50 queries. Only with tens of cases per slice does the paraphrase slice separate from the others.

With four or five queries per slice the 95% intervals overlap almost completely. The same observed rates with 50 queries per slice would separate the paraphrase slice.

Retrieval by answer, 14 queries (application output) Map

1: evidence, wrong answer

Context assembly or generation failed.

9: evidence, correct

Working; keep as regression cases.

3: no evidence, wrong

Retrieval misses: the largest fixable bucket.

1: no evidence, correct

Answered from memory; fails when the fact changes.

Consistency and security (application output) Chart

Tool tasks, per attempt0.75 share
Tool tasks, all 3 runs pass0.5 share
Under attack, task correct0.8 share
Injected instruction obeyed1 of 5 attack cases0.2 share

Per-attempt success overstates reliability; utility under attack says nothing about how many attacks worked.

Inspect seven things in order: cases (Golden set), retrieval (recall@k by slice with an interval), the Retrieval-by-answer table, abstention, tool tasks (All-runs-pass consistency (pass^k)), security (injected instructions obeyed beside utility), and Cost per supported answer with latency.
Then pick the largest fixable bucket on a slice with enough cases, change one thing, write the expected movement first, and judge on held-out cases.

Assumptions

The recorded result is a synthetic fixture standing in for an executed run; every number in this lesson is one the code prints, and none is a measurement of a real system.
Slices of four or five queries are too small to rank methods; they are enough to decide where to collect more data and what to try first.
Prices and latencies are hypothetical placeholders used to show the arithmetic, not provider quotes.

Application and workplace decision · 10 min

One recorded evaluation (a synthetic fixture): 14 answerable queries in three slices, two unanswerable questions, four tool tasks run three times each, five injection cases.
The code computes recall@3 with a Wilson interval, recall per slice, the retrieval-by-answer table, abstention, latency, cost per supported answer, all-runs-pass and attack results.
Before reading on, write three claims the data supports, two it does not, and the one experiment you would run next.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
K, COST_PER_QUERY = 3, 0.012  # context size passed to the model; hypothetical EUR per query
RUN = {  # recorded retrieval run (synthetic fixture): slice -> (gold rank, answer correct, latency ms); rank 0 = not in top 10
    "identifier": [(1, 1, 820), (1, 1, 790), (2, 1, 905), (1, 0, 850)],
    "keyword": [(1, 1, 880), (2, 1, 910), (1, 1, 860), (1, 1, 840), (4, 0, 1210)],
    "paraphrase": [(0, 0, 1180), (0, 1, 1020), (5, 0, 1250), (2, 1, 990), (1, 1, 900)],
}
UNANSWERABLE_ABSTAINED = [True, False]  # two questions whose answer is not in the corpus
TOOL_TRIALS = {"t1": [1, 1, 1], "t2": [1, 0, 1], "t3": [1, 1, 1], "t4": [0, 1, 0]}  # 3 repeated runs per task
ATTACKS = [(1, 0), (1, 0), (1, 1), (0, 0), (1, 0)]  # (task still correct, injected instruction obeyed)
def wilson(hits, n, z=1.96):
    p = hits / n
    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
    half = z * np.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return centre - half, centre + half
rows = [r for v in RUN.values() for r in v]
inctx = [1 <= r[0] <= K for r in rows]
lo, hi = wilson(sum(inctx), len(rows))
print(f"recall@{K}: {sum(inctx)}/{len(rows)} = {np.mean(inctx):.2f}  (95% Wilson interval {lo:.2f}-{hi:.2f})")
for name, v in RUN.items():
    print(f"  {name:<11} recall@{K} {np.mean([1 <= r[0] <= K for r in v]):.2f}  n={len(v)}")
cells = {(i, bool(r[1])): 0 for i in (True, False) for r in rows}
for i, r in zip(inctx, rows):
    cells[(i, bool(r[1]))] += 1
print("evidence in context / answer correct:", {("evidence" if i else "no evidence", "correct" if c else "wrong"): n for (i, c), n in cells.items()})
supported = cells[(True, True)]
total_cost = COST_PER_QUERY * (len(rows) + len(UNANSWERABLE_ABSTAINED))
print(f"abstained on {sum(UNANSWERABLE_ABSTAINED)}/{len(UNANSWERABLE_ABSTAINED)} unanswerable; median latency {np.median([r[2] for r in rows]):.0f} ms")
print(f"cost per supported answer EUR {total_cost / supported:.4f} (per query EUR {COST_PER_QUERY})")
single = np.mean([x for t in TOOL_TRIALS.values() for x in t])
print(f"tool tasks: per-attempt {single:.2f}, all 3 runs pass {np.mean([all(t) for t in TOOL_TRIALS.values()]):.2f}")
print(f"attacks: task still correct {np.mean([a for a, _ in ATTACKS]):.2f}, injected instruction obeyed {np.mean([b for _, b in ATTACKS]):.2f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 29 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The exact sign test, derived and computed by handadvanced7 minWhy only discordant queries count, the binomial null distribution behind the p-value, one test worked by hand, and its link to McNemar's test.

Pairing: every query lands in one of four cells, and only two of them count Hierarchy

One query run through A and Bhit or miss for each
- both hitTie: droppedsays nothing about which is better
- both missTie: droppeda hard query, not a worse configuration
- only B hitsWin for Bcounts toward W
- only A hitsLoss for Bcounts toward n_d - W

The question is whether B succeeds more often on the population these queries were drawn from. Under no difference each discordant query is a fair coin; the p-value is the chance of a split at least as lopsided as the observed one. Comparing two overall rates as independent samples wastes most of this information.

The exact sign test on discordant queries Formula

W \sim \mathrm{Binomial}\!\left(n_d, \tfrac12\right) \text{ under } H_0, \qquad p = \min\!\Big(1,\; 2 \sum_{j=0}^{\min(w,\,l)} \binom{n_d}{j} \Big/ 2^{\,n_d}\Big)

$n_d = w + l$: discordant queries: one configuration succeeds and the other fails; ties are dropped
$H_0$: no difference: each discordant query is equally likely to favour either configuration
$W$: wins for B among the discordant queries, a fair-coin count under H_0
$\sum_{j \le \min(w,l)}$: probability of a split at least as lopsided as the observed one, in one direction
$2 \times$: two-sided: a lopsided split in favour of A would be just as surprising

Conditioning on the discordant count is what removes query difficulty from the comparison. It is the exact McNemar test: statsmodels' mcnemar uses the exact version by default and returns the same p-values (checked in 0.15.0).

Sign test by hand: an online shop's search, B against A on 30 queries (synthetic) Table

Step	Computation	Value
Count	B wins 7, B loses 2, ties 21	n_d = 9
Null distribution	W ~ Binomial(9, 1/2); 2^9 equally likely win/loss sequences	512 sequences
One tail: 2 losses or fewer	C(9,0) + C(9,1) + C(9,2) = 1 + 9 + 36	46 / 512 = 0.090
Two-sided p-value	2 x 46 / 512	0.180
Reading	A 7 to 2 split happens about one time in five with no real difference	Not evidence at 5%

Verified in Python (math.comb and scipy.stats.binomtest give 0.1797). The 21 ties carry no information about which configuration is better, however many there are.

The exact sign test, drawn Illustration

3 wins, 1 loss (4 discordant): two-sided p = 0.625.

6 wins, 2 losses (8 discordant): two-sided p = 0.289.

12 wins, 4 losses (16 discordant): two-sided p = 0.077.

24 wins, 8 losses (32 discordant): two-sided p = 0.007, below 0.05.

as or more extreme than observed (both tails): the p-value
less extreme splits

Exact binomial probabilities computed here (the same test as the hands-on lab). Every panel has the same 75% win share; only the number of discordant queries changes.

Ties are dropped. With no real difference, wins among the discordant queries follow a fair coin; the shaded tails are the p-value. The same 75% win share is noise at 4 discordant queries and a clear signal at 32.

Two configurations on the same 14 queries (optional lab) Table

Slice	A	B
Identifier	4/4	4/4
Keyword	4/5	5/5
Paraphrase	2/5	4/5
Overall	10/14	13/14

B wins 3, loses 0, ties 11. Exact two-sided sign test p = 0.25; six wins with no losses would be needed for p < 0.05. Comparing several configurations on the same queries and keeping the best inflates its score: confirm the winner on held-out queries.

The Wilson interval, derived and computed by handadvanced6 minWhere the Wilson interval comes from, every term computed on one audit, why the plain normal interval fails at small n, and the library default to override.

The Wilson interval for a proportion Formula

\frac{\hat p + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat p(1-\hat p)}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}

$\hat p = h/n$: observed share, for example 10 hits of 14 queries
$z$: 1.96 for a 95% interval
$\frac{z^2}{2n}$: pulls the centre towards 0.5, which keeps the interval inside 0 to 1 at small n
$\sqrt{p(1-p)/n}$: standard error of a share of n independent queries; it depends on the unknown true p
$(\hat p - p)^2 \le z^2\,p(1-p)/n$: the derivation: keep every p a two-sided z-test would not reject. A quadratic in p; its two roots are the Wilson bounds

For 10 of 14 it gives 0.45 to 0.88 (application output). One query more or less moves the estimate by about seven points.

Wilson interval by hand: a clinical-coding assistant right on 17 of 20 notes (synthetic) Table

Step	Computation	Value
Observed share	17 / 20	0.850
Shrinkage term	z^2 / n = 3.8416 / 20	0.192
Centre	(0.850 + 0.096) / 1.192	0.794
Spread	sqrt(0.85 x 0.15 / 20 + 3.8416 / 1600) = sqrt(0.006375 + 0.002401)	0.0937
Half-width	1.96 x 0.0937 / 1.192	0.154
95% Wilson interval	0.794 - 0.154 to 0.794 + 0.154	0.640 to 0.948
Normal (Wald) interval, for contrast	0.85 +/- 1.96 x sqrt(0.85 x 0.15 / 20)	0.694 to 1.006: past 1

Verified against statsmodels proportion_confint with method='wilson' (0.15.0); its default normal method returns the Wald interval clipped at 1.00. The centre moves from 0.85 towards 0.5: that is the Wilson correction at work.

Wilson against the plain normal (Wald) interval Comparison

Criterion	Wilson	Wald (normal)plugs the observed share into the SE
Centre	Pulled from the observed share towards 0.5	The observed share
Stays inside 0 to 1	Always	No: 0.694 to 1.006 for 17 of 20
At 0 of n or n of n	Never zero width	Collapses to a single point
Coverage at small n	Close to the nominal 95%	Often well below 95%

Both assume queries are independent draws from the traffic you care about. Twenty notes from one ward or one coder are not twenty independent draws, and no interval repairs a biased sample.

Slice sample sizes and poweradvanced5 minHow many queries a slice needs for a given precision, computed for two rates, the power of the paired test by query count, and what to write down before the run.

Queries per slice for a target precision Formula

n \approx \frac{z^2\, p(1-p)}{h^2}

$h$: the 95% half-width you can live with, for example plus or minus 0.10
$p$: the rate you expect on the slice; p = 0.5 is the worst case and gives the largest n
$z$: 1.96 for 95%

Solve the normal half-width z sqrt(p(1 - p)/n) = h for n. Halving h needs four times the queries, because precision grows only with the square root of n.

Queries needed per slice: a bank's complaint-routing assistant, by product (illustrative) Table

Expected rate on the slice	Half-width 0.20	Half-width 0.10	Half-width 0.05
0.5 (worst case)	25	97	385
0.8	16	62	246
Check at 50 of 62 (Wilson)	-	0.69 to 0.89	-
Five queries per slice, as in the application	half-width about 0.44 at p = 0.5	-	-

Computed in Python, rounded up. Cards, mortgages and savings each need their own count: a slice-level claim needs slice-level n, and the overall n does not transfer to any one product.

How many labelled queries a paired comparison needs Illustration

B fixes 50% of A's misses, breaks 2% of its hits (recall +0.136)
B fixes 30% of A's misses, breaks 5% of its hits (recall +0.055)
B fixes 15% of A's misses, breaks 3% of its hits (recall +0.024)

Seeded simulation, 200 replicate evaluations per point; A has recall 0.70 per query and B's fixes and breaks are independent across queries. Plan the query count from the smallest change you would act on.

Power of the exact sign test at α = 0.05 (two-sided). With 14 queries even the largest effect is detected only 1% of the time. Reaching 80% takes about 100, 400, 1000 queries for the three effects.

Power of the exact sign test for three sizes of true improvement. Fourteen queries almost never detect any of them; plan the query count from the smallest change you would act on.

Write these down before the run Sequence

Hypothesis and slice
Which change, on which slice, moving which metric at the production k.
Smallest change that matters
The improvement that would change the decision, in points of recall or supported answers.
Query count
From a power estimate for that change, not from what is already labelled.
Test and threshold
Paired test on the same queries, two-sided, the significance level and the regression list.
Decision rule
What you ship, revert or collect next for each possible outcome.

Supported and unsupported claims3 minFive claims from the hand-over note, checked against what the numbers can carry.

What this evaluation does and does not support Table

Claim	Supported?	Why
Evidence goes missing on paraphrase queries	Yes, as a lead	0.40 versus 1.00 and 0.80, small slices
Most wrong answers lack evidence	Yes	3 of 4 wrong answers had no evidence in context
Tool use is reliable	No	All-runs-pass 0.50 against 0.75 per attempt
Lexical retrieval is good enough	No	Slices of 4-5 queries cannot rank methods
The injection risk is low	No	1 of 5 obeyed; any successful attack is a defect

What blocks a sign-off regardless of averages Overview

A successful injection
Fix with code-level controls, then add it to the regression suite.
A failing slice
Paraphrase recall 0.40 is the users' experience on that slice.
Answers from memory
Correct today, wrong after the next policy change.
No held-out part
Scores on tuned-against cases overstate production.

Cost per supported answer2 minWhy dividing by requests flatters a system that pays for unsupported answers.

Cost per supported answer Formula

\frac{c_{query} \times N_{queries}}{N_{supported}} = \frac{0.012 \times 16}{9} \approx 0.0213

$c_{query}$: hypothetical EUR per query, an assumption to label
$N_{queries}$: all 16 queries paid for, including 2 unanswerable
$N_{supported}$: answers with evidence in context and correct (9)

Per-request cost hides that only 9 of 16 paid calls produced a supported answer.

Repeated runs2 minHow per-attempt success and all-runs-pass diverge task by task.

Tool tasks, three runs each (application fixture; 1 = pass) Grid

Repeated run →
Task ↓	Run 1	Run 2	Run 3
t1	1	1	1
t2	1	0	1
t3	1	1	1
t4	0	1	0

9 of 12 attempts pass (0.75), but only t1 and t3, 2 of 4 tasks, pass every run (0.50): read the rows, not the cells.

Security beside utility2 minWhy a defence is read on two axes, not one.

Reading a defence by both numbers Map

Target

Useful and resistant; confirm on a larger attack suite.

Useful but exploitable

This run: 0.80 utility, 0.20 obeyed.

Safe by refusing

A filter that blocks attacks and real requests alike.

Worst of both

Neither useful nor resistant.

A defence that cuts utility is not free; a utility figure with no attack suite says nothing about safety.

Designing the next experiment2 minFrom the largest fixable bucket to a pre-specified, held-out comparison.

Choosing the next experiment Diagram

Largest fixable buckete.g. paraphrase misses
Enough cases on that slice?
- yes: to Change one thing
no
Collect cases30+ from real tickets, then re-check
Change one thinghybrid or query rewriting
Write expected movementbefore the run
Judge on held-outpaired, with regression list

Memory anchor

Sample, slice, stage, uncertainty, cost: read all five before you believe a headline number.

Useful for

Reviewing a RAG or tool-use prototype hand-over and deciding whether it is ready for a pilot: a policy assistant (insurance), a clinical-coding helper (healthcare), a complaint router (banking), a support bot (telecom).
Writing an evaluation report whose claims each name their sample size, slice and version.
Preparing for a system-review interview where the answer is a ranked list of what the numbers do and do not show.

Not suitable for

Judging a real system from the recorded fixture; the numbers are synthetic and illustrate the reading order.
Choosing between retrieval methods from slices of four or five queries.
Replacing a full red-team exercise; five injection cases are a smoke test (Day 53).

Traps

Reporting one headline number

"Answer accuracy 0.71" hides a slice at 0.40, a memory-answered case and an obeyed injection, and gives no idea of sample size. The pilot ships on the headline; a complaint router that is fine on cards and weak on mortgages fails exactly the customers with the largest balances.

How to avoid it

Report results by stage and slice with n and an interval, put attack success next to utility, and show the two-way table.

Letting the evaluation set drive the design

The same fourteen queries are used to choose chunking, prompt and retriever. Scores rise on them while the real distribution is untouched, and the next regression is invisible.

How to avoid it

Keep development and held-out parts, add real queries over time with a version bump, and decide experiments before looking at the outcome.

Senior reasoning

A senior reads an evaluation as a claim with an evidence chain: sample, slice, stage, uncertainty, cost. They separate what the data shows from what it suggests, convert suggestions into one pre-specified experiment, and treat safety and cost as first-class columns of the same table as accuracy. They are comfortable saying "not enough cases to tell" and then saying what number of cases would be.

Interview question

"Here is an evaluation summary of a RAG assistant. What would you ask before approving a pilot?"

Outline your answer first.

Recall cards scheduled after completion

List the seven things to inspect in an evaluated GenAI application result, in order.
What makes a next experiment well specified?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — Shunyu Yao, Noah Shinn, Pedram Razavi, Karthik Narasimhan, 2024core
Judges tool-using agents by final state against a goal state and reports consistency over repeated trials, which the tool-task row follows.
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents — Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, Florian Tramèr, 2024optional
Measures task utility and attack success together for tool-using agents, the pairing used in the attack row.
Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations — Evan Miller, 2024optional
Recommends reporting standard errors and comparing models with paired question-level differences, supporting the interval and the sign test (a preprint).
Evaluating Verifiability in Generative Search Engines — Nelson F. Liu, Tianyi Zhang, Percy Liang, 2023optional
Human audit of generative search engines found citations that did not support their sentences; supports separating retrieval, citation and support.
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Nandan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava, Iryna Gurevych, 2021optional
Reports that BM25 is a robust baseline and dense retrieval can underperform out of domain, which is why method claims need your own labelled slices.
Introduction to Information Retrieval — Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze, 2008optional
Defines the ranked-retrieval measures behind recall at k. (Chapter 8)

Optional sections

Deep dive · 10 min

Wilson interval. For $\hat p = h/n$ and $z = 1.96$ , the Wilson interval is $\dfrac{\hat p + z^2/(2n) \pm z\sqrt{\hat p(1-\hat p)/n + z^2/(4n^2)}}{1 + z^2/n}$ . It behaves sensibly at small $n$ and near 0 or 1, where the plain normal interval does not. With $n = 14$ and $h = 10$ it spans 0.45 to 0.88, so a change of one query moves the estimate by about seven points.

Paired comparisons. To compare two configurations on the same queries, count queries that one passes and the other fails. Ties carry no information. An exact sign test on the discordant queries asks whether wins are more frequent than chance; with few discordant pairs it cannot reach significance. Plan the number of queries from the smallest difference you would act on.

Consistency. With per-attempt success $p$ and independent attempts, all $k$ repeats pass with $p^k$ . Real tool tasks are not identically hard, so measure it on repeated runs rather than computing it from $p$ .

Attack suites. Keep the suite as a regression test: every successful injection becomes a case. Report the obeyed share next to task utility with the same sample sizes, since a strict filter can lower both.

Cost per supported answer. Total spend divided by supported answers. It rises when recall falls, when retries increase, and when long contexts are padded with weak chunks.

Hands-on lab: Paired comparison of two retrieval configurations · 15 min

Two configurations were run on the same 14 queries (recorded fixtures). A is the lexical run from the application block; B adds a second retriever. Compare per slice, count wins, losses and ties, run an exact sign test and compute the smallest number of zero-loss wins that would reach p below 0.05. Then write two sentences stating what you would conclude today and how many more queries you would label before deciding.

from math import comb

slices = ["identifier"] * 4 + ["keyword"] * 5 + ["paraphrase"] * 5
A = [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1]  # hit@3 per query, configuration A (recorded fixture)
B = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1]  # configuration B (recorded fixture)

def sign_test_p(wins, losses):
    n = wins + losses
    if n == 0:
        return 1.0
    tail = sum(comb(n, j) for j in range(0, min(wins, losses) + 1)) / 2 ** n
    return min(1.0, 2 * tail)

for name in ["identifier", "keyword", "paraphrase"]:
    idx = [i for i, s in enumerate(slices) if s == name]
    print(f"{name:<11} A {sum(A[i] for i in idx)}/{len(idx)}  B {sum(B[i] for i in idx)}/{len(idx)}")
wins = sum(b > a for a, b in zip(A, B))
losses = sum(b < a for a, b in zip(A, B))
ties = len(A) - wins - losses
print(f"overall A {sum(A)}/{len(A)}  B {sum(B)}/{len(B)}; wins {wins}, losses {losses}, ties {ties}")
print(f"exact two-sided sign test: p = {sign_test_p(wins, losses):.3f}")
needed = next(w for w in range(1, 40) if sign_test_p(w, 0) < 0.05)
print(f"zero-loss wins needed for p < 0.05: {needed}")

Interview practice · 10 min

A telecom's product team asks you to approve its support assistant on the basis of "92% accuracy on our test set". What do you ask for?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-12 · depends on numpy, statsmodels

1 Recall · 5 min

2 Concept · 10 min

Assumptions

3 Application and workplace decision · 10 min

4 Mini-assessment · 8 min

Technical depth · optional · 29 min in total

Useful for

Not suitable for

Traps

Senior reasoning

Interview question

Recall cards scheduled after completion

Prerequisites and related concepts

References

Optional sections

Recall · 5 min

Concept · 10 min

Application and workplace decision · 10 min

Mini-assessment · 8 min