LLMs, GenAI, RAG, agents, evaluation · insight-1589
Report code-generation pass@k from n samples with the unbiased estimator, at the k users actually get
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Applied AI, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A code or SQL assistant is scored by "any of k samples passes the tests", using k samples per task exactly.
Action
Generate n at least k samples per task, count c passing, and compute 1 minus C(n-c, k) over C(n, k); choose k to match the product, for example pass@1 when the user sees one answer.
Reason
The estimator averages over every size-k subset of the n samples, which removes the single-draw variance and the bias of the closed-form shortcut.
Boundary
Execution-checked generation: code, SQL, formulas with unit tests.
Symptom or risk
The estimate has high variance, and the common shortcut of plugging the pass@1 rate into 1-(1-p)^k is biased, so comparisons between models flip with the seed.
Exceptions
Tasks where users cannot test multiple candidates should report pass@1 only.
Trade-offs
n samples per task multiply cost.
Pitfalls
Reporting pass@100 as if it described a one-shot product.
Example
Hypothetical: n=10, c=3, k=5 gives 1 - 21/252 = 0.917, while 1-(1-0.3)^5 gives 0.832.
Lessons and concepts
Claims and their sources
Chen et al. define pass@k, note that computing it from exactly k samples has high variance, use n at least k samples with an unbiased estimator, and show the 1-(1-p)^k shortcut is biased.