Skip to content
ML Academy

Introduced on day 50

All-runs-pass consistency (pass^k)

The share of tasks that succeed on every one of k repeated runs, reported next to per-attempt success because single-run success overstates the reliability users experience.

In detail

If one attempt succeeds with probability p and attempts are independent, all k pass with p^k, so p = 0.9 gives 0.729 for k = 3. This differs from pass@k, the chance that at least one attempt succeeds, which answers the opposite question and flatters the system. Real tasks vary in difficulty, so measure it on repeated runs.