Pass@k and Pass^k

Both metrics start from the same experiment: run a task several times with a stochastic system (a language model sampled at nonzero temperature, or an agent that makes different tool calls on different runs) and record success or failure for each run. They differ in what they ask of those runs.

• pass@k: the probability that at least one of \(k\) runs succeeds. It measures capability when a failed run can be thrown away, for example because a unit test or verifier picks the working candidate.
• pass^k (read “pass hat k” or “pass to the k”): the probability that all \(k\) runs succeed. It measures reliability when every run takes effect, for example an agent that edits a customer record each time it runs.

Chen et al. defined pass@k for code generation; Yao et al. introduced pass^k in the τ-bench agent benchmark to report consistency over repeated trials.

The idealized forms

Let \(p\) be the probability that one run of a given task succeeds, and assume runs are independent. Then

\[ \text{pass@}k = 1-(1-p)^k, \qquad \text{pass}^k = p^k. \]

For \(p=0.3\) and \(k=5\), pass@5 \(=1-0.7^5 \approx 0.83\) and pass^5 \(=0.3^5 \approx 0.0024\). The same agent looks capable under one metric and unusable under the other. Both equal \(p\) at \(k=1\).

Estimating from a finite sample

In practice \(p\) is unknown. Run \(n \ge k\) trials per task and count the \(c\) successes. Plugging \(\hat p=c/n\) into the formulas above is biased for \(k>1\). Chen et al. use the unbiased estimator

\[ \widehat{\text{pass@}k} = 1-\binom{n-c}{k}\Big/\binom{n}{k}, \]

which is the probability that a random subset of \(k\) of the \(n\) recorded runs contains at least one success. The pass^k analog is the probability that a random \(k\)-subset is all successes:

\[ \widehat{\text{pass}^k} = \binom{c}{k}\Big/\binom{n}{k}. \]

Both are then averaged over tasks. For one task with \(n=10\) and \(c=3\) (hypothetical counts), the program gives:

k	pass@k	pass^k estimator	plug-in \(\hat p^k\)
1	0.3000	0.3000	0.3000
2	0.5333	0.0667	0.0900
3	0.7083	0.0083	0.0270
5	0.9167	0.0000	0.0024

At \(k=5\) the estimator gives exactly zero, because only 3 of the 10 runs succeeded and no 5-run subset can be all successes. The plug-in estimate \(0.3^5\) is positive. This one-sample difference illustrates how the estimators differ; bias means a difference in their expectations over repeated samples, not that one observed estimate is nonzero. Take \(k\) no larger than \(n\), and use enough trials that \(c\) is not tiny.

Averaging over tasks changes the picture

Metrics are averaged across tasks, and tasks differ in difficulty, so the average hides the spread. Compare two hypothetical agents on five tasks with \(n=8\) trials each. Agent A succeeds on all 8 trials of three tasks and on none of the other two (successes per task 8, 8, 8, 0, 0). Agent B succeeds 5, 5, 4, 3, 3 times. Their mean single-run success is 0.60 and 0.50. At \(k=4\):

	pass@4	pass^4
Agent A	0.600	0.600
Agent B	0.969	0.031

B has the higher pass@4 because it can almost always find a success within four tries; A has the far higher pass^4 because on the tasks it can do, it does them every time. Which agent is better depends on the question: pass@k favors B for a setup with a verifier and retries, pass^k favors A for unattended use.

Choosing between them

Use pass@k when a cheap reliable check exists (tests, a validator, a human choosing among drafts) and cost grows with \(k\). Use pass^k when failures are not filtered and each run’s side effects are real. Report \(k\), \(n\), and the sampling settings (temperature, tool access, random seeds) with the number, since both metrics change with them. The formulas assume independent runs within each task at its fixed success probability. Tasks can have different probabilities without violating that assumption: averaging \(p_t^k\) over tasks differs from raising the average success rate to \(k\), as Agent A illustrates. Correlated runs within a task are a separate issue and require estimating the joint success pattern rather than applying the independent-run formulas.

References

• Chen et al., Evaluating Large Language Models Trained on Code — https://arxiv.org/abs/2107.03374
• Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — https://arxiv.org/abs/2406.12045