Evaluation & Quality
pass@k
How often at least one of a set number of AI attempts solves a task.
Example
A coding test checks whether at least one of five suggested solutions works.
Why people use it
It checks whether repeated attempts produce at least one working solution.
What you'll hear
“Did any of the five attempts work?”
What this means for you
State how many attempts were allowed when reporting the score.
Can you control it?
No
No direct control. This describes a wider issue, concept or result rather than something you can simply switch on or off in a tool.
Common questions
- Is pass@5 comparable to single-attempt success?
- Not directly. More attempts provide more chances and consume more resources.
- Does one working attempt mean the system chose it?
- No. The measure can count success even if there is no reliable way to identify the successful attempt.
- Can repeated attempts all make the same mistake?
- Yes. More attempts do not guarantee useful variety.