Advanced Concepts
Inference-time scaling
Improving performance by giving a model more computation, samples, or search during inference.
Example
A service improves difficult-answer quality by allocating more computation during inference.
Why it matters
It can improve difficult-task performance while continuing to use the same trained model.
You'll hear it
“How much better does it get if we double the thinking time?”
Can you control it?
Not directly
No, not directly. Inference-time scaling describes a learned property, observed result, or broader concept rather than a normal user setting. You may influence it indirectly by changing the tool, data, prompt, or workflow.
Common questions
- Does performance improve at a constant rate?
- Not necessarily. Additional effort may bring smaller gains or reach a point where it stops helping.
- Can a smaller model benefit from this?
- Sometimes. Extra effort can help, but it does not guarantee the abilities of a stronger model.
- How should a team test the benefit?
- Use the same task set at different effort levels and compare correct completions, waiting time and cost.