Model Serving & Economics
Latency budget
The maximum response time a product can tolerate for a given operation.
Example
A voice assistant must respond within its 700-millisecond latency budget.
Why it matters
It helps teams fit AI work into an experience that still feels responsive.
You'll hear it
“We have two seconds before the caller thinks we've gone away.”
Can you control it?
Sometimes
Sometimes. Some products or workflows let you influence a latency budget, while others keep it fixed or manage it automatically. Check the settings and documentation for the tool you are using.
Common questions
- Does the budget cover only the model?
- It should reflect the whole wait that matters, including network delays and other services.
- What if an answer cannot finish in time?
- The product can show progress, offer a shorter response or move the work into the background.
- Should every task have the same limit?
- No. People may accept a longer wait for a detailed report than for a live conversation.