Model Serving & Economics
Latency budget
The maximum response time a product can tolerate for a given operation.
Example
A voice assistant must respond within its 700-millisecond latency budget.
Why people use it
Teams use “Latency budget” when they need to balance model quality, reliability, speed, and cost.
What you'll hear
“How does Latency budget affect latency, quality, or cost at scale?”