Infrastructure & Compute
Inference compute
The processing power consumed when a trained model generates outputs.
Example
A serving team measures the compute needed to answer 10,000 requests each minute.
Why it matters
It helps plan the ongoing resources needed after a model has been trained.
You'll hear it
“Each reply still needs processing even though training is finished.”
Can you control it?
Set by the team
The people building or running the AI choose this setup. An everyday user generally needs their help to change how this part works.
Common questions
- Can a longer answer use more resources?
- Often, yes. But system design and the type of work also matter.
- Does cached information always remove all processing?
- No. Some parts may be reused while other work still needs to happen.
- Can demand change after launch?
- Yes. More users or more complex requests can increase ongoing needs.