Infrastructure & Compute
Memory bandwidth
How quickly data can move between memory and a processor.
Example
Limited memory bandwidth slows inference even though the processor can calculate faster.
Why it matters
It helps explain why a powerful processor can still wait for information.
You'll hear it
“The processor is ready, but the data isn't arriving fast enough.”
Can you control it?
Set by the team
The people building or running the AI choose this setup. An everyday user generally needs their help to change how this part works.
Common questions
- Is bandwidth the same as memory capacity?
- No. One concerns transfer speed; the other concerns how much can be stored.
- Can a larger model need more bandwidth?
- It can, especially when calculations repeatedly access many stored values.
- Does advertised peak bandwidth guarantee actual task speed?
- No. Access patterns and other system limits affect the achieved rate.