Generative AI & LLMs
Speculative decoding
Speeding up AI writing by having one system propose pieces that the main AI checks before accepting.
Example
A smaller AI drafts a continuation that the main AI checks.
Why people use it
It can speed up writing by letting a faster system propose pieces for a larger one to check.
What you'll hear
“Let the smaller system draft the next few pieces.”
What this means for you
Measure the whole response time with the actual service and task.
Can you control it?
Developer-only
The people building or running the AI choose this setup. An everyday user generally needs their help to change how this part works.
Common questions
- Does the draft AI system alone determine the final result?
- No. The target checking procedure controls which draft tokens are accepted.
- Does a rejected draft mean the request fails?
- No. The larger system can continue with different pieces instead.
- Will it speed up every request equally?
- No. The benefit depends on how many proposed pieces are accepted and the computing setup.