Model Training & Adaptation
Knowledge distillation
Transferring useful behavior from a teacher model into a smaller student model.
Example
A teacher's output probabilities guide a compact student model during training.
Why it matters
It helps fit useful capabilities into tighter memory or computing budgets.
You'll hear it
“We need the compact version to retain the distinctions that matter.”
Can you control it?
Set by the team
The people building or running the AI choose this setup. An everyday user generally needs their help to change how this part works.
Common questions
- Can the student learn from hints as well as final answers?
- Yes. Some methods use richer guidance about the teacher's choices, not just its selected response.
- Does the teacher need to be correct every time?
- No, but unreliable guidance can limit or misdirect what the student learns.
- Can the transfer change how the system behaves on unfamiliar work?
- Yes. Matching familiar examples does not guarantee equivalent behavior elsewhere.