Model Training & Adaptation
Knowledge distillation
Transferring useful behavior from a teacher model into a smaller student model.
Example
A teacher's output probabilities guide a compact student model during training.
Why people use it
Knowing how “Knowledge distillation” works helps teams adapt models deliberately and diagnose training problems.
What you'll hear
“Would Knowledge distillation improve the model for our specific use case?”