Model Training & Adaptation
Distillation
Training a smaller model to imitate the behavior of a larger or stronger model.
Example
A smaller model learns to imitate the useful behavior of a larger model.
Why people use it
Teams use “Distillation” when they need to adapt models deliberately and diagnose training problems.
What you'll hear
“Would Distillation improve the model for our specific use case?”