Knowledge distillation transfers behavior from a teacher model to a student model. Training may use the teacher’s probability distributions, generated answers, labeled examples or a combination of signals. The aim is often to retain useful task performance while reducing inference cost, latency or deployment requirements.
A team might use a large model to label thousands of support tickets, then train a smaller model on those labels so it can run quickly and cheaply in production. Check the teacher model's terms of use before using its outputs as training data, since some providers restrict that. The result is often a small language model built for one job.
A distilled model may lose capabilities that are not represented in the training data, so evaluate it on the tasks and edge cases that matter. Distillation is different from quantization: one changes learned behavior through training, while the other changes numerical representation. Related: Small Language Model, Model Quantization.