Distillation
Also: knowledge distillation
Training one model with help from another. A smaller model may learn some of a larger model's skills, but can lose others.
The fuller definition
Distillation transfers aspects of a teacher model's behavior to a student model through a training process, often to obtain a smaller or cheaper system.
Supply instructions and examples each time. Nothing in the model changes.
Fetch current documents and put them in context. Facts can change daily.
Train the parameters on examples. Changes behaviour and style; poor at keeping facts current.
Most legal tasks need the first two. Fine-tuning is a project, with data, evaluation, and maintenance, and it is the wrong tool for facts that change.
A concrete example
A smaller model learns from examples or output distributions generated by a larger model.
The distinction that matters
The student does not necessarily retain all of the teacher's capabilities or reliability.
Related terms
Training
Adjusting a model with examples and a learning rule. The aim is to improve how it handles tasks.
Training & generationFine-tuning
More training on a chosen set of examples. It changes a model to suit a task or style.
Reliability & securityEvaluation (eval)
Testing an AI tool on a set of tasks to see how well it actually does.
Where it is taught
How models learn from examplesSource and further reading
Hugging Face · LLM courseThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.