Direct preference optimization (DPO)
A way to train a model with pairs of answers. Each pair shows which answer people preferred.
The fuller definition
Direct preference optimization is a training approach that uses pairs of preferred and dispreferred responses to adjust a model directly under a preference objective.
Supply instructions and examples each time. Nothing in the model changes.
Fetch current documents and put them in context. Facts can change daily.
Train the parameters on examples. Changes behaviour and style; poor at keeping facts current.
Most legal tasks need the first two. Fine-tuning is a project, with data, evaluation, and maintenance, and it is the wrong tool for facts that change.
A concrete example
A training set identifies which of two answers better follows a specified instruction.
The distinction that matters
It is a particular training method, not a name for manually choosing the better answer in chat.
Related terms
Fine-tuning
More training on a chosen set of examples. It changes a model to suit a task or style.
Training & generationReinforcement learning from human feedback (RLHF)
Training that uses people's ratings of model answers. It helps steer the model toward replies people prefer.
Training & generationTraining
Adjusting a model with examples and a learning rule. The aim is to improve how it handles tasks.
Where it is taught
How models learn from examplesSource and further reading
Hugging Face · DPO trainerThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.