Model Training & Adaptation
Direct preference optimization
DPO
A technique that trains a model directly from preferred and rejected response pairs.
Example
Training directly increases the probability of preferred responses over rejected alternatives.
Why it matters
It offers a way to learn from preference comparisons without adding a separate reward-scoring stage.
You'll hear it
“This reply was preferred, but the comparison needs a clear reason.”
Can you control it?
Set by the team
The people building or running the AI choose this setup. An everyday user generally needs their help to change how this part works.
Common questions
- Does preferred always mean correct?
- No. Preference labels can reflect style, assumptions or mistaken judgments.
- Can inconsistent comparisons confuse the learning?
- Yes. Conflicting examples make the desired behavior harder to learn.
- Does it require a live reviewer for every new reply?
- No. Preference pairs can be prepared before training begins.