Deep RL from Human Preferences
/AI/5 min read
Nine hundred comparisons — one hour of someone's attention — taught a simulated robot to backflip. That is 11,111× fewer human decisions than scoring every step, and the trick is what the comparison is asked about.
Reinforcement learning needs a reward: a number after every action saying how well that went. Where there is a natural score — a game, a clearly measurable outcome — this is fine.
For anything else it is the hardest part of the problem. Write a reward for "do a backflip" and you find yourself specifying joint angles, rotation thresholds and landing conditions, and the agent finds some spinning-on-the-ground behaviour that maximises your formula and looks nothing like a backflip.
This paper replaced the written reward with a learned one, and its lasting contribution is what it asked people to do.
The unit of human work
Not "score this action". Two design choices, both about human perception rather than machine learning.
Short clips, not single states. A snapshot of a robot mid-air says nothing about whether it is doing the right thing — motion only exists over time. So the unit is a clip of one or two seconds, the shortest window in which a person can tell what is happening.
Comparison, not scoring. A person is shown two clips and asked which is better. Not how good either is — just which one. That is a judgement people make consistently; absolute numbers are not.
Both choices are about making the question answerable, and they are the reason the labelling budget works out.
How little labelling that is
An agent training for ten million environment steps, with nine hundred comparisons of two-second clips:
| environment steps | 10,000,000 |
| steps a human actually saw | 216,000 |
| share of the run observed | 2.16% |
| human decisions required | |
|---|---|
| a reward for every step | 10,000,000 |
| one preference per pair of clips | 900 |
| 11,111× fewer |
At roughly four seconds of attention per comparison, nine hundred comparisons is one hour of one person's time. That hour produced a backflip that no hand-written reward function had managed.
How a preference becomes a reward
The reward model outputs a number for each step. A clip's score is the sum of those numbers over its steps, and the probability that a person prefers clip A is:
P(A preferred) = exp(sum of A) / ( exp(sum of A) + exp(sum of B) )
With per-step predictions of [2, 1, 2] against [1, 1, 0]:
| sum of A | 5 |
| sum of B | 2 |
| P(A preferred) | 95.3% |
The formula depends only on the difference, which means absolute reward values carry no information:
| gap between the sums | P(preferred) |
|---|---|
| 0 | 50.0% |
| 1 | 73.1% |
| 2 | 88.1% |
| 3 | 95.3% |
Training minimises −log P(the clip the human chose). If the human agreed with the model, the loss here is 0.049. If they picked B, it is 3.049 — 63× larger. A confident wrong prediction is punished hard, a hedged one mildly, which is what makes the model calibrate rather than just rank.
The clever part: nobody says which step was good
Look again at what the human provided. "Clip A was better." That is all.
The model produces a number for every individual step, and those numbers are never labelled. They are inferred — fitted so that their sums explain which clips people preferred, across hundreds of comparisons.
That is credit assignment, solved by the reward model rather than by the person. A step that appears in many preferred clips accumulates a high value; one that appears in rejected clips does not. The human never had to decide that frame 34 was the good bit, which is fortunate, because they could not have.
Three things running at once
The loop is asynchronous, and this is what makes the tiny labelling budget sufficient.
- The agent trains against the current reward model, continuously.
- The reward model retrains on all comparisons collected so far, continuously.
- A human labels clips sampled from what the agent is doing now.
Because the clips come from current behaviour, the labels always describe the region of the space the policy actually occupies. There is no need to label good behaviour in advance, or to cover the space — only to keep correcting the model where the agent has arrived.
Two refinements matter. Clips are chosen where the reward model is most uncertain, so each comparison is maximally informative. And an ensemble of reward models provides that uncertainty estimate, as well as making the learned reward harder to exploit — a policy would have to fool all of them at once.
What it became
The published result — a backflip from about 900 comparisons in under an hour, and complex behaviours from feedback on well under 1% of interactions — was the proof that the approach worked at all.
The structure transferred directly. Replace clips of a simulated robot with candidate text responses, and the same three components appear: demonstrations of preference, a reward model trained by comparison, a policy optimised against it. The vocabulary changed and the method did not.
What to take away
The paper's insight is about the interface to the human, not the algorithm.
Ask for a comparison rather than a score, over a window long enough to see behaviour, sampled from what the agent is doing right now — and one hour of attention replaces ten million judgements. Everything about learning per-step rewards from whole-clip preferences follows from having asked the answerable question.