RLHF data
RLHF data is information used to turn human judgments into a training signal for a model. A common approach collects preferences between candidate outputs and trains a reward model from them. Demonstrations may support an earlier supervised stage, but demonstrations and preference comparisons serve different roles.
Also known as: Reinforcement learning from human feedback data, RLHF
How it works
Reviewers receive a task, candidate outputs, and criteria for judgment. Their rankings or choices are collected with the relevant context. In the InstructGPT work, human demonstrations and output rankings contributed to different stages of instruction-following training. Other workflows can use feedback differently.
Quality depends on clear instructions, appropriate expertise, consistent treatment of ambiguity, and checks for disagreement. Recording only a winning answer can omit information needed to interpret the comparison.
Why it matters for licensing
A recipient needs to understand the feedback protocol, task distribution, and rights covering prompts, outputs, and annotations. Feedback data is not interchangeable with ordinary customer satisfaction scores or any dataset containing human text.
Example
Fictional example: Reviewers compare two draft responses to a maintenance question under a rubric prioritizing correctness and appropriate uncertainty. Disagreements are recorded and examined rather than silently treated as unanimous preference.
Limitations and misconceptions
Human preferences can be inconsistent, context-dependent, or biased. A learned reward model can generalize imperfectly and can be exploited by optimization. RLHF does not guarantee truthfulness, safety, or correctness on every task.
Questions to ask
- What instructions and expertise informed the judgments?
- Are comparison context, disagreements, and uncertainty preserved?
- Which rights cover the prompts, outputs, and feedback?