Stanford Human Preferences (SHP)

SHP is a dataset of 385K collective human preferences over responses to questions/instructions across 18 subject-area subreddits (from cooking to legal advice), intended for training RLHF reward models and NLG evaluation models such as SteamSHP. Each example is a Reddit self-post plus a pair of top-level comments, where one comment is labeled more preferred by Reddit users collectively. The preference is inferred structurally, not by an annotator: if comment A was written no later than comment B yet still has a higher Reddit score, A is taken to be more preferred (writing earlier would otherwise confer a visibility advantage). All responses are naturally occurring and human-written, distinguishing SHP from machine-written preference sets like Anthropic HH-RLHF. Pairs are only included when the post score is at least 10 and each comment score is at least 2, and labels are randomized to a roughly 50/50 A/B distribution. It was created at Stanford using the techniques from the ICML 2022 paper 'Understanding Dataset Difficulty with V-Usable Information'.