GPQA

GPQA (Graduate-Level Google-Proof Q&A) is an evaluation dataset of graduate-level multiple-choice science questions written and validated by domain experts who hold or are pursuing PhDs in biology, physics, and chemistry. The authors collected 564 questions and released 546 after removing an 18-question held-out set (GPQA Extended), and additionally publish two curated subsets: GPQA Main (448 questions, the recommended primary set) and the highest-quality GPQA Diamond (198 questions). The questions are deliberately 'Google-proof': PhD experts reach roughly 65% accuracy while highly skilled non-experts with unrestricted web access (averaging 37 minutes per question) reach only 34%, and the authors' strongest GPT-4 chain-of-thought baseline reached 39% (25% is random chance for four choices). Each record contains the question, one correct answer, three incorrect answer choices, an expert-written explanation, and subdomain / high-level-domain labels. The dataset is distributed as CSV and is gated on Hugging Face (users must accept terms agreeing not to reveal examples online), and it embeds a canary string so it can be filtered out of foundation-model training corpora.