SKU RQA-5009 · Sold by External
MMLU-Pro
Product specifications
| SKU | RQA-5009 |
|---|---|
| Data type | Reasoning QA |
| Volume | 12032 multiple-choice questions (test split; plus 70 validation CoT few-shot examples) |
| Size on disk | ~4.2 MB (parquet: test 4,144,185 B + validation 42,857 B) |
| Format | Parquet |
| Access model | PUBLIC LICENSE |
| Pricing | Free · open dataset (HF, MIT) |
| Quality score | — |
| License | MIT License |
MMLU-Pro is a more robust and challenging successor to MMLU, comprising 12,032 test multiple-choice questions across 14 disciplines (e.g., business, math, physics, chemistry, law, health, psychology, engineering). Each question expands the answer set from 4 options up to 10 options and is filtered to require deliberate multi-step reasoning rather than shallow pattern matching, substantially lowering accuracy and prompt-sensitivity relative to MMLU. Questions are drawn from the original MMLU plus STEM websites (stemez), TheoremQA, and SciBench, and each record carries a subject category and a source tag. The dataset is distributed on Hugging Face as Parquet with a 12,032-example `test` split and a 70-example `validation` split that supplies 5-shot chain-of-thought (`cot_content`) demonstrations. Scoring is multiple-choice exact match on the gold answer letter (A-J). It was released by TIGER-Lab and introduced in the NeurIPS 2024 paper arXiv:2406.01574.