SKU RQA-5005 · Sold by External
StrategyQA
Product specifications
| SKU | RQA-5005 |
|---|---|
| Data type | Reasoning QA |
| Volume | 2,290 questions (1,603 train + 687 test) |
| Size on disk | ~530 KB (parquet download size; 530,106 bytes) |
| Format | parquet |
| Access model | PUBLIC LICENSE |
| Pricing | Free · open-source license (MIT) |
| Quality score | — |
| License | MIT License |
StrategyQA is a reasoning QA benchmark of open-domain yes/no questions where the required reasoning steps are not stated explicitly and must be inferred as a 'strategy' (e.g., 'Did the Royal Air Force fight in the Boxer Rebellion?'). Each question is paired with a supporting set of facts and a boolean gold answer. This Hugging Face mirror (ChilleD/StrategyQA) exposes 2,290 examples across a train split (1,603) and a test split (687), with fields qid, term, description, question, answer, and facts. The original benchmark from Geva et al. (TACL 2021) contains 2,780 examples, each additionally annotated with a question decomposition into reasoning steps and Wikipedia evidence paragraphs, which are not present in this mirror. The paper reports human accuracy of 87% versus roughly 66% for the best baseline model. Scoring is standard boolean accuracy: the predicted yes/no answer is compared for exact match against the gold answer.