ReCoRD

• 07/15/2019 SuperGLUE added ReCoRD in its evaluation suite.
• 03/17/2019 ReCoRD is now a shared task in the COIN workshop at EMNLP 2019.

Reading Comprehension with Commonsense Reasoning Dataset (ReCoRD) is a large-scale reading comprehension dataset which requires commonsense reasoning. ReCoRD consists of queries automatically generated from CNN/Daily Mail news articles; the answer to each query is a text span from a summarizing passage of the corresponding news. The goal of ReCoRD is to evaluate a machine's ability of commonsense reasoning in reading comprehension. ReCoRD is pronounced as [ˈrɛkərd].

ReCoRD contains 120,000+ queries from 70,000+ news articles. Each query has been validated by crowdworkers. Unlike existing reading comprehension datasets, ReCoRD contains a large portion of queries requiring commonsense reasoning, thus presenting a good challenge for future research to bridge the gap between human and machine commonsense reading comprehension.

Browse examples in ReCoRD in a friendly way:

We've built a few resources to help you get started with the dataset.

Download a copy of the dataset in JSON format:

• Training Set (217 MB)
• Dev Set (24 MB)

Read the following Readme to get familiar with the data structure.

• Readme

To evaluate your models, we have also made available the evaluation script we will use for official evaluation, along with a sample prediction file that the script will take as input. To run the evaluation, use python evaluate.py <path_to_dev> <path_to_predictions>.

• Evaluation Script
• Sample Prediction File

Once you have a built a model that works to your expectations on the dev set, you submit it to get official scores on the dev and a hidden test set. To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to submit your model so that we can run it on the test set for you. Here's a tutorial walking you through official evaluation of your model:Submission Tutorial

ReCoRD contains passages from two domains. We make them public under the following licenses:

• Passages and queries collected from the CNN/Daily Mail dataset are under Apache license.
• Passages and queries crawled from Internet Archive are subject to the Term of Use.

Ask us questions at our google group or at zsheng2@jhu.edu.

We thank the SQuAD team for allowing us to use their code and templates for generating this website.