TurkQA Data Set

Christopher Malon and Bing Bai

(C) 2013 NEC Laboratories America

This release contains the training and testing data used in the paper

C. Malon and B. Bai. Answer Extraction by Recursive Parse Tree Descent. Proceedings of the Workshop on Continuous Vector Space Models and their Compositionality, ACL 2013, 110-118.

Please cite the paper if you use the data set in your research.

License

Question and answer data in "results/" is for non-commercial use, released under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 license.

The sentences in "problems/" are from Wikipedia, released under the Creative Commons Attribution-ShareAlike 3.0 license.

Copies of these licenses appear in the subdirectories.

Download the data set here or here.

Synopsis

TurkQA consists of a selection of sentences from English Wikipedia articles, with questions and answers crowdsourced from workers on Amazon Mechanical Turk.

Turks were shown the sentence at the beginning of a random Wikipedia article, and instructed to write four questions.

The questions may be of two types:

Yes/no questions
Short answer questions, answered by a substring from the original sentence

The Turks were given the rules:

Anyone who understands English should be able to answer the question, by reading the given sentence. No background knowledge or common sense should be necessary.
You must write four questions. The same phrase should not be used as the answer for two different questions.
Do not use pronouns (he, she, it) in your questions.
No more than two of the questions may be yes/no.

Assignments of sentences to Turks were random. If a Turk found an assignment too difficult, it could be traded for another one.

The original sentences are collected in the "problems" subdirectory, with the Turk worker submissions in the "results" subdirectory.

The lines of a result file alternate between questions and answers. Questions appear in plain text. Answers consist of the string "yes" or "no", or else two numerical indices, separated by a space, indicating the range of characters from the original sentence to be taken as the answer.

Due to sentence splitting errors, some problems consist of multiple sentences.

For inquiries, contact Christopher Malon (last name at this site).

Amazon Mechanical Turk is a trademark of Amazon Technologies, Inc.