Proudly built by:

ProPara Dataset

The ProPara dataset is designed to train and test comprehension of simple paragraphs describing processes, e.g., photosynthesis. We treat the comprehension task as that of predicting, tracking, and answering questions about how entities change during the process. The dataset contains 488 paragraphs and 3,300 sentences. Each paragraph is richly annotated with the locations of all the main entities (the "participants") at every time step (sentence) during the process (~81,000 annotations), stored in a "grid" (participant x sentence).

ProPara's paragraphs are natural (authored by crowdsourcing) rather than synthetic (e.g,. in bAbI). Workers were given a prompt (e.g., "What happens during photosynthesis?") and then asked to author a series of sentences describing the sequence of events in the process. Subsequently, the participants involved were identified and their locations during the process annotated by other workers. Locations were constrained to be spans of the original process text, or the special symbols "?" (unknown location) or "-" (does not exist). A simplified example is shown below.

    participant-grid-simple    

The direct task is, given a paragraph and list of participants, predict the contents of the grid (i.e., the locations of all participants after all steps of the process. However, given that many participants are irrelevant to each sentence, we use a more targeted end task that is a deterministic computation over the grid, as follows: For each participant E in each process, answer the following 10 questions:

  1. Is E created in the process?
  2. Is E destroyed in the process?
  3. Is E moved in the process?
  4. (If E is created:) When is E created (step #)?
  5. (If E is destroyed:) When is E destroyed?
  6. (If E is moved:) When is E moved?
  7. (If E is created:) Where is E created (location)?
  8. (If E is destroyed:) Where is E destroyed?
  9. (If E is moved:) Where is E moved from?
  10. (If E is moved:) Where is E moved to?

If an entity E's location changes from "-" (doesn't exist) to something else (exists) in consecutive timesteps, E has been created at that timestep. Similarly if it changes from something else to "-", E has been destroyed. Finally if the location changes in subsequent time steps, E has been moved. An algorithmic scan of the grid can thus answer these questions. If there are multiple answers to a question (e.g., multiple moves), we compare the predicted and gold answers and compute F1 as the "accuracy" of the answer. For location comparison, a predicted location is correct if it is identical to, or a subphrase of, the gold location. As there are multiple annotations for the gold location, the score is the fraction that the predicted location agreed with.

Dataset Download

The dataset is most conveniently stored and browsed as a Google Spreadsheet. To view or download the dataset, please click here.

ProPara Repository

The ProPara repository in Github can be accessed here.

Paper

Further details and experimental results are described in the paper:

B. Dalvi Mishra, L. Huang, N. Tandon, W. Yih, P. Clark. Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension. In Proc. NAACL, 2018.

Contact

If you have questions, please do not hesitate to contact the authors at: {bhavanad,nikett,scottyih,peterc}@allenai.org, and warrior.fu@gmail.com.