ProPara aims to promote the research in natural language understanding in the context of procedural text. This requires identifying the actions described in the paragraph and tracking state changes happening to the entities involved. We treat the comprehension task as that of predicting, tracking, and answering questions about how entities change during the procedure. The dataset contains 488 paragraphs and 3,300 sentences. Each paragraph is richly annotated with the existence and locations of all the main entities (the “participants”) at every time step (sentence) throughout the procedure (~81,000 annotations).
ProPara paragraphs are natural (authored by crowdsourcing) rather than synthetic (e.g,. in bAbI). Workers were given a prompt (e.g., “What happens during photosynthesis?”) and then asked to author a series of sentences describing the sequence of events in the procedure. From these sentences, participant entities and their existence and locations were identified. The goal of the challenge is to predict the existence and location of each participant, based on sentences in the paragraph.
The main task is: given a paragraph and list of participants, predict the contents of the grid (i.e., the locations of all participants after all steps of the process). However, given that many participants are irrelevant to each sentence, we use a more targeted end task that is a deterministic computation over the grid, as described below. For each paragraph, answer the following 4 questions (we also provide sample answers for the example paragraph above):
More information can be found on the Leaderboard and evaluator codebase webpages.
In 2019 we created an auxiliary dataset for ProPara, called the Dependency Graph (DG) Dataset, that records the dependencies between different steps in a process. For example, for the earlier paragraph, the DG dataset includes the annotation that step 1 (“Roots absorb water”) enables step 2 (“The water flows to the leaf”) by moving the water to the roots. The DG task is to predict the correct dependencies between steps in the ProPara paragraphs. Evaluation is by measuring the F1 score of predicting all the elements in the dependencies, compared with the gold DG annotations on the test set. For further details, see:
B. Dalvi Mishra, N. Tandon, A. Bosselut, W. Yih, P. Clark. Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Text. In Proc. EMNLP, 2019.
The DG dataset is available as additional tabs in the ProPara spreadsheet below.
Further details and experimental results are described in the following papers:
B. Dalvi Mishra, L. Huang, N. Tandon, W. Yih, P. Clark. Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension. In Proc. NAACL, 2018.
N. Tandon, B. Dalvi Mishra, J. Grus, W. Yih, A. Bosselut, P. Clark. Reasoning about Actions and State Changes by Injecting Commonsense Knowledge. In Proc. EMNLP, 2018.
B. Dalvi Mishra, N. Tandon, A. Bosselut, W. Yih, P. Clark. Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Text. In Proc. EMNLP, 2019.
If you have questions, please do not hesitate to contact the authors at: {bhavanad,nikett,scottyih,peterc}@allenai.org.
Details | Created | Recall |
---|---|---|
1 MeeT Janvijay Singh, Fan Bai, Zhen Wang | 2/14/2023 | 67% |
2 CGLI Kaixin Ma, Filip Ilievski, Jonathan Francis, Eric Nyberg and Alessandro Oltramari | 5/14/2022 | 70% |
3 LEMON Qi Shi, Qian Liu, Bei Chen, Yu Zhang, Ting Liu, Jian-Guang Lou | 1/17/2022 | 70% |
4 MeeT Janvijay Singh, Fan Bai, Zhen Wang | 10/29/2022 | 69% |
5 SKIP Ghazaleh Kazeminejad from University of Colorado Boulder | 12/3/2022 | 68% |