OpenDebateEvidence: A Massive-Scale Argument Mining and Summarization Dataset
0
Sign in to get full access
Overview
- This paper introduces a new dataset called OpenDebateEvidence, which is a large-scale argument mining and summarization dataset.
- The dataset contains over 1.6 million arguments extracted from online debate forums, along with crowd-sourced annotations for various argument quality attributes.
- The dataset is designed to support research in areas like argument mining, argument quality assessment, and argument-based text summarization.
Plain English Explanation
The researchers behind this paper have created a new dataset called OpenDebateEvidence that is focused on online debates. This dataset contains over 1.6 million individual arguments that have been extracted from online debate forums. These arguments have also been annotated by crowdsourced workers, who have provided ratings on various aspects of the argument quality.
This dataset is intended to be a valuable resource for researchers working on argument mining, argument quality assessment, and argument-based text summarization. By having access to this large dataset of annotated arguments, researchers can develop and test new techniques in these important areas of artificial intelligence and natural language processing.
Technical Explanation
The OpenDebateEvidence dataset was created by scraping over 100,000 debate threads from several online debate forums. The researchers then used a combination of rule-based and machine learning-based techniques to extract individual arguments from these debate threads. In total, they were able to extract over 1.6 million unique arguments.
Each argument in the dataset was then annotated by crowdsourced workers on a variety of attributes, including the argument's convincingness, logic, factual accuracy, and overall quality. This annotation process involved having multiple workers review each argument and provide ratings on these different dimensions.
The resulting dataset provides a rich resource for researchers working on argument mining, where the goal is to automatically identify and extract arguments from text. It also supports research on argument quality assessment, where the aim is to develop models that can evaluate the strength and persuasiveness of arguments. Finally, the dataset can be used to advance research on argument-based text summarization, which involves generating summaries of texts based on the key arguments present.
Critical Analysis
The OpenDebateEvidence dataset represents a significant advance in the field of argument mining and analysis. By providing a large-scale dataset of annotated arguments, it enables researchers to develop and test more robust and accurate models in this area.
However, the paper does acknowledge some limitations of the dataset. For example, the arguments come from online debate forums, which may not be representative of arguments in other domains, such as academic or policy discussions. Additionally, the annotation process, while comprehensive, may still contain some degree of subjectivity and bias.
Another potential issue is that the dataset only includes the text of the arguments, without any associated contextual information, such as the topic of the debate or the identities of the participants. This contextual information could be valuable for certain types of analysis.
Despite these limitations, the OpenDebateEvidence dataset is a valuable contribution to the field and is likely to spur further advancements in argument mining, argument quality assessment, and argument-based text summarization. Researchers should carefully consider the dataset's strengths and limitations when designing their studies and interpreting their findings.
Conclusion
The OpenDebateEvidence dataset represents a significant step forward in the field of argument mining and analysis. By providing a large-scale dataset of annotated arguments, it enables researchers to develop and test more robust and accurate models in this area. The dataset has the potential to support advancements in a variety of important applications, such as assisted debate builders, argument quality assessment, and argument-based text summarization. While the dataset has some limitations, it represents a valuable contribution to the field and is likely to spur further research and innovation in this important area of artificial intelligence and natural language processing.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
OpenDebateEvidence: A Massive-Scale Argument Mining and Summarization Dataset
Allen Roush, Yusuf Shabazz, Arvind Balaji, Peter Zhang, Stefano Mezza, Markus Zhang, Sanjay Basu, Sriram Vishwanath, Mehdi Fatemi, Ravid Shwartz-Ziv
We introduce OpenDebateEvidence, a comprehensive dataset for argument mining and summarization sourced from the American Competitive Debate community. This dataset includes over 3.5 million documents with rich metadata, making it one of the most extensive collections of debate evidence. OpenDebateEvidence captures the complexity of arguments in high school and college debates, providing valuable resources for training and evaluation. Our extensive experiments demonstrate the efficacy of fine-tuning state-of-the-art large language models for argumentative abstractive summarization across various methods, models, and datasets. By providing this comprehensive resource, we aim to advance computational argumentation and support practical applications for debaters, educators, and researchers. OpenDebateEvidence is publicly available to support further research and innovation in computational argumentation. Access it here: https://huggingface.co/datasets/Yusuf5/OpenCaselist
Read more7/8/2024
0
Which Side Are You On? A Multi-task Dataset for End-to-End Argument Summarisation and Evaluation
Hao Li, Yuping Wu, Viktor Schlegel, Riza Batista-Navarro, Tharindu Madusanka, Iqra Zahid, Jiayan Zeng, Xiaochi Wang, Xinran He, Yizhi Li, Goran Nenadic
With the recent advances of large language models (LLMs), it is no longer infeasible to build an automated debate system that helps people to synthesise persuasive arguments. Previous work attempted this task by integrating multiple components. In our work, we introduce an argument mining dataset that captures the end-to-end process of preparing an argumentative essay for a debate, which covers the tasks of claim and evidence identification (Task 1 ED), evidence convincingness ranking (Task 2 ECR), argumentative essay summarisation and human preference ranking (Task 3 ASR) and metric learning for automated evaluation of resulting essays, based on human feedback along argument quality dimensions (Task 4 SQE). Our dataset contains 14k examples of claims that are fully annotated with the various properties supporting the aforementioned tasks. We evaluate multiple generative baselines for each of these tasks, including representative LLMs. We find, that while they show promising results on individual tasks in our benchmark, their end-to-end performance on all four tasks in succession deteriorates significantly, both in automated measures as well as in human-centred evaluation. This challenge presented by our proposed dataset motivates future research on end-to-end argument mining and summarisation. The repository of this project is available at https://github.com/HaoBytes/ArgSum-Datatset
Read more8/21/2024
💬
0
Assisted Debate Builder with Large Language Models
Elliot Faugier, Fr'ed'eric Armetta, Angela Bonifati, Bruno Yun
We introduce ADBL2, an assisted debate builder tool. It is based on the capability of large language models to generalise and perform relation-based argument mining in a wide-variety of domains. It is the first open-source tool that leverages relation-based mining for (1) the verification of pre-established relations in a debate and (2) the assisted creation of new arguments by means of large language models. ADBL2 is highly modular and can work with any open-source large language models that are used as plugins. As a by-product, we also provide the first fine-tuned Mistral-7B large language model for relation-based argument mining, usable by ADBL2, which outperforms existing approaches for this task with an overall F1-score of 90.59% across all domains.
Read more5/24/2024
0
DebateQA: Evaluating Question Answering on Debatable Knowledge
Rongwu Xu, Xuan Qi, Zehan Qi, Wei Xu, Zhijiang Guo
The rise of large language models (LLMs) has enabled us to seek answers to inherently debatable questions on LLM chatbots, necessitating a reliable way to evaluate their ability. However, traditional QA benchmarks assume fixed answers are inadequate for this purpose. To address this, we introduce DebateQA, a dataset of 2,941 debatable questions, each accompanied by multiple human-annotated partial answers that capture a variety of perspectives. We develop two metrics: Perspective Diversity, which evaluates the comprehensiveness of perspectives, and Dispute Awareness, which assesses if the LLM acknowledges the question's debatable nature. Experiments demonstrate that both metrics align with human preferences and are stable across different underlying models. Using DebateQA with two metrics, we assess 12 popular LLMs and retrieval-augmented generation methods. Our findings reveal that while LLMs generally excel at recognizing debatable issues, their ability to provide comprehensive answers encompassing diverse perspectives varies considerably.
Read more8/6/2024