MAVEN-Fact: A Large-scale Event Factuality Detection Dataset
0
Sign in to get full access
Overview
- Maven-Fact is a large-scale dataset for event factuality detection
- It contains over 2 million annotated events from news articles
- Events are labeled as either factual, non-factual, or uncertain
Plain English Explanation
The Maven-Fact dataset is a collection of over 2 million events extracted from news articles, with each event labeled as either factual, non-factual, or uncertain. This allows machine learning models to be trained to recognize when an event described in text is real and true, versus when it is fictional or uncertain.
The dataset was constructed by gathering news articles from a variety of reputable sources, and then using natural language processing techniques to identify and extract the key events mentioned in the text. Each event was then manually reviewed and assigned a label indicating whether it was a true, factual event, a non-factual or fictional event, or an event where the factuality was unclear or uncertain.
Having a large, high-quality dataset like Maven-Fact is valuable for developing AI systems that can accurately determine the factual status of events described in text. This could be useful for applications like fact-checking, event tracking, and improving the reliability of information extraction from news sources.
Technical Explanation
The Maven-Fact dataset was constructed by first gathering a diverse set of news articles from reputable sources. Natural language processing techniques were then used to identify and extract the key events mentioned in the text of these articles.
Each extracted event was then manually reviewed by human annotators, who assigned it a label indicating whether the event was factual, non-factual, or uncertain in its factuality. The authors developed detailed guidelines to ensure consistent and reliable labeling.
The final Maven-Fact dataset contains over 2 million annotated events, making it one of the largest resources of its kind. The dataset is balanced in terms of the distribution of factual, non-factual, and uncertain labels, allowing for effective training and evaluation of machine learning models.
Critical Analysis
The Maven-Fact dataset represents a significant advance in the field of event factuality detection. By providing a large, high-quality dataset, the authors enable the development of more accurate and robust AI systems for this task.
However, the paper acknowledges that the dataset is limited to news articles in the English language. Expanding the dataset to cover a broader range of text sources and languages could further enhance its utility.
Additionally, the authors note that the manual annotation process, while rigorous, may still contain some subjectivity and inconsistencies. Exploring ways to further improve the annotation process or integrate automatic pre-labeling could help address this limitation.
Conclusion
The Maven-Fact dataset represents a significant contribution to the field of event factuality detection. By providing a large-scale, high-quality dataset of annotated events, the authors have enabled the development of more accurate and reliable AI systems for tasks such as fact-checking, event tracking, and information extraction from news sources.
While the dataset has some limitations, the overall impact of Maven-Fact is likely to be substantial, as it paves the way for advancements in the understanding and verification of the factual content of textual information.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
MAVEN-Fact: A Large-scale Event Factuality Detection Dataset
Chunyang Li, Hao Peng, Xiaozhi Wang, Yunjia Qi, Lei Hou, Bin Xu, Juanzi Li
Event Factuality Detection (EFD) task determines the factuality of textual events, i.e., classifying whether an event is a fact, possibility, or impossibility, which is essential for faithfully understanding and utilizing event knowledge. However, due to the lack of high-quality large-scale data, event factuality detection is under-explored in event understanding research, which limits the development of EFD community. To address these issues and provide faithful event understanding, we introduce MAVEN-Fact, a large-scale and high-quality EFD dataset based on the MAVEN dataset. MAVEN-Fact includes factuality annotations of 112,276 events, making it the largest EFD dataset. Extensive experiments demonstrate that MAVEN-Fact is challenging for both conventional fine-tuned models and large language models (LLMs). Thanks to the comprehensive annotations of event arguments and relations in MAVEN, MAVEN-Fact also supports some further analyses and we find that adopting event arguments and relations helps in event factuality detection for fine-tuned models but does not benefit LLMs. Furthermore, we preliminarily study an application case of event factuality detection and find it helps in mitigating event-related hallucination in LLMs. Our dataset and codes can be obtained from url{https://github.com/lcy2723/MAVEN-FACT}
Read more7/23/2024
🤔
0
MAVEN-Arg: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation
Xiaozhi Wang, Hao Peng, Yong Guan, Kaisheng Zeng, Jianhui Chen, Lei Hou, Xu Han, Yankai Lin, Zhiyuan Liu, Ruobing Xie, Jie Zhou, Juanzi Li
Understanding events in texts is a core objective of natural language understanding, which requires detecting event occurrences, extracting event arguments, and analyzing inter-event relationships. However, due to the annotation challenges brought by task complexity, a large-scale dataset covering the full process of event understanding has long been absent. In this paper, we introduce MAVEN-Arg, which augments MAVEN datasets with event argument annotations, making the first all-in-one dataset supporting event detection, event argument extraction (EAE), and event relation extraction. As an EAE benchmark, MAVEN-Arg offers three main advantages: (1) a comprehensive schema covering 162 event types and 612 argument roles, all with expert-written definitions and examples; (2) a large data scale, containing 98,591 events and 290,613 arguments obtained with laborious human annotation; (3) the exhaustive annotation supporting all task variants of EAE, which annotates both entity and non-entity event arguments in document level. Experiments indicate that MAVEN-Arg is quite challenging for both fine-tuned EAE models and proprietary large language models (LLMs). Furthermore, to demonstrate the benefits of an all-in-one dataset, we preliminarily explore a potential application, future event prediction, with LLMs. MAVEN-Arg and codes can be obtained from https://github.com/THU-KEG/MAVEN-Argument.
Read more6/21/2024
🛸
0
Fin-Fact: A Benchmark Dataset for Multimodal Financial Fact Checking and Explanation Generation
Aman Rangapur, Haoran Wang, Ling Jian, Kai Shu
Fact-checking in financial domain is under explored, and there is a shortage of quality dataset in this domain. In this paper, we propose Fin-Fact, a benchmark dataset for multimodal fact-checking within the financial domain. Notably, it includes professional fact-checker annotations and justifications, providing expertise and credibility. With its multimodal nature encompassing both textual and visual content, Fin-Fact provides complementary information sources to enhance factuality analysis. Its primary objective is combating misinformation in finance, fostering transparency, and building trust in financial reporting and news dissemination. By offering insightful explanations, Fin-Fact empowers users, including domain experts and end-users, to understand the reasoning behind fact-checking decisions, validating claim credibility, and fostering trust in the fact-checking process. The Fin-Fact dataset, along with our experimental codes is available at https://github.com/IIT-DM/Fin-Fact/.
Read more5/3/2024
0
EX-FEVER: A Dataset for Multi-hop Explainable Fact Verification
Huanhuan Ma, Weizhi Xu, Yifan Wei, Liuji Chen, Liang Wang, Qiang Liu, Shu Wu, Liang Wang
Fact verification aims to automatically probe the veracity of a claim based on several pieces of evidence. Existing works are always engaging in accuracy improvement, let alone explainability, a critical capability of fact verification systems. Constructing an explainable fact verification system in a complex multi-hop scenario is consistently impeded by the absence of a relevant, high-quality dataset. Previous datasets either suffer from excessive simplification or fail to incorporate essential considerations for explainability. To address this, we present EXFEVER, a pioneering dataset for multi-hop explainable fact verification. With over 60,000 claims involving 2-hop and 3-hop reasoning, each is created by summarizing and modifying information from hyperlinked Wikipedia documents. Each instance is accompanied by a veracity label and an explanation that outlines the reasoning path supporting the veracity classification. Additionally, we demonstrate a novel baseline system on our EX-FEVER dataset, showcasing document retrieval, explanation generation, and claim verification, and validate the significance of our dataset. Furthermore, we highlight the potential of utilizing Large Language Models in the fact verification task. We hope our dataset could make a significant contribution by providing ample opportunities to explore the integration of natural language explanations in the domain of fact verification.
Read more6/17/2024