MultiSocial: Multilingual Benchmark of Machine-Generated Text Detection of Social-Media Texts
0
Sign in to get full access
Overview
- This paper introduces the MultiSocial benchmark, a multilingual dataset for evaluating machine-generated text detection on social media texts.
- The dataset includes posts in 7 languages (English, Spanish, French, German, Italian, Arabic, and Russian) and covers multiple social media platforms.
- The authors also propose several strong baseline models for the task of distinguishing between human-written and machine-generated social media posts.
Plain English Explanation
The paper focuses on the problem of detecting machine-generated text on social media. As large language models and other text generation technologies have become more advanced, it's becoming increasingly difficult to tell whether online posts are written by humans or computers. This can have significant implications, as machine-generated content is often used to spread misinformation or manipulate public discourse.
To address this challenge, the researchers created the MultiSocial benchmark - a collection of social media posts in 7 different languages, some of which were written by humans and others generated by machines. By training and evaluating machine learning models on this dataset, the researchers aim to develop more robust techniques for identifying machine-generated content in the real world.
The paper also presents several baseline models that perform this task with reasonable accuracy, laying the groundwork for further research and development in this important area. As language technologies continue to evolve, being able to reliably distinguish between human and machine-generated text will only become more crucial for maintaining the integrity of online discourse.
Technical Explanation
The MultiSocial dataset contains over 200,000 social media posts across 7 languages: English, Spanish, French, German, Italian, Arabic, and Russian. The posts were collected from platforms like Twitter, Reddit, and online forums, and then annotated as either human-written or machine-generated using a combination of human evaluation and automated detection.
The authors propose several baseline models for the task of classifying posts as human or machine-generated. These include a simple logistic regression model, a transformers-based model, and an ensemble approach that combines multiple models. The models are evaluated using standard metrics like accuracy, precision, recall, and F1-score.
The results show that the transformers-based model achieves the best overall performance, with F1-scores ranging from 0.80 to 0.90 across the different language subsets. However, the authors note that detecting machine-generated content remains a challenging task, especially for more contextual or nuanced social media posts.
Critical Analysis
The MultiSocial benchmark is a valuable contribution to the field, as it provides a standardized dataset for evaluating machine-generated text detection in a multilingual, social media context. The authors have made a concerted effort to collect and annotate a high-quality dataset that reflects the diversity of real-world social media content.
That said, the paper does not delve deeply into potential biases or limitations of the dataset. For example, the authors do not discuss whether the distribution of human-written and machine-generated posts is representative of real-world social media usage, or whether certain types of content or user demographics are overrepresented.
Additionally, while the baseline models provide a solid starting point, the paper does not explore more advanced techniques that could potentially improve performance, such as few-shot learning or domain adaptation approaches. Further research in this direction could yield valuable insights.
Conclusion
The MultiSocial benchmark and the baseline models presented in this paper represent a significant step forward in the effort to detect machine-generated content on social media. As language models continue to advance, being able to reliably distinguish between human and machine-generated text will become increasingly important for maintaining the integrity of online discourse and combating the spread of misinformation.
The authors have laid the groundwork for further research and development in this area, and the MultiSocial dataset is a valuable resource for the broader AI and natural language processing communities. By continuing to refine and expand upon this work, researchers can contribute to the development of more robust and effective techniques for identifying machine-generated content in the wild.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
MultiSocial: Multilingual Benchmark of Machine-Generated Text Detection of Social-Media Texts
Dominik Macko, Jakub Kopal, Robert Moro, Ivan Srba
Recent LLMs are able to generate high-quality multilingual texts, indistinguishable for humans from authentic human-written ones. Research in machine-generated text detection is however mostly focused on the English language and longer texts, such as news articles, scientific papers or student essays. Social-media texts are usually much shorter and often feature informal language, grammatical errors, or distinct linguistic items (e.g., emoticons, hashtags). There is a gap in studying the ability of existing methods in detection of such texts, reflected also in the lack of existing multilingual benchmark datasets. To fill this gap we propose the first multilingual (22 languages) and multi-platform (5 social media platforms) dataset for benchmarking machine-generated text detection in the social-media domain, called MultiSocial. It contains 472,097 texts, of which about 58k are human-written and approximately the same amount is generated by each of 7 multilingual LLMs. We use this benchmark to compare existing detection methods in zero-shot as well as fine-tuned form. Our results indicate that the fine-tuned detectors have no problem to be trained on social-media texts and that the platform selection for training matters.
Read more6/19/2024
0
MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platforms
Yiqiao Jin, Minje Choi, Gaurav Verma, Jindong Wang, Srijan Kumar
Social media platforms are hubs for multimodal information exchange, encompassing text, images, and videos, making it challenging for machines to comprehend the information or emotions associated with interactions in online spaces. Multimodal Large Language Models (MLLMs) have emerged as a promising solution to these challenges, yet they struggle to accurately interpret human emotions and complex content such as misinformation. This paper introduces MM-Soc, a comprehensive benchmark designed to evaluate MLLMs' understanding of multimodal social media content. MM-Soc compiles prominent multimodal datasets and incorporates a novel large-scale YouTube tagging dataset, targeting a range of tasks from misinformation detection, hate speech detection, and social context generation. Through our exhaustive evaluation on ten size-variants of four open-source MLLMs, we have identified significant performance disparities, highlighting the need for advancements in models' social understanding capabilities. Our analysis reveals that, in a zero-shot setting, various types of MLLMs generally exhibit difficulties in handling social media tasks. However, MLLMs demonstrate performance improvements post fine-tuning, suggesting potential pathways for improvement. Our code and data are available at https://github.com/claws-lab/MMSoc.git.
Read more9/4/2024
0
M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection
Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohanned Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, Preslav Nakov
The advent of Large Language Models (LLMs) has brought an unprecedented surge in machine-generated text (MGT) across diverse channels. This raises legitimate concerns about its potential misuse and societal implications. The need to identify and differentiate such content from genuine human-generated text is critical in combating disinformation, preserving the integrity of education and scientific fields, and maintaining trust in communication. In this work, we address this problem by introducing a new benchmark based on a multilingual, multi-domain, and multi-generator corpus of MGTs -- M4GT-Bench. The benchmark is compiled of three tasks: (1) mono-lingual and multi-lingual binary MGT detection; (2) multi-way detection where one need to identify, which particular model generated the text; and (3) mixed human-machine text detection, where a word boundary delimiting MGT from human-written content should be determined. On the developed benchmark, we have tested several MGT detection baselines and also conducted an evaluation of human performance. We see that obtaining good performance in MGT detection usually requires an access to the training data from the same domain and generators. The benchmark is available at https://github.com/mbzuai-nlp/M4GT-Bench.
Read more6/28/2024
🔎
0
Deepfake Text Detection in the Wild
Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang, Longyue Wang, Linyi Yang, Shuming Shi, Yue Zhang
Large language models (LLMs) have achieved human-level text generation, emphasizing the need for effective AI-generated text detection to mitigate risks like the spread of fake news and plagiarism. Existing research has been constrained by evaluating detection methods on specific domains or particular language models. In practical scenarios, however, the detector faces texts from various domains or LLMs without knowing their sources. To this end, we build a comprehensive testbed by gathering texts from diverse human writings and texts generated by different LLMs. Empirical results show challenges in distinguishing machine-generated texts from human-authored ones across various scenarios, especially out-of-distribution. These challenges are due to the decreasing linguistic distinctions between the two sources. Despite challenges, the top-performing detector can identify 86.54% out-of-domain texts generated by a new LLM, indicating the feasibility for application scenarios. We release our resources at https://github.com/yafuly/MAGE.
Read more5/22/2024