EkoHate: Abusive Language and Hate Speech Detection for Code-switched Political Discussions on Nigerian Twitter
2404.18180
0
0
Abstract
Nigerians have a notable online presence and actively discuss political and topical matters. This was particularly evident throughout the 2023 general election, where Twitter was used for campaigning, fact-checking and verification, and even positive and negative discourse. However, little or none has been done in the detection of abusive language and hate speech in Nigeria. In this paper, we curated code-switched Twitter data directed at three musketeers of the governorship election on the most populous and economically vibrant state in Nigeria; Lagos state, with the view to detect offensive speech in political discussions. We developed EkoHate -- an abusive language and hate speech dataset for political discussions between the three candidates and their followers using a binary (normal vs offensive) and fine-grained four-label annotation scheme. We analysed our dataset and provided an empirical evaluation of state-of-the-art methods across both supervised and cross-lingual transfer learning settings. In the supervised setting, our evaluation results in both binary and four-label annotation schemes show that we can achieve 95.1 and 70.3 F1 points respectively. Furthermore, we show that our dataset adequately transfers very well to three publicly available offensive datasets (OLID, HateUS2020, and FountaHate), generalizing to political discussions in other regions like the US.
Create account to get full access
Overview
- This research paper introduces EkoHate, a dataset for detecting abusive language and hate speech in code-switched political discussions on Nigerian Twitter.
- The dataset includes tweets in English, Pidgin English, and Nigerian Yoruba, which are common languages used in political discourse on Nigerian Twitter.
- The paper also presents baseline models for hate speech and abusive language detection on the EkoHate dataset, and discusses the challenges of working with code-switched data.
Plain English Explanation
This research paper focuses on the problem of detecting abusive language and hate speech in online political discussions, specifically on Nigerian Twitter. The researchers created a new dataset called EkoHate, which contains tweets written in a mix of English, Pidgin English, and Nigerian Yoruba. This type of "code-switching" between languages is common in political conversations on Nigerian Twitter, but can make it more difficult to automatically identify harmful or hateful content.
To address this challenge, the researchers developed baseline machine learning models to detect abusive language and hate speech in the EkoHate dataset. This can help social media platforms and moderators better understand and respond to problematic content in these complex, multilingual discussions. The paper also discusses the unique challenges of working with code-switched data, which could inform future research in this area.
Technical Explanation
The EkoHate dataset was created by collecting tweets related to Nigerian politics, and then manually annotating them for the presence of abusive language and hate speech. The dataset includes tweets in English, Pidgin English, and Nigerian Yoruba, which are common languages used in political discourse on Nigerian Twitter.
To establish baselines for hate speech and abusive language detection on the EkoHate dataset, the researchers experimented with several machine learning models, including logistic regression, support vector machines, and transformer-based models like BERT. They explored the performance of these models on the overall task, as well as on subsets of the data based on language.
The results showed that transformer-based models like BERT generally outperformed the more traditional machine learning approaches, especially for detecting hate speech. However, the researchers also found that the performance of all models decreased when evaluated on code-switched data, compared to single-language subsets. This highlights the challenges of working with multilingual, code-switched text, which is common in many real-world online conversations.
Critical Analysis
The EkoHate dataset and the baseline models presented in this paper represent an important step forward in addressing the problem of abusive language and hate speech detection in code-switched political discussions. By focusing on the unique linguistic landscape of Nigerian Twitter, the researchers have identified a crucial gap in the existing literature and taken steps to fill it.
That said, the paper does acknowledge several limitations of the current work. For example, the dataset is relatively small, and the manual annotation process may have introduced some biases. Additionally, the baseline models, while informative, do not represent the state-of-the-art in hate speech and abusive language detection. Further research is needed to develop more robust and effective models for these challenging, multilingual scenarios.
It would also be valuable to explore the broader societal implications of this type of research. While the technical work is important, it is crucial to consider how these systems could be used, by whom, and with what potential consequences for marginalized communities. Responsible development and deployment of such technologies should be a key priority.
Conclusion
This research paper introduces the EkoHate dataset and baseline models for detecting abusive language and hate speech in code-switched political discussions on Nigerian Twitter. By focusing on this unique linguistic landscape, the researchers have made an important contribution to the field of online abuse detection, which has traditionally been dominated by work on English-only datasets.
The challenges identified in this paper, particularly around the difficulties of working with code-switched data, will likely be relevant to many other real-world settings where multiple languages are used in online conversations. As such, the insights from this work could help inform future research and the development of more robust, inclusive systems for identifying and mitigating harmful online content.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
š¬
A multilingual dataset for offensive language and hate speech detection for hausa, yoruba and igbo languages
Saminu Mohammad Aliyu, Gregory Maksha Wajiga, Muhammad Murtala
0
0
The proliferation of online offensive language necessitates the development of effective detection mechanisms, especially in multilingual contexts. This study addresses the challenge by developing and introducing novel datasets for offensive language detection in three major Nigerian languages: Hausa, Yoruba, and Igbo. We collected data from Twitter and manually annotated it to create datasets for each of the three languages, using native speakers. We used pre-trained language models to evaluate their efficacy in detecting offensive language in our datasets. The best-performing model achieved an accuracy of 90%. To further support research in offensive language detection, we plan to make the dataset and our models publicly available.
6/7/2024
NaijaHate: Evaluating Hate Speech Detection on Nigerian Twitter Using Representative Data
Manuel Tonneau, Pedro Vitor Quinta de Castro, Karim Lasri, Ibrahim Farouq, Lakshminarayanan Subramanian, Victor Orozco-Olvera, Samuel P. Fraiberger
0
0
To address the global issue of online hate, hate speech detection (HSD) systems are typically developed on datasets from the United States, thereby failing to generalize to English dialects from the Majority World. Furthermore, HSD models are often evaluated on non-representative samples, raising concerns about overestimating model performance in real-world settings. In this work, we introduce NaijaHate, the first dataset annotated for HSD which contains a representative sample of Nigerian tweets. We demonstrate that HSD evaluated on biased datasets traditionally used in the literature consistently overestimates real-world performance by at least two-fold. We then propose NaijaXLM-T, a pretrained model tailored to the Nigerian Twitter context, and establish the key role played by domain-adaptive pretraining and finetuning in maximizing HSD performance. Finally, owing to the modest performance of HSD systems in real-world conditions, we find that content moderators would need to review about ten thousand Nigerian tweets flagged as hateful daily to moderate 60% of all hateful content, highlighting the challenges of moderating hate speech at scale as social media usage continues to grow globally. Taken together, these results pave the way towards robust HSD systems and a better protection of social media users from hateful content in low-resource settings.
6/26/2024
Exploring Boundaries and Intensities in Offensive and Hate Speech: Unveiling the Complex Spectrum of Social Media Discourse
Abinew Ali Ayele, Esubalew Alemneh Jalew, Adem Chanie Ali, Seid Muhie Yimam, Chris Biemann
0
0
The prevalence of digital media and evolving sociopolitical dynamics have significantly amplified the dissemination of hateful content. Existing studies mainly focus on classifying texts into binary categories, often overlooking the continuous spectrum of offensiveness and hatefulness inherent in the text. In this research, we present an extensive benchmark dataset for Amharic, comprising 8,258 tweets annotated for three distinct tasks: category classification, identification of hate targets, and rating offensiveness and hatefulness intensities. Our study highlights that a considerable majority of tweets belong to the less offensive and less hate intensity levels, underscoring the need for early interventions by stakeholders. The prevalence of ethnic and political hatred targets, with significant overlaps in our dataset, emphasizes the complex relationships within Ethiopia's sociopolitical landscape. We build classification and regression models and investigate the efficacy of models in handling these tasks. Our results reveal that hate and offensive speech can not be addressed by a simplistic binary classification, instead manifesting as variables across a continuous range of values. The Afro-XLMR-large model exhibits the best performances achieving F1-scores of 75.30%, 70.59%, and 29.42% for the category, target, and regression tasks, respectively. The 80.22% correlation coefficient of the Afro-XLMR-large model indicates strong alignments.
4/19/2024
IndoToxic2024: A Demographically-Enriched Dataset of Hate Speech and Toxicity Types for Indonesian Language
Lucky Susanto, Musa Izzanardi Wijanarko, Prasetia Anugrah Pratama, Traci Hong, Ika Idris, Alham Fikri Aji, Derry Wijaya
0
0
Hate speech poses a significant threat to social harmony. Over the past two years, Indonesia has seen a ten-fold increase in the online hate speech ratio, underscoring the urgent need for effective detection mechanisms. However, progress is hindered by the limited availability of labeled data for Indonesian texts. The condition is even worse for marginalized minorities, such as Shia, LGBTQ, and other ethnic minorities because hate speech is underreported and less understood by detection tools. Furthermore, the lack of accommodation for subjectivity in current datasets compounds this issue. To address this, we introduce IndoToxic2024, a comprehensive Indonesian hate speech and toxicity classification dataset. Comprising 43,692 entries annotated by 19 diverse individuals, the dataset focuses on texts targeting vulnerable groups in Indonesia, specifically during the hottest political event in the country: the presidential election. We establish baselines for seven binary classification tasks, achieving a macro-F1 score of 0.78 with a BERT model (IndoBERTweet) fine-tuned for hate speech classification. Furthermore, we demonstrate how incorporating demographic information can enhance the zero-shot performance of the large language model, gpt-3.5-turbo. However, we also caution that an overemphasis on demographic information can negatively impact the fine-tuned model performance due to data fragmentation.
6/28/2024