Toxicity Classification in Ukrainian

Overview

• The paper aims to create a labeled toxicity classification corpus for the Ukrainian language, which currently lacks such resources.
• The researchers explore cross-lingual knowledge transfer techniques, such as translating from an English corpus, filtering toxic samples using keywords, and annotating with crowdsourcing.
• They compare large language model (LLM) prompting and other cross-lingual transfer approaches with and without fine-tuning to determine the most robust and efficient baselines for toxicity classification in Ukrainian.

Plain English Explanation

Detecting toxic or harmful content online is an important task, especially as large language models (LLMs) become more advanced and widely used. However, not all languages have available datasets of labeled toxic and non-toxic text that can be used to train models for this task. Ukrainian is one such language that lacks these resources.

The researchers in this study aimed to address this gap by exploring different techniques to create a labeled toxicity classification corpus for Ukrainian. They tried three main approaches:

1. Translating from English: They took an existing English corpus of toxic and non-toxic text and translated it into Ukrainian.
2. Filtering with keywords: They used known keywords associated with toxic content to identify and extract potentially toxic Ukrainian text samples.
3. Crowdsourcing annotations: They had human annotators review and label Ukrainian text samples as toxic or non-toxic.

Once they had built this Ukrainian toxicity dataset, the researchers then compared different methods for using it to train models that can detect toxic Ukrainian text. They looked at how well large language models (like those used for translation) can be used directly for this task, as well as how much performance improves when the models are further fine-tuned on the Ukrainian toxicity data.

The goal was to find the most effective and efficient ways to build toxicity detection capabilities for the Ukrainian language, which could then be applied to make online platforms and communities safer and more inclusive for Ukrainian users.

Technical Explanation

The paper investigates techniques for creating a labeled toxicity classification corpus for the Ukrainian language, which currently lacks such resources. The researchers explore three main approaches:

1. Translating from English: They translate an existing English toxicity classification corpus (e.g., RTP-LX) into Ukrainian.
2. Filtering with keywords: They use a set of known toxic keywords (e.g., from Towards Building a Robust Toxicity Predictor) to identify and extract potentially toxic Ukrainian text samples.
3. Crowdsourcing annotations: They employ crowdsourcing to have human annotators label Ukrainian text samples as toxic or non-toxic, similar to the approach used in Constant Hate: Analyzing Toxicity Across Reddit Topics.

With the resulting Ukrainian toxicity classification corpus, the researchers then evaluate and compare different cross-lingual transfer learning techniques for toxicity detection, including:

• LLM prompting: Using large language models (e.g., multilingual BERT) directly for toxicity classification without fine-tuning
• Fine-tuning: Further training the LLMs on the created Ukrainian toxicity dataset

The paper provides insights into the most robust and efficient baselines for toxicity classification in the Ukrainian language, which can inform the development of safe and fair large language models, as discussed in Ukrainian Texts Classification: Exploration of Cross-Lingual Knowledge Transfer and Cross-Lingual Named Entity Corpus for Slavic Languages.

Critical Analysis

The paper presents a valuable contribution in addressing the lack of toxicity classification resources for the Ukrainian language. The researchers' exploration of different data creation techniques, including translation, keyword-based filtering, and crowdsourcing, provides a comprehensive approach to building a labeled corpus.

However, the paper does not provide a detailed analysis of the quality and potential biases in the created dataset. The researchers mention that the crowdsourcing approach may introduce biases, but they do not quantify or discuss these issues in depth. Additionally, the performance of the cross-lingual transfer learning techniques could be influenced by the accuracy and reliability of the translated or filtered samples, which is not thoroughly examined.

Further research could focus on evaluating the created Ukrainian toxicity dataset more rigorously, potentially by comparing it to expert-annotated samples or analyzing the consistency and fairness of the crowdsourced labels. Exploring alternative data creation methods, such as unsupervised approaches to identify toxic content, could also help strengthen the research and provide a more robust foundation for toxicity detection in the Ukrainian language.

Conclusion

This study fills an important gap by creating a labeled toxicity classification corpus for the Ukrainian language, which currently lacks such resources. The researchers explore various cross-lingual knowledge transfer techniques, including translation, keyword-based filtering, and crowdsourcing, to build this dataset.

The findings provide insights into the most robust and efficient baselines for toxicity classification in Ukrainian, which can inform the development of safe and fair large language models. This work contributes to the broader effort of making online platforms and communities more inclusive and welcoming for Ukrainian users.