MultiHateClip: A Multilingual Benchmark Dataset for Hateful Video Detection on YouTube and Bilibili

Overview

• A new multilingual dataset called MultiHateClip for detecting hateful videos on YouTube and Bilibili
• Includes videos in 4 languages (English, Chinese, Spanish, Arabic) with annotations for hateful content
• Designed to advance research on multimodal and multilingual approaches to hateful video detection

Plain English Explanation

MultiHateClip is a dataset that can be used to train and evaluate systems for detecting hateful videos on popular video platforms like YouTube and Bilibili. The dataset includes videos in 4 different languages - English, Chinese, Spanish, and Arabic - and each video has been annotated to indicate whether it contains hateful content.

The goal of this dataset is to drive progress in hateful video detection by providing a common benchmark that researchers can use. Existing datasets for this task have been limited in their language coverage or have focused only on text-based hate speech detection. MultiHateClip aims to advance the field by enabling the development of multimodal and multilingual approaches that can effectively identify hateful content across different languages and video platforms.

Technical Explanation

The MultiHateClip dataset contains a total of 15,000 videos from YouTube and Bilibili, with 3,750 videos in each of the 4 target languages (English, Chinese, Spanish, Arabic). Each video has been annotated by trained human raters to indicate whether it contains hateful content.

The researchers used a multi-stage sampling process to ensure the dataset is representative of the types of hateful content found on these platforms. They first identified relevant keywords and channels associated with hateful content, then sampled videos based on these signals while also including a portion of "non-hateful" videos to provide negative examples.

The paper also introduces a multimodal, multilingual framework for hateful video detection that leverages both the visual and audio-textual content of the videos. This approach is designed to generalize across languages and video platforms, going beyond prior work that has focused primarily on text-based hate speech detection.

Critical Analysis

The MultiHateClip dataset represents an important step forward in enabling more robust and generalizable hateful video detection systems. By including videos across multiple languages, the dataset can help drive the development of multilingual models that can operate effectively in diverse cultural contexts.

However, the dataset is limited to only 4 languages, which may not reflect the full linguistic diversity of hateful content online. Additionally, the annotation process, while rigorous, could potentially introduce annotator biases that impact the reliability of the labels.

Further research is needed to understand how well the models trained on MultiHateClip generalize to real-world hateful video detection scenarios, and to explore ways of mitigating potential biases in the dataset and annotation process.

Conclusion

The MultiHateClip dataset represents an important contribution to the field of hateful video detection. By providing a multilingual, multimodal benchmark, the dataset can enable the development of more robust and generalizable systems for identifying hateful content on video platforms. While the dataset has some limitations, it is a valuable resource that can help advance research in this critical area.