Towards Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results

Overview

• This paper explores the role of humor in human-AI interaction and introduces a novel dataset to study spontaneous humor in real-world settings.
• Current humor detection methods have been limited to staged data, which may not accurately reflect how humor is used in natural conversations.
• The researchers create the Passau-Spontaneous Football Coach Humor (Passau-SFCH) dataset, which contains 11 hours of recordings annotated for humor and its dimensions.
• They experiment with various machine learning models, including Transformers, convolutional neural networks, and expert-designed features, to analyze the performance of different modalities (text, audio, video) for spontaneous humor recognition.
• The researchers also investigate multimodal approaches to humor recognition, including decision-level fusion and a novel multimodal architecture.

Plain English Explanation

Humor is an important part of how humans interact with each other and how we experience the world. Being able to understand humor automatically could help make interactions between humans and AI systems more natural and comfortable. However, most current methods for detecting humor have only been tested on staged, or pre-planned, data, which may not accurately represent how humor is used in real-life situations.

To address this, the researchers in this paper created a new dataset called the Passau-Spontaneous Football Coach Humor (Passau-SFCH) dataset. This dataset contains about 11 hours of recordings of spontaneous humor, where the humor was not planned or staged. The researchers annotated the dataset to indicate when humor was present and what kind of humor it was (e.g., positive or negative sentiment, directed at oneself or others).

Using this dataset, the researchers experimented with different machine learning models to see which ones were best at automatically detecting and analyzing spontaneous humor. They tried models that looked at the text, audio, and video of the recordings, as well as models that combined information from multiple sources (called "multimodal" models).

The researchers found that facial expressions were the most promising for automatically detecting the sentiment (positive or negative) of humor, while the text-based features were best for detecting the direction of the humor (whether it was directed at oneself or others). They also developed a new multimodal model that performed the best overall at recognizing spontaneous humor.

Technical Explanation

The researchers conducted a series of experiments to analyze the performance of different modalities (text, audio, video) and multimodal approaches for spontaneous humor recognition using the Passau-SFCH dataset.

For the unimodal experiments, they employed pretrained Transformers, convolutional neural networks, and expert-designed features. The text-based models leveraged linguistic cues, the audio models utilized acoustic features, and the video models analyzed facial expressions and body language.

In the multimodal experiments, the researchers explored different fusion strategies, including decision-level fusion and a novel multimodal Transformer (MulT) approach. The decision-level fusion combined the predictions from the unimodal models, while the MulT architecture learned cross-modal relationships to make the final predictions.

The results suggest that for automatic analysis of humor sentiment, facial expressions are the most promising modality, while humor direction can be best modeled using text-based features. The researchers found that multimodal approaches generally outperformed unimodal models, with their novel multimodal architecture yielding the best overall performance.

Critical Analysis

The researchers make a valuable contribution by introducing the Passau-SFCH dataset, which provides a more naturalistic setting for studying spontaneous humor compared to previous datasets that focused on staged or planned humor. This is a significant step forward, as the ability to recognize humor in real-world interactions is crucial for developing more natural and engaging human-AI interfaces.

However, the dataset is relatively small, with only 11 hours of recordings, and may not capture the full diversity of spontaneous humor that can occur in different contexts. Additionally, the annotations for humor and its dimensions (sentiment and direction) were performed by human raters, which could introduce subjective biases.

The researchers acknowledge these limitations and suggest that future work should focus on expanding the dataset size and exploring more advanced multimodal fusion techniques. It would also be interesting to investigate how the models perform on different types of spontaneous humor, such as irony, sarcasm, or puns, and how the models might generalize to other real-world settings beyond the specific football coaching context.

Conclusion

This paper presents an important step towards enabling AI systems to better understand and respond to spontaneous human humor in natural interactions. The introduction of the Passau-SFCH dataset and the experiments with various machine learning models provide valuable insights into the strengths and limitations of different modalities and fusion strategies for humor recognition.

The findings suggest that a multimodal approach, which combines text, audio, and video features, has the most promise for accurately detecting and analyzing spontaneous humor in real-world settings. This research lays the foundation for further advancements in humor-aware AI systems that can engage in more natural, enjoyable, and meaningful interactions with humans.