EmpathicStories++: A Multimodal Dataset for Empathy towards Personal Experiences
2405.15708
0
0
Abstract
Modeling empathy is a complex endeavor that is rooted in interpersonal and experiential dimensions of human interaction, and remains an open problem within AI. Existing empathy datasets fall short in capturing the richness of empathy responses, often being confined to in-lab or acted scenarios, lacking longitudinal data, and missing self-reported labels. We introduce a new multimodal dataset for empathy during personal experience sharing: the EmpathicStories++ dataset (https://mitmedialab.github.io/empathic-stories-multimodal/) containing 53 hours of video, audio, and text data of 41 participants sharing vulnerable experiences and reading empathically resonant stories with an AI agent. EmpathicStories++ is the first longitudinal dataset on empathy, collected over a month-long deployment of social robots in participants' homes, as participants engage in natural, empathic storytelling interactions with AI agents. We then introduce a novel task of predicting individuals' empathy toward others' stories based on their personal experiences, evaluated in two contexts: participants' own personal shared story context and their reflections on stories they read. We benchmark this task using state-of-the-art models to pave the way for future improvements in contextualized and longitudinal empathy modeling. Our work provides a valuable resource for further research in developing empathetic AI systems and understanding the intricacies of human empathy within genuine, real-world settings.
Create account to get full access
Overview
- This paper introduces a new multimodal dataset called EmpathicStories++ that can be used to study empathy towards personal experiences.
- The dataset includes textual stories, audio recordings of the stories, and annotations of empathetic reactions from listeners.
- The goal is to enable research on understanding and generating empathetic responses to personal narratives across different modalities.
Plain English Explanation
The researchers created a new dataset called EmpathicStories++ that contains personal stories told by people, along with audio recordings of those stories and ratings of how empathetic the listeners were in response. Empathy is the ability to understand and share the feelings of another person.
The purpose of this dataset is to help researchers study how people can become more empathetic towards others' personal experiences, especially when those experiences are conveyed through different mediums like text and audio. By analyzing the dataset, scientists can gain insights into the multimodal cues [https://aimodels.fyi/papers/arxiv/imagery-as-inquiry-exploring-multimodal-dataset-conversational] that trigger empathetic responses from listeners.
This could lead to advancements in conversational AI systems that can better understand and respond empathetically to people sharing their personal stories. It could also help develop training for humans to become more emotionally attuned and compassionate towards others' experiences, which has important societal implications.
Technical Explanation
The EmpathicStories++ dataset contains over 3,000 personal narratives collected from diverse adult participants. Each story is provided as text and as an audio recording. After listening to the stories, human raters annotated their empathetic reactions on multiple dimensions, including emotional, cognitive, and behavioral empathy.
The researchers designed the dataset to enable the development of multimodal machine learning models that can understand and generate empathetic responses to personal experiences. They envision applications in areas like conversational AI and training for improving human empathy.
The dataset includes demographic information about the story tellers and raters, enabling analysis of how factors like gender, age, and cultural background influence empathetic responses. The researchers used state-of-the-art annotation procedures and quality control measures to ensure the reliability of the empathy ratings.
Critical Analysis
The EmpathicStories++ dataset represents an important step forward in enabling multimodal research on empathy. By including both text and audio modalities, as well as multidimensional empathy annotations, the dataset provides a rich resource for exploring how people convey and respond to personal experiences.
However, the dataset is limited to Western, educated, industrialized, rich, and democratic (WEIRD) populations, so the generalizability of the findings may be constrained. Additionally, the empathy ratings are subjective and could be influenced by rater biases. Further research is needed to validate the dataset across more diverse samples and to develop more objective empathy measurement techniques.
It would also be valuable to expand the dataset to include other modalities, such as video or physiological signals, to gain a more comprehensive understanding of empathetic processes. Longitudinal data tracking how empathy evolves over time in response to personal stories could provide additional insights.
Conclusion
The EmpathicStories++ dataset represents an important contribution to the field of empathy research. By providing a multimodal resource for studying how people convey and respond to personal experiences, the dataset has the potential to drive advances in conversational AI, human-AI interaction, and our broader understanding of the factors that shape empathetic responses. While the dataset has some limitations, it opens up new avenues for exploring the complex and multifaceted nature of empathy.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
EmPO: Theory-Driven Dataset Construction for Empathetic Response Generation through Preference Optimization
Ondrej Sotolar
0
0
Empathetic response generation is a desirable aspect of conversational agents, crucial for facilitating engaging and emotionally intelligent multi-turn conversations between humans and machines. Leveraging large language models for this task has shown promising results, yet challenges persist in ensuring both the empathetic quality of the responses and retention of the generalization performance of the models. In this paper, we propose a novel approach where we construct theory-driven preference datasets and use them to align LLMs with preference optimization algorithms to address these challenges. To measure empathetic response generation, we employ the EmpatheticDialogues dataset, assessing empathy with the diff-EPITOME and BERTscore metrics, and evaluate the generalization performance on the MMLU benchmark. We make all datasets, source code, and models publicly available.
6/28/2024
🔎
Empathy Detection from Text, Audiovisual, Audio or Physiological Signals: Task Formulations and Machine Learning Methods
Md Rakibul Hasan, Md Zakir Hossain, Shreya Ghosh, Aneesh Krishna, Tom Gedeon
0
0
Empathy indicates an individual's ability to understand others. Over the past few years, empathy has drawn attention from various disciplines, including but not limited to Affective Computing, Cognitive Science and Psychology. Detecting empathy has potential applications in society, healthcare and education. Despite being a broad and overlapping topic, the avenue of empathy detection leveraging Machine Learning remains underexplored from a systematic literature review perspective. We collected 828 papers from 10 well-known databases, systematically screened them and analysed the final 61 papers. Our analyses reveal several prominent task formulations $-$ including empathy on localised utterances or overall expressions, unidirectional or parallel empathy, and emotional contagion $-$ in monadic, dyadic and group interactions. Empathy detection methods are summarised based on four input modalities $-$ text, audiovisual, audio and physiological signals $-$ thereby presenting modality-specific network architecture design protocols. We discuss challenges, research gaps and potential applications in the Affective Computing-based empathy domain, which can facilitate new avenues of exploration. We further enlist the public availability of datasets and codes. We believe that our work is a stepping stone to developing a robust empathy detection system that can be deployed in practice to enhance the overall well-being of human life.
6/27/2024
EMMI -- Empathic Multimodal Motivational Interviews Dataset: Analyses and Annotations
Lucie Galland, Catherine Pelachaud, Florian Pecune
0
0
The study of multimodal interaction in therapy can yield a comprehensive understanding of therapist and patient behavior that can be used to develop a multimodal virtual agent supporting therapy. This investigation aims to uncover how therapists skillfully blend therapy's task goal (employing classical steps of Motivational Interviewing) with the social goal (building a trusting relationship and expressing empathy). Furthermore, we seek to categorize patients into various ``types'' requiring tailored therapeutic approaches. To this intent, we present multimodal annotations of a corpus consisting of simulated motivational interviewing conversations, wherein actors portray the roles of patients and therapists. We introduce EMMI, composed of two publicly available MI corpora, AnnoMI and the Motivational Interviewing Dataset, for which we add multimodal annotations. We analyze these annotations to characterize functional behavior for developing a virtual agent performing motivational interviews emphasizing social and empathic behaviors. Our analysis found three clusters of patients expressing significant differences in behavior and adaptation of the therapist's behavior to those types. This shows the importance of a therapist being able to adapt their behavior depending on the current situation within the dialog and the type of user.
6/26/2024
Can Machines Resonate with Humans? Evaluating the Emotional and Empathic Comprehension of LMs
Muhammad Arslan Manzoor, Yuxia Wang, Minghan Wang, Preslav Nakov
0
0
Empathy plays a pivotal role in fostering prosocial behavior, often triggered by the sharing of personal experiences through narratives. However, modeling empathy using NLP approaches remains challenging due to its deep interconnection with human interaction dynamics. Previous approaches, which involve fine-tuning language models (LMs) on human-annotated empathic datasets, have had limited success. In our pursuit of improving empathy understanding in LMs, we propose several strategies, including contrastive learning with masked LMs and supervised fine-tuning with Large Language Models (LLMs). While these methods show improvements over previous methods, the overall results remain unsatisfactory. To better understand this trend, we performed an analysis which reveals a low agreement among annotators. This lack of consensus hinders training and highlights the subjective nature of the task. We also explore the cultural impact on annotations. To study this, we meticulously collected story pairs in Urdu language and find that subjectivity in interpreting empathy among annotators appears to be independent of cultural background. The insights from our systematic exploration of LMs' understanding of empathy suggest that there is considerable room for exploration in both task formulation and modeling.
6/18/2024