The Psychosocial Impacts of Generative AI Harms
2405.01740
0
29
š¤
Abstract
The rapid emergence of generative Language Models (LMs) has led to growing concern about the impacts that their unexamined adoption may have on the social well-being of diverse user groups. Meanwhile, LMs are increasingly being adopted in K-20 schools and one-on-one student settings with minimal investigation of potential harms associated with their deployment. Motivated in part by real-world/everyday use cases (e.g., an AI writing assistant) this paper explores the potential psychosocial harms of stories generated by five leading LMs in response to open-ended prompting. We extend findings of stereotyping harms analyzing a total of 150K 100-word stories related to student classroom interactions. Examining patterns in LM-generated character demographics and representational harms (i.e., erasure, subordination, and stereotyping) we highlight particularly egregious vignettes, illustrating the ways LM-generated outputs may influence the experiences of users with marginalized and minoritized identities, and emphasizing the need for a critical understanding of the psychosocial impacts of generative AI tools when deployed and utilized in diverse social contexts.
Get summaries of the top AI research delivered straight to your inbox:
Overview
- This paper explores the potential psychosocial harms of stories generated by five leading language models (LMs) in response to open-ended prompts.
- The researchers analyze a dataset of 150,000 100-word stories related to student classroom interactions, examining patterns in character demographics and representational harms (e.g., erasure, subordination, stereotyping).
- The goal is to highlight how LM-generated outputs may influence the experiences of users with marginalized and minoritized identities, and to emphasize the need for a critical understanding of the psychosocial impacts of generative AI tools in diverse social contexts.
Plain English Explanation
As generative language models become more widely used, there is growing concern about the potential negative impacts on diverse user groups. This is especially true in education, where these models are being adopted in K-20 schools and one-on-one student settings with limited investigation into the possible harms.
In this study, the researchers wanted to understand how stories generated by five leading language models might affect the experiences of users with marginalized or minoritized identities. They created a dataset of 150,000 100-word stories about student classroom interactions and analyzed them for patterns in character demographics and representational harms, such as erasure, subordination, and stereotyping.
The researchers found concerning examples of how the language models' outputs could negatively impact the experiences of users from diverse backgrounds. This highlights the need for a critical understanding of the psychosocial impacts of generative AI tools, especially when they are used in educational and other social contexts.
Technical Explanation
The researchers used five leading language models (LMs) to generate 150,000 100-word stories in response to open-ended prompts related to student classroom interactions. They then analyzed the stories for patterns in character demographics and representational harms, such as erasure, subordination, and stereotyping.
The analysis revealed concerning examples of how the LM-generated outputs could negatively impact the experiences of users with marginalized and minoritized identities. For instance, certain stories may erase the presence of underrepresented groups, subordinate them to dominant groups, or reinforce harmful stereotypes.
The researchers argue that these findings highlight the need for a critical understanding of the psychosocial impacts of generative AI tools, particularly when they are deployed in educational and other social contexts where they may shape the experiences and perceptions of diverse user groups.
Critical Analysis
The paper provides a valuable exploration of the potential harms associated with the widespread adoption of generative language models, particularly in educational settings. The researchers acknowledge the limitations of their study, which focused on a specific set of prompts and language models, and call for further research to validate and expand on their findings.
One potential area for concern is the reliance on manual annotation to identify representational harms in the generated stories. While the researchers employed multiple annotators and established inter-rater reliability, there may be inherent biases or inconsistencies in this approach that could influence the results.
Additionally, the paper does not delve deeply into the technical mechanisms underlying the biases and harms observed in the LM-generated outputs. A more detailed analysis of the model architectures, training data, and other factors that contribute to these issues could provide valuable insights for developing mitigation strategies.
Overall, the research presented in this paper is an important step in understanding the societal impacts of generative AI and highlights the need for a more critical and comprehensive approach to the deployment of these technologies, particularly in sensitive domains like education.
Conclusion
This paper provides a thought-provoking exploration of the potential psychosocial harms associated with the widespread adoption of generative language models, especially in educational settings. The researchers analyze a large dataset of LM-generated stories and identify concerning patterns of representational harms that could negatively impact the experiences of users with marginalized and minoritized identities.
The findings underscore the need for a more critical and comprehensive understanding of the societal impacts of these technologies, as well as the development of mitigation strategies to ensure that generative AI tools are deployed in a responsible and equitable manner. As these technologies continue to evolve and become more integrated into our daily lives, it is crucial that we carefully consider their broader implications and work to address the potential harms they may pose to diverse user groups.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
š¬
Laissez-Faire Harms: Algorithmic Biases in Generative Language Models
Evan Shieh, Faye-Marie Vassel, Cassidy Sugimoto, Thema Monroe-White
0
0
The rapid deployment of generative language models (LMs) has raised concerns about social biases affecting the well-being of diverse consumers. The extant literature on generative LMs has primarily examined bias via explicit identity prompting. However, prior research on bias in earlier language-based technology platforms, including search engines, has shown that discrimination can occur even when identity terms are not specified explicitly. Studies of bias in LM responses to open-ended prompts (where identity classifications are left unspecified) are lacking and have not yet been grounded in end-consumer harms. Here, we advance studies of generative LM bias by considering a broader set of natural use cases via open-ended prompting. In this laissez-faire setting, we find that synthetically generated texts from five of the most pervasive LMs (ChatGPT3.5, ChatGPT4, Claude2.0, Llama2, and PaLM2) perpetuate harms of omission, subordination, and stereotyping for minoritized individuals with intersectional race, gender, and/or sexual orientation identities (AI/AN, Asian, Black, Latine, MENA, NH/PI, Female, Non-binary, Queer). We find widespread evidence of bias to an extent that such individuals are hundreds to thousands of times more likely to encounter LM-generated outputs that portray their identities in a subordinated manner compared to representative or empowering portrayals. We also document a prevalence of stereotypes (e.g. perpetual foreigner) in LM-generated outputs that are known to trigger psychological harms that disproportionately affect minoritized individuals. These include stereotype threat, which leads to impaired cognitive performance and increased negative self-perception. Our findings highlight the urgent need to protect consumers from discriminatory harms caused by language models and invest in critical AI education programs tailored towards empowering diverse consumers.
4/17/2024
New!Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech Generators
Wiebke Hutiri, Oresiti Papakyriakopoulos, Alice Xiang
0
0
The rapid and wide-scale adoption of AI to generate human speech poses a range of significant ethical and safety risks to society that need to be addressed. For example, a growing number of speech generation incidents are associated with swatting attacks in the United States, where anonymous perpetrators create synthetic voices that call police officers to close down schools and hospitals, or to violently gain access to innocent citizens' homes. Incidents like this demonstrate that multimodal generative AI risks and harms do not exist in isolation, but arise from the interactions of multiple stakeholders and technical AI systems. In this paper we analyse speech generation incidents to study how patterns of specific harms arise. We find that specific harms can be categorised according to the exposure of affected individuals, that is to say whether they are a subject of, interact with, suffer due to, or are excluded from speech generation systems. Similarly, specific harms are also a consequence of the motives of the creators and deployers of the systems. Based on these insights we propose a conceptual framework for modelling pathways to ethical and safety harms of AI, which we use to develop a taxonomy of harms of speech generators. Our relational approach captures the complexity of risks and harms in sociotechnical AI systems, and yields a taxonomy that can support appropriate policy interventions and decision making for the responsible development and release of speech generation models.
5/16/2024
š¤
Frontier AI Ethics: Anticipating and Evaluating the Societal Impacts of Generative Agents
Seth Lazar
0
0
Some have criticised Generative AI Systems for replicating the familiar pathologies of already widely-deployed AI systems. Other critics highlight how they foreshadow vastly more powerful future systems, which might threaten humanity's survival. The first group says there is nothing new here; the other looks through the present to a perhaps distant horizon. In this paper, I instead pay attention to what makes these particular systems distinctive: both their remarkable scientific achievement, and the most likely and consequential ways in which they will change society over the next five to ten years. In particular, I explore the potential societal impacts and normative questions raised by the looming prospect of 'Generative Agents', in which multimodal large language models (LLMs) form the executive centre of complex, tool-using AI systems that can take unsupervised sequences of actions towards some goal.
4/11/2024
New!Simulating Policy Impacts: Developing a Generative Scenario Writing Method to Evaluate the Perceived Effects of Regulation
Julia Barnett, Kimon Kieslich, Nicholas Diakopoulos
0
0
The rapid advancement of AI technologies yields numerous future impacts on individuals and society. Policy-makers are therefore tasked to react quickly and establish policies that mitigate those impacts. However, anticipating the effectiveness of policies is a difficult task, as some impacts might only be observable in the future and respective policies might not be applicable to the future development of AI. In this work we develop a method for using large language models (LLMs) to evaluate the efficacy of a given piece of policy at mitigating specified negative impacts. We do so by using GPT-4 to generate scenarios both pre- and post-introduction of policy and translating these vivid stories into metrics based on human perceptions of impacts. We leverage an already established taxonomy of impacts of generative AI in the media environment to generate a set of scenario pairs both mitigated and non-mitigated by the transparency legislation of Article 50 of the EU AI Act. We then run a user study (n=234) to evaluate these scenarios across four risk-assessment dimensions: severity, plausibility, magnitude, and specificity to vulnerable populations. We find that this transparency legislation is perceived to be effective at mitigating harms in areas such as labor and well-being, but largely ineffective in areas such as social cohesion and security. Through this case study on generative AI harms we demonstrate the efficacy of our method as a tool to iterate on the effectiveness of policy on mitigating various negative impacts. We expect this method to be useful to researchers or other stakeholders who want to brainstorm the potential utility of different pieces of policy or other mitigation strategies.
5/17/2024