Breaking News: Case Studies of Generative AI's Use in Journalism
2406.13706
0
0
Abstract
Journalists are among the many users of large language models (LLMs). To better understand the journalist-AI interactions, we conduct a study of LLM usage by two news agencies through browsing the WildChat dataset, identifying candidate interactions, and verifying them by matching to online published articles. Our analysis uncovers instances where journalists provide sensitive material such as confidential correspondence with sources or articles from other agencies to the LLM as stimuli and prompt it to generate articles, and publish these machine-generated articles with limited intervention (median output-publication ROUGE-L of 0.62). Based on our findings, we call for further research into what constitutes responsible use of AI, and the establishment of clear guidelines and best practices on using LLMs in a journalistic context.
Create account to get full access
Overview
- This paper presents case studies on the use of generative AI in journalism, exploring the potential benefits and challenges.
- The researchers investigated how generative AI models can assist journalists in tasks like news writing, summarization, and fact-checking.
- The paper discusses the ethical considerations and potential biases that must be addressed when using these AI systems in a journalistic context.
Plain English Explanation
This research paper looks at how generative AI, which can create human-like text, is being used in the field of journalism. The authors studied several real-world examples of news organizations using AI-powered tools to help with tasks like writing articles, summarizing information, and verifying facts.
The key focus is on understanding both the potential advantages and the potential risks of incorporating this type of AI technology into journalistic workflows. On the positive side, generative AI could help journalists be more efficient and productive by automating some of the more repetitive or time-consuming aspects of their work. However, there are also concerns about AI-generated content introducing new biases or inaccuracies that could mislead readers.
The paper encourages the journalism industry to thoughtfully consider how to best leverage these advanced AI models while also maintaining high standards of accuracy, fairness, and transparency. It's important that the use of generative AI in news reporting is done in a way that upholds the core principles of quality journalism.
Technical Explanation
The researchers conducted a series of case studies to investigate how generative AI models are currently being applied in news media organizations. They examined several real-world examples, including:
- Using a large language model to generate initial drafts of news articles on routine topics like sports recaps or weather reports.
- Employing AI-powered text summarization to quickly distill key points from long documents or transcripts.
- Applying natural language processing to automatically fact-check claims and identify potential misinformation.
The researchers found that these generative AI systems can provide significant efficiency gains for newsrooms, freeing up journalists to focus on higher-level reporting and analysis. However, the paper also cautions that these models can introduce new biases and inaccuracies if not properly monitored and controlled.
Critical Analysis
The paper acknowledges several important limitations and challenges with using generative AI in journalism. For example, the potential for language models to perpetuate harmful biases must be carefully evaluated, as AI-generated content could inadvertently reinforce societal prejudices.
Additionally, the debiasing of conversational language models for use in news production is an area that requires further research and development. Journalists will need robust tools to identify and mitigate any biases or factual errors introduced by the AI systems.
Overall, the paper strikes a balanced tone, highlighting both the promise and the perils of generative AI in journalism. The researchers encourage news organizations to proceed cautiously and thoughtfully as they explore the integration of these powerful technologies.
Conclusion
This research provides valuable insights into the current state of generative AI's use in journalism. While the technology shows promise in enhancing newsroom efficiency and productivity, there are also significant ethical and accuracy concerns that must be addressed.
As news media organizations continue to experiment with these AI tools, it will be crucial for them to maintain strong editorial oversight, transparent processes, and a commitment to journalistic integrity. By doing so, they can unlock the benefits of generative AI while safeguarding the quality and reliability of the news they produce.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
Quantifying Generative Media Bias with a Corpus of Real-world and Generated News Articles
Filip Trhlik, Pontus Stenetorp
0
0
Large language models (LLMs) are increasingly being utilised across a range of tasks and domains, with a burgeoning interest in their application within the field of journalism. This trend raises concerns due to our limited understanding of LLM behaviour in this domain, especially with respect to political bias. Existing studies predominantly focus on LLMs undertaking political questionnaires, which offers only limited insights into their biases and operational nuances. To address this gap, our study establishes a new curated dataset that contains 2,100 human-written articles and utilises their descriptions to generate 56,700 synthetic articles using nine LLMs. This enables us to analyse shifts in properties between human-authored and machine-generated articles, with this study focusing on political bias, detecting it using both supervised models and LLMs. Our findings reveal significant disparities between base and instruction-tuned LLMs, with instruction-tuned models exhibiting consistent political bias. Furthermore, we are able to study how LLMs behave as classifiers, observing their display of political bias even in this role. Overall, for the first time within the journalistic domain, this study outlines a framework and provides a structured dataset for quantifiable experiments, serving as a foundation for further research into LLM political bias and its implications.
6/18/2024
💬
Bias of AI-Generated Content: An Examination of News Produced by Large Language Models
Xiao Fang, Shangkun Che, Minjia Mao, Hongzhe Zhang, Ming Zhao, Xiaohang Zhao
0
0
Large language models (LLMs) have the potential to transform our lives and work through the content they generate, known as AI-Generated Content (AIGC). To harness this transformation, we need to understand the limitations of LLMs. Here, we investigate the bias of AIGC produced by seven representative LLMs, including ChatGPT and LLaMA. We collect news articles from The New York Times and Reuters, both known for their dedication to provide unbiased news. We then apply each examined LLM to generate news content with headlines of these news articles as prompts, and evaluate the gender and racial biases of the AIGC produced by the LLM by comparing the AIGC and the original news articles. We further analyze the gender bias of each LLM under biased prompts by adding gender-biased messages to prompts constructed from these news headlines. Our study reveals that the AIGC produced by each examined LLM demonstrates substantial gender and racial biases. Moreover, the AIGC generated by each LLM exhibits notable discrimination against females and individuals of the Black race. Among the LLMs, the AIGC generated by ChatGPT demonstrates the lowest level of bias, and ChatGPT is the sole model capable of declining content generation when provided with biased prompts.
4/5/2024
💬
Exploring the Potential of the Large Language Models (LLMs) in Identifying Misleading News Headlines
Md Main Uddin Rony, Md Mahfuzul Haque, Mohammad Ali, Ahmed Shatil Alam, Naeemul Hassan
0
0
In the digital age, the prevalence of misleading news headlines poses a significant challenge to information integrity, necessitating robust detection mechanisms. This study explores the efficacy of Large Language Models (LLMs) in identifying misleading versus non-misleading news headlines. Utilizing a dataset of 60 articles, sourced from both reputable and questionable outlets across health, science & tech, and business domains, we employ three LLMs- ChatGPT-3.5, ChatGPT-4, and Gemini-for classification. Our analysis reveals significant variance in model performance, with ChatGPT-4 demonstrating superior accuracy, especially in cases with unanimous annotator agreement on misleading headlines. The study emphasizes the importance of human-centered evaluation in developing LLMs that can navigate the complexities of misinformation detection, aligning technical proficiency with nuanced human judgment. Our findings contribute to the discourse on AI ethics, emphasizing the need for models that are not only technically advanced but also ethically aligned and sensitive to the subtleties of human interpretation.
5/7/2024
🏅
Evaluating the Capabilities of LLMs for Supporting Anticipatory Impact Assessment
Mowafak Allaham, Nicholas Diakopoulos
0
0
Gaining insight into the potential negative impacts of emerging Artificial Intelligence (AI) technologies in society is a challenge for implementing anticipatory governance approaches. One approach to produce such insight is to use Large Language Models (LLMs) to support and guide experts in the process of ideating and exploring the range of undesirable consequences of emerging technologies. However, performance evaluations of LLMs for such tasks are still needed, including examining the general quality of generated impacts but also the range of types of impacts produced and resulting biases. In this paper, we demonstrate the potential for generating high-quality and diverse impacts of AI in society by fine-tuning completion models (GPT-3 and Mistral-7B) on a diverse sample of articles from news media and comparing those outputs to the impacts generated by instruction-based (GPT-4 and Mistral-7B-Instruct) models. We examine the generated impacts for coherence, structure, relevance, and plausibility and find that the generated impacts using Mistral-7B, a small open-source model fine-tuned on impacts from the news media, tend to be qualitatively on par with impacts generated using a more capable and larger scale model such as GPT-4. Moreover, we find that impacts produced by instruction-based models had gaps in the production of certain categories of impacts in comparison to fine-tuned models. This research highlights a potential bias in the range of impacts generated by state-of-the-art LLMs and the potential of aligning smaller LLMs on news media as a scalable alternative to generate high quality and more diverse impacts in support of anticipatory governance approaches.
5/22/2024