I'm Not Sure, But...: Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust

Read original: arXiv:2405.00623 - Published 5/16/2024 by Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan

💬

Overview

Large language models (LLMs) can produce convincing yet incorrect outputs, which can mislead users who rely on them
There have been calls for LLMs to communicate their uncertainty to users, but little research on how users perceive and act on this uncertainty
This study examines how different natural language expressions of uncertainty impact user reliance, trust, and performance on a task

Plain English Explanation

The paper explores how expressing uncertainty can help reduce overreliance on large language models. When LLMs produce incorrect outputs, users may mistakenly believe they are correct, especially if the LLM doesn't indicate its own uncertainty.

The researchers conducted an experiment where participants answered medical questions using a fictional search engine powered by an LLM. Some participants saw the LLM's responses with different types of uncertainty expressions, like "I'm not sure, but..." or "It's not clear, but..." The researchers then looked at how this affected the participants' trust in the system, their tendency to agree with the answers, and their overall accuracy.

They found that first-person expressions of uncertainty (like "I'm not sure") decreased participants' confidence in the system and their agreement with the answers, while improving their accuracy. This suggests that expressing uncertainty can help users be less overreliant on incorrect LLM outputs. However, the effects were weaker for more general uncertainty expressions.

The study highlights the importance of carefully designing how LLMs communicate their uncertainty to users, and testing this before deploying the models at scale. Different phrasings can have significantly different impacts on user behavior and decision-making.

Technical Explanation

The researchers conducted a large-scale, pre-registered human-subject experiment with 404 participants. Participants were asked to answer medical questions using a fictional search engine powered by an LLM. Some participants had access to the LLM's responses, while others did not.

The researchers examined how different natural language expressions of uncertainty affected the participants' reliance, trust, and overall task performance. They tested two types of uncertainty expressions: first-person ("I'm not sure, but...") and general ("It's not clear, but...").

Using both behavioral and self-reported measures, the study found that first-person expressions of uncertainty decreased participants' confidence in the system and their tendency to agree with the system's answers. Importantly, this also increased their answer accuracy. An exploratory analysis suggested this was due to reduced, but not fully eliminated, overreliance on incorrect answers.

While the researchers observed similar effects for general uncertainty expressions, these effects were weaker and not statistically significant. This highlights the importance of the specific language used to communicate uncertainty.

Critical Analysis

The study provides valuable insights into how users perceive and respond to LLMs' expressions of uncertainty. However, there are a few limitations and areas for further research:

The experiment used a fictional search engine, so the findings may not fully generalize to real-world LLM deployments. Further research is needed to validate the results in more realistic settings.
The study focused on medical questions, but the impacts of uncertainty expressions may vary across different domains. Exploring a wider range of tasks and applications would be beneficial.
The researchers note that the effects of general uncertainty expressions were weaker and not statistically significant. More research is needed to understand the nuances of how different phrasings of uncertainty affect user behavior.
The study did not examine long-term effects or how users' perceptions of the LLM might change over repeated interactions. Longitudinal research would provide valuable insights.

Overall, this study is an important step in understanding how to harness the power of LLM uncertainty to improve user decision-making and reduce overreliance. Careful design and testing of uncertainty expressions will be crucial as these models become more widely deployed.

Conclusion

This study demonstrates that how LLMs communicate their uncertainty can have a significant impact on user behavior and performance. Expressing uncertainty in a first-person manner (e.g., "I'm not sure") seems to be more effective at reducing overreliance on incorrect answers, compared to more general uncertainty expressions.

As LLMs become more widely used, it will be important for developers to carefully consider how to quantify and communicate the uncertainty of these models. User testing and iterative design will be crucial to ensuring LLMs are deployed in a way that supports informed and accurate decision-making.

This research represents an important step towards harnessing the power of LLM uncertainty to improve the reliability and trustworthiness of these powerful language models.

This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!

Follow @aimodelsfyi on 𝕏 →

Related Papers

💬

I'm Not Sure, But...: Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust

Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan

Widely deployed large language models (LLMs) can produce convincing yet incorrect outputs, potentially misleading users who may rely on them as if they were correct. To reduce such overreliance, there have been calls for LLMs to communicate their uncertainty to end users. However, there has been little empirical work examining how users perceive and act upon LLMs' expressions of uncertainty. We explore this question through a large-scale, pre-registered, human-subject experiment (N=404) in which participants answer medical questions with or without access to responses from a fictional LLM-infused search engine. Using both behavioral and self-reported measures, we examine how different natural language expressions of uncertainty impact participants' reliance, trust, and overall task performance. We find that first-person expressions (e.g., I'm not sure, but...) decrease participants' confidence in the system and tendency to agree with the system's answers, while increasing participants' accuracy. An exploratory analysis suggests that this increase can be attributed to reduced (but not fully eliminated) overreliance on incorrect answers. While we observe similar effects for uncertainty expressed from a general perspective (e.g., It's not clear, but...), these effects are weaker and not statistically significant. Our findings suggest that using natural language expressions of uncertainty may be an effective approach for reducing overreliance on LLMs, but that the precise language used matters. This highlights the importance of user testing before deploying LLMs at scale.

5/16/2024

Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty

Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, Maarten Sap

As natural language becomes the default interface for human-AI interaction, there is a need for LMs to appropriately communicate uncertainties in downstream applications. In this work, we investigate how LMs incorporate confidence in responses via natural language and how downstream users behave in response to LM-articulated uncertainties. We examine publicly deployed models and find that LMs are reluctant to express uncertainties when answering questions even when they produce incorrect responses. LMs can be explicitly prompted to express confidences, but tend to be overconfident, resulting in high error rates (an average of 47%) among confident responses. We test the risks of LM overconfidence by conducting human experiments and show that users rely heavily on LM generations, whether or not they are marked by certainty. Lastly, we investigate the preference-annotated datasets used in post training alignment and find that humans are biased against texts with uncertainty. Our work highlights new safety harms facing human-LM interactions and proposes design recommendations and mitigating strategies moving forward.

7/11/2024

Perceptions of Linguistic Uncertainty by Language Models and Humans

Catarina G Belem, Markelle Kelly, Mark Steyvers, Sameer Singh, Padhraic Smyth

Uncertainty expressions such as ``probably'' or ``highly unlikely'' are pervasive in human language. While prior work has established that there is population-level agreement in terms of how humans interpret these expressions, there has been little inquiry into the abilities of language models to interpret such expressions. In this paper, we investigate how language models map linguistic expressions of uncertainty to numerical responses. Our approach assesses whether language models can employ theory of mind in this setting: understanding the uncertainty of another agent about a particular statement, independently of the model's own certainty about that statement. We evaluate both humans and 10 popular language models on a task created to assess these abilities. Unexpectedly, we find that 8 out of 10 models are able to map uncertainty expressions to probabilistic responses in a human-like manner. However, we observe systematically different behavior depending on whether a statement is actually true or false. This sensitivity indicates that language models are substantially more susceptible to bias based on their prior knowledge (as compared to humans). These findings raise important questions and have broad implications for human-AI alignment and AI-AI communication.

7/23/2024

Finetuning Language Models to Emit Linguistic Expressions of Uncertainty

Arslan Chaudhry, Sridhar Thiagarajan, Dilan Gorur

Large language models (LLMs) are increasingly employed in information-seeking and decision-making tasks. Despite their broad utility, LLMs tend to generate information that conflicts with real-world facts, and their persuasive style can make these inaccuracies appear confident and convincing. As a result, end-users struggle to consistently align the confidence expressed by LLMs with the accuracy of their predictions, often leading to either blind trust in all outputs or a complete disregard for their reliability. In this work, we explore supervised finetuning on uncertainty-augmented predictions as a method to develop models that produce linguistic expressions of uncertainty. Specifically, we measure the calibration of pre-trained models and then fine-tune language models to generate calibrated linguistic expressions of uncertainty. Through experiments on various question-answering datasets, we demonstrate that LLMs are well-calibrated in assessing their predictions, and supervised finetuning based on the model's own confidence leads to well-calibrated expressions of uncertainty, particularly for single-claim answers.

9/19/2024