AI Reliance and Decision Quality: Fundamentals, Interdependence, and the Effects of Interventions
0
🤖
Sign in to get full access
Overview
- The paper examines the relationship between reliance on AI recommendations and the quality of decisions made with AI assistance.
- It argues that reliance and decision quality are often conflated in current research on AI-assisted decision-making.
- The paper aims to disentangle and formalize the relationship between reliance and decision quality, and characterize the conditions for achieving human-AI complementarity.
Plain English Explanation
When humans use AI systems to help make decisions, the expectation is that the human can override the AI's wrong recommendations. However, in practice, humans often cannot assess the correctness of the AI's advice and end up following bad recommendations or ignoring good ones.
The way humans rely on AI recommendations can significantly impact the quality of the final decision, but these two factors - reliance and decision quality - are often not clearly distinguished in the existing research. This paper tries to separate and clarify the relationship between reliance and decision quality, and identify the circumstances where humans and AI can work together effectively.
To illustrate this, the paper proposes a visual framework that shows how reliance and decision quality are connected. This framework can help interpret findings from studies on interventions like explanations that aim to improve human-AI collaboration.
The key message is that researchers and developers need to be mindful of the difference between how much humans rely on AI and the ultimate quality of the decisions made, as these are related but distinct factors in AI-assisted decision-making.
Technical Explanation
The paper starts by noting that a central promise of having a human-in-the-loop for AI-assisted decision-making is that the human can complement the AI system by overriding its incorrect recommendations. However, in reality, humans often struggle to assess the correctness of the AI's advice and end up adhering to wrong recommendations or overriding correct ones.
To disentangle the relationship between reliance and decision quality, the authors propose a formal decision-theoretic framework. This framework models the human's decision-making process as a function of the AI's recommendation, the human's assessment of the recommendation, and the human's willingness to rely on the AI.
Using this framework, the authors characterize the conditions under which human-AI complementarity can be achieved. They show that high reliance does not necessarily lead to high decision quality, and vice versa. The framework also allows the authors to interpret the effects of interventions like explanations on both reliance and decision quality.
The paper presents a visual representation of the framework, which the authors use to illustrate the distinct implications of different reliance strategies on decision quality. This visual tool can help researchers and practitioners reason about and compare the effects of design choices on human-AI interaction.
Critical Analysis
The paper makes a compelling case for the need to distinguish between reliance behavior and decision quality in the context of AI-assisted decision-making. By formalizing this distinction, the authors provide a useful framework for analyzing and interpreting empirical findings in this domain.
One limitation of the work is that it does not directly address the underlying cognitive and behavioral factors that drive human reliance on AI recommendations. Understanding these factors could help inform the design of better human-AI collaboration systems.
Additionally, the paper focuses on binary decision tasks, whereas many real-world decisions involve more complex, multi-faceted choices. Extending the framework to handle such scenarios could enhance its applicability.
Future research could also explore the role of trust, transparency, and explainability in shaping reliance and decision quality. Investigating how these factors interact with the proposed framework could yield valuable insights.
Conclusion
This paper highlights the importance of distinguishing between reliance on AI recommendations and the ultimate quality of decisions made with AI assistance. By formalizing the relationship between these two factors, the authors provide a valuable decision-theoretic framework for analyzing and interpreting human-AI interaction.
The proposed visual framework can serve as a useful tool for researchers and practitioners to reason about the distinct implications of different reliance strategies on decision quality. This work lays the groundwork for further investigation into the cognitive and behavioral aspects of human-AI complementarity in decision-making.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
🤖
0
AI Reliance and Decision Quality: Fundamentals, Interdependence, and the Effects of Interventions
Jakob Schoeffer, Johannes Jakubik, Michael Voessing, Niklas Kuehl, Gerhard Satzger
In AI-assisted decision-making, a central promise of having a human-in-the-loop is that they should be able to complement the AI system by overriding its wrong recommendations. In practice, however, we often see that humans cannot assess the correctness of AI recommendations and, as a result, adhere to wrong or override correct advice. Different ways of relying on AI recommendations have immediate, yet distinct, implications for decision quality. Unfortunately, reliance and decision quality are often inappropriately conflated in the current literature on AI-assisted decision-making. In this work, we disentangle and formalize the relationship between reliance and decision quality, and we characterize the conditions under which human-AI complementarity is achievable. To illustrate how reliance and decision quality relate to one another, we propose a visual framework and demonstrate its usefulness for interpreting empirical findings, including the effects of interventions like explanations. Overall, our research highlights the importance of distinguishing between reliance behavior and decision quality in AI-assisted decision-making.
Read more8/29/2024
0
A Decision Theoretic Framework for Measuring AI Reliance
Ziyang Guo, Yifan Wu, Jason Hartline, Jessica Hullman
Humans frequently make decisions with the aid of artificially intelligent (AI) systems. A common pattern is for the AI to recommend an action to the human who retains control over the final decision. Researchers have identified ensuring that a human has appropriate reliance on an AI as a critical component of achieving complementary performance. We argue that the current definition of appropriate reliance used in such research lacks formal statistical grounding and can lead to contradictions. We propose a formal definition of reliance, based on statistical decision theory, which separates the concepts of reliance as the probability the decision-maker follows the AI's recommendation from challenges a human may face in differentiating the signals and forming accurate beliefs about the situation. Our definition gives rise to a framework that can be used to guide the design and interpretation of studies on human-AI complementarity and reliance. Using recent AI-advised decision making studies from literature, we demonstrate how our framework can be used to separate the loss due to mis-reliance from the loss due to not accurately differentiating the signals. We evaluate these losses by comparing to a baseline and a benchmark for complementary performance defined by the expected payoff achieved by a rational decision-maker facing the same decision task as the behavioral decision-makers.
Read more5/14/2024
➖
0
New!Questioning AI: Promoting Decision-Making Autonomy Through Reflection
Simon WS Fischer
Decision-making is increasingly supported by machine recommendations. In healthcare, for example, a clinical decision support system is used by the physician to find a treatment option for a patient. In doing so, people can rely too much on these systems, which impairs their own reasoning process. The European AI Act addresses the risk of over-reliance and postulates in Article 14 on human oversight that people should be able to remain aware of the possible tendency of automatically relying or over-relying on the output. Similarly, the EU High-Level Expert Group identifies human agency and oversight as the first of seven key requirements for trustworthy AI. The following position paper proposes a conceptual approach to generate machine questions about the decision at hand, in order to promote decision-making autonomy. This engagement in turn allows for oversight of recommender systems. The systematic and interdisciplinary investigation (e.g., machine learning, user experience design, psychology, philosophy of technology) of human-machine interaction in relation to decision-making provides insights to questions like: how to increase human oversight and calibrate over- and under-reliance on machine recommendations; how to increase decision-making autonomy and remain aware of other possibilities beyond automated suggestions that repeat the status-quo?
Read more9/17/2024
0
Towards Optimizing Human-Centric Objectives in AI-Assisted Decision-Making With Offline Reinforcement Learning
Zana Buc{c}inca, Siddharth Swaroop, Amanda E. Paluch, Susan A. Murphy, Krzysztof Z. Gajos
Imagine if AI decision-support tools not only complemented our ability to make accurate decisions, but also improved our skills, boosted collaboration, and elevated the joy we derive from our tasks. Despite the potential to optimize a broad spectrum of such human-centric objectives, the design of current AI tools remains focused on decision accuracy alone. We propose offline reinforcement learning (RL) as a general approach for modeling human-AI decision-making to optimize human-AI interaction for diverse objectives. RL can optimize such objectives by tailoring decision support, providing the right type of assistance to the right person at the right time. We instantiated our approach with two objectives: human-AI accuracy on the decision-making task and human learning about the task and learned decision support policies from previous human-AI interaction data. We compared the optimized policies against several baselines in AI-assisted decision-making. Across two experiments (N=316 and N=964), our results demonstrated that people interacting with policies optimized for accuracy achieve significantly better accuracy -- and even human-AI complementarity -- compared to those interacting with any other type of AI support. Our results further indicated that human learning was more difficult to optimize than accuracy, with participants who interacted with learning-optimized policies showing significant learning improvement only at times. Our research (1) demonstrates offline RL to be a promising approach to model human-AI decision-making, leading to policies that may optimize human-centric objectives and provide novel insights about the AI-assisted decision-making space, and (2) emphasizes the importance of considering human-centric objectives beyond decision accuracy in AI-assisted decision-making, opening up the novel research challenge of optimizing human-AI interaction for such objectives.
Read more4/16/2024