Leveraging Counterfactual Paths for Contrastive Explanations of POMDP Policies

Introduction

The paper discusses the importance of Explainable Artificial Intelligence (XAI) in enabling transparency and trust in AI systems, especially in mission-critical roles such as search and rescue. Partially observable Markov decision processes (POMDPs) provide a flexible framework for reasoning over state and transition uncertainty and lend themselves well to explanations. Contrastive explanations and counterfactuals can help make systems more interpretable and gain the trust of end users.

The authors aim to explore the use of user-given counterfactuals to provide contrastive explanations for POMDP policies in the context of a Search and Rescue (SAR) POMDP example. They provide an overview of explainable planning, which can be organized into model-based algorithm-agnostic and algorithm-specific explanations. Many works have sought to increase the explainability of planning in the context of MDPs by offering contrastive explanations in terms of expected feature occupancy.

POMDPs introduce an added layer of complexity by reasoning over uncertainty in the partially observable state. Several works have explored interactions between the autonomy and the end user to enhance trust, while others provide explanations in terms of the POMDP components. The authors propose a methodology for generating contrastive explanations for a SAR POMDP by contrasting policies against user-proposed counterfactuals via feature expectations.

Methodology

The paper proposes a method for explaining optimal policies in Partially Observable Markov Decision Processes (POMDPs) by comparing them to user-provided counterfactual paths. The approach focuses on the search and rescue (SAR) domain, where a human rescuer collaborates with an autonomous agent to find a missing person.

Key points:

1. The user can express a counterfactual path, which is translated into an open-loop policy.
2. The performance of the user's policy is compared to the optimal policy to form the basis of an explanation.
3. Feature expectations and factored rewards are used to represent the components contributing to the reward, providing insight into the problem objectives satisfied by the algorithm.
4. The feature expectation is defined as the expected sum of discounted feature occupancy functions over time, given a policy and initial belief.
5. The value of a policy can be expressed as the dot product of feature weights and feature expectations.
6. Explanations are generated by calculating feature expectations for both the optimal closed-loop policy and the user's open-loop policy, then translating the differences into plain language.
7. This method can be applied to any domain where a user provides a counterfactual policy, as long as feature expectations can be defined and calculated.

The approach aims to provide intuitive explanations for the executed path in the SAR POMDP domain by leveraging the visual nature of the problem and comparing the optimal policy to user-proposed alternatives.

Case Studies

The paper demonstrates an approach for explaining optimal policies in a search and rescue partially observable Markov decision process (SAR POMDP). The robot searches an n x n grid for a partially observable stationary target while visiting regions of interest, subject to a limited battery capacity.

Two case studies are presented:

1. Observable and unobservable objectives: The readily observable objective (cell of interest) and more valuable partially observable objective (hidden target) do not align. Feature expectations show the optimal policy finds the target more often than the user policy, leading to higher value despite visiting the cell of interest less frequently.

2. Resource constraints: Battery constraints prevent reaching a high-reward cell of interest proposed by the user policy. Feature expectations indicate the optimal policy finds the target more often and avoids battery depletion compared to the user policy.

Contrastive explanations comparing feature expectations of the optimal and user policies are proposed to justify the optimal policy in plain language, accounting for uncertainty in target location and problem constraints like limited battery.

Conclusion and Future Direction

The paper presents an approach for explaining paths generated by optimal solutions to POMDP search and rescue problems. The method uses a counterfactual user path as the basis for an open-loop policy, which is contrasted against an optimal policy through feature expectations. However, the approach assumes that the user maintains an open-loop policy that does not change with new information, which may not accurately capture user reasoning. Accounting for policy changes due to new information and the influence of observations on user's reasoning could lead to more effective POMDP explanations. The paper suggests that future methods should consider closed-loop reasoning while requiring limited user input. Additionally, providing proactive explanations of executed paths automatically in anticipation of user confusion could reduce user workload and dependence on domain-specific user feedback.