On Measuring Faithfulness or Self-consistency of Natural Language Explanations

Overview

• Large language models (LLMs) can provide post-hoc or Chain-of-Thought (CoT) explanations for their predictions, but these explanations may not always be faithful to the model's underlying reasoning.
• Recent work has designed tests to judge the faithfulness of these explanations, but this paper argues that these tests actually measure the self-consistency of the model's outputs, rather than faithfulness to its inner workings.
• The paper makes three key contributions:
  1. Clarifying that the existing faithfulness tests are actually self-consistency tests
  2. Constructing a Comparative Consistency Bank to compare these self-consistency tests across 11 LLMs and 5 tasks
  3. Introducing a new self-consistency measure called CC-SHAP that examines how a model's inputs contribute to both its predictions and explanations

Plain English Explanation

Large language models (LLMs) like GPT-3 and InstructGPT can try to explain their predictions by providing post-hoc or Chain-of-Thought (CoT) explanations. However, these explanations may not always accurately reflect the model's true reasoning process.

Researchers have developed tests to evaluate how faithful these explanations are to the model's inner workings. But this paper argues that these "faithfulness tests" are actually just measuring how consistent the model's outputs are with each other, rather than how faithful they are to the model's actual decision-making process.

The paper makes three key contributions:

1. Clarifying the nature of faithfulness tests: The authors explain that the existing faithfulness tests are really just tests of the model's self-consistency, not its faithfulness to its internal logic.

2. Comparative Consistency Bank: The authors create a set of 11 different LLMs and 5 different tasks, and use a variety of self-consistency tests to compare the models' behaviors.

3. CC-SHAP: The authors introduce a new self-consistency measure called CC-SHAP. This metric looks at how the model's inputs contribute to both its final predictions and the explanations it provides. This gives a more fine-grained view of the model's self-consistency than the existing faithfulness tests.

By taking this more nuanced look at model consistency, the authors hope to bring us closer to truly understanding the faithfulness of LLM explanations and how they relate to the models' internal decision-making processes.

Technical Explanation

The paper addresses the challenge of evaluating the faithfulness of post-hoc or Chain-of-Thought (CoT) explanations provided by large language models (LLMs). Recent work has developed tests that aim to judge the faithfulness of these explanations, but the authors argue that these tests actually measure the self-consistency of the model's outputs, rather than their faithfulness to the model's inner workings.

The paper makes three key contributions:

1. Clarifying the status of faithfulness tests: The authors characterize the existing faithfulness tests as measures of self-consistency, rather than true faithfulness. They argue that these tests examine how well the model's outputs (predictions and explanations) align with each other, but do not necessarily reflect the model's actual reasoning process.

2. Comparative Consistency Bank: To further explore this issue, the authors construct a Comparative Consistency Bank that compares the behavior of 11 different LLMs across 5 different tasks. They use a variety of self-consistency tests to evaluate the models' outputs.

3. CC-SHAP: The authors introduce a new self-consistency measure called CC-SHAP (Comparative Consistency SHAP). This metric looks at how the model's inputs contribute to both its final predictions and the explanations it provides. By examining this fine-grained relationship between inputs, predictions, and explanations, CC-SHAP aims to bring us closer to understanding the faithfulness of LLM explanations.

The authors' key insight is that the existing faithfulness tests do not actually measure faithfulness to the model's internal reasoning, but rather its self-consistency at the output level. The introduction of CC-SHAP represents a step towards a more nuanced understanding of LLM explanations and their relationship to the models' underlying decision-making processes.

Critical Analysis

The authors make a compelling case that the existing "faithfulness tests" for LLM explanations are actually measures of self-consistency, rather than true faithfulness to the models' internal workings. This is an important distinction, as it highlights the limitations of these tests and the need for more sophisticated approaches to evaluating explanation faithfulness.

The authors' construction of the Comparative Consistency Bank and the introduction of the CC-SHAP metric are valuable contributions that can help advance the field. By providing a common framework for evaluating LLM self-consistency across a range of models and tasks, the Bank allows for more systematic comparisons. And the CC-SHAP metric's focus on the relationship between inputs, predictions, and explanations is a step towards a more fine-grained understanding of LLM behavior.

That said, the paper does not address some potential limitations and areas for further research. For example, it's unclear how well the CC-SHAP metric captures the full complexity of LLM explanations, or how it might be affected by factors like task difficulty or model architecture. Additionally, the paper does not explore potential ways to directly measure faithfulness to the models' internal reasoning, which remains an important challenge.

Overall, this paper makes a significant contribution by clarifying the limitations of existing faithfulness tests and proposing new approaches for evaluating LLM explanations. By encouraging a more critical and nuanced understanding of these models, the authors help pave the way for more robust and reliable explanation methods in the future.

Conclusion

This paper challenges the current understanding of faithfulness tests for large language model (LLM) explanations, arguing that these tests actually measure the self-consistency of the model's outputs rather than their faithfulness to the underlying reasoning process.

The authors make three key contributions to address this issue:

1. Clarifying the status of faithfulness tests: They characterize existing tests as measures of self-consistency, not true faithfulness.
2. Comparative Consistency Bank: They create a framework to compare the self-consistency of 11 LLMs across 5 tasks.
3. CC-SHAP: They introduce a new fine-grained self-consistency metric that examines the relationship between model inputs, predictions, and explanations.

By taking this more nuanced approach, the authors bring us closer to understanding the true faithfulness of LLM explanations and their connection to the models' internal decision-making. This work represents an important step forward in evaluating and improving the transparency and reliability of these powerful language models.