Are self-explanations from Large Language Models faithful?

Overview

• Large language models (LLMs) excel at many tasks and can provide self-explanations for their reasoning
• However, these self-explanations may be convincing but wrong, leading to unjustified confidence in the model
• It's important to measure whether self-explanations truly reflect the model's behavior, a concept called "interpretability-faithfulness"
• This is challenging because the ground truth is inaccessible and many LLMs only have an inference API
• To address this, the researchers propose using self-consistency checks to measure faithfulness

Plain English Explanation

Large language models are incredibly capable at various tasks, and they can even explain their own reasoning, which is known as "self-explanations." However, these self-explanations can sometimes be convincing but actually wrong, causing people to trust the model's outputs more than they should.

To address this issue, the researchers wanted to find a way to measure whether these self-explanations genuinely reflect how the model is actually behaving. They call this "interpretability-faithfulness," but it's challenging to do because the true inner workings of the model are not accessible, and many of these powerful language models can only be used through an online interface.

To overcome this, the researchers suggest using "self-consistency checks." The idea is that if a model says certain words are important for making a prediction, then it shouldn't be able to make that prediction without those words. By checking this self-consistency, the researchers can get a sense of how faithful the model's self-explanations really are.

This approach of using self-consistency checks to evaluate faithfulness hasn't been successfully applied to the various types of self-explanations that large language models can provide, such as counterfactuals, feature attribution, and redaction explanations. The researchers' results show that faithfulness depends on the specific explanation, model, and task - so self-explanations shouldn't be blindly trusted in general.

Technical Explanation

The researchers investigate the "interpretability-faithfulness" of the self-explanations provided by large language models (LLMs). Interpretability-faithfulness refers to how well the model's self-explanations align with its actual behavior. This is important because convincing but incorrect self-explanations can lead to unjustified trust in the model's outputs.

Measuring interpretability-faithfulness is challenging, as the ground truth of the model's internal reasoning is inaccessible, and many LLMs only provide an inference API. To address this, the researchers propose using "self-consistency checks" - for example, verifying that an LLM cannot make a prediction without the words it claims are important.

While self-consistency checks are a common approach to evaluating faithfulness, the researchers note that this technique has not been successfully applied to the various types of self-explanations that LLMs can provide, such as counterfactuals, feature attribution, and redaction explanations.

The researchers' results demonstrate that faithfulness is dependent on the specific explanation, model, and task. For example, they find that for sentiment classification, counterfactuals are more faithful for the Llama2 model, feature attribution is more faithful for Mistral, and redaction is more faithful for Falcon 40B. This suggests that self-explanations should not be trusted in general without careful evaluation.

Critical Analysis

The researchers have made an important contribution by highlighting the need to carefully evaluate the faithfulness of self-explanations provided by large language models. Their proposed use of self-consistency checks is a reasonable approach, though it does have some limitations.

One potential issue is that self-consistency checks may not capture all aspects of faithfulness. For example, a model could pass a self-consistency test but still have subtle biases or flaws in its reasoning that are not detected. Additionally, the researchers note that the ground truth of the model's internal reasoning is inaccessible, which means there may be aspects of faithfulness that cannot be fully evaluated.

It's also worth considering whether self-explanations are the most appropriate way for large language models to communicate their reasoning. Alternative approaches, such as concept-based explanations, may be more faithful and interpretable. The researchers could have delved deeper into these alternative explanation methods and their potential advantages.

Overall, the researchers have raised an important issue and proposed a reasonable approach to address it. However, further research is needed to fully understand the limitations of self-explanations and develop more reliable methods for evaluating the faithfulness of large language models.

Conclusion

This research highlights the critical need to carefully evaluate the faithfulness of self-explanations provided by large language models. While these models excel at many tasks and can generate compelling self-explanations, the researchers demonstrate that these self-explanations may not always align with the model's true behavior.

By proposing the use of self-consistency checks to measure interpretability-faithfulness, the researchers have provided a valuable tool for assessing the reliability of large language model outputs. Their finding that faithfulness is dependent on the specific explanation, model, and task underscores the importance of not blindly trusting self-explanations and instead subjecting them to rigorous evaluation.

As large language models become increasingly influential in a wide range of applications, understanding their limitations and potential flaws is crucial. This research represents an important step towards developing more transparent and trustworthy AI systems that can be reliably deployed in real-world scenarios.