Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model

Overview

• This paper explores a new approach called "markup-and-mask" for generating rationales that explain the reasoning behind open-book question answering systems.
• The key idea is to augment the input passage with free-text markup that allows each sentence to stand on its own, and then select a sub-span of the marked-up passage as the rationale.
• To train this system without annotations, the authors leverage in-context learning, using a frozen pretrained language model as a "teacher" to generate silver-annotated data.
• The student model is then fine-tuned on the subset of rationales that led to correct answers, creating a trustworthy pipeline where the rationale acts as a bottleneck between the passage and the answer.

Plain English Explanation

Explainable AI systems should not only provide accurate answers, but also justify their reasoning in a way that humans can understand and verify. This paper proposes a new approach called "markup-and-mask" to generate these kinds of explanations.

The key idea is to start with a passage of text, and add free-text annotations or "markup" to each sentence. This markup helps the sentence stand on its own, without relying on the context of the full passage. Then, the system selects a subset of the marked-up sentences as the "rationale" - the explanation for why the answer was chosen.

To train this system without having lots of annotated data, the authors use a clever trick. They take a large, pretrained language model and use it as a "teacher" to generate example rationales. They then train a smaller "student" model to mimic the teacher's rationales, but with the constraint that the rationale must lead to the correct answer. This ensures the student model is being honest and transparent in its reasoning.

The result is a pipeline system where the rationale acts as a bottleneck between the input passage and the final answer. This makes the system more trustworthy, since humans can inspect the rationale to understand and verify the model's reasoning.

Technical Explanation

The paper introduces a new approach for generating rationales in open-book question answering systems called "markup-and-mask". This combines aspects of extractive and free-text explanations.

In the "markup" phase, the input passage is augmented with free-text annotations that allow each sentence to stand on its own, without relying on the broader context. Then, in the "masking" phase, a sub-span of the marked-up passage is selected as the rationale.

To train this system without annotations, the authors leverage in-context learning. They use a frozen, pretrained language model as a "teacher" to generate silver-annotated data. Specifically, they send a series of prompts to the teacher model, which responds with potential rationales. The authors then fine-tune a smaller "student" model to mimic the teacher's rationales, but with the constraint that the rationale must lead to the correct answer.

This creates a trustworthy pipeline system, where the rationale acts as a bottleneck between the passage and the final answer. The untrusted teacher model operates without such constraints, while the student model is "honest" in the sense that its rationale must justify its output.

Critical Analysis

The markup-and-mask approach proposed in this paper is a novel and promising direction for building explainable question answering systems. The use of in-context learning to generate silver-annotated data is a clever way to overcome the challenge of obtaining human-annotated rationales. This allows the system to be trained without the need for costly and time-consuming manual annotation.

However, the authors acknowledge that the markup-and-mask rationales may not always be as informative as human-written explanations. There is also the potential risk of the student model "gaming" the system by generating rationales that appear plausible but do not truly reflect its reasoning.

Further research is needed to explore the robustness of this approach and its ability to scale to more complex question answering tasks. The authors suggest several potential directions, such as incorporating additional constraints or leveraging self-supervised techniques to improve the quality of the rationales.

Overall, this paper presents an intriguing approach to the important problem of building trustworthy and explainable AI systems. While there are still challenges to address, the markup-and-mask technique offers a promising step towards more transparent and accountable question answering systems.

Conclusion

This paper introduces a new "markup-and-mask" approach for generating rationales in open-book question answering systems. By augmenting input passages with free-text markup and leveraging in-context learning to generate silver-annotated data, the authors demonstrate a way to train explainable AI systems without the need for costly human annotations.

The resulting pipeline model is "honest" in the sense that the rationale acts as a bottleneck between the input and the final answer, making the system's reasoning more transparent and trustworthy. While further research is needed to address the limitations and scale the approach, this work represents an important step forward in the quest for AI systems that can not only provide accurate answers, but also justify their reasoning in a way that humans can understand and verify.