Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluation

Overview

• This paper explores the use of large language models (LLMs) with long context capabilities to automatically generate synthetic reading comprehension data from entire books.
• Previous efforts relied on crowdsourcing, but the emergence of transformers with 1 million+ token context size now enables fully automatic approaches.
• The goal is to test LLM capabilities to analyze, understand, and reason over problems requiring detailed comprehension of long text, such as questions about character arcs, themes, or consequences of early actions.
• The authors propose a pipeline for automatic data generation including question generation, answering, and model scoring using an "Evaluator".
• They find that a relative approach comparing model answers in a pairwise fashion and ranking with a Bradley-Terry model provides more consistent and differentiating scoring than an absolute scorer.
• LLMs from different model families also show moderate agreement in their ratings.

Plain English Explanation

The researchers wanted to see if powerful language models could be used to automatically create reading comprehension datasets from entire books, rather than relying on crowd-sourcing like previous efforts. The idea is to test these models' ability to deeply understand long stretches of text, such as by asking questions about the characters' journeys, the broader themes of a story, or how earlier events impact later parts of the plot.

To do this, the researchers developed a complete system that can generate questions, get answers from language models, and then evaluate the quality of those answers. Interestingly, they found that comparing the answers from different models in a relative way (i.e. ranking them against each other) works better for scoring than just judging each answer individually. They also noticed that language models from different families tend to agree moderately on their evaluations.

The researchers tested their approach using an existing high-quality reading comprehension dataset, and found that using an entire book as context helps the language models perform better than just relying on their general knowledge or retrieving relevant passages. This suggests these models are developing impressive capabilities for deeply understanding lengthy narratives.

Technical Explanation

The paper proposes a pipeline for automatically generating synthetic reading comprehension datasets from entire books, leveraging the long context capabilities of modern transformers.

The key components are:

1. Question Generation: The system automatically generates questions about the content of the books.
2. Answer Generation: Language models are used to provide answers to the generated questions.
3. Evaluator: A scoring mechanism is developed to assess the quality of the model-generated answers.

The authors find that a relative scoring approach, where answers are compared pairwise between models and ranked using a Bradley-Terry model, provides more consistent and differentiating results than an absolute scoring mechanism. They also observe moderate agreement between language models from different families in their evaluations.

The system is tested on the NarrativeQA dataset, where it shows strong agreement with human judgments and even identifies errors in the original dataset. Importantly, the authors demonstrate that using an entire book as context leads to superior reading comprehension performance compared to baseline approaches that only leverage parametric knowledge or retrieve relevant passages.

Critical Analysis

The paper presents a compelling approach to automatically generate high-quality reading comprehension datasets from books, addressing the limitations of previous crowd-sourcing efforts. By leveraging the long context capabilities of modern language models, the authors are able to test these models' ability to deeply understand and reason over lengthy narratives.

One potential limitation is the reliance on a single dataset, NarrativeQA, for grounding and evaluation. It would be valuable to see how the system performs on a broader range of reading comprehension benchmarks, especially those that cover different genres or styles of text.

Additionally, the authors note that their relative scoring approach introduces some subjective elements, as the rankings depend on the specific set of models being compared. It could be worthwhile to explore alternative scoring mechanisms that provide more objective, absolute measures of answer quality.

Finally, while the paper demonstrates the advantages of using full-book context, it would be interesting to further investigate the specific types of questions or reasoning tasks that benefit most from this long-range understanding, versus those that can be adequately addressed using more local contextual information.

Overall, this research represents an important step forward in empowering language models to engage with and comprehend lengthy, complex narratives, with promising applications in areas like educational assessment, literary analysis, and interactive storytelling.

Conclusion

This paper presents a novel approach for automatically generating high-quality reading comprehension datasets from entire books, leveraging the long-context capabilities of modern language models. By developing a holistic pipeline for question generation, answer production, and relative model scoring, the authors demonstrate the ability of these models to deeply analyze and reason over lengthy narratives.

The findings suggest that using full-book context can significantly improve reading comprehension performance compared to more limited approaches, and that language models from different families exhibit moderate agreement in their evaluations. This work lays the groundwork for further advancements in applying large language models to complex, long-form understanding tasks with broad applications in education, literary studies, and beyond.