UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations

Overview

• This paper explores the task of "uncommonsense abductive reasoning" - reasoning about unusual, unexpected, and unlikely situations to generate an explanation that makes the unexpected outcome more likely.
• The researchers curate a new English language corpus called "UNcommonsense" to investigate this ability, which differs from existing commonsense reasoning datasets focused on everyday situations.
• They compare the performance of large language models and human explainers, finding that human-written explanations enhanced with model outputs achieve the highest quality.
• The paper also experiments with imitation learning algorithms to train open and accessible language models on this task, showing improvements over the standard supervised fine-tuning approach.

Plain English Explanation

The paper focuses on a type of reasoning called "uncommonsense abductive reasoning." This involves trying to explain surprising or unlikely events, rather than the more common everyday situations that existing commonsense reasoning research has looked at.

To study this, the researchers created a new dataset called "UNcommonsense" with examples of unusual, unexpected, and unlikely scenarios. They then compared how well large language models and human explainers could come up with explanations that made these unexpected outcomes seem more plausible.

The key finding is that the best results came from a hybrid approach, where human-written explanations were enhanced using outputs from the language models. This allowed them to balance specificity and diversity in the explanations.

The researchers also experimented with new training methods, called "imitation learning," to teach language models to perform this kind of uncommonsense reasoning. These methods outperformed the standard fine-tuning approach, suggesting they're a promising direction for developing more capable commonsense reasoning systems.

Technical Explanation

The paper introduces the task of "uncommonsense abductive reasoning," where the goal is to generate an explanation that makes an unexpected or unlikely outcome more plausible given some context. This differs from existing work on commonsense reasoning, which has focused on more common, everyday situations.

To study this, the researchers curate a new dataset called "UNcommonsense" containing examples of unusual scenarios with unexpected outcomes. They then compare the performance of large language models and human explainers on this task, finding that model-augmented human-written explanations achieve the highest quality.

Additionally, the paper experiments with several imitation learning algorithms to train open and accessible language models on this task. These methods consistently outperform the standard supervised fine-tuning approach, as evaluated by human raters.

Critical Analysis

The paper provides a valuable contribution by introducing the "uncommonsense abductive reasoning" task and the associated UNcommonsense dataset. This expands on the typical commonsense reasoning benchmarks, which tend to focus on more commonplace situations.

However, the paper acknowledges some limitations of the dataset and task. For example, the scenarios may still be somewhat constrained and not fully representative of the breadth of unexpected real-world situations. Additionally, the evaluation relies on human ratings, which could be subjective and may not capture all aspects of commonsense reasoning ability.

Further research could explore broader evaluations of reasoning behavior beyond just the quality of generated explanations. Investigating the underlying reasoning processes and commonsense knowledge used by both humans and models could provide additional insights.

Conclusion

This paper makes an important contribution by introducing the task of "uncommonsense abductive reasoning" and a corresponding dataset to study the ability of language models and humans to reason about unexpected and unlikely situations. The key findings suggest that a hybrid approach, combining model outputs with human-written explanations, can achieve the highest quality results.

Additionally, the experiments with imitation learning techniques show promise for developing more capable commonsense reasoning systems in an open and accessible way. Overall, this work expands the frontiers of commonsense reasoning research and points to exciting directions for improving the reasoning abilities of artificial intelligence systems.