MSciNLI: A Diverse Benchmark for Scientific Natural Language Inference

Overview

• Describes a new dataset called MSciNLI for the task of scientific Natural Language Inference (NLI)
• MSciNLI contains 132,320 sentence pairs from 5 new scientific domains, expanding on the existing SciNLI dataset
• Establishes strong baselines using fine-tuned Pre-trained Language Models (PLMs) and prompting Large Language Models (LLMs)
• Demonstrates the challenge of MSciNLI for both types of models, and that domain shift degrades performance

Plain English Explanation

The paper introduces a new dataset called MSciNLI for the task of scientific Natural Language Inference (NLI). NLI is the task of predicting the semantic relationship between two sentences, such as whether one sentence entails, contradicts, or is neutral with respect to another.

The researchers created MSciNLI by extracting 132,320 sentence pairs from research papers across 5 new scientific domains, expanding on the existing SciNLI dataset which was limited to the computational linguistics domain. This diversity allows for the study of how models perform across different scientific fields.

The researchers then established strong baseline performance on MSciNLI using two types of models: fine-tuned Pre-trained Language Models (PLMs) and prompted Large Language Models (LLMs). They found that MSciNLI is challenging for both types of models, with the best PLM and LLM baselines achieving Macro F1 scores of 77.21% and 51.77% respectively.

Furthermore, the researchers showed that a model's performance degrades when applied to a different scientific domain, demonstrating the diverse characteristics of the domains represented in MSciNLI. Finally, they showed that using MSciNLI and SciNLI in an intermediate task transfer learning setting can improve performance on downstream tasks in the scientific domain.

Technical Explanation

The paper introduces a new dataset called MSciNLI for the task of scientific Natural Language Inference (NLI). The NLI task involves predicting the semantic relationship (entailment, contradiction, or neutrality) between two sentences extracted from research articles.

The researchers created MSciNLI by extracting 132,320 sentence pairs from research papers across 5 new scientific domains: biology, chemistry, computer science, materials science, and physics. This expands on the existing SciNLI dataset, which was limited to the computational linguistics domain.

The researchers established strong baseline performance on MSciNLI using two types of models:

1. Pre-trained Language Models (PLMs): The researchers fine-tuned PLMs such as BERT, RoBERTa, and ELECTRA on the MSciNLI dataset, achieving the highest Macro F1 score of 77.21%.

2. Large Language Models (LLMs): The researchers prompted LLMs such as GPT-3 and PaLM on the MSciNLI task, achieving the highest Macro F1 score of 51.77%.

The results demonstrate that MSciNLI is a challenging dataset for both types of models, illustrating the diversity and complexity of the scientific NLI task.

Furthermore, the researchers showed that domain shift degrades the performance of scientific NLI models, indicating the diverse characteristics of the different scientific domains represented in the dataset.

Finally, the researchers used both the SciNLI and MSciNLI datasets in an intermediate task transfer learning setting, and showed that this can improve the performance of downstream tasks in the scientific domain.

Critical Analysis

The paper presents a valuable contribution to the field of scientific NLI by introducing the MSciNLI dataset, which expands the diversity of domains beyond the existing SciNLI dataset. This allows for the study of domain shift and the development of more robust models that can generalize across scientific fields.

However, the paper does not delve into potential limitations or caveats of the dataset or the baseline models. For example, it would be helpful to understand the potential biases or skewed distributions within the dataset, and how that might impact model performance.

Additionally, the paper could have discussed potential challenges in annotating the sentence pairs or in deriving the dataset from research papers. Understanding these challenges could provide insights into improving the dataset or the NLI task itself.

Finally, the paper could have explored more avenues for further research, such as investigating the use of domain-specific knowledge or multimodal information (e.g., incorporating figures and tables) to improve scientific NLI performance.

Conclusion

The paper introduces a new dataset called MSciNLI for the task of scientific Natural Language Inference (NLI), which expands on the existing SciNLI dataset by including sentence pairs from 5 new scientific domains. The researchers establish strong baselines using fine-tuned Pre-trained Language Models and prompted Large Language Models, demonstrating the challenging nature of the task.

The paper's key contribution is the expansion of the scientific NLI task to a more diverse set of domains, enabling the study of domain shift and the development of more robust models. The findings suggest that scientific NLI remains a challenging problem, and that further research is needed to improve model performance and generalization across scientific fields.