MAmmoTH2: Scaling Instructions from the Web

Overview

• This paper explores a new approach to improve the reasoning abilities of large language models (LLMs) through instruction tuning.
• The key factors for successful instruction tuning are data quality and scalability.
• The researchers propose a method to efficiently harvest 10 million naturally occurring instruction data from the pre-training web corpus, without relying on costly human annotation or GPT-4 distillation.
• The resulting MAmmoTH2 models show significant performance boosts on reasoning benchmarks, outperforming previous instruction tuning approaches.
• Further training on public instruction tuning datasets yields the MAmmoTH2-Plus model, which achieves state-of-the-art results on various reasoning and chatbot tasks.

Plain English Explanation

The paper describes a new way to make large language models better at reasoning and problem-solving. The key is instruction tuning, which means training the models on a large number of examples of instructions and the corresponding responses.

The researchers found that the quality and quantity of the instruction data are crucial for this process. Most previous approaches relied on either human-sourced data or distilling from the powerful GPT-4 model. However, both of these methods can be expensive and time-consuming.

Instead, the researchers developed a clever way to automatically extract high-quality instruction-response pairs from the internet data used to train the original language models. This involved recalling relevant documents, extracting the pairs, and then refining them using open-source language models.

The resulting MAmmoTH2 models showed a big boost in their ability to reason and solve problems, without needing any additional training on specialized datasets. For example, the performance on a math reasoning test went up from 11% to 34%, and on a common sense reasoning test from 36% to 67%.

By training the MAmmoTH2 models further on public instruction datasets, the researchers were able to create the even more capable MAmmoTH2-Plus model, which sets new state-of-the-art results on several reasoning and chatbot benchmarks.

Overall, this work demonstrates a new, more efficient way to build high-quality instruction tuning data for improving the reasoning abilities of large language models, without the need for expensive human labeling or distillation from the most advanced models.

Technical Explanation

The key steps of the researchers' approach are:

1. Recalling relevant documents: The researchers first retrieve a large set of potentially relevant documents from the pre-training web corpus, using an information retrieval system.

2. Extracting instruction-response pairs: They then extract pairs of instructions and the corresponding responses from the recalled documents, using heuristics and open-source language models to identify the relevant text.

3. Refining the extracted pairs: Finally, the extracted pairs are further refined using additional filtering and scoring with open-source LLMs, to ensure high quality.

This process allows the researchers to efficiently harvest a dataset of 10 million naturally occurring instruction-response pairs, without the need for costly human annotation or distillation from GPT-4.

The researchers then fine-tune base LLMs on this dataset, creating the MAmmoTH2 model family. Experiments show that the MAmmoTH2 models significantly outperform previous instruction tuning approaches on a range of reasoning benchmarks, such as MATH and GSM8K.

Further training the MAmmoTH2 models on public instruction tuning datasets, such as those used in the TextSquare project, yields the MAmmoTH2-Plus model, which achieves state-of-the-art performance on several reasoning and chatbot tasks.

Critical Analysis

The researchers acknowledge that their approach has some limitations. For example, the quality of the extracted instruction-response pairs is still dependent on the accuracy of the retrieval and filtering heuristics, and could be further improved.

Additionally, the researchers did not directly compare the performance of the MAmmoTH2 models to those trained on human-annotated or GPT-4 distilled data. It would be interesting to see how the different data sources and approaches compare in terms of both model performance and the diversity/quality of the resulting instruction-following capabilities.

Further research could also explore ways to enhance the generalization abilities of the MAmmoTH2 models, beyond just the reasoning benchmarks considered in this paper. Investigating the model's robustness, common sense understanding, and ability to follow open-ended instructions would provide a more comprehensive evaluation.

Overall, this work presents a promising new approach to instruction tuning that could significantly improve the reasoning and problem-solving abilities of large language models, without the need for costly data collection and curation processes.

Conclusion

This paper introduces a novel paradigm for efficiently harvesting high-quality instruction-response data from the web, to enhance the reasoning capabilities of large language models through instruction tuning.

By automatically extracting and refining 10 million natural instruction examples, the researchers were able to create the MAmmoTH2 models, which outperformed previous instruction tuning approaches on a range of reasoning benchmarks. Further training on public instruction datasets led to the state-of-the-art MAmmoTH2-Plus model.

This work demonstrates the potential for scalable, high-quality instruction data to be a key driver for improving the general intelligence of large language models, without the need for costly human annotation or distillation from the most advanced models. The insights from this research could pave the way for more efficient and effective instruction tuning approaches in the future.