Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

Overview

• Researchers have created a large, diverse corpus of English text called Dolma, containing over 3 trillion tokens from various sources including web content, scientific papers, code, books, social media, and encyclopedias.
• The goal is to facilitate more open and reproducible research on language models by providing a well-documented dataset for pretraining.
• The researchers share insights from their data curation process and release open-source tools to enable others to build similar large-scale datasets.

Plain English Explanation

The paper discusses the creation of a new dataset called Dolma, which is a massive collection of English text from many different sources. This includes things like websites, academic papers, computer code, books, social media, and encyclopedias. In total, the dataset contains over 3 trillion individual words or "tokens."

The main reason the researchers built Dolma is to support more open and scientific research on language models - the large AI systems that are trained on huge amounts of text data to understand and generate human language. Currently, the datasets used to train the best-performing language models are often kept secret by the companies that develop them. This makes it hard for other researchers to study how the training data affects the models' capabilities and limitations.

By openly sharing the Dolma dataset and documenting how it was created, the researchers hope to enable more people to study language models and advance the field. They also release the tools they used to build Dolma, so others can create similar large-scale datasets for their own research.

Throughout the process, the researchers learned important lessons about best practices for curating diverse, high-quality data for training language models. They share these insights in the paper, providing a roadmap for others who want to build their own large text corpora.

Technical Explanation

The paper presents the Dolma dataset, a 3-trillion-token English corpus, and the researchers' approach to its creation. Dolma was constructed from a wide variety of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials.

The researchers extensively document Dolma, including its design principles, construction details, and a summary of its contents. They share analyses and experimental results on intermediate states of Dolma to highlight important data curation practices, such as filtering low-quality or redundant content.

Additionally, the researchers open-source their data curation toolkit to enable reproduction of their work and support further research in large-scale data curation for language models. This aligns with their goal of facilitating more open and reproducible scientific research on language modeling, which is often hindered by the lack of transparency around commercial models' training data.

The paper's findings emphasize the value of curating diverse, high-quality datasets like Dolma to advance the state of the art in language modeling. By providing this resource, the researchers hope to enable researchers to better understand how training data impacts model capabilities and limitations, leading to more robust and generalizable language models.

Critical Analysis

The paper makes a compelling case for the importance of open and reproducible research in language modeling, a field that has been dominated by commercial models with opaque training data. The creation of the Dolma dataset is a significant step towards addressing this problem, as it provides a large-scale, well-documented corpus for researchers to use in their studies.

One potential limitation of the Dolma dataset is that, like many web-based corpora, it may contain biases and artifacts that could influence the performance and behavior of language models trained on it. The researchers acknowledge this issue and mention plans to further analyze and mitigate such biases in future work.

Additionally, while the dataset is made available to the research community, the sheer size of Dolma (3 trillion tokens) may pose challenges for some researchers in terms of storage, processing, and experimentation. The researchers could potentially explore ways to make the dataset more accessible, such as providing subsets or sampling techniques.

Overall, the Dolma dataset and the researchers' approach to its creation represent an important contribution to the field of language modeling. By embracing open science and sharing their insights, the researchers have laid the groundwork for more collaborative and impactful research in this area.

Conclusion

The paper presents the Dolma dataset, a large and diverse English corpus of over 3 trillion tokens, as a means to facilitate more open and reproducible research on language modeling. By openly sharing this dataset and the tools used to create it, the researchers aim to enable other researchers to better understand the impact of training data on language model capabilities and limitations.

The open-sourcing of the Dolma dataset and the researchers' data curation toolkit is a significant step towards promoting transparency and collaboration in the field of language modeling. This work has the potential to drive advancements in the development of more robust and generalizable language models, ultimately benefiting a wide range of applications that rely on natural language processing.