QuRating: Selecting High-Quality Data for Training Language Models

Overview

• This paper introduces a method called QuRating for selecting high-quality pre-training data for large language models (LLMs).
• Existing methods for data selection rely on simple heuristics, but QuRating aims to capture more nuanced human intuitions about data quality.
• The researchers investigate four key qualities - writing style, required expertise, facts & trivia, and educational value - and find that LLMs can discern these qualities, particularly when making pairwise judgments.
• They train a QuRater model to learn scalar ratings from pairwise judgments and use it to annotate a 260B training corpus with quality ratings for each of the four criteria.
• By sampling data based on these quality ratings, the researchers are able to train smaller language models that outperform baselines in terms of perplexity and in-context learning performance.

Plain English Explanation

When training large language models, the quality of the data used for pretraining is very important. Existing methods for selecting this data often rely on simple rules, like only choosing data from reputable websites. However, the researchers behind this paper argue that we can capture more nuanced human intuitions about what makes high-quality data.

They identify four key qualities that they think are important: writing style, required expertise, facts & trivia, and educational value. The researchers find that language models are actually quite good at recognizing these qualities, especially when comparing two pieces of text side-by-side.

Using this insight, the researchers train a model called a "QuRater" that can learn to rate text on these four dimensions. They then use the QuRater to assess the quality of a huge 260 billion token training corpus, giving each piece of text a score for each of the four qualities.

Next, the researchers experiment with only using the highest-quality data, as scored by the QuRater, to train smaller language models. They find that these models actually perform better than models trained on a random sample of the full dataset. The best-performing model uses the data scored highest for educational value.

Beyond just selecting the data, the researchers also experiment with using the quality ratings to guide the training curriculum - that is, the order in which the model sees the data. They find this can also improve performance without changing the underlying dataset.

Overall, this research suggests that carefully considering the quality and characteristics of training data, rather than just the quantity, can lead to more capable language models.

Technical Explanation

The key innovation in this paper is the QuRating method for selecting high-quality pretraining data for large language models. Existing approaches often use simple heuristics like only choosing data from reputable websites, but the researchers argue this misses more nuanced aspects of data quality.

To address this, the researchers first identify four key qualities they believe are important: writing style, required expertise, facts & trivia, and educational value. They then conduct experiments showing that large language models can in fact discern these qualities, particularly when making pairwise comparisons between texts.

Building on this, the researchers train a "QuRater" model that can learn scalar ratings for each of the four quality dimensions from pairwise human judgments. They then use this QuRater to annotate a massive 260 billion token training corpus with quality scores for each piece of text.

The researchers experiment with training smaller 1.3 billion parameter language models, but only using the highest-scoring data according to the different quality dimensions. They find that sampling based on these quality ratings, especially educational value, leads to better performance in terms of perplexity and in-context learning compared to baselines.

Additionally, the researchers experiment with using the quality ratings to construct a training curriculum, sequencing the data to improve performance without changing the underlying dataset. They provide extensive analysis of the quality ratings, discussing their characteristics, biases, and broader implications.

Critical Analysis

One limitation of this work is that the researchers only evaluate the performance of the language models on standard benchmarks, rather than real-world tasks. It would be valuable to see how the models perform in more applied settings, to better understand the practical benefits of their data selection approach.

Additionally, the researchers acknowledge that their quality ratings may contain biases, and they do not provide a thorough analysis of the sources and implications of these biases. A more rigorous examination of the potential pitfalls and ethical considerations of using these ratings would strengthen the paper.

Another area for further research would be to explore how the QuRating method could be expanded beyond the four qualities examined here. There may be other important dimensions of data quality that could be incorporated to produce even more robust and capable language models.

Overall, however, this paper represents an important step forward in moving beyond simplistic heuristics for data selection and towards more principled approaches to building high-quality training corpora. The researchers' emphasis on capturing nuanced human intuitions about text quality is a valuable contribution to the field of language model development.

Conclusion

This paper introduces QuRating, a novel method for selecting high-quality pretraining data for large language models. By identifying four key qualities - writing style, required expertise, facts & trivia, and educational value - and training a model to assess these qualities, the researchers are able to produce smaller language models that outperform baselines.

Beyond just data selection, the researchers also demonstrate the value of using these quality ratings to guide the training curriculum, further improving model performance. While the work has some limitations, it represents an important advance in moving towards more principled approaches to building capable and reliable language models.

As the use of large language models becomes increasingly widespread, ensuring the quality and trustworthiness of the data they are trained on will only grow in importance. The insights and techniques presented in this paper offer a promising path forward in this critical area of research.