On the Effect of (Near) Duplicate Subwords in Language Modelling

Overview

• Tokenization is a core part of language models (LMs)
• It involves splitting a character sequence into subwords and assigning them arbitrary indices
• While typically lossless, this process can reduce the sample efficiency of LM training
• The paper studies the impact of "near duplicate" subwords on LM training efficiency

Plain English Explanation

Tokenization is an important step in how language models (LMs) work. It involves breaking up a piece of text into smaller pieces, called "subwords", and assigning each subword a unique number. This is done before the text is used to train the LM.

Typically, this tokenization process doesn't lose any information. However, it could make it harder for LMs to learn efficiently. This is because tokenization removes some character-level details, which could make it difficult for the LM to recognize similarities between related subwords, like "now" and "Now".

The researchers in this paper wanted to understand how much this loss of character-level information impacts LM training. They did this by:

1. Creating an experiment where each subword had a "duplicate" - an identical copy. This gave them an upper bound on how much LMs could improve if they could perfectly recognize these near-duplicate subwords.

2. Studying how LMs perform when trained on naturally occurring "near duplicate" subwords (rather than the artificial duplicates).

The key findings were:

• LMs need about 17% more data when trained on the fully duplicated subwords, suggesting there is room for improvement.
• However, when looking at naturally occurring near duplicates, merging them actually hurts LM performance.

This implies that while tokenization can reduce training efficiency, the natural near duplicates may not be as similar as expected, limiting the potential gains from better handling them.

Technical Explanation

The paper investigates the impact of "near duplicate" subwords on the training efficiency of language models (LMs). Tokenization is a core component of how LMs work, where a character sequence is split into subwords and assigned arbitrary indices before being used to train the model.

While typically lossless, the researchers hypothesize that this tokenization process may reduce the sample efficiency of LM training. By removing character-level information, it could make it harder for LMs to generalize across similar subwords, such as "now" and "Now".

To study this, the researchers first design an experiment that gives an upper bound on how much LMs could improve if they could perfectly generalize across near duplicate subwords. They do this by creating a fully duplicated vocabulary, where each subword has an identical copy. Experimentally, they find that LMs need roughly 17% more data when trained in this fully duplicated setting.

Next, the researchers investigate the impact of naturally occurring near duplicate subwords on LM performance. Here, they find that merging these near duplicates considerably hurts LM performance. This suggests that while subword duplication negatively impacts LM training efficiency, the naturally occurring near duplicates may not be as similar as anticipated, limiting the potential for performance improvements.

Critical Analysis

The paper provides a systematic analysis of how subword tokenization can impact the training efficiency of language models. The key experimental approach of creating a fully duplicated vocabulary to establish an upper bound on potential gains is novel and insightful.

However, the paper does not explore the reasons why naturally occurring near duplicates may not be as similar as expected. It would be valuable to understand the specific linguistic or contextual factors that contribute to this. Additionally, the paper does not discuss potential methods for better handling near duplicates, beyond simply merging them.

Further research could investigate more sophisticated techniques for identifying and leveraging near duplicate subwords, potentially drawing on work in systematic analysis of subwords for cross-lingual transfer, multi-word tokenization, or learning mutually informed representations of characters and subwords. Exploring the impacts on different types of language models and tasks, as well as across different languages, could also yield additional insights.

Conclusion

This paper provides a detailed analysis of the impact of "near duplicate" subwords on the training efficiency of language models. While the tokenization process used in LMs can reduce sample efficiency, the researchers find that naturally occurring near duplicates may not be as similar as expected, limiting the potential gains from better handling them.

The experimental approach of creating a fully duplicated vocabulary offers an insightful upper bound on possible improvements, but further research is needed to understand the factors contributing to the lack of similarity in natural near duplicates. Exploring more sophisticated techniques for identifying and leveraging these subwords could yield valuable improvements in LM training and performance.