Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage Retrieval

Overview

• This paper proposes a new pre-training method called "Bag-of-Word (BoW) Prediction" for improving the performance of dense passage retrieval models.
• The key idea is to pre-train the model to predict a bag of words from the input text, rather than trying to reconstruct the entire input through a decoder-based approach like traditional masked auto-encoders.
• The authors show that this BoW prediction pre-training leads to better retrieval performance compared to other pre-training methods, without the need for a decoder module.

Plain English Explanation

The paper introduces a new way to pre-train dense retrieval models, which are used to quickly find relevant information from large text databases. The standard approach is to use a masked auto-encoder, where the model tries to reconstruct the original text from a partially masked version.

However, the authors argue that this decoder-based approach is not necessary. Instead, they propose pre-training the model to simply predict a "bag of words" - the set of words that appear in the text, without worrying about the order. This is simpler and more efficient, yet still teaches the model useful representations for retrieval.

The authors show that this Bag-of-Word (BoW) Prediction pre-training leads to better retrieval performance compared to other methods, like using a traditional auto-encoder or no pre-training at all. It's like teaching a search engine to understand the key concepts in a document, rather than trying to memorize the entire text.

Technical Explanation

The key technical innovation in this paper is the Bag-of-Word (BoW) Prediction pre-training approach. Instead of using a decoder to reconstruct the full input text, as in a standard masked auto-encoder pre-training, the authors train the model to predict the set of words that appear in the input.

Specifically, they randomly mask out some tokens in the input text, and then train the model to predict a binary vector indicating the presence or absence of each word in the vocabulary. This BoW prediction task encourages the model to learn representations that capture the key semantic concepts in the text, without needing to memorize the exact wording.

The authors evaluate this BoW Prediction pre-training on several dense passage retrieval benchmarks, and show that it outperforms other pre-training approaches, including BERT-style masked language modeling and contrastive learning. They argue that the simpler BoW objective is more effective for learning useful representations for retrieval tasks.

Critical Analysis

The authors provide a thorough evaluation of their BoW Prediction pre-training approach, considering its performance on multiple retrieval datasets and comparing to strong baselines. The results are impressive, showing clear benefits over other pre-training methods.

However, the paper does not explore the limitations of this approach in depth. For example, it's not clear how the BoW Prediction pre-training would perform on more complex language understanding tasks beyond retrieval, where the order and structure of the text may be more important.

Additionally, the authors do not investigate the interpretability or explainability of the learned representations, which could be an interesting area for further research. Understanding how the BoW Prediction objective shapes the model's internal representations could provide insights into its strengths and weaknesses.

Overall, this is a well-executed study that makes a compelling case for the BoW Prediction pre-training approach. The simplicity and strong empirical results are compelling, and the work opens up interesting directions for future research on efficient pre-training methods for retrieval and beyond.

Conclusion

This paper presents a novel pre-training method called Bag-of-Word (BoW) Prediction that improves the performance of dense passage retrieval models. By training the model to predict the set of words present in the input, rather than trying to reconstruct the full text, the authors show significant gains in retrieval accuracy compared to other pre-training approaches.

The key insight is that the BoW Prediction objective is simpler and more efficient than traditional decoder-based pre-training, yet still allows the model to learn useful representations for finding relevant information in large text databases. This work demonstrates the potential benefits of rethinking pre-training objectives to better suit the target task, and could inspire similar approaches for other language understanding problems.