CoVR: Learning Composed Video Retrieval from Web Video Captions

Overview

• This paper introduces a new task called Composed Image Retrieval (CoIR) and a related task called Composed Video Retrieval (CoVR).
• Most existing CoIR approaches require manually annotated datasets, which are expensive to create and limit scalability.
• The authors propose a scalable automatic dataset creation methodology to generate CoIR and CoVR triplets (query image/video, modification text, target image/video) using video-caption pairs.
• The authors create a new large-scale dataset called WebVid-CoVR using this methodology and introduce a new benchmark for the CoVR task.
• The authors demonstrate that a model trained on the WebVid-CoVR dataset can effectively transfer to and improve state-of-the-art performance on the CoIR task in a zero-shot setup.

Plain English Explanation

The paper focuses on a task called Composed Image Retrieval (CoIR), where the goal is to find an image in a database that is relevant to both a text query and an image query. For example, if the text query is "a person with a hat" and the image query is a person without a hat, the target image should be the same person wearing a hat.

Most existing approaches to CoIR require manually curated datasets, where each example consists of an image, a text description of how to modify the image, and the resulting target image. However, manually creating these datasets is time-consuming and expensive, which limits their scalability.

To address this, the authors propose a method to automatically generate these datasets using video-caption pairs. They find pairs of videos with similar captions, and then use a large language model to generate the text describing how to modify one video to get the other. This allows them to create a large dataset of CoIR triplets (query image, modification text, target image) called WebVid-CoVR.

The authors also introduce a new benchmark for a related task called Composed Video Retrieval (CoVR), where the goal is to find a video in a database that is relevant to both a text query and a video query. They manually annotate a CoVR evaluation set and provide baseline results.

Finally, the authors show that a model trained on the WebVid-CoVR dataset can effectively transfer to the CoIR task, leading to improved state-of-the-art performance on existing CoIR benchmarks like CIRR and FashionIQ in a zero-shot setup.

Technical Explanation

The key contributions of this paper are:

1. Automatic Dataset Creation: The authors propose a scalable methodology to automatically generate CoIR and CoVR triplets using video-caption pairs from the WebVid2M dataset. They mine paired videos with similar captions, and use a large language model to generate the corresponding modification text.

2. WebVid-CoVR Dataset: Applying this methodology to WebVid2M, the authors create a new large-scale dataset called WebVid-CoVR, containing 1.6 million CoVR triplets.

3. CoVR Benchmark: The authors introduce a new benchmark for the CoVR task, including a manually annotated evaluation set and baseline results.

4. Zero-Shot CoIR Transfer: The authors demonstrate that a model trained on the WebVid-CoVR dataset can effectively transfer to the CoIR task, leading to improved state-of-the-art performance on the CIRR and FashionIQ benchmarks in a zero-shot setup.

The automatic dataset creation methodology leverages video-caption pairs to generate CoIR and CoVR triplets. First, they mine pairs of videos with similar captions from the WebVid2M dataset. Then, they use a large language model to generate the text describing how to modify one video to get the other, creating the modification text component of the triplet.

For the CoVR benchmark, the authors manually annotate a evaluation set to measure the performance of models on this task. They provide baseline results using a multimodal retrieval model trained on the WebVid-CoVR dataset.

The authors further demonstrate that the knowledge gained from training on the WebVid-CoVR dataset can be effectively transferred to the CoIR task, leading to state-of-the-art performance on the CIRR and FashionIQ benchmarks in a zero-shot setup.

Critical Analysis

The authors present a novel approach to generating large-scale datasets for the Composed Image Retrieval (CoIR) and Composed Video Retrieval (CoVR) tasks, which is a significant contribution to the field. The automatic dataset creation methodology is scalable and allows for the generation of a much larger dataset (WebVid-CoVR) compared to previous manually curated datasets.

However, the authors acknowledge some potential limitations of their approach. For example, the modification texts generated by the language model may not always accurately describe the differences between the paired videos, which could introduce noise into the dataset. Additionally, the WebVid2M dataset used as the source for the video-caption pairs may not be representative of all types of composed image or video retrieval tasks.

The authors also do not provide a detailed analysis of the types of modifications or the diversity of the generated triplets in the WebVid-CoVR dataset. It would be valuable to understand the characteristics of the dataset and how it compares to manually curated datasets in terms of the complexity and variety of the modifications.

Furthermore, while the authors demonstrate impressive zero-shot transfer performance to the CoIR task, it would be interesting to see how the WebVid-CoVR dataset and trained models perform on a broader range of composed retrieval tasks and benchmarks. Expanding the evaluation to include more diverse datasets and scenarios could provide a more comprehensive understanding of the strengths and limitations of the proposed approach.

Overall, the paper presents an innovative solution to the challenge of dataset creation for composed retrieval tasks, and the authors' work is a valuable contribution to the field. However, further investigation into the quality and characteristics of the generated dataset, as well as more extensive evaluation, could strengthen the impact of the research.

Conclusion

This paper introduces a novel approach to automatically generating large-scale datasets for Composed Image Retrieval (CoIR) and Composed Video Retrieval (CoVR) tasks. The authors' scalable methodology leverages video-caption pairs to create a new dataset called WebVid-CoVR, containing 1.6 million triplets.

The authors also introduce a new benchmark for the CoVR task and demonstrate that a model trained on the WebVid-CoVR dataset can effectively transfer to the CoIR task, leading to improved state-of-the-art performance on existing benchmarks in a zero-shot setup.

This research represents a significant advancement in the field of composed retrieval, as it addresses the key challenge of dataset creation and paves the way for more scalable and diverse applications of these techniques. The authors' work has the potential to catalyze further progress in this area and inspire new directions for research in multimodal learning and retrieval.