HyCIR: Boosting Zero-Shot Composed Image Retrieval with Synthetic Labels

Overview

• This paper proposes a new method called Hybrid CIR (HyCIR) to improve zero-shot Composed Image Retrieval (ZS-CIR) performance.
• ZS-CIR aims to retrieve images based on a query image with text, without using expensive triplet-labeled training datasets.
• The authors introduce a new label Synthesis pipeline for CIR (SynCir) to generate synthetic CIR triplets, using only unlabeled images.
• HyCIR uses a hybrid training strategy that combines ZS-CIR supervision and synthetic CIR triplets to improve performance.

Plain English Explanation

The paper focuses on a task called Composed Image Retrieval (CIR), where the goal is to find relevant images based on a query that includes both an image and some text. Current zero-shot CIR (ZS-CIR) methods try to solve this problem without using expensive labeled training data, but they still lag behind methods that do use that data.

To address this, the authors propose a new approach called Hybrid CIR (HyCIR). The key idea is to use a technique called SynCir to automatically generate synthetic training data, in the form of image-text pairs. This synthetic data is then combined with the existing zero-shot supervision to train a more powerful CIR model.

The SynCir process works by first finding visually similar image pairs, then generating relevant text descriptions for those pairs using language models. This creates realistic-looking CIR training examples without the need for manual labeling.

By using both the synthetic data and the original zero-shot training, the HyCIR model is able to learn a better image-to-text mapping that is tailored for the CIR task. This allows it to outperform previous zero-shot CIR methods on common benchmarks, getting state-of-the-art results.

Technical Explanation

The paper introduces a new approach called Hybrid CIR (HyCIR) to improve the performance of zero-shot Composed Image Retrieval (ZS-CIR). ZS-CIR aims to retrieve relevant images given a query containing both an image and some text, without requiring expensive triplet-labeled training data.

The key innovation is the SynCir pipeline, which automatically generates synthetic CIR training triplets from unlabeled image data. First, visually similar image pairs are extracted. Then, relevant text descriptions are generated for each pair using a vision-language model and a large language model (LLM). Finally, the data is filtered based on semantic similarity to create high-quality synthetic triplets.

To leverage this synthetic data, HyCIR uses a hybrid training strategy. It combines the original ZS-CIR supervision, which uses only the query image and text, with additional training on the synthetic CIR triplets. Two forms of contrastive learning are used: one to learn a general image-to-text mapping from large-scale unlabeled data, and another to specialize that mapping for the CIR task using the synthetic triplets.

The authors evaluate HyCIR on common CIR benchmarks like CIRR and CIRCO, and show that it achieves state-of-the-art zero-shot performance, outperforming previous ZS-CIR methods. This demonstrates the value of automatically generating synthetic training data to bridge the gap between zero-shot and fully-supervised CIR approaches.

Critical Analysis

The paper presents a clever and effective approach to improving zero-shot Composed Image Retrieval (ZS-CIR) performance by leveraging automatically generated synthetic training data. The SynCir pipeline for creating these synthetic triplets is a key technical contribution, as it allows the authors to sidestep the need for expensive manual labeling.

That said, the authors acknowledge that there is still a gap between the performance of ZS-CIR and fully-supervised CIR methods. While HyCIR represents a significant step forward, further research may be needed to completely close this gap. Additionally, the synthetic data generation process, while effective, could potentially introduce biases or other issues that impact model performance.

It would also be interesting to see how HyCIR compares to other few-shot or semi-supervised techniques for CIR that aim to reduce reliance on fully-labeled training data. Exploring the trade-offs between different approaches in this space could yield additional insights.

Overall, the HyCIR method represents an important advance in zero-shot Composed Image Retrieval, and the authors' thoughtful approach to leveraging synthetic data is a model for future work in this area.

Conclusion

This paper proposes a new Hybrid CIR (HyCIR) method that uses automatically generated synthetic training data to boost the performance of zero-shot Composed Image Retrieval (ZS-CIR). By combining ZS-CIR supervision with synthetic CIR triplets created through the SynCir pipeline, HyCIR is able to achieve state-of-the-art zero-shot results on standard benchmarks.

The key innovation is the ability to sidestep the need for expensive manually-labeled training data, which has been a major limitation of previous CIR approaches. This work demonstrates the power of leveraging large language models and other recent advances in AI to automatically generate high-quality synthetic training data, paving the way for more efficient and effective Composed Image Retrieval systems.