Taming Self-Training for Open-Vocabulary Object Detection

Overview

• Recent studies show promising performance in open-vocabulary object detection (OVD) using pseudo labels (PLs) from pre-trained vision and language models (VLMs)
• However, teacher-student self-training, a powerful technique to leverage PLs, is rarely explored for OVD
• This work identifies two challenges in using self-training for OVD: noisy PLs from VLMs and frequent distribution changes of PLs
• To address these challenges, the researchers propose SAS-Det, a method that tames self-training for OVD

Plain English Explanation

Object detection is the task of identifying and localizing objects in an image. Open-vocabulary object detection aims to detect a wide range of objects, even ones not seen during training. Recent advances have shown that using pseudo labels (PL) - predictions from pre-trained vision and language models - can boost the performance of open-vocabulary object detectors.

One powerful technique to leverage these PLs is called self-training. In self-training, the model uses its own predictions to "teach" itself and improve over time. However, self-training has rarely been used for open-vocabulary object detection. The researchers identified two key challenges:

1. The PLs from vision and language models can be noisy and inaccurate, which can degrade the model's performance if used directly.
2. In open-vocabulary detection, the distribution of PLs is constantly changing as the model sees new objects, making the self-training process unstable.

To address these issues, the researchers developed a new method called SAS-Det. The key ideas are:

1. A "split-and-fusion" detection head that separates the model into an "open" branch for novel objects and a "closed" branch for familiar objects. This helps reduce the impact of noisy PLs.
2. A strategy to periodically update the teacher model, which reduces the frequency of changes in the PL distribution and stabilizes the training process.

Through extensive experiments, the researchers showed that SAS-Det outperforms other open-vocabulary detection models and achieves strong performance on challenging benchmarks like COCO and LVIS.

Technical Explanation

The researchers propose SAS-Det, a method that tames self-training for open-vocabulary object detection (OVD). SAS-Det addresses two key challenges in using self-training for OVD:

1. Noisy Pseudo Labels (PLs): PLs from pre-trained vision and language models (VLMs) can be noisy and inaccurate, which can degrade the model's performance if used directly.
2. Frequent Distribution Changes of PLs: Unlike in closed-set tasks, the PL distributions in OVD are solely determined by the teacher model. As the teacher sees new objects, the PL distribution changes frequently, making the self-training process unstable.

To address these challenges, SAS-Det introduces two key components:

1. Split-and-Fusion (SAF) Head: The detection head is split into an "open" branch and a "closed" branch. The open branch focuses on learning from noisy PLs for novel objects, while the closed branch learns from clean ground-truth labels for familiar objects. The outputs of the two branches are then fused to leverage complementary knowledge.
2. Periodic Update Strategy: Instead of continuously updating the teacher model, SAS-Det updates the teacher model periodically. This reduces the frequency of changes in the PL distribution, stabilizing the self-training process.

The researchers evaluate SAS-Det on the COCO and LVIS benchmarks, and show that it outperforms recent open-vocabulary object detection and self-training methods. SAS-Det achieves 37.4 AP50 and 29.1 APr on the novel categories of COCO and LVIS, respectively.

Critical Analysis

The researchers have identified and addressed two important challenges in using self-training for open-vocabulary object detection, which is a promising but underexplored area. The proposed SAS-Det method shows strong performance on challenging benchmarks, demonstrating the effectiveness of the split-and-fusion head and periodic update strategy.

However, the paper does not provide a deep analysis of the trade-offs and limitations of the approach. For example, it's unclear how the split-and-fusion head impacts the model complexity and inference time. Additionally, the periodic update strategy may not be optimal for all scenarios, and the researchers could explore more adaptive approaches to updating the teacher model.

Further research could also investigate the generalization of SAS-Det to other open-vocabulary tasks, such as segmentation, and explore ways to make the method more robust to noisy and dynamic pseudo labels.

Conclusion

This work presents SAS-Det, a method that tames self-training for open-vocabulary object detection. By addressing the challenges of noisy pseudo labels and frequent distribution changes, SAS-Det achieves state-of-the-art performance on COCO and LVIS benchmarks. The split-and-fusion detection head and periodic update strategy are key innovations that could inspire future research in open-vocabulary learning and self-training. As the field of open-vocabulary perception continues to advance, methods like SAS-Det will play an important role in enabling AI systems to robustly detect and recognize a wide range of objects in the real world.