ZeroI2V: Zero-Cost Adaptation of Pre-trained Transformers from Image to Video

Overview

• This paper presents a new paradigm called "ZeroI2V" for efficiently adapting image models to video recognition tasks.
• The key ideas are:
  1. Using spatial-temporal dual-headed attention to give image transformers the ability to model temporal dynamics in videos without extra parameters or compute.
  2. A linear adaptation strategy that uses lightweight adapters to transfer frozen image models to videos, with the adapters merged back into the original model for zero extra cost during inference.

Plain English Explanation

The paper focuses on the challenge of taking image recognition models and adapting them to work well for video recognition tasks. Image-to-video adaptation is an efficient approach, as image models have a lot of parameters and can learn useful features that transfer to video. However, directly fine-tuning these models on video data can be inefficient and unnecessary.

To address this, the authors propose a new "ZeroI2V" paradigm. The core ideas are:

1. Capturing video dynamics: The model uses a special type of self-attention called "spatial-temporal dual-headed attention" to let the image transformer efficiently learn to model the temporal aspects of videos, without needing any extra parameters or compute.

2. Bridging the image-video gap: The model uses lightweight "linear adapters" that are trained to transfer the frozen image model to work well on video data. Importantly, these adapters can then be merged back into the original model, so there is zero extra cost during final inference.

By combining these two key ideas, the ZeroI2V model can match or outperform previous state-of-the-art methods for video recognition, while being much more efficient in terms of parameters and inference time.

Technical Explanation

The paper begins by noting the efficiency of adapting powerful image recognition models, like transformers, to video tasks through image-to-video adaptation. However, directly fine-tuning these models on video data can be wasteful.

To overcome this, the authors propose the "ZeroI2V" paradigm, with two key technical contributions:

1. Spatial-Temporal Dual-Headed Attention (STDHA): The core idea is to extend the self-attention mechanism of the image transformer to efficiently model temporal dynamics in videos. This is done by having two separate attention heads - one for spatial attention and one for temporal attention. This STDHA module can be added to the image transformer with zero extra parameters or computation.

2. Linear Adaptation: To bridge the domain gap between images and videos, the authors propose a lightweight "linear adapter" module. These adapters are densely placed and trained to transfer the frozen image model to work well on video data. Importantly, after training, these adapters can be structurally reparameterized and merged back into the original model, resulting in zero extra cost during final inference.

Experiments on various video recognition benchmarks show that ZeroI2V can match or exceed previous state-of-the-art methods, while being significantly more parameter and compute efficient.

Critical Analysis

The paper presents a clever and efficient approach to adapting powerful image models to video recognition tasks. The core ideas of STDHA and linear adaptation are well-motivated and show strong empirical results.

However, one potential limitation is that the approach may struggle to capture more complex temporal dynamics that require deeper reasoning about the video content. The authors acknowledge this, suggesting that the linear adapters may have limited expressivity for certain video tasks.

Additionally, while the zero-cost inference is a major advantage, the training process does involve some overhead with the adapter modules. It would be interesting to see how this compares to alternative approaches like object-centric image-to-video adaptation or instruction-guided diffusion model adaptation.

Overall, the ZeroI2V paradigm represents an important step forward in efficient video recognition, and the authors' insights around spatial-temporal attention and linear adaptation could inspire further research in this direction.

Conclusion

This paper introduces the ZeroI2V paradigm, a new approach for efficiently adapting powerful image recognition models to video tasks. The key ideas are using spatial-temporal attention to capture video dynamics, and lightweight linear adapters to bridge the image-video domain gap, all while maintaining zero extra cost during final inference.

The results show that ZeroI2V can match or outperform previous state-of-the-art video recognition methods, while being significantly more parameter and compute efficient. This work represents an important advance in the field of multimodal video transfer learning, opening up new possibilities for deploying high-performance video models in resource-constrained settings.