Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer

Introduction

The provided text discusses the challenges of class-incremental learning (CIL) compared to traditional deep learning. In traditional deep learning, the model can access all the data at once and learning is performed on a static dataset. However, in real-world applications, data usually arrives in a stream format with new classes, requiring the model to learn continuously.

The primary objective of CIL is to enable the model to learn continuously from non-stationary data streams, facilitating adaptation to new classes and mitigating catastrophic forgetting. Several methods have been proposed to alleviate catastrophic forgetting, such as replay-based, regularization-based, and isolation-based approaches. However, these methods assume that models are trained from scratch, while ignoring the potential benefits of using a strong pre-trained model in the CIL setting.

The text summarizes recent research on using pre-trained vision transformer models for continual incremental learning (CIL). Key points:

• Pre-trained vision transformer models have shown excellent performance on various vision tasks and are being explored for CIL.
• CIL methods based on pre-trained models can achieve significant performance improvements over traditional SOTA methods trained from scratch.
• CIL with pre-trained models typically uses parameter-efficient tuning (PET) approaches like adapters, prompts, and semantic soft projection (SSF), rather than fully fine-tuning the entire model.
• Prompt-based PET methods require constructing a pool of task-specific prompts, which increases storage and computation overhead.
• Other PET methods like adapter tuning are shown to be better continual learners than prompt tuning or SSF, avoiding catastrophic forgetting of old classes.
• The paper proposes a new approach that fine-tunes only the adapter and a task-specific classifier, without regularization, and retrains the classifier using semantic shift-based prototype updates.
• Extensive experiments demonstrate this simple but effective approach achieves state-of-the-art performance on CIL benchmarks.

Related Work

The provided text discusses various approaches to class-incremental learning (CIL) and parameter-efficient tuning (PET) on pre-trained models.

CIL requires the model to be continuously updated with new class instances while retaining old knowledge. Traditional CIL methods can be categorized into replay-based, regularization-based, and parameter isolation-based. Replay-based methods involve retaining or generating samples of previous classes. Regularization-based methods add constraints to limit the update of parameters important for old classes. Isolation-based methods focus on updating only a subset of parameters to mitigate catastrophic forgetting. Methods have also been proposed to expand the network's representative capacity without compromising existing knowledge.

PET refers to inserting and fine-tuning specific sub-modules within a pre-trained network, rather than performing full fine-tuning. This approach has shown effective transfer learning results in both NLP and vision transformer models. Prompt-based PET methods have also been applied to vision-language models.

The text also discusses CIL methods that involve pre-trained models, such as L2P, DualPrompt, CODAPrompt, SLCA, Adam, and LAE. These methods utilize the strong feature representation ability of pre-trained models and employ various strategies to learn new tasks while retaining old knowledge.

Methodology

The paper introduces a class-incremental learning (CIL) formulation where a neural network must learn a series of training sessions while avoiding catastrophic forgetting of previous classes. The authors propose a baseline approach that incrementally tunes a shared adapter module without parameter constraints, which outperforms other parameter-efficient tuning methods like prompt tuning and scale-and-shift factors.

The authors analyze why the adapter-based approach is effective for CIL. They show that constraining parameter updates to mitigate forgetting can hinder the plasticity needed to learn new classes, as the sensitivity of parameters is similar across tasks. The authors then introduce a semantic shift estimation technique to align classifiers across learning sessions without access to past samples.

Experimental results on several CIL benchmarks demonstrate the effectiveness of the proposed baseline approach, which achieves state-of-the-art performance compared to existing parameter-efficient tuning methods.

Experiments

Conclusion

Appendix

A. More experiments on parameter sensitivity

B. More Ablation Experiments

More Implementation Details