Class-Incremental Learning: A Survey

Read original: arXiv:2302.03648 - Published 7/16/2024 by Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, Ziwei Liu

↗️

Overview

This paper discusses the challenge of class-incremental learning (CIL) in deep learning models, where new classes emerge over time, requiring the model to continuously acquire new knowledge without forgetting previous knowledge.
The paper surveys recent advances in CIL and provides a comprehensive evaluation of 17 CIL methods on benchmark image classification tasks.
The paper also highlights the importance of fair comparison by aligning the memory budget in evaluation, as well as the need for memory-agnostic performance measures.

Plain English Explanation

Deep learning models like convolutional neural networks (CNNs) and vision transformers have achieved impressive results in many visual tasks. However, our world is constantly changing, and new classes of objects or concepts can emerge over time. This requires the learning system to continuously acquire new knowledge without forgetting what it has learned before.

This challenge is known as class-incremental learning (CIL). When training the model on new class instances, it can suffer from a problem called "catastrophic forgetting," where the model forgets the characteristics of the previously learned classes, and its performance dramatically decreases.

The research community has proposed many methods to tackle this issue of catastrophic forgetting. This paper provides a comprehensive survey of these recent advances in CIL and evaluates 17 different CIL methods on benchmark image classification tasks. The paper also highlights the importance of fair comparisons by considering the memory budget used by each method, as well as the need for memory-agnostic performance measures.

Technical Explanation

The paper first provides an overview of the class-incremental learning (CIL) problem, where a model needs to learn new classes of data over time without forgetting the previously learned ones. This is a challenging task, as directly training the model on new class instances can lead to catastrophic forgetting, where the model's performance on the old classes drastically decreases.

The paper then surveys recent advances in CIL, categorizing the methods into several groups, such as feature expansion-enhanced compression, Bayesian learning-driven prototypical contrastive loss, and decision boundary-aware knowledge consolidation. The paper provides a detailed technical explanation of these methods and their key ideas.

To evaluate the CIL methods, the paper conducts a rigorous and unified evaluation on benchmark image classification tasks, considering 17 different CIL algorithms. The evaluation not only focuses on the overall performance but also emphasizes the importance of fair comparisons by aligning the memory budget used by each method. Additionally, the paper advocates for the use of memory-agnostic performance measures, which do not directly depend on the memory budget.

Critical Analysis

The paper highlights an important and practical problem in the field of deep learning – the need for models to continuously acquire new knowledge without forgetting the previously learned information. This is a challenging task, and the paper's comprehensive survey of recent CIL methods provides valuable insights into the current state of the research.

One potential limitation of the paper is that it does not delve deeply into the underlying reasons for the catastrophic forgetting problem, nor does it provide a thorough discussion of the inherent trade-offs and challenges in designing effective CIL algorithms. Additionally, while the paper advocates for fair comparisons and memory-agnostic performance measures, it could have provided more detailed analysis and justification for these recommendations.

Furthermore, the paper could have explored the potential implications of CIL in real-world applications, such as robotics, medical imaging, or autonomous systems, where the ability to learn continuously is crucial. Addressing these aspects could have strengthened the paper's contribution and provided a more holistic perspective on the problem and its significance.

Conclusion

This paper presents a comprehensive survey of recent advances in class-incremental learning (CIL), a critical problem in deep learning where models need to continuously acquire new knowledge without forgetting the previously learned information. The paper evaluates 17 different CIL methods on benchmark image classification tasks and highlights the importance of fair comparisons by aligning the memory budget used by each method, as well as the need for memory-agnostic performance measures.

The findings of this paper provide valuable insights for researchers and practitioners working on developing robust and adaptive deep learning models that can harness neural unit dynamics for effective and scalable class-incremental learning. By addressing the challenge of catastrophic forgetting, the research community can work towards building learning systems that can truly adapt and grow with the ever-changing world.

This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!

Follow @aimodelsfyi on 𝕏 →

Related Papers

↗️

Class-Incremental Learning: A Survey

Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, Ziwei Liu

Deep models, e.g., CNNs and Vision Transformers, have achieved impressive achievements in many vision tasks in the closed world. However, novel classes emerge from time to time in our ever-changing world, requiring a learning system to acquire new knowledge continually. Class-Incremental Learning (CIL) enables the learner to incorporate the knowledge of new classes incrementally and build a universal classifier among all seen classes. Correspondingly, when directly training the model with new class instances, a fatal problem occurs -- the model tends to catastrophically forget the characteristics of former ones, and its performance drastically degrades. There have been numerous efforts to tackle catastrophic forgetting in the machine learning community. In this paper, we survey comprehensively recent advances in class-incremental learning and summarize these methods from several aspects. We also provide a rigorous and unified evaluation of 17 methods in benchmark image classification tasks to find out the characteristics of different algorithms empirically. Furthermore, we notice that the current comparison protocol ignores the influence of memory budget in model storage, which may result in unfair comparison and biased results. Hence, we advocate fair comparison by aligning the memory budget in evaluation, as well as several memory-agnostic performance measures. The source code is available at https://github.com/zhoudw-zdw/CIL_Survey/

7/16/2024

Addressing Imbalance for Class Incremental Learning in Medical Image Classification

Xuze Hao, Wenqian Ni, Xuhao Jiang, Weimin Tan, Bo Yan

Deep convolutional neural networks have made significant breakthroughs in medical image classification, under the assumption that training samples from all classes are simultaneously available. However, in real-world medical scenarios, there's a common need to continuously learn about new diseases, leading to the emerging field of class incremental learning (CIL) in the medical domain. Typically, CIL suffers from catastrophic forgetting when trained on new classes. This phenomenon is mainly caused by the imbalance between old and new classes, and it becomes even more challenging with imbalanced medical datasets. In this work, we introduce two simple yet effective plug-in methods to mitigate the adverse effects of the imbalance. First, we propose a CIL-balanced classification loss to mitigate the classifier bias toward majority classes via logit adjustment. Second, we propose a distribution margin loss that not only alleviates the inter-class overlap in embedding space but also enforces the intra-class compactness. We evaluate the effectiveness of our method with extensive experiments on three benchmark datasets (CCH5000, HAM10000, and EyePACS). The results demonstrate that our approach outperforms state-of-the-art methods.

7/23/2024

DiffClass: Diffusion-Based Class Incremental Learning

Zichong Meng, Jie Zhang, Changdi Yang, Zheng Zhan, Pu Zhao, Yanzhi Wang

Class Incremental Learning (CIL) is challenging due to catastrophic forgetting. On top of that, Exemplar-free Class Incremental Learning is even more challenging due to forbidden access to previous task data. Recent exemplar-free CIL methods attempt to mitigate catastrophic forgetting by synthesizing previous task data. However, they fail to overcome the catastrophic forgetting due to the inability to deal with the significant domain gap between real and synthetic data. To overcome these issues, we propose a novel exemplar-free CIL method. Our method adopts multi-distribution matching (MDM) diffusion models to unify quality and bridge domain gaps among all domains of training data. Moreover, our approach integrates selective synthetic image augmentation (SSIA) to expand the distribution of the training data, thereby improving the model's plasticity and reinforcing the performance of our method's ultimate component, multi-domain adaptation (MDA). With the proposed integrations, our method then reformulates exemplar-free CIL into a multi-domain adaptation problem to implicitly address the domain gap problem to enhance model stability during incremental training. Extensive experiments on benchmark class incremental datasets and settings demonstrate that our method excels previous exemplar-free CIL methods and achieves state-of-the-art performance.

7/23/2024

Exploiting Fine-Grained Prototype Distribution for Boosting Unsupervised Class Incremental Learning

Jiaming Liu, Hongyuan Liu, Zhili Qin, Wei Han, Yulu Fan, Qinli Yang, Junming Shao

The dynamic nature of open-world scenarios has attracted more attention to class incremental learning (CIL). However, existing CIL methods typically presume the availability of complete ground-truth labels throughout the training process, an assumption rarely met in practical applications. Consequently, this paper explores a more challenging problem of unsupervised class incremental learning (UCIL). The essence of addressing this problem lies in effectively capturing comprehensive feature representations and discovering unknown novel classes. To achieve this, we first model the knowledge of class distribution by exploiting fine-grained prototypes. Subsequently, a granularity alignment technique is introduced to enhance the unsupervised class discovery. Additionally, we proposed a strategy to minimize overlap between novel and existing classes, thereby preserving historical knowledge and mitigating the phenomenon of catastrophic forgetting. Extensive experiments on the five datasets demonstrate that our approach significantly outperforms current state-of-the-art methods, indicating the effectiveness of the proposed method.

8/20/2024