PASS++: A Dual Bias Reduction Framework for Non-Exemplar Class-Incremental Learning
0
Sign in to get full access
Overview
- This paper proposes a new framework called PASS++ for non-exemplar class-incremental learning.
- Class-incremental learning is the task of learning new classes without forgetting previously learned ones.
- The key ideas of PASS++ are: 1) reducing bias towards old classes, and 2) alleviating forgetting of new classes.
Plain English Explanation
The paper introduces a new approach called PASS++ for a machine learning problem called "class-incremental learning." In this problem, the AI model needs to learn new classes of objects over time without forgetting what it has learned before.
The key innovations of PASS++ are:
- Reducing bias towards old classes: The model has a tendency to favor predicting the old classes it has learned, which can make it hard to learn new classes well. PASS++ tries to address this bias.
- Alleviating forgetting of new classes: As the model learns new classes, it can sometimes start to forget the information it had previously learned about older classes. PASS++ aims to prevent this forgetting.
By addressing these two challenges, PASS++ aims to help AI models learn new classes effectively over time without losing their knowledge of previous classes.
Technical Explanation
The PASS++ framework consists of two main components:
-
Prototypical Augmentation for Self-Supervision (PASS): This component helps reduce the model's bias towards old classes. It does this by generating synthetic training samples for the new classes using a self-supervised approach. This encourages the model to learn more balanced representations of the old and new classes.
-
Class-Incremental Learning (CIL): This component tackles the issue of forgetting old classes as new ones are learned. It does this by using a "lightweight memory bank" to store and selectively replay samples from previous classes during training on new classes. This helps the model retain knowledge of old classes.
The paper evaluates PASS++ on several standard class-incremental learning benchmarks and shows that it outperforms previous state-of-the-art methods. The results demonstrate the effectiveness of the dual bias reduction approach used in PASS++.
Critical Analysis
The paper provides a comprehensive technical explanation of the PASS++ framework and its components. However, it does not discuss potential limitations or areas for future research in depth.
One potential limitation is that the self-supervised approach used in PASS may not work as well for more complex or diverse datasets. The effectiveness of the selective replay mechanism in CIL may also depend on the specific characteristics of the dataset and task.
Additionally, the paper does not compare PASS++ to other recently proposed class-incremental learning techniques, such as those that leverage Bayesian learning or contrastive losses. Exploring how PASS++ performs relative to these newer methods could provide further insights.
Conclusion
In summary, the PASS++ framework introduces a dual-pronged approach to address the key challenges in non-exemplar class-incremental learning: reducing bias towards old classes and alleviating forgetting of new classes. The technical evaluation shows promising results, but further research is needed to understand the broader applicability and limitations of the method.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
PASS++: A Dual Bias Reduction Framework for Non-Exemplar Class-Incremental Learning
Fei Zhu, Xu-Yao Zhang, Zhen Cheng, Cheng-Lin Liu
Class-incremental learning (CIL) aims to recognize new classes incrementally while maintaining the discriminability of old classes. Most existing CIL methods are exemplar-based, i.e., storing a part of old data for retraining. Without relearning old data, those methods suffer from catastrophic forgetting. In this paper, we figure out two inherent problems in CIL, i.e., representation bias and classifier bias, that cause catastrophic forgetting of old knowledge. To address these two biases, we present a simple and novel dual bias reduction framework that employs self-supervised transformation (SST) in input space and prototype augmentation (protoAug) in deep feature space. On the one hand, SST alleviates the representation bias by learning generic and diverse representations that can transfer across different tasks. On the other hand, protoAug overcomes the classifier bias by explicitly or implicitly augmenting prototypes of old classes in the deep feature space, which poses tighter constraints to maintain previously learned decision boundaries. We further propose hardness-aware prototype augmentation and multi-view ensemble strategies, leading to significant improvements. The proposed framework can be easily integrated with pre-trained models. Without storing any samples of old classes, our method can perform comparably with state-of-the-art exemplar-based approaches which store plenty of old data. We hope to draw the attention of researchers back to non-exemplar CIL by rethinking the necessity of storing old samples in CIL.
Read more7/22/2024
0
DiffClass: Diffusion-Based Class Incremental Learning
Zichong Meng, Jie Zhang, Changdi Yang, Zheng Zhan, Pu Zhao, Yanzhi Wang
Class Incremental Learning (CIL) is challenging due to catastrophic forgetting. On top of that, Exemplar-free Class Incremental Learning is even more challenging due to forbidden access to previous task data. Recent exemplar-free CIL methods attempt to mitigate catastrophic forgetting by synthesizing previous task data. However, they fail to overcome the catastrophic forgetting due to the inability to deal with the significant domain gap between real and synthetic data. To overcome these issues, we propose a novel exemplar-free CIL method. Our method adopts multi-distribution matching (MDM) diffusion models to unify quality and bridge domain gaps among all domains of training data. Moreover, our approach integrates selective synthetic image augmentation (SSIA) to expand the distribution of the training data, thereby improving the model's plasticity and reinforcing the performance of our method's ultimate component, multi-domain adaptation (MDA). With the proposed integrations, our method then reformulates exemplar-free CIL into a multi-domain adaptation problem to implicitly address the domain gap problem to enhance model stability during incremental training. Extensive experiments on benchmark class incremental datasets and settings demonstrate that our method excels previous exemplar-free CIL methods and achieves state-of-the-art performance.
Read more7/23/2024
↗️
0
Class-Incremental Learning: A Survey
Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, Ziwei Liu
Deep models, e.g., CNNs and Vision Transformers, have achieved impressive achievements in many vision tasks in the closed world. However, novel classes emerge from time to time in our ever-changing world, requiring a learning system to acquire new knowledge continually. Class-Incremental Learning (CIL) enables the learner to incorporate the knowledge of new classes incrementally and build a universal classifier among all seen classes. Correspondingly, when directly training the model with new class instances, a fatal problem occurs -- the model tends to catastrophically forget the characteristics of former ones, and its performance drastically degrades. There have been numerous efforts to tackle catastrophic forgetting in the machine learning community. In this paper, we survey comprehensively recent advances in class-incremental learning and summarize these methods from several aspects. We also provide a rigorous and unified evaluation of 17 methods in benchmark image classification tasks to find out the characteristics of different algorithms empirically. Furthermore, we notice that the current comparison protocol ignores the influence of memory budget in model storage, which may result in unfair comparison and biased results. Hence, we advocate fair comparison by aligning the memory budget in evaluation, as well as several memory-agnostic performance measures. The source code is available at https://github.com/zhoudw-zdw/CIL_Survey/
Read more7/16/2024
0
Exploiting Fine-Grained Prototype Distribution for Boosting Unsupervised Class Incremental Learning
Jiaming Liu, Hongyuan Liu, Zhili Qin, Wei Han, Yulu Fan, Qinli Yang, Junming Shao
The dynamic nature of open-world scenarios has attracted more attention to class incremental learning (CIL). However, existing CIL methods typically presume the availability of complete ground-truth labels throughout the training process, an assumption rarely met in practical applications. Consequently, this paper explores a more challenging problem of unsupervised class incremental learning (UCIL). The essence of addressing this problem lies in effectively capturing comprehensive feature representations and discovering unknown novel classes. To achieve this, we first model the knowledge of class distribution by exploiting fine-grained prototypes. Subsequently, a granularity alignment technique is introduced to enhance the unsupervised class discovery. Additionally, we proposed a strategy to minimize overlap between novel and existing classes, thereby preserving historical knowledge and mitigating the phenomenon of catastrophic forgetting. Extensive experiments on the five datasets demonstrate that our approach significantly outperforms current state-of-the-art methods, indicating the effectiveness of the proposed method.
Read more8/20/2024