Boosting Consistency in Dual Training for Long-Tailed Semi-Supervised Learning
0
Sign in to get full access
Overview
- Addresses the challenge of long-tailed semi-supervised learning, where there is a large imbalance in the number of samples for different classes
- Proposes a novel consistency-based training approach to boost performance on the long-tail classes
- Leverages dual training, where a main model and an auxiliary model learn to reinforce each other's predictions on both labeled and unlabeled data
Plain English Explanation
In machine learning, sometimes we have datasets where there are many more examples of some classes (the "head" classes) than others (the "long-tail" classes). This can make it challenging for models to learn the rarer classes effectively. This paper presents a new approach to tackle this problem in the context of semi-supervised learning, where we have both labeled and unlabeled data.
The key idea is to use a "dual training" setup, where we have two models that learn to reinforce each other's predictions. One model is the main model that we care about, and the other is an auxiliary model. The models take turns trying to predict the labels of both the labeled data and the unlabeled data. Whenever they disagree on the unlabeled data, the models are updated to make their predictions more consistent.
This consistency-based training helps the models learn robust representations, especially for the long-tail classes, without requiring additional labeled data for those classes. The authors show that this approach outperforms previous methods on several long-tailed semi-supervised learning benchmarks.
Technical Explanation
The paper proposes a novel consistency-based training approach for long-tailed semi-supervised learning, called Boosting Consistency in Dual Training (BCDT).
BCDT leverages a dual training framework, where a main model and an auxiliary model are trained to reinforce each other's predictions on both labeled and unlabeled data. Specifically, the two models take turns predicting the labels of the unlabeled data, and when they disagree, the models are updated to make their predictions more consistent.
This consistency-based training helps the models learn better representations, especially for the long-tail classes, without requiring additional labeled data for those classes. The authors show that BCDT outperforms previous semi-supervised learning methods on several long-tailed benchmarks, including Three Heads Are Better Than One: Complementary Learning, Towards Realistic Long-Tailed Semi-Supervised Learning, and Extracting a Clean Balanced Subset from a Noisy Long-Tailed Dataset.
Critical Analysis
The paper provides a well-designed and thorough evaluation of the proposed BCDT approach, comparing it to multiple state-of-the-art semi-supervised learning methods on long-tailed datasets. The authors acknowledge that BCDT relies on the assumption that the unlabeled data follows a similar distribution to the labeled data, which may not always be the case in real-world settings.
Additionally, the paper does not explore the effects of different hyperparameter choices or the sensitivity of the method to the ratio of labeled to unlabeled data. Further research could investigate these aspects to better understand the practical limitations and robustness of the BCDT approach.
The authors also briefly mention that BCDT could be combined with other techniques, such as Diverse Teacher-Students for Deep Safe Semi-Supervised Learning or Frequency-Based Matcher for Long-Tailed Semantic Segmentation, to potentially further improve performance on long-tailed semi-supervised learning tasks. Exploring such combinations could be a promising direction for future work.
Conclusion
The Boosting Consistency in Dual Training (BCDT) approach presented in this paper offers a novel and effective solution to the challenge of long-tailed semi-supervised learning. By leveraging a consistency-based training scheme between a main model and an auxiliary model, BCDT is able to learn robust representations, especially for the underrepresented long-tail classes, without requiring additional labeled data. The strong empirical results demonstrate the potential of this approach to advance the state of the art in semi-supervised learning for imbalanced datasets, which is an important problem with real-world applications.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
Boosting Consistency in Dual Training for Long-Tailed Semi-Supervised Learning
Kai Gan, Tong Wei, Min-Ling Zhang
While long-tailed semi-supervised learning (LTSSL) has received tremendous attention in many real-world classification problems, existing LTSSL algorithms typically assume that the class distributions of labeled and unlabeled data are almost identical. Those LTSSL algorithms built upon the assumption can severely suffer when the class distributions of labeled and unlabeled data are mismatched since they utilize biased pseudo-labels from the model. To alleviate this problem, we propose a new simple method that can effectively utilize unlabeled data from unknown class distributions through Boosting cOnsistency in duAl Training (BOAT). Specifically, we construct the standard and balanced branch to ensure the performance of the head and tail classes, respectively. Throughout the training process, the two branches incrementally converge and interact with each other, eventually resulting in commendable performance across all classes. Despite its simplicity, we show that BOAT achieves state-of-the-art performance on a variety of standard LTSSL benchmarks, e.g., an averaged 2.7% absolute increase in test accuracy against existing algorithms when the class distributions of labeled and unlabeled data are mismatched. Even when the class distributions are identical, BOAT consistently outperforms many sophisticated LTSSL algorithms. We carry out extensive ablation studies to tease apart the factors that are the most important to the success of BOAT. The source code is available at https://github.com/Gank0078/BOAT.
Read more6/21/2024
0
Three Heads Are Better Than One: Complementary Experts for Long-Tailed Semi-supervised Learning
Chengcheng Ma, Ismail Elezi, Jiankang Deng, Weiming Dong, Changsheng Xu
We address the challenging problem of Long-Tailed Semi-Supervised Learning (LTSSL) where labeled data exhibit imbalanced class distribution and unlabeled data follow an unknown distribution. Unlike in balanced SSL, the generated pseudo-labels are skewed towards head classes, intensifying the training bias. Such a phenomenon is even amplified as more unlabeled data will be mislabeled as head classes when the class distribution of labeled and unlabeled datasets are mismatched. To solve this problem, we propose a novel method named ComPlementary Experts (CPE). Specifically, we train multiple experts to model various class distributions, each of them yielding high-quality pseudo-labels within one form of class distribution. Besides, we introduce Classwise Batch Normalization for CPE to avoid performance degradation caused by feature distribution mismatch between head and non-head classes. CPE achieves state-of-the-art performances on CIFAR-10-LT, CIFAR-100-LT, and STL-10-LT dataset benchmarks. For instance, on CIFAR-10-LT, CPE improves test accuracy by over 2.22% compared to baselines. Code is available at https://github.com/machengcheng2016/CPE-LTSSL.
Read more4/4/2024
⛏️
0
Towards Realistic Long-tailed Semi-supervised Learning in an Open World
Yuanpeng He, Lijian Li
Open-world long-tailed semi-supervised learning (OLSSL) has increasingly attracted attention. However, existing OLSSL algorithms generally assume that the distributions between known and novel categories are nearly identical. Against this backdrop, we construct a more emph{Realistic Open-world Long-tailed Semi-supervised Learning} (textbf{ROLSSL}) setting where there is no premise on the distribution relationships between known and novel categories. Furthermore, even within the known categories, the number of labeled samples is significantly smaller than that of the unlabeled samples, as acquiring valid annotations is often prohibitively costly in the real world. Under the proposed ROLSSL setting, we propose a simple yet potentially effective solution called dual-stage post-hoc logit adjustments. The proposed approach revisits the logit adjustment strategy by considering the relationships among the frequency of samples, the total number of categories, and the overall size of data. Then, it estimates the distribution of unlabeled data for both known and novel categories to dynamically readjust the corresponding predictive probabilities, effectively mitigating category bias during the learning of known and novel classes with more selective utilization of imbalanced unlabeled data. Extensive experiments on datasets such as CIFAR100 and ImageNet100 have demonstrated performance improvements of up to 50.1%, validating the superiority of our proposed method and establishing a strong baseline for this task. For further researches, the anonymous link to the experimental code is at href{https://github.com/heyuanpengpku/ROLSSL}{textcolor{brightpink}{https://github.com/heyuanpengpku/ROLSSL}}
Read more5/24/2024
❗
0
Flexible Distribution Alignment: Towards Long-tailed Semi-supervised Learning with Proper Calibration
Emanuel Sanchez Aimar, Nathaniel Helgesen, Yonghao Xu, Marco Kuhlmann, Michael Felsberg
Long-tailed semi-supervised learning (LTSSL) represents a practical scenario for semi-supervised applications, challenged by skewed labeled distributions that bias classifiers. This problem is often aggravated by discrepancies between labeled and unlabeled class distributions, leading to biased pseudo-labels, neglect of rare classes, and poorly calibrated probabilities. To address these issues, we introduce Flexible Distribution Alignment (FlexDA), a novel adaptive logit-adjusted loss framework designed to dynamically estimate and align predictions with the actual distribution of unlabeled data and achieve a balanced classifier by the end of training. FlexDA is further enhanced by a distillation-based consistency loss, promoting fair data usage across classes and effectively leveraging underconfident samples. This method, encapsulated in ADELLO (Align and Distill Everything All at Once), proves robust against label shift, significantly improves model calibration in LTSSL contexts, and surpasses previous state-of-of-art approaches across multiple benchmarks, including CIFAR100-LT, STL10-LT, and ImageNet127, addressing class imbalance challenges in semi-supervised learning. Our code is available at https://github.com/emasa/ADELLO-LTSSL.
Read more7/16/2024