Split-Ensemble: Efficient OOD-aware Ensemble via Task and Model Splitting
2312.09148
0
0
Abstract
Uncertainty estimation is crucial for machine learning models to detect out-of-distribution (OOD) inputs. However, the conventional discriminative deep learning classifiers produce uncalibrated closed-set predictions for OOD data. A more robust classifiers with the uncertainty estimation typically require a potentially unavailable OOD dataset for outlier exposure training, or a considerable amount of additional memory and compute to build ensemble models. In this work, we improve on uncertainty estimation without extra OOD data or additional inference costs using an alternative Split-Ensemble method. Specifically, we propose a novel subtask-splitting ensemble training objective, where a common multiclass classification task is split into several complementary subtasks. Then, each subtask's training data can be considered as OOD to the other subtasks. Diverse submodels can therefore be trained on each subtask with OOD-aware objectives. The subtask-splitting objective enables us to share low-level features across submodels to avoid parameter and computational overheads. In particular, we build a tree-like Split-Ensemble architecture by performing iterative splitting and pruning from a shared backbone model, where each branch serves as a submodel corresponding to a subtask. This leads to improved accuracy and uncertainty estimation across submodels under a fixed ensemble computation budget. Empirical study with ResNet-18 backbone shows Split-Ensemble, without additional computation cost, improves accuracy over a single model by 0.8%, 1.8%, and 25.5% on CIFAR-10, CIFAR-100, and Tiny-ImageNet, respectively. OOD detection for the same backbone and in-distribution datasets surpasses a single model baseline by, correspondingly, 2.2%, 8.1%, and 29.6% mean AUROC.
Create account to get full access
Overview
ā¢ This paper introduces a novel approach called "Split-Ensemble" that addresses the challenge of out-of-distribution (OOD) detection in machine learning models.
ā¢ The key idea is to split the overall task into multiple sub-tasks and train separate models on each sub-task, then combine these models into an ensemble that is more robust to OOD inputs.
Plain English Explanation
ā¢ Machine learning models can struggle to identify inputs that are very different from the data they were trained on, known as out-of-distribution (OOD) inputs. This can lead to unreliable or unsafe predictions in real-world applications.
ā¢ The "Split-Ensemble" approach aims to improve OOD detection by dividing the overall task into simpler sub-tasks, training individual models on each sub-task, and then combining these models into an ensemble.
ā¢ By splitting the task and model, the ensemble can leverage the specialized expertise of each sub-model to better identify OOD inputs that may fool a single, more general model. This makes the overall system more robust and reliable.
ā¢ The authors demonstrate the effectiveness of their Split-Ensemble approach on several benchmark datasets, showing improved OOD detection performance compared to traditional ensemble methods.
Technical Explanation
ā¢ The key innovation in this paper is the "Split-Ensemble" architecture, which consists of:
- Task Splitting: Dividing the overall task into multiple sub-tasks, each with its own training data and model.
- Model Splitting: Training separate models for each sub-task, allowing each model to specialize in its particular domain.
- Ensemble Integration: Combining the outputs of the sub-models into a final ensemble prediction.
ā¢ The authors hypothesize that this task and model splitting approach can better capture the diverse characteristics of OOD inputs, as each sub-model is trained on a specific aspect of the problem.
ā¢ To evaluate their approach, the authors conduct experiments on several OOD detection benchmarks, including MultiOOD, EntropyStop, and Noisy Elephant. They compare the performance of their Split-Ensemble against traditional ensemble methods and other OOD detection techniques.
ā¢ The results show that the Split-Ensemble approach consistently outperforms the baselines, demonstrating its effectiveness in identifying OOD inputs across different datasets and problem settings.
Critical Analysis
ā¢ The authors provide a thorough evaluation of their Split-Ensemble approach, comparing it to a wide range of state-of-the-art OOD detection methods on several benchmark datasets.
ā¢ However, the paper does not discuss the potential computational overhead or training complexity of the Split-Ensemble approach, which may be a concern for real-world deployment.
ā¢ Additionally, the authors could have explored the interpretability of the Split-Ensemble model, as understanding how the sub-models contribute to the final OOD prediction could provide valuable insights.
ā¢ Further research could also investigate the optimal number of sub-tasks and the impact of different task-splitting strategies on the overall performance of the Split-Ensemble.
Conclusion
ā¢ The Split-Ensemble approach introduced in this paper offers a promising solution to the challenge of out-of-distribution (OOD) detection in machine learning.
ā¢ By splitting the overall task and model into specialized sub-tasks and sub-models, the ensemble can better capture the diverse characteristics of OOD inputs and make more reliable predictions.
ā¢ The experimental results demonstrate the effectiveness of the Split-Ensemble approach, suggesting it could have significant implications for improving the robustness and safety of machine learning systems in real-world applications.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
Continual Unsupervised Out-of-Distribution Detection
Lars Doorenbos, Raphael Sznitman, Pablo M'arquez-Neila
0
0
Deep learning models excel when the data distribution during training aligns with testing data. Yet, their performance diminishes when faced with out-of-distribution (OOD) samples, leading to great interest in the field of OOD detection. Current approaches typically assume that OOD samples originate from an unconcentrated distribution complementary to the training distribution. While this assumption is appropriate in the traditional unsupervised OOD (U-OOD) setting, it proves inadequate when considering the place of deployment of the underlying deep learning model. To better reflect this real-world scenario, we introduce the novel setting of continual U-OOD detection. To tackle this new setting, we propose a method that starts from a U-OOD detector, which is agnostic to the OOD distribution, and slowly updates during deployment to account for the actual OOD distribution. Our method uses a new U-OOD scoring function that combines the Mahalanobis distance with a nearest-neighbor approach. Furthermore, we design a confidence-scaled few-shot OOD detector that outperforms previous methods. We show our method greatly improves upon strong baselines from related fields.
6/5/2024
Pursuing Feature Separation based on Neural Collapse for Out-of-Distribution Detection
Yingwen Wu, Ruiji Yu, Xinwen Cheng, Zhengbao He, Xiaolin Huang
0
0
In the open world, detecting out-of-distribution (OOD) data, whose labels are disjoint with those of in-distribution (ID) samples, is important for reliable deep neural networks (DNNs). To achieve better detection performance, one type of approach proposes to fine-tune the model with auxiliary OOD datasets to amplify the difference between ID and OOD data through a separation loss defined on model outputs. However, none of these studies consider enlarging the feature disparity, which should be more effective compared to outputs. The main difficulty lies in the diversity of OOD samples, which makes it hard to describe their feature distribution, let alone design losses to separate them from ID features. In this paper, we neatly fence off the problem based on an aggregation property of ID features named Neural Collapse (NC). NC means that the penultimate features of ID samples within a class are nearly identical to the last layer weight of the corresponding class. Based on this property, we propose a simple but effective loss called OrthLoss, which binds the features of OOD data in a subspace orthogonal to the principal subspace of ID features formed by NC. In this way, the features of ID and OOD samples are separated by different dimensions. By optimizing the feature separation loss rather than purely enlarging output differences, our detection achieves SOTA performance on CIFAR benchmarks without any additional data augmentation or sampling, demonstrating the importance of feature separation in OOD detection. The code will be published.
5/29/2024
š¤æ
Diversifying Deep Ensembles: A Saliency Map Approach for Enhanced OOD Detection, Calibration, and Accuracy
Stanislav Dereka, Ivan Karpukhin, Maksim Zhdanov, Sergey Kolesnikov
0
0
Deep ensembles are capable of achieving state-of-the-art results in classification and out-of-distribution (OOD) detection. However, their effectiveness is limited due to the homogeneity of learned patterns within ensembles. To overcome this issue, our study introduces Saliency Diversified Deep Ensemble (SDDE), a novel approach that promotes diversity among ensemble members by leveraging saliency maps. Through incorporating saliency map diversification, our method outperforms conventional ensemble techniques and improves calibration in multiple classification and OOD detection tasks. In particular, the proposed method achieves state-of-the-art OOD detection quality, calibration, and accuracy on multiple benchmarks, including CIFAR10/100 and large-scale ImageNet datasets.
6/17/2024
š
MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities
Hao Dong, Yue Zhao, Eleni Chatzi, Olga Fink
0
0
Detecting out-of-distribution (OOD) samples is important for deploying machine learning models in safety-critical applications such as autonomous driving and robot-assisted surgery. Existing research has mainly focused on unimodal scenarios on image data. However, real-world applications are inherently multimodal, which makes it essential to leverage information from multiple modalities to enhance the efficacy of OOD detection. To establish a foundation for more realistic Multimodal OOD Detection, we introduce the first-of-its-kind benchmark, MultiOOD, characterized by diverse dataset sizes and varying modality combinations. We first evaluate existing unimodal OOD detection algorithms on MultiOOD, observing that the mere inclusion of additional modalities yields substantial improvements. This underscores the importance of utilizing multiple modalities for OOD detection. Based on the observation of Modality Prediction Discrepancy between in-distribution (ID) and OOD data, and its strong correlation with OOD performance, we propose the Agree-to-Disagree (A2D) algorithm to encourage such discrepancy during training. Moreover, we introduce a novel outlier synthesis method, NP-Mix, which explores broader feature spaces by leveraging the information from nearest neighbor classes and complements A2D to strengthen OOD detection performance. Extensive experiments on MultiOOD demonstrate that training with A2D and NP-Mix improves existing OOD detection algorithms by a large margin. Our source code and MultiOOD benchmark are available at https://github.com/donghao51/MultiOOD.
5/28/2024