Evidential uncertainty sampling for active learning
2309.12494
0
0
Abstract
Recent studies in active learning, particularly in uncertainty sampling, have focused on the decomposition of model uncertainty into reducible and irreducible uncertainties. In this paper, the aim is to simplify the computational process while eliminating the dependence on observations. Crucially, the inherent uncertainty in the labels is considered, the uncertainty of the oracles. Two strategies are proposed, sampling by Klir uncertainty, which tackles the exploration-exploitation dilemma, and sampling by evidential epistemic uncertainty, which extends the concept of reducible uncertainty within the evidential framework, both using the theory of belief functions. Experimental results in active learning demonstrate that our proposed method can outperform uncertainty sampling.
Create account to get full access
Overview
- Examines the problem of active learning with "rich labels" - labels that contain more information than just class assignments
- Proposes a new uncertainty sampling method that quantifies the evidential uncertainty in these rich labels
- Demonstrates the approach on image classification and medical diagnosis tasks
Plain English Explanation
This paper looks at a type of active learning where the labels we get for training data contain more information than just the class the example belongs to. For example, in a medical diagnosis task, the label might not just say the patient has a certain disease, but also provide details about the symptoms, test results, and other relevant information.
The key idea is that we can use this additional "rich" label information to better quantify the uncertainty in our model's predictions. The paper proposes a new uncertainty sampling method that looks at the evidential uncertainty in these rich labels, rather than just the model's confidence in the class prediction.
This evidential uncertainty metric can then be used to select the most informative examples for the model to learn from in an active learning setting. The authors show that this approach outperforms standard active learning methods on both image classification and medical diagnosis tasks, as the rich label information allows the model to focus on the most uncertain and informative examples.
Technical Explanation
The paper begins by introducing the problem of active learning with "rich labels" - labels that provide more information than just the class assignment. They formalize this using the Dempster-Shafer theory of evidence, which allows for the representation of aleatoric and epistemic uncertainty.
The key contribution is a new uncertainty sampling method called "evidential uncertainty sampling" (EUS). EUS computes the evidential uncertainty in the rich label information, rather than just the model's confidence in the class prediction. This provides a more nuanced view of the model's uncertainty.
The authors evaluate EUS on both image classification and medical diagnosis tasks. They show that EUS outperforms standard active learning baselines, as the rich label information allows the model to focus on the most informative and uncertain examples. The paper also includes ablation studies and visualizations to provide additional insights.
Critical Analysis
One limitation noted by the authors is that EUS requires the rich label information to be represented in a specific Dempster-Shafer evidence structure. This may limit its applicability to settings where the label information cannot be easily encoded in this format.
Additionally, the paper does not explore the impact of label noise or label errors on the performance of EUS. Real-world datasets often contain mislabeled examples, which could potentially skew the evidential uncertainty estimates and lead to suboptimal active learning performance.
Further research could also investigate the robustness of EUS to different types of rich label information, as well as its scalability to larger datasets and more complex models. Exploring ways to relax the Dempster-Shafer representation requirement could also broaden the applicability of the approach.
Conclusion
This paper presents a novel active learning method that leverages "rich label" information to quantify the evidential uncertainty in model predictions. By focusing on the most informationally uncertain examples, the approach outperforms standard active learning baselines on both image classification and medical diagnosis tasks.
The paper makes a compelling case for the benefits of incorporating richer label information into the active learning process. While there are some limitations to the current implementation, the overall approach represents an important step towards more effective and efficient active learning systems, with potential applications in a wide range of domains.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
💬
Uncertainty for Active Learning on Graphs
Dominik Fuchsgruber, Tom Wollschlager, Bertrand Charpentier, Antonio Oroz, Stephan Gunnemann
0
0
Uncertainty Sampling is an Active Learning strategy that aims to improve the data efficiency of machine learning models by iteratively acquiring labels of data points with the highest uncertainty. While it has proven effective for independent data its applicability to graphs remains under-explored. We propose the first extensive study of Uncertainty Sampling for node classification: (1) We benchmark Uncertainty Sampling beyond predictive uncertainty and highlight a significant performance gap to other Active Learning strategies. (2) We develop ground-truth Bayesian uncertainty estimates in terms of the data generating process and prove their effectiveness in guiding Uncertainty Sampling toward optimal queries. We confirm our results on synthetic data and design an approximate approach that consistently outperforms other uncertainty estimators on real datasets. (3) Based on this analysis, we relate pitfalls in modeling uncertainty to existing methods. Our analysis enables and informs the development of principled uncertainty estimation on graphs.
5/3/2024
Label-wise Aleatoric and Epistemic Uncertainty Quantification
Yusuf Sale, Paul Hofman, Timo Lohr, Lisa Wimmer, Thomas Nagler, Eyke Hullermeier
0
0
We present a novel approach to uncertainty quantification in classification tasks based on label-wise decomposition of uncertainty measures. This label-wise perspective allows uncertainty to be quantified at the individual class level, thereby improving cost-sensitive decision-making and helping understand the sources of uncertainty. Furthermore, it allows to define total, aleatoric, and epistemic uncertainty on the basis of non-categorical measures such as variance, going beyond common entropy-based measures. In particular, variance-based measures address some of the limitations associated with established methods that have recently been discussed in the literature. We show that our proposed measures adhere to a number of desirable properties. Through empirical evaluation on a variety of benchmark data sets -- including applications in the medical domain where accurate uncertainty quantification is crucial -- we establish the effectiveness of label-wise uncertainty quantification.
6/5/2024
🌿
Transductive Active Learning: Theory and Applications
Jonas Hubotter, Bhavya Sukhija, Lenart Treven, Yarden As, Andreas Krause
0
0
We generalize active learning to address real-world settings with concrete prediction targets where sampling is restricted to an accessible region of the domain, while prediction targets may lie outside this region. We analyze a family of decision rules that sample adaptively to minimize uncertainty about prediction targets. We are the first to show, under general regularity assumptions, that such decision rules converge uniformly to the smallest possible uncertainty obtainable from the accessible data. We demonstrate their strong sample efficiency in two key applications: Active few-shot fine-tuning of large neural networks and safe Bayesian optimization, where they improve significantly upon the state-of-the-art.
5/24/2024
Uncertainty-aware Evidential Fusion-based Learning for Semi-supervised Medical Image Segmentation
Yuanpeng He, Lijian Li
0
0
Although the existing uncertainty-based semi-supervised medical segmentation methods have achieved excellent performance, they usually only consider a single uncertainty evaluation, which often fails to solve the problem related to credibility completely. Therefore, based on the framework of evidential deep learning, this paper integrates the evidential predictive results in the cross-region of mixed and original samples to reallocate the confidence degree and uncertainty measure of each voxel, which is realized by emphasizing uncertain information of probability assignments fusion rule of traditional evidence theory. Furthermore, we design a voxel-level asymptotic learning strategy by introducing information entropy to combine with the fused uncertainty measure to estimate voxel prediction more precisely. The model will gradually pay attention to the prediction results with high uncertainty in the learning process, to learn the features that are difficult to master. The experimental results on LA, Pancreas-CT, ACDC and TBAD datasets demonstrate the superior performance of our proposed method in comparison with the existing state of the arts.
4/12/2024