Soft Label PU Learning
2405.01990
0
0
📶
Abstract
PU learning refers to the classification problem in which only part of positive samples are labeled. Existing PU learning methods treat unlabeled samples equally. However, in many real tasks, from common sense or domain knowledge, some unlabeled samples are more likely to be positive than others. In this paper, we propose soft label PU learning, in which unlabeled data are assigned soft labels according to their probabilities of being positive. Considering that the ground truth of TPR, FPR, and AUC are unknown, we then design PU counterparts of these metrics to evaluate the performances of soft label PU learning methods within validation data. We show that these new designed PU metrics are good substitutes for the real metrics. After that, a method that optimizes such metrics is proposed. Experiments on public datasets and real datasets for anti-cheat services from Tencent games demonstrate the effectiveness of our proposed method.
Create account to get full access
Overview
- This paper introduces a new approach called "soft label PU learning" for the classification problem where only some positive samples are labeled.
- Existing PU learning methods treat all unlabeled samples equally, but in many real-world tasks, some unlabeled samples are more likely to be positive than others based on common sense or domain knowledge.
- The proposed soft label PU learning approach assigns soft labels to unlabeled data based on their probability of being positive.
- The paper also introduces new PU counterparts of performance metrics like TPR, FPR, and AUC, which can be used to evaluate soft label PU learning methods within the validation data.
- Experiments on public datasets and real-world anti-cheat datasets demonstrate the effectiveness of the proposed approach.
Plain English Explanation
In many real-world classification problems, we only have access to some of the positive samples, while the rest of the data is unlabeled. This is known as the positive-unlabeled (PU) learning problem.
Existing PU learning methods treat all unlabeled samples equally, assuming they are a mix of positive and negative examples. However, in many cases, we can make educated guesses about which unlabeled samples are more likely to be positive based on common sense or domain knowledge.
The soft label PU learning approach proposed in this paper addresses this by assigning "soft labels" to the unlabeled data. These soft labels represent the probability of each unlabeled sample being positive, rather than treating them all the same.
Additionally, the researchers designed new PU counterparts of common performance metrics like true positive rate (TPR), false positive rate (FPR), and area under the curve (AUC). These new metrics can be used to evaluate the performance of soft label PU learning methods within the validation data, as the ground truth of these metrics is unknown.
The paper then presents a method that optimizes these new PU metrics, and experiments show that this approach is effective on both public datasets and real-world anti-cheat datasets from Tencent games.
Technical Explanation
The key innovation in this paper is the introduction of "soft label PU learning," which assigns probabilistic soft labels to the unlabeled data based on their likelihood of being positive. This is in contrast to existing PU learning methods, which treat all unlabeled samples equally.
To evaluate the performance of soft label PU learning methods, the researchers designed new PU counterparts of common classification metrics like TPR, FPR, and AUC. These new metrics can be used within the validation data, as the ground truth of the original metrics is unknown in the PU learning setting.
The paper then presents a method that optimizes these new PU metrics, which the authors show is effective on both public datasets and real-world anti-cheat datasets from Tencent games. The experiments demonstrate the advantages of the soft label approach compared to traditional PU learning methods.
Critical Analysis
The paper presents a novel and promising approach to the PU learning problem, particularly in cases where we have some domain knowledge or intuition about the unlabeled data. By assigning soft labels to the unlabeled samples, the method can potentially capture more information and lead to better performance.
However, the paper does not extensively explore the limitations of the soft label approach or the potential downsides of the new PU metrics. For example, the reliability and robustness of the PU metrics under different data distributions or noise levels could be further investigated.
Additionally, the paper focuses on the classification setting, but it would be interesting to see how the soft label approach could be extended to other PU learning tasks, such as reinforcement learning or unsupervised prompt learning.
Overall, the paper presents a valuable contribution to the PU learning field, and the proposed prompt-based pseudo-labeling strategy could be a promising avenue for further research and development.
Conclusion
This paper introduces a new approach called "soft label PU learning" that addresses the classification problem where only a subset of positive samples are labeled. By assigning probabilistic soft labels to the unlabeled data based on their likelihood of being positive, the method can leverage additional information and potentially outperform traditional PU learning techniques.
The paper also presents new PU counterparts of common performance metrics, which can be used to evaluate soft label PU learning methods within the validation data. Experiments on both public datasets and real-world anti-cheat datasets demonstrate the effectiveness of the proposed approach.
While the paper focuses on the classification setting, the soft label concept and the new PU metrics could have broader applications in other PU learning tasks, such as reinforcement learning and unsupervised prompt learning. Further research is needed to explore the limitations and potential extensions of this work.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
⛏️
Positive Unlabeled Contrastive Learning
Anish Acharya, Sujay Sanghavi, Li Jing, Bhargav Bhushanam, Dhruv Choudhary, Michael Rabbat, Inderjit Dhillon
0
0
Self-supervised pretraining on unlabeled data followed by supervised fine-tuning on labeled data is a popular paradigm for learning from limited labeled examples. We extend this paradigm to the classical positive unlabeled (PU) setting, where the task is to learn a binary classifier given only a few labeled positive samples, and (often) a large amount of unlabeled samples (which could be positive or negative). We first propose a simple extension of standard infoNCE family of contrastive losses, to the PU setting; and show that this learns superior representations, as compared to existing unsupervised and supervised approaches. We then develop a simple methodology to pseudo-label the unlabeled samples using a new PU-specific clustering scheme; these pseudo-labels can then be used to train the final (positive vs. negative) classifier. Our method handily outperforms state-of-the-art PU methods over several standard PU benchmark datasets, while not requiring a-priori knowledge of any class prior (which is a common assumption in other PU methods). We also provide a simple theoretical analysis that motivates our methods.
4/1/2024
Meta-learning for Positive-unlabeled Classification
Atsutoshi Kumagai, Tomoharu Iwata, Yasuhiro Fujiwara
0
0
We propose a meta-learning method for positive and unlabeled (PU) classification, which improves the performance of binary classifiers obtained from only PU data in unseen target tasks. PU learning is an important problem since PU data naturally arise in real-world applications such as outlier detection and information retrieval. Existing PU learning methods require many PU data, but sufficient data are often unavailable in practice. The proposed method minimizes the test classification risk after the model is adapted to PU data by using related tasks that consist of positive, negative, and unlabeled data. We formulate the adaptation as an estimation problem of the Bayes optimal classifier, which is an optimal classifier to minimize the classification risk. The proposed method embeds each instance into a task-specific space using neural networks. With the embedded PU data, the Bayes optimal classifier is estimated through density-ratio estimation of PU densities, whose solution is obtained as a closed-form solution. The closed-form solution enables us to efficiently and effectively minimize the test classification risk. We empirically show that the proposed method outperforms existing methods with one synthetic and three real-world datasets.
6/7/2024
PUAL: A Classifier on Trifurcate Positive-Unlabeled Data
Xiaoke Wang, Xiaochen Yang, Rui Zhu, Jing-Hao Xue
0
0
Positive-unlabeled (PU) learning aims to train a classifier using the data containing only labeled-positive instances and unlabeled instances. However, existing PU learning methods are generally hard to achieve satisfactory performance on trifurcate data, where the positive instances distribute on both sides of the negative instances. To address this issue, firstly we propose a PU classifier with asymmetric loss (PUAL), by introducing a structure of asymmetric loss on positive instances into the objective function of the global and local learning classifier. Then we develop a kernel-based algorithm to enable PUAL to obtain non-linear decision boundary. We show that, through experiments on both simulated and real-world datasets, PUAL can achieve satisfactory classification on trifurcate data.
6/3/2024
Positive Label Is All You Need for Multi-Label Classification
Zhixiang Yuan, Kaixin Zhang, Tao Huang
0
0
Multi-label classification (MLC) faces challenges from label noise in training data due to annotating diverse semantic labels for each image. Current methods mainly target identifying and correcting label mistakes using trained MLC models, but still struggle with persistent noisy labels during training, resulting in imprecise recognition and reduced performance. Our paper addresses label noise in MLC by introducing a positive and unlabeled multi-label classification (PU-MLC) method. To counteract noisy labels, we directly discard negative labels, focusing on the abundance of negative labels and the origin of most noisy labels. PU-MLC employs positive-unlabeled learning, training the model with only positive labels and unlabeled data. The method incorporates adaptive re-balance factors and temperature coefficients in the loss function to address label distribution imbalance and prevent over-smoothing of probabilities during training. Additionally, we introduce a local-global convolution module to capture both local and global dependencies in the image without requiring backbone retraining. PU-MLC proves effective on MLC and MLC with partial labels (MLC-PL) tasks, demonstrating significant improvements on MS-COCO and PASCAL VOC datasets with fewer annotations. Code is available at: https://github.com/TAKELAMAG/PU-MLC.
4/17/2024