FedNoisy: Federated Noisy Label Learning Benchmark
0
Sign in to get full access
Overview
- This paper introduces FedNoisy, a benchmark for evaluating Federated Learning (FL) algorithms in the presence of noisy labels.
- Federated Learning allows training machine learning models on data distributed across many devices without centralizing the data, but can be impacted by noisy or unreliable labels provided by participants.
- FedNoisy provides a standardized dataset and evaluation protocol to test the robustness of FL algorithms to noisy labels, enabling more rigorous comparisons.
Plain English Explanation
FedNoisy: Federated Noisy Label Learning Benchmark is a new benchmark for evaluating how well Federated Learning (FL) algorithms can handle unreliable or noisy label data. In Federated Learning, machine learning models are trained on data spread across many devices rather than centralized, but the data labels provided by participants may not always be accurate.
The FedNoisy benchmark provides a standardized dataset and evaluation process to test how robust different FL algorithms are to these noisy labels. This allows researchers to more rigorously compare the performance of various FL techniques in more realistic, noisy data conditions. By having a common benchmark, it becomes easier to assess the strengths and weaknesses of different FL approaches when dealing with unreliable label information from distributed participants.
Technical Explanation
FedNoisy: Federated Noisy Label Learning Benchmark presents a benchmark for evaluating Federated Learning (FL) algorithms under noisy label conditions. The authors create a standardized dataset, based on the CIFAR-10 image classification task, where a portion of the labels are randomly corrupted.
They then define an evaluation protocol that simulates a federated setting, with multiple clients holding partitions of the dataset. The goal is to train an accurate classifier while dealing with the unreliable labels contributed by some clients. The authors test several state-of-the-art FL algorithms, including Blind Federated Learning, FedRobust, and Personalized Wireless Federated Learning, on this FedNoisy benchmark.
The results show that existing FL algorithms can struggle with noisy labels, highlighting the need for more robust techniques. The authors also demonstrate the potential of ranking-based client selection strategies to improve FL performance in the presence of label noise.
Critical Analysis
The FedNoisy benchmark provides a valuable tool for assessing the noise-resilience of Federated Learning algorithms. By creating a standardized dataset and evaluation protocol, the authors enable more rigorous comparisons between different FL techniques in the presence of unreliable label data.
However, the authors acknowledge that the FedNoisy dataset may not fully capture the complexity of real-world noisy label scenarios. The random corruption used to introduce noise may not reflect the more nuanced patterns of label errors that could arise in practical applications. Further research is needed to develop benchmarks that better mimic real-world noisy label distributions.
Additionally, the authors only test a limited set of FL algorithms on the FedNoisy benchmark. While the results highlight the need for more robust techniques, there may be other promising approaches that could perform well on this task but were not included in the evaluation.
Despite these limitations, the FedNoisy benchmark represents an important step forward in assessing the practical viability of Federated Learning in the face of noisy or unreliable label data. As the field of Federated Learning continues to evolve, this benchmark can serve as a valuable tool for driving the development of more robust and reliable algorithms.
Conclusion
FedNoisy: Federated Noisy Label Learning Benchmark introduces a standardized benchmark for evaluating Federated Learning algorithms in the presence of noisy label data. By providing a common dataset and evaluation protocol, the authors enable more rigorous comparisons of different FL techniques in realistic, noisy conditions.
The results highlight the need for more robust FL algorithms that can effectively handle unreliable label information contributed by distributed participants. While the benchmark has some limitations, it represents an important step forward in assessing the practical applicability of Federated Learning in real-world scenarios. As the field continues to evolve, the FedNoisy benchmark can serve as a valuable tool to drive the development of more reliable and trustworthy Federated Learning systems.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
FedNoisy: Federated Noisy Label Learning Benchmark
Siqi Liang, Jintao Huang, Junyuan Hong, Dun Zeng, Jiayu Zhou, Zenglin Xu
Federated learning has gained popularity for distributed learning without aggregating sensitive data from clients. But meanwhile, the distributed and isolated nature of data isolation may be complicated by data quality, making it more vulnerable to noisy labels. Many efforts exist to defend against the negative impacts of noisy labels in centralized or federated settings. However, there is a lack of a benchmark that comprehensively considers the impact of noisy labels in a wide variety of typical FL settings. In this work, we serve the first standardized benchmark that can help researchers fully explore potential federated noisy settings. Also, we conduct comprehensive experiments to explore the characteristics of these data settings and unravel challenging scenarios on the federated noisy label learning, which may guide method development in the future. We highlight the 20 basic settings for more than 5 datasets proposed in our benchmark and standardized simulation pipeline for federated noisy label learning. We hope this benchmark can facilitate idea verification in federated learning with noisy labels. texttt{FedNoisy} is available at codeword{https://github.com/SMILELab-FL/FedNoisy}.
Read more5/31/2024
0
Collaboratively Learning Federated Models from Noisy Decentralized Data
Haoyuan Li, Mathias Funk, Nezihe Merve Gurel, Aaqib Saeed
Federated learning (FL) has emerged as a prominent method for collaboratively training machine learning models using local data from edge devices, all while keeping data decentralized. However, accounting for the quality of data contributed by local clients remains a critical challenge in FL, as local data are often susceptible to corruption by various forms of noise and perturbations, which compromise the aggregation process and lead to a subpar global model. In this work, we focus on addressing the problem of noisy data in the input space, an under-explored area compared to the label noise. We propose a comprehensive assessment of client input in the gradient space, inspired by the distinct disparity observed between the density of gradient norm distributions of models trained on noisy and clean input data. Based on this observation, we introduce a straightforward yet effective approach to identify clients with low-quality data at the initial stage of FL. Furthermore, we propose a noise-aware FL aggregation method, namely Federated Noise-Sifting (FedNS), which can be used as a plug-in approach in conjunction with widely used FL strategies. Our extensive evaluation on diverse benchmark datasets under different federated settings demonstrates the efficacy of FedNS. Our method effortlessly integrates with existing FL strategies, enhancing the global model's performance by up to 13.68% in IID and 15.85% in non-IID settings when learning from noisy decentralized data.
Read more9/5/2024
0
FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models
Rui Ye, Rui Ge, Xinyu Zhu, Jingyi Chai, Yaxin Du, Yang Liu, Yanfeng Wang, Siheng Chen
Federated learning has enabled multiple parties to collaboratively train large language models without directly sharing their data (FedLLM). Following this training paradigm, the community has put massive efforts from diverse aspects including framework, performance, and privacy. However, an unpleasant fact is that there are currently no realistic datasets and benchmarks for FedLLM and previous works all rely on artificially constructed datasets, failing to capture properties in real-world scenarios. Addressing this, we propose FedLLM-Bench, which involves 8 training methods, 4 training datasets, and 6 evaluation metrics, to offer a comprehensive testbed for the FedLLM community. FedLLM-Bench encompasses three datasets (e.g., user-annotated multilingual dataset) for federated instruction tuning and one dataset (e.g., user-annotated preference dataset) for federated preference alignment, whose scale of client number ranges from 38 to 747. Our datasets incorporate several representative diversities: language, quality, quantity, instruction, length, embedding, and preference, capturing properties in real-world scenarios. Based on FedLLM-Bench, we conduct experiments on all datasets to benchmark existing FL methods and provide empirical insights (e.g., multilingual collaboration). We believe that our FedLLM-Bench can benefit the FedLLM community by reducing required efforts, providing a practical testbed, and promoting fair comparisons. Code and datasets are available at https://github.com/rui-ye/FedLLM-Bench.
Read more6/10/2024
0
FedClust: Tackling Data Heterogeneity in Federated Learning through Weight-Driven Client Clustering
Md Sirajul Islam, Simin Javaherian, Fei Xu, Xu Yuan, Li Chen, Nian-Feng Tzeng
Federated learning (FL) is an emerging distributed machine learning paradigm that enables collaborative training of machine learning models over decentralized devices without exposing their local data. One of the major challenges in FL is the presence of uneven data distributions across client devices, violating the well-known assumption of independent-and-identically-distributed (IID) training samples in conventional machine learning. To address the performance degradation issue incurred by such data heterogeneity, clustered federated learning (CFL) shows its promise by grouping clients into separate learning clusters based on the similarity of their local data distributions. However, state-of-the-art CFL approaches require a large number of communication rounds to learn the distribution similarities during training until the formation of clusters is stabilized. Moreover, some of these algorithms heavily rely on a predefined number of clusters, thus limiting their flexibility and adaptability. In this paper, we propose {em FedClust}, a novel approach for CFL that leverages the correlation between local model weights and the data distribution of clients. {em FedClust} groups clients into clusters in a one-shot manner by measuring the similarity degrees among clients based on the strategically selected partial weights of locally trained models. We conduct extensive experiments on four benchmark datasets with different non-IID data settings. Experimental results demonstrate that {em FedClust} achieves higher model accuracy up to $sim$45% as well as faster convergence with a significantly reduced communication cost up to 2.7$times$ compared to its state-of-the-art counterparts.
Read more7/11/2024