Improving Privacy-Preserving Vertical Federated Learning by Efficient Communication with ADMM
2207.10226
0
0
⚙️
Abstract
Federated learning (FL) enables distributed resource-constrained devices to jointly train shared models while keeping the training data local for privacy purposes. Vertical FL (VFL), which allows each client to collect partial features, has attracted intensive research efforts recently. We identified the main challenges that existing VFL frameworks are facing: the server needs to communicate gradients with the clients for each training step, incurring high communication cost that leads to rapid consumption of privacy budgets. To address these challenges, in this paper, we introduce a VFL framework with multiple heads (VIM), which takes the separate contribution of each client into account, and enables an efficient decomposition of the VFL optimization objective to sub-objectives that can be iteratively tackled by the server and the clients on their own. In particular, we propose an Alternating Direction Method of Multipliers (ADMM)-based method to solve our optimization problem, which allows clients to conduct multiple local updates before communication, and thus reduces the communication cost and leads to better performance under differential privacy (DP). We provide the user-level DP mechanism for our framework to protect user privacy. Moreover, we show that a byproduct of VIM is that the weights of learned heads reflect the importance of local clients. We conduct extensive evaluations and show that on four vertical FL datasets, VIM achieves significantly higher performance and faster convergence compared with the state-of-the-art. We also explicitly evaluate the importance of local clients and show that VIM enables functionalities such as client-level explanation and client denoising. We hope this work will shed light on a new way of effective VFL training and understanding.
Create account to get full access
Overview
- Federated learning (FL) allows distributed devices to jointly train shared models while keeping training data local for privacy
- Vertical FL (VFL) enables clients to collect partial features, but existing VFL frameworks require high communication costs between clients and server
- This paper introduces a VFL framework called VIM that decomposes the optimization objective into sub-objectives, reducing communication costs and improving performance under differential privacy (DP)
Plain English Explanation
Federated learning is a way for many different devices, like smartphones or computers, to work together to train a shared machine learning model without having to share their private data. This is useful for protecting people's privacy.
One type of federated learning is called vertical federated learning (VFL). In VFL, each device or "client" only has access to some of the features or characteristics of the data, not the full dataset. The paper explains that existing VFL frameworks have a big problem - the central server needs to constantly communicate with the clients to update the model, which uses up a lot of the clients' privacy budgets.
To solve this, the researchers created a new VFL framework called VIM. VIM is able to break down the overall optimization problem into smaller sub-problems that the server and clients can work on separately. This means they don't need to communicate as often, saving the clients' privacy. VIM also has another benefit - it can figure out which clients are the most important for training the model, which could be useful for understanding the data.
The paper shows that VIM outperforms other state-of-the-art VFL methods on several datasets, converging faster and achieving higher accuracy, all while better protecting the clients' privacy.
Technical Explanation
The key innovation in this paper is the VIM framework for vertical federated learning (VFL). Existing VFL approaches require the central server to communicate gradients with clients for each training step, which incurs high communication costs and quickly consumes the clients' privacy budgets under differential privacy (DP).
To address these challenges, the authors propose VIM, which decomposes the VFL optimization objective into sub-objectives that can be tackled iteratively by the server and clients separately. They use an Alternating Direction Method of Multipliers (ADMM)-based optimization method to solve this problem, allowing clients to perform multiple local updates before communicating with the server. This reduces the overall communication and improves DP performance.
A key benefit of VIM is that the learned weights of the model's "heads" (output layers) reflect the importance of each client's local data. This enables functionalities like client-level explanation and denoising, which can help understand and improve the learning process.
The authors extensively evaluate VIM on four VFL datasets and show it outperforms state-of-the-art methods in terms of performance and convergence speed, while better preserving client privacy.
Critical Analysis
The paper provides a thorough technical description of the VIM framework and its advantages over prior VFL approaches. However, there are a few areas that could be explored further:
-
Scalability: The authors demonstrate VIM's effectiveness on relatively small datasets. It would be important to evaluate its performance as the number of clients and amount of data grows, to ensure VIM can scale to real-world applications.
-
Heterogeneity: The paper does not delve deeply into how VIM handles heterogeneous data or model architectures across clients. This is a common challenge in federated learning that could be worth investigating further.
-
Interpretability: While the paper highlights VIM's ability to provide client-level explanations, more details on the interpretability and explainability of the learned model would be valuable for building trust in the system.
-
Practical Considerations: The paper focuses on the theoretical and experimental aspects of VIM. Discussing potential deployment challenges, such as communication failures or client dropouts, could help bridge the gap to real-world applications.
Overall, the VIM framework represents an important step forward in vertical federated learning, with promising results in terms of performance, privacy, and interpretability. Further research addressing the points above could help solidify VIM's position as a leading solution in this domain.
Conclusion
This paper introduces a novel vertical federated learning (VFL) framework called VIM that significantly improves upon existing VFL approaches. VIM decomposes the VFL optimization objective into sub-objectives that can be tackled iteratively by the server and clients separately, reducing communication costs and improving performance under differential privacy.
A key benefit of VIM is that the learned model weights reflect the importance of each client's local data, enabling functionalities like client-level explanation and denoising. Extensive evaluations show that VIM outperforms state-of-the-art VFL methods in terms of both accuracy and convergence speed.
This work represents an important advancement in federated learning, as it tackles the critical challenge of reducing communication overhead while preserving privacy. The insights and techniques developed in this paper could have far-reaching impacts on the deployment of federated learning systems in real-world applications.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
📊
Scalable Vertical Federated Learning via Data Augmentation and Amortized Inference
Conor Hassan, Matthew Sutton, Antonietta Mira, Kerrie Mengersen
0
0
Vertical federated learning (VFL) has emerged as a paradigm for collaborative model estimation across multiple clients, each holding a distinct set of covariates. This paper introduces the first comprehensive framework for fitting Bayesian models in the VFL setting. We propose a novel approach that leverages data augmentation techniques to transform VFL problems into a form compatible with existing Bayesian federated learning algorithms. We present an innovative model formulation for specific VFL scenarios where the joint likelihood factorizes into a product of client-specific likelihoods. To mitigate the dimensionality challenge posed by data augmentation, which scales with the number of observations and clients, we develop a factorized amortized variational approximation that achieves scalability independent of the number of observations. We showcase the efficacy of our framework through extensive numerical experiments on logistic regression, multilevel regression, and a novel hierarchical Bayesian split neural net model. Our work paves the way for privacy-preserving, decentralized Bayesian inference in vertically partitioned data scenarios, opening up new avenues for research and applications in various domains.
5/8/2024
Vertical Federated Learning for Effectiveness, Security, Applicability: A Survey
Mang Ye, Wei Shen, Bo Du, Eduard Snezhko, Vassili Kovalev, Pong C. Yuen
0
0
Vertical Federated Learning (VFL) is a privacy-preserving distributed learning paradigm where different parties collaboratively learn models using partitioned features of shared samples, without leaking private data. Recent research has shown promising results addressing various challenges in VFL, highlighting its potential for practical applications in cross-domain collaboration. However, the corresponding research is scattered and lacks organization. To advance VFL research, this survey offers a systematic overview of recent developments. First, we provide a history and background introduction, along with a summary of the general training protocol of VFL. We then revisit the taxonomy in recent reviews and analyze limitations in-depth. For a comprehensive and structured discussion, we synthesize recent research from three fundamental perspectives: effectiveness, security, and applicability. Finally, we discuss several critical future research directions in VFL, which will facilitate the developments in this field. We provide a collection of research lists and periodically update them at https://github.com/shentt67/VFL_Survey.
6/5/2024
Vertical Federated Learning Hybrid Local Pre-training
Wenguo Li, Xinling Guo, Xu Jiao, Tiancheng Huang, Xiaoran Yan, Yao Yang
0
0
Vertical Federated Learning (VFL), which has a broad range of real-world applications, has received much attention in both academia and industry. Enterprises aspire to exploit more valuable features of the same users from diverse departments to boost their model prediction skills. VFL addresses this demand and concurrently secures individual parties from exposing their raw data. However, conventional VFL encounters a bottleneck as it only leverages aligned samples, whose size shrinks with more parties involved, resulting in data scarcity and the waste of unaligned data. To address this problem, we propose a novel VFL Hybrid Local Pre-training (VFLHLP) approach. VFLHLP first pre-trains local networks on the local data of participating parties. Then it utilizes these pre-trained networks to adjust the sub-model for the labeled party or enhance representation learning for other parties during downstream federated learning on aligned data, boosting the performance of federated models. The experimental results on real-world advertising datasets, demonstrate that our approach achieves the best performance over baseline methods by large margins. The ablation study further illustrates the contribution of each technique in VFLHLP to its overall performance.
5/22/2024
Communication-efficient Vertical Federated Learning via Compressed Error Feedback
Pedro Valdeira, Jo~ao Xavier, Cl'audia Soares, Yuejie Chi
0
0
Communication overhead is a known bottleneck in federated learning (FL). To address this, lossy compression is commonly used on the information communicated between the server and clients during training. In horizontal FL, where each client holds a subset of the samples, such communication-compressed training methods have recently seen significant progress. However, in their vertical FL counterparts, where each client holds a subset of the features, our understanding remains limited. To address this, we propose an error feedback compressed vertical federated learning (EFVFL) method to train split neural networks. In contrast with previous communication-compressed methods for vertical FL, EFVFL does not require a vanishing compression error for the gradient norm to converge to zero for smooth nonconvex problems. By leveraging error feedback, our method can achieve a $mathcal{O}(1/T)$ convergence rate in the full-batch case, improving over the state-of-the-art $mathcal{O}(1/sqrt{T})$ rate under $mathcal{O}(1/sqrt{T})$ compression error, and matching the rate of uncompressed methods. Further, when the objective function satisfies the Polyak-{L}ojasiewicz inequality, our method converges linearly. In addition to improving convergence rates, our method also supports the use of private labels. Numerical experiments show that EFVFL significantly improves over the prior art, confirming our theoretical results.
6/21/2024