FedCVT: Semi-supervised Vertical Federated Learning with Cross-view Training

Overview

• Federated learning allows multiple parties to build machine learning models collaboratively without exposing their data.
• Vertical federated learning (VFL) enables parties to build a joint model based on distributed features of aligned samples.
• VFL requires a sufficient amount of aligned samples, but in reality, the set of aligned samples may be small.
• The paper proposes Federated Cross-view Training (FedCVT), a semi-supervised learning approach that improves the performance of the VFL model with limited aligned samples.

Plain English Explanation

Federated learning is a way for multiple organizations to work together on a machine learning model without sharing their private data. In vertical federated learning, the organizations each have different features (or attributes) about the same set of samples, and they can build a joint model using this distributed data. However, VFL requires that the organizations have a sufficient number of samples where the features are aligned - meaning they have the same set of features for those samples.

In reality, the number of aligned samples may be small, leaving a lot of non-aligned data unused. To address this, the researchers propose Federated Cross-view Training (FedCVT). FedCVT is a semi-supervised approach that can improve the VFL model's performance even when there are limited aligned samples.

FedCVT does this by:

1. Estimating representations for missing features: It can fill in the gaps where one organization is missing certain features for a sample.
2. Predicting pseudo-labels for unlabeled samples: It can generate labels for samples that don't have true labels, expanding the training data.
3. Training three classifiers jointly: It trains three different models simultaneously, each focusing on a different "view" of the data, to improve the overall performance.

Importantly, FedCVT does all of this without the organizations having to share their original data or model parameters, preserving data privacy. The researchers show that FedCVT significantly outperforms the basic VFL approach in experiments on several datasets.

Technical Explanation

Vertical federated learning (VFL) enables multiple parties to collaboratively build a joint machine learning model based on distributed features of aligned samples. However, VFL requires all parties to share a sufficient amount of aligned samples. In reality, the set of aligned samples may be small, leaving the majority of the non-aligned data unused.

To address this, the authors propose Federated Cross-view Training (FedCVT), a semi-supervised learning approach that improves the performance of the VFL model with limited aligned samples. FedCVT has three key components:

1. Feature Imputation: FedCVT estimates representations for missing features using a deep neural network, allowing it to leverage non-aligned samples.
2. Pseudo-labeling: FedCVT predicts pseudo-labels for unlabeled samples to expand the training set.
3. Cross-view Training: FedCVT trains three classifiers jointly based on different views of the expanded training set (original features, imputed features, and pseudo-labels) to improve the VFL model's performance.

Importantly, FedCVT does not require parties to share their original data or model parameters, preserving data privacy. The authors evaluate FedCVT on NUS-WIDE, Vehicle, and CIFAR10 datasets, demonstrating that it significantly outperforms the vanilla VFL approach that only utilizes aligned samples.

Critical Analysis

The paper presents a novel and promising approach to improving the performance of vertical federated learning models when the set of aligned samples is limited. By incorporating feature imputation, pseudo-labeling, and cross-view training, FedCVT is able to leverage non-aligned samples and expand the effective training set without compromising data privacy.

However, the authors do acknowledge some limitations of their approach. For example, the performance of FedCVT may degrade if the underlying feature distributions across parties are too different, making accurate feature imputation difficult. Additionally, the pseudo-labeling process could introduce noise, potentially harming model performance if not properly controlled.

Further research could explore ways to adaptively adjust the balance between using imputed features and pseudo-labels, based on the characteristics of the dataset and the parties involved. It would also be valuable to investigate the robustness of FedCVT to various data distribution shifts or adversarial attacks, given the importance of preserving data confidentiality in federated learning settings.

Conclusion

The Federated Cross-view Training (FedCVT) approach proposed in this paper represents a significant advancement in vertical federated learning. By leveraging semi-supervised techniques to overcome the challenge of limited aligned samples, FedCVT can improve the performance of VFL models while preserving data privacy. The experimental results demonstrate the effectiveness of this approach, and the insights gained from this research could inform the development of more robust and versatile federated learning solutions in the future.