Nearly Tight Black-Box Auditing of Differentially Private Machine Learning

Overview

• This paper presents a new method for auditing the privacy guarantees of the Differentially Private Stochastic Gradient Descent (DP-SGD) algorithm in a black-box setting.
• The authors empirically estimate the privacy leakage from DP-SGD using membership inference attacks, and show that their estimates are much closer to the theoretical DP bounds compared to previous work.
• The key insight is to craft worst-case initial model parameters, as DP-SGD's privacy analysis is agnostic to the choice of the initial model parameters.

Plain English Explanation

Differential privacy is an important concept in machine learning that helps protect the privacy of the data used to train models. DP-SGD is a popular algorithm for training machine learning models with differential privacy guarantees.

In this paper, the authors present a new way to audit the privacy of DP-SGD models. They use a technique called "membership inference attacks" to estimate how much information about the training data is being leaked by the model. Unlike previous work, their estimates are much closer to the theoretical privacy bounds promised by differential privacy.

The key insight is that the privacy analysis of DP-SGD doesn't actually depend on the initial parameters of the model. So the authors craft "worst-case" initial parameters to get a more realistic estimate of the privacy leakage.

For models trained with a theoretical privacy parameter of 10.0 on the MNIST and CIFAR-10 datasets, the authors' auditing procedure yielded empirical estimates of around 7 and 5, respectively. This is much tighter than previous work, which could only achieve tight audits in unrealistic "white-box" settings where the adversary had full access to the model's internal parameters.

The authors' auditing procedure can be used to more easily detect bugs or violations of the promised differential privacy guarantees. It also provides valuable insights that can be used to further improve the privacy analysis of DP-SGD.

Technical Explanation

The paper presents a new auditing procedure for the Differentially Private Stochastic Gradient Descent (DP-SGD) algorithm in a black-box setting. Unlike previous work, the authors' estimates of the privacy leakage from DP-SGD are much closer to the theoretical DP bounds.

The key insight is to craft worst-case initial model parameters, as DP-SGD's privacy analysis is agnostic to the choice of the initial model parameters. The authors empirically estimate the privacy leakage using membership inference attacks, which aim to determine whether a given data point was part of the training set.

For models trained with a theoretical privacy parameter of 10.0 on the MNIST and CIFAR-10 datasets, the authors' auditing procedure yields empirical estimates of 7.21 and 6.95, respectively, on 1,000-record samples, and 6.48 and 4.96 on the full datasets. This is in contrast to previous work, which could only achieve tight audits in stronger (i.e., less realistic) white-box models that allow the adversary to access the model's inner parameters and insert arbitrary gradients.

The authors' auditing procedure can be used to more easily detect bugs and DP violations, and offers valuable insight into how the privacy analysis of DP-SGD can be further improved. The paper also discusses the limitations of the approach and suggests areas for future research, such as extending the auditing procedure to other differential privacy mechanisms.

Critical Analysis

The paper presents a novel and promising approach for auditing the privacy guarantees of DP-SGD models in a black-box setting. The authors' key insight of crafting worst-case initial model parameters to get more realistic privacy estimates is a clever way to address a limitation in the theoretical privacy analysis of DP-SGD.

However, the paper does not address several important caveats and limitations. For example, the auditing procedure relies on membership inference attacks, which have been shown to have limitations and biases. It's unclear how robust the authors' estimates would be to more sophisticated attacks or different attack models.

Additionally, the paper only considers a single dataset (MNIST) and a single hyperparameter setting (ε=10.0) for DP-SGD. It's important to evaluate the auditing procedure's performance across a wider range of datasets, model architectures, and privacy parameters to fully understand its strengths and weaknesses.

The paper also does not address the potential implications of this work for the broader field of differential privacy. For example, it would be valuable to understand how the authors' findings relate to uncertainty quantification in DP-SGD or the linear scaling rule for private hyperparameter tuning.

Overall, the paper presents an intriguing new approach for auditing DP-SGD, but more research is needed to fully understand its strengths, limitations, and broader significance. Readers should approach the findings with appropriate caution and critical thinking.

Conclusion

This paper introduces a new method for auditing the privacy guarantees of the Differentially Private Stochastic Gradient Descent (DP-SGD) algorithm in a black-box setting. The authors' key insight is to craft worst-case initial model parameters, which allows them to obtain empirical estimates of the privacy leakage that are much closer to the theoretical DP bounds compared to previous work.

The authors' auditing procedure can be used to more easily detect bugs or violations of the promised differential privacy guarantees, and it also provides valuable insights that can be used to further improve the privacy analysis of DP-SGD. While the paper presents a promising new approach, more research is needed to fully understand its strengths, limitations, and broader implications for the field of differential privacy.