Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model

Overview

• This paper examines privacy guarantees for machine learning models trained using differentially private optimizers like DP-SGD, where only the final model is released and not the intermediate model updates.
• The authors identify a significant gap between the lower bound provided by empirical privacy auditing and the theoretical upper bound provided by privacy accounting in this "hidden state" threat model.
• To address this gap, the authors propose a novel auditing approach where the adversary crafts a gradient sequence to maximize the privacy loss of the final model without accessing the intermediate models.

Plain English Explanation

The paper looks at machine learning models that are trained with special techniques to protect the privacy of the data used for training. These techniques, called differentially private optimizers, add noise to the training process to make it harder for an attacker to figure out the original data.

In this work, the researchers are interested in a specific scenario where only the final trained model is released, and the intermediate steps of the training process are kept secret. They found that in this "hidden state" scenario, the actual privacy guarantee (measured through experiments) is much weaker than the theoretical privacy guarantee that the techniques are supposed to provide.

To better understand this gap, the researchers developed a new way to "audit" the privacy of the final model. Instead of just looking at the final model, they have the attacker try to craft a sequence of gradients (the steps taken during training) that would maximally reduce the privacy of the final model. By doing this, they can get a better sense of the true privacy guarantee in this hidden state scenario.

The researchers found that when the crafted gradient is used at every training step, releasing only the final model does not actually improve privacy at all. However, when the crafted gradient is not used at every step, they found some evidence that privacy can still be "amplified" to some degree, even in the hidden state scenario, although not as much as in simpler cases.

Technical Explanation

The paper explores the privacy guarantees of machine learning models trained using differentially private stochastic gradient descent (DP-SGD), where only the final trained model is released and the intermediate model updates are not publicly accessible (the "hidden state" threat model).

The authors identify a significant gap between the lower bound on privacy loss provided by empirical auditing and the theoretical upper bound provided by privacy accounting in this hidden state scenario. To challenge this gap, they propose a novel auditing approach where the adversary crafts a gradient sequence to maximize the privacy loss of the final model, without accessing the intermediate models.

Experimentally, the authors demonstrate that this crafted gradient approach consistently outperforms prior attempts at auditing the hidden state model. When the crafted gradient is inserted at every optimization step, their results imply that releasing only the final model does not amplify privacy, providing a novel negative result. On the other hand, when the crafted gradient is not inserted at every step, the authors show strong evidence that a privacy amplification phenomenon emerges in the general non-convex setting (albeit weaker than in convex regimes), suggesting that existing privacy upper bounds can be improved.

These findings have important implications for understanding the privacy guarantees of machine learning models trained with differentially private techniques, especially in scenarios where only the final model is released. The authors' novel auditing approach provides a more comprehensive way to assess the true privacy risks in such settings.

Critical Analysis

The paper makes a valuable contribution by highlighting the significant gap between theoretical privacy guarantees and empirical privacy risks in the "hidden state" threat model for differentially private machine learning. The authors' proposed auditing approach using crafted gradients is a novel and insightful technique that provides a more comprehensive assessment of privacy risks in this scenario.

However, the paper also acknowledges several limitations and caveats. The authors note that their results only apply to the specific threat model where the adversary cannot access the intermediate model updates, and that the privacy amplification effect they observe is weaker in the general non-convex setting compared to simpler convex regimes. Further research may be needed to fully understand the implications of these findings and explore alternative threat models or auditing techniques.

Additionally, while the authors' auditing approach is a significant advancement, it still relies on the ability of the adversary to craft a gradient sequence that maximizes the privacy loss. In practice, this may be a challenging task, and the effectiveness of the approach may depend on the specific machine learning task and dataset. Exploring alternative auditing methods could further strengthen the understanding of privacy guarantees in these settings.

Overall, the paper presents important insights and a novel technical contribution to the field of differentially private machine learning. However, as with any research, there are areas that warrant further investigation and refinement to fully understand the implications and limitations of the findings.

Conclusion

This paper tackles the important challenge of understanding the privacy guarantees of machine learning models trained with differentially private techniques, particularly in the "hidden state" scenario where only the final model is released. By proposing a novel auditing approach using crafted gradients, the authors are able to identify a significant gap between theoretical and empirical privacy risks in this setting.

The key findings of the paper have important implications for the development and deployment of privacy-preserving machine learning systems. The negative result showing that releasing only the final model does not amplify privacy, as well as the evidence of weaker privacy amplification in the general non-convex setting, suggest that existing theoretical privacy bounds may need to be re-evaluated. Additionally, the authors' auditing technique provides a more comprehensive way to assess the true privacy risks in these scenarios.

Overall, this work contributes valuable insights and methodological advancements to the field of differentially private machine learning, with potential implications for the responsible development and deployment of privacy-preserving AI systems.