When Machine Learning Models Leak: An Exploration of Synthetic Training Data
2310.08775
0
0
🏋️
Abstract
We investigate an attack on a machine learning model that predicts whether a person or household will relocate in the next two years, i.e., a propensity-to-move classifier. The attack assumes that the attacker can query the model to obtain predictions and that the marginal distribution of the data on which the model was trained is publicly available. The attack also assumes that the attacker has obtained the values of non-sensitive attributes for a certain number of target individuals. The objective of the attack is to infer the values of sensitive attributes for these target individuals. We explore how replacing the original data with synthetic data when training the model impacts how successfully the attacker can infer sensitive attributes.
Create account to get full access
Overview
- This research paper investigates an attack on a machine learning model that predicts whether a person or household will relocate in the next two years.
- The attack assumes the attacker can query the model to obtain predictions and has access to the public data distribution used to train the model.
- The attacker also has access to the non-sensitive attribute values for a set of target individuals.
- The goal of the attack is to infer the sensitive attribute values for those target individuals.
- The paper explores how replacing the original training data with synthetic data impacts the attacker's ability to infer sensitive attributes.
Plain English Explanation
The researchers looked at a way for attackers to try to guess personal information about people, even if they don't have direct access to that information. They focused on a machine learning model that predicts whether someone will move in the next two years.
The researchers assumed the attacker could interact with the model and see its predictions, and also had access to the overall distribution of the data used to train the model (but not the specific data itself). Additionally, the attacker knew some non-sensitive details about certain people.
Using this information, the attacker's goal was to try to figure out the sensitive personal details about those people - details the attacker wasn't supposed to have access to. The researchers explored how replacing the original training data with synthetic data might impact the attacker's ability to infer those sensitive details.
Technical Explanation
The researchers designed an attack that assumes the attacker can query the propensity-to-move classifier model to obtain predictions, and also has access to the public marginal distribution of the data used to train the model.
Additionally, the attacker is assumed to have obtained the values of non-sensitive attributes for a set of target individuals. The goal of the attack is to infer the values of sensitive attributes for these target individuals.
The researchers explore how replacing the original training data with synthetic data impacts the attacker's ability to successfully infer sensitive attributes. This relates to concepts like the transpose attack and SEEP training dynamics.
Critical Analysis
The paper acknowledges some limitations, such as the assumption that the attacker has access to the public data distribution used to train the model. In a real-world scenario, this information may not always be readily available.
Additionally, the researchers only explore replacing the original data with synthetic data, but do not investigate other potential countermeasures, such as privacy-preserving debiasing techniques or shake-to-leak fine-tuning. Further research could examine a broader range of defense mechanisms.
Conclusion
This research paper investigates an attack on a machine learning model that predicts whether a person or household will relocate in the next two years. The attack assumes the attacker can query the model, has access to the public data distribution, and knows some non-sensitive details about target individuals.
The researchers explore how replacing the original training data with synthetic data impacts the attacker's ability to infer sensitive attributes about the target individuals. This work highlights the importance of considering potential attacks and developing robust defenses to protect the privacy of individuals in machine learning systems.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
Synthetic Data Outliers: Navigating Identity Disclosure
Carolina Trindade, Lu'is Antunes, T^ania Carvalho, Nuno Moniz
0
0
Multiple synthetic data generation models have emerged, among which deep learning models have become the vanguard due to their ability to capture the underlying characteristics of the original data. However, the resemblance of the synthetic to the original data raises important questions on the protection of individuals' privacy. As synthetic data is perceived as a means to fully protect personal information, most current related work disregards the impact of re-identification risk. In particular, limited attention has been given to exploring outliers, despite their privacy relevance. In this work, we analyze the privacy of synthetic data w.r.t the outliers. Our main findings suggest that outliers re-identification via linkage attack is feasible and easily achieved. Furthermore, additional safeguards such as differential privacy can prevent re-identification, albeit at the expense of the data utility.
6/6/2024
🏋️
Transpose Attack: Stealing Datasets with Bidirectional Training
Guy Amit, Mosh Levy, Yisroel Mirsky
0
0
Deep neural networks are normally executed in the forward direction. However, in this work, we identify a vulnerability that enables models to be trained in both directions and on different tasks. Adversaries can exploit this capability to hide rogue models within seemingly legitimate models. In addition, in this work we show that neural networks can be taught to systematically memorize and retrieve specific samples from datasets. Together, these findings expose a novel method in which adversaries can exfiltrate datasets from protected learning environments under the guise of legitimate models. We focus on the data exfiltration attack and show that modern architectures can be used to secretly exfiltrate tens of thousands of samples with high fidelity, high enough to compromise data privacy and even train new models. Moreover, to mitigate this threat we propose a novel approach for detecting infected models.
5/20/2024
Reconstruction Attacks on Machine Unlearning: Simple Models are Vulnerable
Martin Bertran, Shuai Tang, Michael Kearns, Jamie Morgenstern, Aaron Roth, Zhiwei Steven Wu
0
0
Machine unlearning is motivated by desire for data autonomy: a person can request to have their data's influence removed from deployed models, and those models should be updated as if they were retrained without the person's data. We show that, counter-intuitively, these updates expose individuals to high-accuracy reconstruction attacks which allow the attacker to recover their data in its entirety, even when the original models are so simple that privacy risk might not otherwise have been a concern. We show how to mount a near-perfect attack on the deleted data point from linear regression models. We then generalize our attack to other loss functions and architectures, and empirically demonstrate the effectiveness of our attacks across a wide range of datasets (capturing both tabular and image data). Our work highlights that privacy risk is significant even for extremely simple model classes when individuals can request deletion of their data from the model.
5/31/2024
🤯
A Linear Reconstruction Approach for Attribute Inference Attacks against Synthetic Data
Meenatchi Sundaram Muthu Selva Annamalai, Andrea Gadotti, Luc Rocher
0
0
Recent advances in synthetic data generation (SDG) have been hailed as a solution to the difficult problem of sharing sensitive data while protecting privacy. SDG aims to learn statistical properties of real data in order to generate artificial data that are structurally and statistically similar to sensitive data. However, prior research suggests that inference attacks on synthetic data can undermine privacy, but only for specific outlier records. In this work, we introduce a new attribute inference attack against synthetic data. The attack is based on linear reconstruction methods for aggregate statistics, which target all records in the dataset, not only outliers. We evaluate our attack on state-of-the-art SDG algorithms, including Probabilistic Graphical Models, Generative Adversarial Networks, and recent differentially private SDG mechanisms. By defining a formal privacy game, we show that our attack can be highly accurate even on arbitrary records, and that this is the result of individual information leakage (as opposed to population-level inference). We then systematically evaluate the tradeoff between protecting privacy and preserving statistical utility. Our findings suggest that current SDG methods cannot consistently provide sufficient privacy protection against inference attacks while retaining reasonable utility. The best method evaluated, a differentially private SDG mechanism, can provide both protection against inference attacks and reasonable utility, but only in very specific settings. Lastly, we show that releasing a larger number of synthetic records can improve utility but at the cost of making attacks far more effective.
5/10/2024