Key Takeaways

• Generative AI has shown to be proficient at making inferences using vast sets of data, which increases the potential of generative AI to identify data that has been potentially deidentified.
• Using anonymized or deidentified data in training sets may not be sufficient, which comes with compliance and regulatory implications.
• While this is a relatively new concept, there are some ways to mitigate the risk of deidentified data being leaked via generative AI.

Introduction

The use of generative AI has increasingly become the subject of concern for the broader public. According to a Deloitte survey, amongst IT and business professionals, data privacy was highlighted as the most significant concern with the use of generative AI. The question arises – is deidentification enough in the face of growing AI capabilities? Can AI see through our masking of data and anonymization efforts?

Generative AI Data Leakage

Generative AI can leak sensitive data if it has “memorized” the data as part of training the model. A study performed in 2021 using GPT-2 successfully and accurately extracted training data, including sensitive data such as names and addresses that were “memorized” by the large language model (LLM). If a model were to be trained using sensitive data, it would therefore be possible for a malicious actor to attempt to extract that sensitive data.

Generative AI Data Inferences

Generative AI can infer new data based on existing inputs. For example, if a user asks, “How do I implement a neural network in Python?” the AI may infer the user is involved in computer science or data science. While this example may seem innocuous, AI is more than capable of using many different data points and arriving at more detailed inferences. If you provide an AI with an individual’s comment history on a platform like Reddit, as this study has shown, the AI can correctly infer personal sensitive information on the individual leaving the comments, even if the sensitive details are not included. In the study’s example, the AI used a combination of comments on traffic complaints, clothing, and TV shows to ascertain the individual’s location, gender, and age.

Deidentification of Data

Deidentifying data via anonymization and masking is a common practice, especially in healthcare.  While the current best practice is to deidentify sensitive information used in training data to limit the potential effects of memorization, this may not be sufficient. Studies have shown the feasibility of attacks on masked training data where an attacker can use both training data extraction attacks and inference attacks could reidentify data that has been deidentified for LLM Training purposes.

To summarize the concern, if an organization’s AI was trained using deidentified data across many different subjects, a malicious attacker could extract enough training data that has been memorized to then use inference techniques to identify the individual the data relates to. If the attacker already has the redacted training data, the task of unmasking the deidentified data becomes much easier.

Mitigation Techniques

• Avoid Using Actual Data: The simplest solution is not to use actual data for training LLMs. If live data must be used, deidentification and redaction help but may not be sufficient.
• Decentralized Training: Training techniques that don’t expose the public-facing model to the raw training data, such as Federated Learning, can reduce the risk of data leakage.
• Implement Access and Change Controls: Logical access and change management controls can prevent attackers from accessing full training data, thus making their inference attacks easier.
• Evaluate Third-Party Security: Ensure that any purchased models have robust security controls and have undergone third-party AI audits.
• Prompt Monitoring Tools: Implement tools to monitor and block malicious prompts, preventing memory extraction and inference attacks.
• Audits and Penetration Testing: Conducting regular security audits and penetration testing can proactively identify vulnerability leakage and inference vulnerabilities, allowing mitigation before an actual event.

Mitigating AI Inference Risks: Best Practices for Securing Deidentified Healthcare Data

Organizations, especially in healthcare, rely on deidentifying data to safeguard their sensitive information. Generative AI has inference capabilities that are exacerbated by risks in AI memorizing training data. By implementing best practice security controls around logical access, change management, and third-party risk management, organizations can mitigate inference attacks.

Protect Your AI Programs Before Threats Strike

We provide comprehensive audits and penetration testing services designed to proactively safeguard your AI initiatives—helping you identify and resolve vulnerabilities before a security event occurs.

Content provided by Andrew Stansfield, LBMC Cybersecurity Manager. He can be reached at andrew.stansfield@lbmc.com.