Key Takeaways
- Generative AI has shown to be proficient at making inferences using vast sets of data, which increases the potential of generative AI to identify data that has been potentially deidentified.
- Using anonymized or deidentified data in training sets may not be sufficient, which comes with compliance and regulatory implications.
- While this is a relatively new concept, there are some ways to mitigate the risk of deidentified data being leaked via generative AI.
Why Deidentified Data Is No Longer Enough
The use of generative AI has increasingly become the subject of concern for the broader public. According to a Deloitte survey, amongst IT and business professionals, data privacy was highlighted as the most significant concern with the use of generative AI.
The question arises—is deidentification enough in the face of growing AI capabilities? As organizations expand the use of generative AI, they need more than technical safeguards alone. LBMC’s AI and generative AI strategy services help organizations evaluate AI readiness, establish governance, assess privacy risks, and implement responsible AI practices that support innovation while protecting sensitive data.
How Generative AI Can Leak Sensitive Data
Generative AI can leak sensitive data if it has “memorized” the data as part of training the model. A study performed in 2021 using GPT-2 successfully and accurately extracted training data, including sensitive data such as names and addresses that were “memorized” by the large language model (LLM). If a model were to be trained using sensitive data, it would therefore be possible for a malicious actor to attempt to extract that sensitive data.
How AI Can Infer Hidden Personal Information
Generative AI can infer new data based on existing inputs. For example, if a user asks, “How do I implement a neural network in Python?” the AI may infer the user is involved in computer science or data science. While this example may seem innocuous, AI is more than capable of using many different data points and arriving at more detailed inferences.
These capabilities are becoming increasingly important as healthcare AI and automation trends reshape how organizations use clinical, operational, and financial data. As AI adoption accelerates, healthcare leaders must balance innovation with strong privacy, governance, and data protection practices.
If you provide an AI with an individual’s comment history on a platform like Reddit, as this study has shown, the AI can correctly infer personal sensitive information on the individual leaving the comments, even if the sensitive details are not included. In the study’s example, the AI used a combination of comments on traffic complaints, clothing, and TV shows to ascertain the individual’s location, gender, and age.
Deidentification of Data
Deidentifying data via anonymization and masking is a common practice, especially in healthcare. While the current best practice is to deidentify sensitive information used in training data to limit the potential effects of memorization, this may not be sufficient. Studies have shown the feasibility of attacks on masked training data where an attacker can use both training data extraction attacks and inference attacks could reidentify data that has been deidentified for LLM Training purposes.
To summarize the concern, if an organization’s AI was trained using deidentified data across many different subjects, a malicious attacker could extract enough training data that has been memorized to then use inference techniques to identify the individual the data relates to. If the attacker already has the redacted training data, the task of unmasking the deidentified data becomes much easier.
The growing ability of AI to infer information from seemingly anonymous datasets changes how organizations should think about data privacy. Traditional deidentification techniques remain valuable, but they should now be considered one component of a broader AI governance strategy that includes technical safeguards, access controls, continuous monitoring, and ongoing risk assessments.
How to Reduce AI Inference Risk
- Avoid Using Actual Data: The simplest solution is not to use actual data for training LLMs. If live data must be used, deidentification and redaction help but may not be sufficient.
- Decentralized Training: Training techniques that don’t expose the public-facing model to the raw training data, such as Federated Learning, can reduce the risk of data leakage.
- Implement Access and Change Controls: Logical access and change management controls can prevent attackers from accessing full training data, thus making their inference attacks easier.
- Evaluate Third-Party Security: Ensure that any purchased models have robust security controls and have undergone third-party AI audits. Third-party AI models should also be evaluated within the context of broader healthcare AI and cybersecurity trends, including evolving regulatory expectations, supply chain risk, cyber resilience, and the growing use of AI across healthcare environments.
- Prompt Monitoring Tools: Implement tools to monitor and block malicious prompts, preventing memory extraction and inference attacks.
- Audits and Penetration Testing: Conducting regular security audits and penetration testing can proactively identify vulnerability leakage and inference vulnerabilities, allowing mitigation before an actual event.
Mitigating AI Inference Risks: Best Practices for Securing Deidentified Healthcare Data
Organizations, especially in healthcare, rely on deidentifying data to safeguard their sensitive information. Generative AI has inference capabilities that are exacerbated by risks in AI memorizing training data. By implementing best practice security controls around logical access, change management, and third-party risk management, organizations can mitigate inference attacks.
Although this discussion focuses on healthcare data, similar governance challenges exist across other business functions. The growing use of AI in accounting demonstrates that organizations must also protect financial information, validate AI outputs, and establish strong oversight wherever sensitive business data is used to train or support AI systems.
Protect Your AI Programs Before Threats Strike
Organizations adopting AI need more than penetration testing — they need a comprehensive strategy for securing data, governing AI systems, and reducing operational risk.
LBMC’s Data and AI services help organizations improve data readiness, establish AI governance, identify privacy risks, and implement solutions that balance innovation with security and regulatory compliance.
Our cybersecurity professionals also provide AI security assessments, audits, and penetration testing designed to identify vulnerabilities before they become business risks.
