Key Takeaways:

• Third-party auditors cannot provide reasonable assurance on a Generative AI’s output, but can provide reasonable assurance on the controls around the development, maintenance, and use of AI by the organization.
• Key controls to implement when using Generative AI within an organization include workforce training controls, logical access controls, third-party risk management controls, and change management controls.
• Broad and specific security frameworks already exist that would allow an internal or external audit of an organization’s Generative AI systems.

The benefits of integrating Generative AI within organizations include giving them a competitive edge. However, the risks of Generative AI can have drastic consequences. Conducting an audit of systems with AI helps provide the organization and its customers with assurance that the data processed through AI systems remains secure.

While the technology is relatively new, existing and new information security frameworks can be used by internal and external auditors to achieve some assurance that Generative AI is being used in a secure manner.

Reasonable Assurance and AI

Reasonable Assurance is the level of confidence an auditor can obtain regarding the completeness and accuracy of a statement. In terms of an audit, this would mean the auditor has gathered enough evidence to support that whatever statement is made is complete, accurate, and free from material misstatement. This can pose a challenge to organizations wishing to audit systems that have AI components where those components rely on the output of AI to deliver the final output. As the output of AI can vary, it would be impossible for an auditor to gain comfort that the output of an AI would be complete, accurate, and void of misstatements. However, that does not mean that reasonable assurance could not be gained on the AI itself and the controls that surround the AI. For example, while an auditor may not be able to reasonably assure the output of an AI used in processing information is complete and accurate, the auditor could validate the prompt used to generate the output is secured from tampering, or that any review process around the output of the AI occurred in line with organizational requirements.

Key Control Areas

Below are several key areas where controls should be implemented to address and mitigate the risks involved when using Generative AI. While this list is not exhaustive, it should provide organizations with some examples of the types of risks that come with AI components and ways they can be addressed. These areas were specifically identified, as they are likely already in effect if an organization is undergoing an audit such as a SOC 2 Type 2, but reviewing them under the lens of AI assures internal and external stakeholders of the appropriate and secure use of generative AI.

Policy & Procedure

The first step in many organizations’ AI security journey is drafting an AI Policy document, which details the organization’s stance on using generative AI. This could be as simple as a broad ban on the use of AI, if that’s the organization’s desire, but could specify the various other controls that will need to be implemented where AI is allowed.

Documenting a procedure around the use of AI brings to life the policy document. This would specify the specific AI solutions allowed. For example, this could be the specific chatbots allowed (ChatGPT, Copilot, etc.), the circumstances for using the chatbots, and whether the organization has its own containerized version of the chatbot that is a prerequisite to use the chatbot. Further, this could also specify the steps required to follow if members of the organization wish to use AI. This procedure would establish the controls needed to follow, such as required approvals, third-party risk management, change management, etc.

Workforce Training

Providing workforce members training around the use of AI serves several purposes, but we’ll highlight the three primary ones below. First and foremost, it provides a platform to communicate the organization’s formally established policies and procedures around AI to the organization as a whole to ensure everyone is made aware of their expectations. The second purpose serves to encourage secure use of AI. By providing the broader organization with specific guidance on the appropriate use of AI, you decrease the incentive to use publicly available AI in an inappropriate fashion. For example, if your organization has its own containerized AI from a third party or custom built AI, by making this known and readily available to workforce members, you decrease the incentive of those workforce members to use other AI in an insecure manner. The third purpose is to encourage the appropriate use of AI. This would aim to discourage blind trust, or over-reliance on generative AI outputs without consideration for potential inaccuracies and limitations.

Third Party Risk Management

Most organizations will acquire the Generative AI component (application, tool, plugin, etc.) from a third party. When developing a custom AI, the Large Language Model (LLM) is often sourced from third parties. As such, ensuring appropriate third-party risk management controls to review and evaluate the third parties providing AIs and LLMs is critical. For example, if the AI source has poor controls around logical access and change management, it is feasible a malicious actor could have compromised the training data used to train the AI you’ve now added to your organization. Another example is if the organization’s data is used to train the model that other organizations use, your organization’s data is potentially being leaked to other parties. Even when training data has been deidentified, there are still privacy concerns where a malicious actor could use the AI to extract and piece together information to reidentify it.  

Logical Access

Once an AI component has been implemented, ensuring logical access to the systems supporting the AI, as well as the AI itself, is essential. This is particularly important around the prompt itself. For example, if an organization’s system that contains the prompt is compromised, the malicious actor who compromised the prompt could significantly change the AI’s output to serve their needs, or to siphon off the data the AI is working with. Further, if the organization is training its own AI, securing the training data prevents malicious actors from inserting their own information into the training data. 

The most problematic prompt-related vulnerabilities with Generative AI have been prompt injections, where an attacker embeds malicious instructions in an otherwise innocuous request. With the rise in the use of agentic AI, where an AI agent has access to many systems containing potentially vast amounts of sensitive information, this is of particular concern. Because AI treats the whole request under the same context, it has a hard time determining what instructions should be trusted or untrusted, which makes mitigation controls difficult.

 A new solution has recently been proposed by Google to address this risk with agentic AI called CaMeL (Capabilities for Machine Learning) where two LLMs are used. CaMeL uses one LLM to translate prompts into code, and then uses another LLM with an interpreter to evaluate whether any of the actions violate predefined security policies. The first LLM does not actually need to view any of the content where prompt injections would normally be placed and only ever sees the initial query. The second LLM only takes actions if the security policies permit. In this way, CaMeL provides a way to implement least-privilege principles into AI tasks. While this technique significantly improves the security of LLM agents against prompt injection attacks and allows security administrators to implement finely tuned security policies, it has some limitations and requires organizations to spend time and effort to identify security policies.

Change Management

Many organizations are moving to include generative AI features into their existing applications. Best practice change management controls, such as code reviews, testing, code scanning, and approvals will likely pick up on unauthorized changes to the AI systems. For example, this could be a chatbot that users interact with to get help on the website. In such circumstances, the organization will likely add its own wording to a user’s prompt to ensure that it can execute in accordance with the organization’s standards. If unintended or malicious changes are made to the prompt, the implications could be detrimental. For example, if a rogue developer changes the prompt’s code to ignore previous instructions and send the user’s details to the individual’s personal email, the developer can start exfiltrating data. This kind of change to the prompt would likely be caught through testing, code review, or approval steps.

AI Audits

As the capability and risks associated with generative AI have increased as the technology develops, security organizations have started publishing frameworks to guide organizations on generative AI security best practices. NIST has issued the AI Risk Management Framework, which defines 13 unique or exacerbated risks related to generative AI, and MITRE has published its ATLAS Matrix, which shows the progression of an attack on AI. While an audit or assessment against these sources is feasible by an internal or external party, the audit would not come with the backing of a governing body like other established audits do. Governing bodies of existing security frameworks, such as HITRUST and CSA Star, have published or are developing formal AI Assessments. In HITRUST’s case, the formal assessment backed by HITRUST is currently available, which organizations can use to evaluate their current AI usage against and demonstrate compliance to their customers.

SOC 2 reports are standard across many industries for service organizations to demonstrate compliance with information security best practices. While AI is not specifically addressed within the COSO framework, all of the above control areas would fall within the SOC TSCs. For organizations that already undergo SOC 2 Type 2 Assessments, including AI Components into the scope of a SOC 2 Audit would be an efficient way to demonstrate the implementation of security best practices on AI systems.

Content provided by Andrew Stansfield, LBMC Cybersecurity Manager. He can be reached at andrew.stansfield@lbmc.com.