AI visibility, governance, and protection begin with an understanding of the current risk landscape. The chart below ranks the most prevalent risks by the frequency of violations observed within organizations. At the top of this list are upstream data policy violations, which once accounted for nearly 100% of alerts but have fallen below 90% as other types of violations have increased in volume. At the other end of the spectrum are malicious code violations (where an AI system returns malicious code), which are the fewest but also the most severe.
In the following list, we explore each of these risks in more detail, including the direction, intent, and severity.
Upstream data policy violation
Direction: Upstream
Intent profile: Insider/productivity/negligence
Severity: Cumulative
What is an upstream data policy violation?
The most common AI risk is a user or agent attempting to send data to an AI system in violation of the organization’s data use policies, including sending confidential information or source code to unapproved services. Organizations typically use such policies to steer their users away from shadow AI and toward approved AI services, and to steer use cases to the appropriate services. For example, some AI services might be authorized only for coding and development tasks, while others are authorized solely to process sensitive data in documents and spreadsheets. Data policy violations originate primarily from trusted insiders who seek to increase their productivity but do so negligently (without knowledge of organizational policy). The risk severity here is cumulative because a consistent stream of exposure of sensitive data to third parties can cause compliance or regulatory liability.
Downstream data policy violation
Direction: Downstream
Intent profile: Insider/productivity/negligence
Severity: Low/moderate
What is a downstream data policy violation?
Downstream data policy violations occur when an AI service returns data that organizational policies prohibit the user or agent from accessing. Such violations occur when an organization uses MCP, retrieval-augmented generation (RAG), or customized models trained on proprietary data. In these cases, an AI app might return some sensitive data from a connected system that the user or agent is not authorized to access. Another less common example of a downstream data policy violation occurs when an organization authorizes an app but restricts its use to a specific use case. For example, users and agents might be allowed to use the app as a general-purpose chatbot, but not to generate source code. Any source code returned by the app would be classified as a downstream violation. As with upstream violations, the primary intent profile for downstream violations is a trusted insider motivated by productivity gain, and the risk severity depends on the type of data and the breadth of access.
Content filtering
Direction: Bidirectional
Intent profile: Insider/behavior
Severity: Low
What is content filtering in the context of AI risk?
Content filtering violations are bidirectional and occur when a user or agent sends a prompt to, or receives a response from, an AI system that contains content deemed inappropriate under organizational policies. These types of violations are similar to URL filtering violations from a secure web gateway (SWG): In the same way that organizations block sexual or violent content on the web, they typically also block such content from being returned by AI apps. Content filtering violations are low in severity because they pose minimal threat to the organization’s network or data assets. Instead, they are primarily HR, legal compliance, or cultural alignment issues.
Prompt injection and jailbreaking
Direction: Bidirectional
Intent profile: Adversarial/evasion
Severity: Critical
What is prompt injection? What is jailbreaking?
Prompt injection and jailbreaking violations occur when a user or agent attempts to bypass guardrails, subvert underlying system instructions, or otherwise force an AI system to perform some restricted action. Such violations result from deliberate adversarial or evasive actions by an insider or an external adversary. In many cases, these are indirect prompt injection attacks, in which an adversary embeds malicious instructions into external data sources accessed by an AI system. The severity is critical because successful exploitation subverts the model control plane, potentially triggering cascading execution paths across connected backend services.
Requests for sensitive data
Direction: Upstream
Intent profile: Adversarial/reconnaissance/harvesting
Severity: High
What are requests for sensitive data in the context of AI risk?
Requests for sensitive data are upstream violations that occur when a user or agent exhibits signs of deliberately attempting to bypass controls to access it. The severity of such a violation is generally higher than that of a downstream data policy violation due to a different intent profile. A downstream data policy violation might occur without deliberate prompting simply because the system has access to data that the user is not authorized to access. By contrast, requests for sensitive data indicate that the requester intends to access such data. Requests for sensitive data, combined with downstream data violations, indicate a critical risk, as they suggest the requester is succeeding in their efforts.
Piracy and copyright
Direction: Downstream
Intent profile: Compliance/accidental
Severity: Low
What are the copyright risks associated with AI?
A piracy and copyright violation is a downstream violation that occurs when an AI application returns protected, copyrighted, or trademarked material without valid third-party licensing or attribution of permission. Such material poses a legal and financial risk to the organization if used inappropriately. Still, it presents a minimal threat to the organization’s network or data assets and is therefore considered low risk.
Malicious code
Direction: Downstream
Intent profile: Adversarial/weaponization
Severity: Critical
What is malicious code in the AI context?
A malicious code violation occurs when an AI application returns malicious code in its response. The malicious code might be some shell commands that the user or agent might execute directly or some malicious code snippet that is incorporated into a broader code base. Autonomous coding agents, which are rapidly gaining popularity, amplify this risk because they generate code much faster than humans and autonomously execute AI-generated code and commands. An AI app might return malicious code as the result of an indirect prompt injection, trojanized AI tooling or models, or even organically. Organic malicious code might be returned as a result of training on vulnerable data, emergent misalignment, or hallucination. The severity is critical because successful execution of malicious code might result in widespread system compromise. Malicious code violations coupled with prompt injection and jailbreaking violations reveal intent and indicate possible success.
