Artificial intelligence is no longer isolated inside experimental models or research environments. AI is now embedded in customer support platforms, SaaS products, internal knowledge systems, developer tools, financial workflows, analytics platforms and autonomous software agents. As these systems gain access to company data, APIs and real-world actions, securing the model alone is no longer enough.

AI security is the discipline of protecting AI-powered applications, models, data pipelines, retrieval systems, connected tools and surrounding infrastructure against manipulation, unauthorized access, information disclosure and unintended actions. Modern AI security therefore combines traditional application security with new attack surfaces created by large language models, machine learning systems and autonomous agents.

This distinction matters. The OWASP GenAI Security Project identifies risks such as prompt injection, sensitive information disclosure, data and model poisoning, excessive agency and other weaknesses that emerge when generative AI becomes part of an application architecture. NIST likewise treats AI risk as a broader lifecycle problem and provides frameworks for identifying and managing risks across the design, development, deployment and operation of AI systems.

For organizations deploying AI in production, the practical question is no longer whether AI introduces additional security considerations. The question is where those security boundaries exist and whether they hold when the system is deliberately manipulated.

What AI Security Actually Means

AI security is sometimes reduced to protecting a model from malicious prompts. That definition is too narrow for modern applications.

A production AI system usually includes several connected components: user interfaces, authentication, application logic, model APIs, retrieval systems, databases, vector stores, external APIs, internal tools and sometimes autonomous agents capable of executing actions. Each connection introduces trust assumptions.

An attacker does not necessarily need to compromise the underlying model. It may be enough to influence the data supplied to it, manipulate the context it receives, exploit weak permissions around a connected tool or convince an application to expose information the user should not be able to access.

For this reason, AI security should be understood as application security around an intelligent and partially unpredictable component.

The model can reason about instructions, combine information from different sources and generate dynamic outputs. But the security architecture surrounding that model must still determine what information it is allowed to receive, what tools it can call and what actions a particular user is authorized to perform.

NIST’s AI Risk Management Framework approaches AI risk through four broad functions — Govern, Map, Measure and Manage — rather than treating security as a single technical control added after deployment. That lifecycle perspective is particularly useful for AI applications because risks can originate in model behavior, application architecture, data sources or operational decisions.

Why AI Applications Create a Different Attack Surface

Traditional web applications generally process deterministic instructions defined by developers. A request reaches an endpoint, application logic evaluates it and the system produces a predictable result.

LLM applications behave differently.

They accept natural-language instructions, interpret context and produce outputs probabilistically. They may also process external documents, websites, emails or other untrusted content. When connected to tools, an AI system can move from simply generating text to affecting the underlying application.

This creates a new layer between user input and application behavior.

A traditional security boundary might ask:

“Is this authenticated user allowed to call this API?”

An AI-powered system must ask additional questions:

“Can untrusted text influence the model into attempting that API call?”

“Can retrieved content alter the model’s instructions?”

“Does the AI have more permissions than the user who triggered it?”

“Could output generated by the model become an instruction to another system?”

Those are architecture questions, not merely prompt-engineering questions.

The Model Is Only One Part of the System

One of the most important principles in AI application security is that the model should not be treated as the entire product.

Consider an AI assistant connected to an internal CRM. The LLM might interpret a request, but the actual security boundary sits across several components: user authentication, retrieval permissions, CRM API authorization, tool execution and the application’s validation layer.

Even a well-behaved model becomes dangerous if it receives access to information the user should never have received in the first place.

Similarly, a model that occasionally produces an unsafe instruction may have limited security impact if the surrounding application independently validates every privileged action.

The architecture determines whether model behavior becomes a security incident.

Data, Tools and Permissions Matter More Than Expected

The most serious AI vulnerabilities often appear where the model interacts with something outside itself.

Data determines what the AI can know. Tools determine what it can do. Permissions determine how far a manipulated workflow can go.

This is why an AI assistant that only summarizes public documentation has a very different risk profile from an agent that can read customer records, modify database entries, send emails and call financial APIs.

As capability increases, authority must become more constrained.

This principle becomes particularly important when organizations deploy agents capable of performing multiple actions autonomously. OWASP describes excessive agency as a situation where damaging actions can result from unexpected, ambiguous or manipulated LLM outputs, often because functionality, permissions or autonomy are broader than necessary.

The Main Security Risks in AI-Powered Applications

There is no single category called an “AI vulnerability.” The real attack surface contains multiple classes of weakness.

Some are familiar from traditional cybersecurity: broken authorization, exposed APIs, insecure secrets and excessive privileges. Others become more important because an LLM can interpret adversarial content and convert it into application behavior.

OWASP’s current GenAI security guidance highlights prompt injection and sensitive information disclosure among the major risks affecting LLM applications. NIST’s adversarial machine learning taxonomy expands the picture further by examining attacker goals, capabilities and attacks across different stages of the machine-learning lifecycle.

Prompt Injection and Instruction Manipulation

Prompt injection occurs when crafted input influences an LLM to behave in an unintended way. OWASP describes it as manipulation of model responses through inputs that alter behavior, including cases capable of bypassing intended safeguards.

The simplest version is direct prompt injection. A user explicitly tries to override application instructions.

The more complex version is indirect prompt injection.

In this scenario, the attacker may not interact directly with the AI at all. Malicious instructions can exist inside content that the system later retrieves or processes. An agent reading webpages, emails, files or knowledge-base documents may encounter adversarial content and interpret it as instructions.

This becomes particularly dangerous when the model has access to privileged tools.

Prompt injection should therefore be treated as a trust-boundary problem, not merely as a text-filtering problem.

Filtering obvious malicious phrases may reduce trivial attacks, but it cannot replace authorization, least privilege and action validation.

Sensitive Data Exposure

AI applications frequently process more information than is visible in their user interfaces.

The model may receive system instructions, retrieved documents, internal records, user history, API responses or application metadata as part of its context. If that information is not correctly isolated, the AI can become another route through which confidential information is exposed.

OWASP identifies sensitive information disclosure as a distinct LLM application risk because AI systems can reveal confidential, proprietary, personal or otherwise protected information through their outputs.

A secure architecture should not depend on the model remembering that a particular record is confidential.

Access control should happen before sensitive information enters the context whenever possible.

If User A is not authorized to access User B’s records, those records should not be retrieved and then given to the model with a prompt asking it not to reveal them.

That reverses the security boundary.

Unsafe Tool and Agent Actions

A text-only chatbot can produce harmful output, but an autonomous agent can convert harmful output into action.

Once an AI system can interact with tools, the consequences of manipulation can change dramatically.

A tool might allow the AI to query internal databases, modify an account, send a message, create a file, interact with infrastructure or trigger another API. The security question then becomes not only whether the AI can be manipulated, but what authority the manipulated AI possesses.

The safest architecture assumes that the model can misunderstand or be influenced by adversarial input.

Sensitive operations should therefore be protected by deterministic controls outside the model: authorization checks, allowlists, constrained parameters, confirmation steps and limited credentials.

A model’s decision should not function as the final authorization layer for a high-impact action.

How AI Security Differs From Traditional Application Security

AI security does not replace traditional application security. It expands it.

An AI-powered SaaS platform can still contain ordinary vulnerabilities such as broken access control, insecure authentication or API authorization failures. In fact, connecting an LLM to these components may make existing weaknesses more consequential because the model creates another interface through which application capabilities can be reached.

Traditional penetration testing usually asks whether an attacker can manipulate HTTP requests, session state, authentication flows, APIs and application logic.

AI security testing adds another dimension: whether an attacker can manipulate meaning.

The system may technically receive a valid request, but the content inside that request can influence downstream model reasoning. Retrieved documents may be technically legitimate data while containing instructions designed to alter agent behavior.

This makes AI testing more contextual.

A scanner can identify an exposed server header or known software vulnerability. It is far less capable of determining whether a carefully structured instruction inside a customer-support ticket can influence an AI agent into accessing a tool beyond the intended workflow.

That is why effective AI and LLM security testing requires both security engineering and adversarial reasoning.

What an AI Security Assessment Should Test

A serious AI security assessment begins by mapping the system rather than immediately attacking prompts.

Researchers need to understand where user input enters, which information is added to model context, where retrieval occurs, which APIs are connected and what actions the system can perform.

The result is an AI trust map.

One boundary may exist between an unauthenticated user and the application. Another between the application and the model. Another between the model and a retrieval database. Another between an AI agent and a privileged tool.

Testing then focuses on whether those boundaries can be crossed.

An assessment should evaluate direct and indirect prompt manipulation, sensitive-data access, authorization behavior, tool permissions, retrieval isolation, context handling, unsafe output consumption and business logic surrounding AI-generated actions.

For systems based on machine learning beyond LLM applications, adversarial testing may also need to consider attacks such as evasion, poisoning and privacy-oriented attacks. NIST’s 2025 adversarial machine learning taxonomy explicitly organizes threats according to lifecycle stage, attacker objectives, capabilities and knowledge, reinforcing that AI security extends beyond conversational prompt attacks.

The key objective is not to generate the largest possible collection of unusual model outputs.

It is to determine whether manipulation can produce meaningful security impact.

AI Security Testing Should Follow Attack Paths, Not Prompt Counts

An AI assessment becomes low-value when success is measured by how many times researchers can make the model say something unexpected.

A strange response is not automatically a vulnerability.

The important question is what comes next.

Can the manipulated output reveal information the user cannot normally access?

Can it influence another trusted component?

Can it activate a tool?

Can it bypass a permission boundary?

Can multiple small weaknesses be combined into a larger attack chain?

This is the same principle ShabuShabu applies in its broader security testing methodology: detection should be followed by validation and practical impact analysis.

An apparently dramatic model response may have almost no security impact if it cannot affect sensitive data or application behavior.

A subtle authorization weakness, by contrast, may be critical even when the model output itself looks harmless.

Building AI Security Into the Product Lifecycle

AI security becomes significantly easier when trust boundaries are designed before the product reaches production.

Retrofitting authorization around an agent that already has broad access to dozens of internal services can be much harder than designing limited tool permissions from the beginning.

NIST’s Generative AI Profile is intended as a companion to the AI Risk Management Framework for organizations designing, developing, using and evaluating generative AI systems, reinforcing the idea that AI risk management should occur throughout the lifecycle rather than only after deployment.

The architecture should begin by defining what the AI actually needs.

If the model only needs read access, it should not receive write credentials.

If it only needs records belonging to the current user, retrieval should enforce that boundary before the information reaches the model.

If an action has meaningful financial, privacy or operational consequences, deterministic application logic should decide whether it is allowed.

If the AI processes untrusted external content, that content should be considered potentially adversarial.

These controls follow a simple principle: the model should operate inside the security architecture, not become the security architecture.

Secure AI Requires Least Privilege

Least privilege is particularly important for AI agents because language models are designed to interpret instructions flexibly.

Giving an agent broad permissions because “it may need them later” can convert a minor prompt-manipulation issue into a significant compromise.

Instead, capabilities should be constrained around the task.

An agent that schedules meetings should not automatically receive unrestricted mailbox access. An AI support assistant should not have administrative account privileges simply because the underlying API makes them convenient. A code assistant should not inherit deployment credentials when read-only repository access is sufficient.

Security architecture should also distinguish the identity of the user from the identity of the AI service.

The agent acts on behalf of someone. That relationship should remain visible to downstream authorization systems.

The safest question for every AI tool is therefore not “Can the model use this?”

It is:

“Under which user, role, context and action is the model permitted to use this?”

Monitoring AI Systems Requires More Than Application Logs

Traditional application logs may show that an API call occurred without explaining why an AI agent decided to make it.

AI systems benefit from richer security telemetry.

Teams may need visibility into the initiating user, relevant model interaction, tools requested, authorization decisions, data sources accessed and resulting high-impact actions.

The goal is not necessarily to retain every piece of model context indefinitely. Excessive logging can itself create privacy and security risks.

The objective is to preserve enough evidence to answer an important incident-response question:

How did this user-controlled input become this system action?

That connection between input, reasoning context and downstream action becomes particularly valuable when investigating indirect prompt injection or agent abuse.

When to Perform AI Security Testing

Waiting until an AI product has been operating in production for months is a poor time to discover that its fundamental permission model is unsafe.

A Security Crash Test or dedicated AI security assessment is especially valuable before launching a new AI-enabled product, introducing retrieval over confidential information, connecting an LLM to internal tools, expanding an agent’s permissions or increasing the level of autonomous action.

Testing is also important after significant architectural changes.

Adding one new tool can materially change the impact of an existing prompt-injection weakness. Connecting an internal knowledge source can introduce an entirely new data boundary. Changing user roles may create authorization paths that did not exist during an earlier assessment.

AI security is therefore not a one-time certification.

It is an ongoing property of the architecture.

AI Security Before Launch: Questions Engineering Teams Should Ask

Before deploying an AI-powered feature, teams should be able to explain exactly which security controls remain effective even when the model behaves unexpectedly.

Who can provide instructions to the AI?

Which content can enter its context?

Can external documents influence decisions?

Which confidential information can the application retrieve?

Are retrieval permissions enforced outside the model?

Which tools can the AI invoke?

Do those tools independently verify authorization?

Can one user cause the AI to access another user’s information?

What happens when the model requests an action outside its intended role?

Can security teams reconstruct sensitive agent actions after an incident?

If these questions do not have clear technical answers, the product probably still contains undefined trust boundaries.

Why Manual AI Security Testing Still Matters

AI security tooling will continue to improve, and automated systems are increasingly useful for generating adversarial inputs, exploring application behavior and identifying suspicious patterns.

But security impact remains contextual.

An automated system may discover that a model follows a particular instruction. A human researcher needs to understand whether that behavior crosses a meaningful security boundary.

The same applies to vulnerability discovery more broadly. NIST’s adversarial AI work provides systematic terminology for attack classes, but applying those concepts to a real product still requires understanding architecture, threat actors and operational impact.

Effective AI red teaming therefore combines automation with manual investigation.

Automation expands coverage.

Human reasoning determines what matters.

AI Security Is Ultimately Application Security

The most important lesson for companies adopting artificial intelligence is surprisingly simple.

AI does not remove traditional security principles.

It makes them more important.

Authentication still matters. Authorization still matters. Least privilege still matters. Input validation, data isolation, logging and incident response still matter.

What changes is the component sitting between those controls.

An LLM can interpret ambiguous instructions, combine information from multiple sources and choose between available actions. That flexibility creates enormous product value, but it also makes weak trust boundaries easier to reach in unexpected ways.

The safest AI architectures assume that model behavior can fail.

They do not assume that every output is correct, that every retrieved document is trustworthy or that every model-selected action should be allowed.

Instead, they build deterministic security controls around a probabilistic system.

That is the foundation of practical AI security in 2026.

Frequently Asked Questions About AI Security

What is AI security?

AI security is the practice of protecting AI-powered systems from manipulation, unauthorized access, information exposure and unsafe actions. It includes the model itself but also covers application logic, user identity, data sources, retrieval systems, APIs, connected tools and autonomous agents.

Is AI security the same as cybersecurity?

AI security is part of cybersecurity, but it introduces additional attack surfaces. Traditional cybersecurity protects applications, infrastructure, identities and data. AI security also evaluates how model behavior, prompts, training or retrieval data, AI agents and probabilistic outputs can interact with those existing security boundaries.

What is the biggest security risk in LLM applications?

There is no universal single risk because impact depends on architecture. Prompt injection is one of the most prominent LLM application risks identified by OWASP, but a prompt-injection weakness becomes substantially more dangerous when the model also has access to sensitive information, privileged tools or excessive permissions.

Can prompt engineering make an AI system secure?

Prompt engineering can influence model behavior, but it should not be treated as a substitute for application-level security controls. Authorization, data isolation, tool permissions and sensitive actions should be enforced outside the model so that a manipulated or unexpected response cannot automatically bypass the security boundary.

What is AI red teaming?

AI red teaming is adversarial testing designed to identify ways an AI system can be manipulated or produce unsafe outcomes. In production applications, effective red teaming should extend beyond the model and examine connected data, application logic, permissions, APIs, tools and real-world actions.

When should an organization conduct AI security testing?

Organizations should consider testing before production launch, before connecting AI to confidential data or privileged tools, when deploying autonomous agents, after significant architectural changes and when expanding an existing AI system’s capabilities or permissions.