Large language models are increasingly embedded inside SaaS platforms, internal copilots, customer-support systems, developer tools, search products, autonomous agents and business workflows. As soon as an LLM can retrieve confidential information, call an API or perform an action on behalf of a user, the security problem extends far beyond the model itself.

LLM security testing is the process of assessing an AI-powered application under controlled adversarial conditions to determine whether attackers can manipulate model behavior, cross authorization boundaries, expose sensitive information, abuse tools or turn an apparently harmless AI interaction into a practical attack path.

This is different from simply testing whether an LLM can produce an undesirable response.

OWASP’s current Top 10 for LLM and Generative AI applications includes risks such as Prompt Injection, Sensitive Information Disclosure, Excessive Agency, System Prompt Leakage, Vector and Embedding Weaknesses and Unbounded Consumption. NIST’s current adversarial machine-learning taxonomy takes an even broader view, organizing AI attacks around lifecycle stages as well as attacker goals, capabilities and knowledge.

For penetration testers, this creates a fundamental shift.

The objective is no longer only to find a vulnerable endpoint.

It is to understand how user-controlled information enters the AI system, how that information affects model reasoning and what the surrounding application allows the model to do next.

What Is LLM Security Testing?

LLM security testing is a specialized form of application security assessment focused on products that use large language models as part of their functionality.

A serious assessment evaluates the entire AI-enabled system rather than treating the underlying model as an isolated target.

That typically means examining the relationship between:

the user,

the application,

the LLM,

retrieval systems,

internal data,

APIs,

connected tools,

agent permissions,

and downstream actions.

The model is only one component.

A perfectly ordinary model response can still participate in a vulnerability if the surrounding application performs an unauthorized operation based on that response.

Likewise, an unusual or manipulated model response may have almost no practical security impact when the application correctly enforces permissions around sensitive actions.

The purpose of an LLM penetration test is therefore to determine where AI behavior intersects with real security boundaries.

Why Traditional Vulnerability Scanning Is Not Enough

Traditional security scanners remain useful in AI-powered applications.

An AI product can still contain exposed services, insecure HTTP configuration, outdated dependencies, authentication weaknesses and ordinary API vulnerabilities.

But scanners have difficulty reasoning about product-specific AI workflows.

Consider a support assistant that can search customer records.

A scanner may confirm that the endpoint itself is functioning correctly.

It is much harder for an automated scanner to understand whether a low-privilege user can phrase a request in a way that causes the assistant to retrieve another customer’s information.

The technical request may be completely valid.

The vulnerability exists in the relationship between language, identity, data and authorization.

This is why manual reasoning becomes critical.

NIST’s AI work treats security and resilience as core elements of trustworthy AI, while its adversarial machine-learning taxonomy explicitly considers attacker objectives and capabilities rather than reducing AI security to a single technical test.

LLM security testing should follow the same principle.

Do not ask only:

“Does the model behave strangely?”

Ask:

“Can an attacker turn that behavior into unauthorized impact?”

Mapping the LLM Application Attack Surface

Before attempting adversarial testing, researchers need to understand how the application works.

This attack-surface mapping phase is one of the most important parts of an AI and LLM security assessment.

The tester needs to know where instructions originate, where sensitive data enters the system, which components can be influenced by users and which capabilities exist downstream from the model.

A useful architecture map usually includes several layers.

User Inputs

The most obvious attack surface is information supplied directly by users.

This may include ordinary chat messages, uploaded files, support requests, search queries, profile information, forms or other content that eventually enters model context.

However, not all input is equally trusted.

An authenticated administrator and an anonymous visitor may interact with the same LLM while having very different legitimate permissions.

Security testing should therefore consider who controls the input and which identity is associated with the resulting AI workflow.

Direct user input is also the primary surface for prompt-injection testing.

OWASP currently defines Prompt Injection as manipulation in which input changes an LLM’s behavior or output in unintended ways.

The penetration tester needs to determine whether that manipulation can cross a meaningful application boundary.

Model Context

The visible user message is usually only a fraction of what the model receives.

An application may also supply:

developer instructions,

system prompts,

conversation history,

user metadata,

retrieved documents,

tool responses,

structured application state,

or internal instructions.

This assembled context represents another attack surface.

Security researchers should understand which information is included automatically and whether users can influence parts of that context indirectly.

For example, a user may control a profile field that later appears inside the model context. A customer may submit a support ticket that an internal AI assistant later reads. A malicious document may enter a knowledge base before being retrieved by a completely different user.

These are all potential routes through which attacker-controlled information can reach the model.

Retrieval Systems

RAG applications add another security layer.

The model may retrieve knowledge from document stores, vector databases, internal search systems or external content before producing a response.

OWASP’s current LLM risk set includes Vector and Embedding Weaknesses, reflecting security problems that can emerge around retrieval and embedding-based architectures.

From a penetration-testing perspective, researchers should ask:

Can one user’s query retrieve another user’s documents?

Are tenant permissions applied before retrieval?

Can untrusted documents influence model behavior?

Can attackers introduce content into the knowledge source?

Does retrieved content retain provenance?

Can a relevant-but-unauthorized document enter model context?

RAG security is not only about prompt injection.

It is also about data authorization.

Tools and External APIs

Tool-enabled LLMs create some of the highest-impact attack surfaces.

A model may be connected to APIs capable of reading information, creating objects, sending messages, changing records or performing operational actions.

The penetration tester needs to determine what happens between the model requesting a tool and the application executing it.

Does the tool independently authenticate the user?

Does it verify authorization?

Does it receive a broad service credential?

Can the model choose unrestricted parameters?

Can a manipulated model invoke an operation the original user should not control?

These questions often reveal more meaningful vulnerabilities than testing model responses alone.

Build a Trust-Boundary Map Before Testing

A useful AI penetration test converts the architecture into trust boundaries.

For example:

User → Application

Can an unauthenticated or low-privilege user influence a protected AI workflow?

Application → Model

Which internal information does the application expose to the model?

Model → RAG

Which documents can be retrieved, and how are permissions applied?

Model → Tool

Which capabilities can the model request?

Tool → Backend

Does the backend verify the original user’s authority independently?

Agent → External System

Can the agent initiate actions that affect systems outside the immediate application?

Each boundary creates a security hypothesis.

Testing becomes much more focused when researchers understand exactly what assumption they are attempting to break.

Testing Prompt and Instruction Boundaries

Prompt injection is one of the first areas most organizations associate with LLM security.

It deserves attention, but high-quality testing goes beyond demonstrating that a model can be influenced.

OWASP currently places Prompt Injection at the top of its LLM application risk list.

The assessment should examine both direct and indirect instruction manipulation.

Direct testing focuses on content supplied through ordinary user-controlled interfaces.

Indirect testing focuses on content the model consumes from elsewhere, such as RAG documents, files or connected systems.

The real objective is to answer a more important question:

What security consequence follows successful instruction manipulation?

Does the system expose protected context?

Can tool usage be influenced?

Can an agent be redirected toward an action outside the intended workflow?

Can one user’s request affect another user’s information?

If not, the security impact may remain limited.

If yes, prompt injection becomes part of a larger attack chain.

Testing Sensitive Information Disclosure

LLM applications routinely process information that should remain confidential.

OWASP identifies Sensitive Information Disclosure as a major LLM application risk, covering confidential information that may affect either the LLM itself or its surrounding application context.

During testing, researchers should map every sensitive data source available to the AI system.

This may include:

customer records,

internal documents,

conversation memory,

system prompts,

API responses,

source code,

employee information,

and application configuration.

The test then determines whether normal or adversarial interactions can expose information beyond the current user’s legitimate access.

Cross-user and cross-tenant testing is especially important.

A model should not become a route around the authorization controls already expected from the SaaS platform.

Testing Authorization Through the AI Layer

One of the most serious architectural weaknesses occurs when user identity disappears inside the AI workflow.

The frontend knows who the user is.

The orchestration service calls an API.

The API sees only a powerful backend service account.

The model now effectively sits between a low-privilege user and a high-privilege service.

This can create an authorization mismatch.

Testing should verify that the permissions of downstream actions remain connected to the original identity.

For example, suppose two test accounts exist:

one ordinary user,

one administrator.

Researchers should determine whether the ordinary user can use the AI interface to trigger functionality that should only exist for the administrator.

This may require no technical bypass of the underlying API.

The vulnerable logic may exist entirely inside the AI orchestration layer.

Testing AI Tools and Function Calling

Tool-enabled AI requires a separate security review.

The model should generally be treated as a component that requests an action rather than a component that authorizes it.

Testing should examine what parameters the model can supply, which identities the tool uses and what validation occurs before the action executes.

A secure tool boundary should assume that model output can be incorrect or adversarially influenced.

The backend should not trust a tool request merely because the LLM generated it.

This becomes especially important when tools can:

modify user data,

access internal systems,

send communications,

change permissions,

manage files,

or trigger financially meaningful actions.

The more capability available to the model, the greater the possible impact of successful manipulation.

OWASP describes Excessive Agency as a vulnerability in which unexpected, ambiguous or manipulated LLM outputs can lead to damaging actions because functionality, permissions or autonomy are broader than necessary.

Testing AI Agents

AI agents require testing at both the model and application levels.

Unlike a basic chatbot, an agent may perform a sequence of actions:

interpret a request,

search for information,

choose a tool,

process the result,

select another tool,

and continue until a task is completed.

This introduces attack chains.

An individual action may appear safe when inspected in isolation.

The sequence can still create dangerous behavior.

Testing should therefore evaluate:

what objectives the agent accepts,

which information can alter those objectives,

which tools are available,

what permissions each tool carries,

how actions are validated,

and when human approval is required.

An agent with limited read-only capabilities presents a fundamentally different security risk from one capable of altering accounts or interacting with production infrastructure.

Security testing needs to reflect that difference.

Testing RAG Applications

RAG security deserves more than a single prompt-injection test.

Researchers should inspect the complete retrieval pipeline.

Retrieval Authorization

Can the current user retrieve only documents they are permitted to access?

This should be tested explicitly using different roles and tenants.

The embedding system’s relevance ranking should never silently replace authorization.

Document Manipulation

Can an attacker control or modify information likely to be retrieved?

If so, the tester should examine whether untrusted content can influence model reasoning or downstream actions.

Cross-Tenant Isolation

In multi-tenant systems, AI retrieval must preserve the same tenant boundaries expected from the rest of the application.

This is a high-value test because AI interfaces may expose information semantically rather than through obvious object identifiers.

Retrieval Context Leakage

Researchers should also inspect whether responses expose excessive fragments of retrieved documents or other context that was not required to answer the user’s request.

The goal is to verify that RAG improves model knowledge without weakening the application’s data-access model.

Testing System Prompt Exposure

System prompts frequently attract attention during AI security testing.

OWASP’s current LLM risk set contains System Prompt Leakage, but its guidance also emphasizes that system prompts should not themselves be treated as secure secret stores.

This changes how testers should assess impact.

If the system prompt contains ordinary behavioral guidance and becomes visible, the impact may be relatively limited.

If the prompt contains credentials, internal secrets or security logic that the application depends on remaining hidden, the root problem is more significant.

The penetration tester should therefore ask:

What did disclosure reveal?

What can the attacker do with that information?

Does the system depend on secrecy rather than real authorization?

Severity should follow the attack path rather than the label.

Testing Improper Output Handling

AI applications may consume model output programmatically.

This is an important distinction.

If the output is shown only as text, the attack surface is relatively limited.

If model-generated output becomes input to another interpreter, API, browser or application component, additional security issues can emerge.

OWASP’s current LLM Top 10 includes Improper Output Handling as a distinct risk category.

Testing should determine whether downstream components treat model output as trusted data.

Any model-generated content entering another security-sensitive system should pass through appropriate validation.

The LLM is not a trusted sanitizer.

Testing Resource and Consumption Abuse

LLM systems introduce operational security risks as well.

Model calls may be expensive.

Agent loops may execute repeatedly.

Large context windows consume resources.

Automated requests can increase infrastructure cost dramatically.

OWASP includes Unbounded Consumption in its current LLM application risk list.

A penetration test does not need to create destructive resource consumption to identify these weaknesses.

Instead, researchers can assess whether the architecture contains reasonable limits on requests, agent iterations, tool use and other expensive actions.

The goal is to identify whether an attacker could create disproportionate cost or workload without performing an unnecessary production-impacting stress test.

Testing Adversarial Machine Learning Risks

Not every AI system is purely an LLM application.

Products may include classifiers, recommendation systems, anomaly detection models or other machine-learning components.

For these systems, testing may need to consider a broader adversarial ML threat model.

NIST AI 100-2e2025 provides a taxonomy covering machine-learning methods, lifecycle stages, attacker goals, capabilities and knowledge, providing a structured vocabulary for evaluating adversarial ML threats.

Depending on the system, security teams may need to assess risks related to inference-time manipulation, training or data poisoning, privacy attacks and other adversarial behaviors.

These assessments require different techniques from ordinary LLM prompt testing.

This is another reason “AI penetration testing” should not be treated as one generic checklist.

The assessment should reflect the actual architecture.

Manual Testing vs Automated LLM Security Scanning

Automation plays an important role in AI security.

Automated tools can generate large numbers of adversarial inputs, detect suspicious responses and explore variations much faster than a human tester.

That makes automation valuable for breadth.

The limitation is context.

An automated system may identify an unusual response but fail to understand whether the result matters.

A human security researcher can connect the behavior to:

user identity,

application permissions,

business workflows,

data sensitivity,

tool capabilities,

and downstream impact.

This leads to a useful division.

Automation finds candidate behaviors.

Manual penetration testing determines which behaviors represent meaningful vulnerabilities.

The strongest AI assessments use both.

LLM Security Testing Should Be Risk-Based

Not every feature deserves the same testing depth.

An AI feature that summarizes public articles does not carry the same risk as an internal agent that can modify production records.

The testing methodology should reflect capability.

A practical model considers at least three dimensions.

Data Sensitivity

What can the AI access?

Public documentation?

Internal business information?

Customer records?

Credentials?

Financial data?

Tool Authority

What can the AI do?

Generate text?

Read documents?

Modify application state?

Call administrative APIs?

Autonomy

How independently can the AI act?

Does every operation require user confirmation?

Can it perform several steps without approval?

Can it decide which tools to use autonomously?

Higher data sensitivity, broader authority and greater autonomy typically justify deeper adversarial testing.

Validating Security Impact Without Creating Unnecessary Risk

Vulnerability validation is essential.

A report saying “the model behaved strangely” gives engineers little information about the actual risk.

The tester needs enough evidence to demonstrate:

what boundary failed,

which user or attacker controls the input,

what permission should have existed,

what information or action became reachable,

and how reproducible the issue is.

But validation should remain proportionate.

Security researchers do not need to download large datasets simply to prove that cross-user access exists.

They do not need to execute destructive actions when a safe test object can demonstrate broken authorization.

They do not need to create operational disruption to prove that an agent has excessive privileges.

ShabuShabu’s testing methodology follows this distinction between identifying a security signal, validating the underlying control and establishing practical impact.

Attack Chaining Matters More Than Individual AI Findings

Some of the most important AI vulnerabilities emerge from combinations.

Consider a hypothetical sequence:

A low-privilege user can influence the model.

The model can access a broad internal search tool.

The search tool uses an overprivileged service identity.

Retrieved data contains confidential records.

Individually, each design choice may appear manageable.

Together, they create an unauthorized data-access path.

This is why AI penetration testing should not end after identifying one prompt weakness.

Researchers need to determine whether that weakness enables movement across additional trust boundaries.

The resulting attack chain often provides more useful remediation information than the initial model behavior.

What an LLM Penetration Testing Report Should Include

A professional report should be usable by both security teams and engineers.

The report should not consist of hundreds of AI conversations with no explanation of impact.

For each confirmed finding, the useful questions are straightforward.

What failed?

Where did it fail?

Who can exploit it?

What does exploitation enable?

Which architecture assumption caused the problem?

What should engineering change?

A strong report may therefore include technical reproduction context, affected functionality, attack-path explanation, security impact and remediation guidance.

For complex AI systems, it is also useful to visualize trust boundaries.

For example:

User → LLM → Retrieval → Sensitive Data

or:

User → LLM → Tool → Privileged API → Business Action

This immediately communicates why a seemingly small prompt-related weakness matters.

How to Prioritize LLM Security Findings

AI assessments can produce substantial volumes of interesting behavior.

Not all of it deserves the same remediation priority.

Severity should consider realistic impact.

A useful prioritization process asks:

Can the attacker reproduce the behavior reliably?

What privileges are required?

What sensitive data or functionality becomes accessible?

Does exploitation affect only the attacker’s session or other users?

Can the issue be combined with another weakness?

Does the model have authority over real-world actions?

Does a deterministic backend control prevent meaningful impact?

This prevents security teams from treating every model policy failure as a critical vulnerability.

It also prevents subtle authorization failures from being underestimated simply because the generated text looks harmless.

Remediation Should Target the Root Cause

AI findings are often fixed at the wrong layer.

An engineering team discovers a prompt that causes unwanted behavior.

The exact phrase is filtered.

The test is repeated.

The original prompt no longer works.

The vulnerability appears closed.

But if the actual problem is excessive tool permissions, weak retrieval authorization or missing backend access control, blocking one prompt does not remove the attack path.

Effective remediation should determine which architectural control failed.

If unauthorized information reached the model, fix access control.

If a tool accepted unauthorized operations, fix tool authorization.

If untrusted documents could change agent behavior, strengthen content isolation and capability boundaries.

If the agent possessed unnecessary privileges, reduce them.

Model-side safeguards remain useful.

They are strongest when operating alongside deterministic controls rather than replacing them.

Retesting AI Security Findings

AI security retesting presents an interesting challenge because model behavior may be probabilistic.

A finding should not necessarily be considered resolved because one particular conversation now produces a different answer.

Retesting should focus on the underlying security property.

If the original vulnerability involved cross-tenant retrieval, confirm that unauthorized documents can no longer enter context.

If the issue involved excessive tool permissions, verify that the backend now rejects unauthorized actions.

If prompt injection previously led to confidential-data exposure, test whether the confidential data is still reachable through alternative variations.

The objective is to verify that the attack path has disappeared.

This is more reliable than verifying that one exact prompt stopped working.

When Companies Should Perform LLM Security Testing

AI security testing becomes particularly valuable whenever an AI system gains meaningful access to information or actions.

The ideal time is before production launch.

Testing is also important when organizations:

connect an LLM to confidential RAG sources,

add AI agents,

introduce new tools,

increase agent autonomy,

connect additional APIs,

expand AI access across multiple tenants,

or materially change authorization architecture.

NIST’s Generative AI Profile is specifically intended to help organizations manage generative AI risks across design, development, use and evaluation rather than treating risk only as a post-deployment concern.

The security principle is simple.

Every time AI capability expands, the threat model should be revisited.

Pre-Launch LLM Security Testing

Pre-launch testing is especially useful because security architecture can still be changed without disrupting a mature product.

Researchers can evaluate the complete workflow before users and sensitive production data depend on it.

A pre-launch Security Crash Test may examine:

how identities reach the AI layer,

which information enters model context,

how RAG permissions work,

which tools are available,

what happens after model output,

and which actions require independent authorization.

This approach often identifies architectural weaknesses that conventional vulnerability scanning would never see.

LLM Security Testing Is Application Security With a New Reasoning Layer

There is a temptation to treat generative AI security as a completely separate discipline.

In practice, many of the most serious problems rely on security principles that already exist.

Authentication.

Authorization.

Least privilege.

Isolation.

Input handling.

Output validation.

Logging.

Data minimization.

Secure APIs.

The difference is that an LLM now sits between some of those controls.

It can interpret ambiguous input, consume untrusted content and choose between possible actions.

That reasoning layer changes the way attackers interact with the system.

It does not eliminate the need for conventional security boundaries.

The safest architecture assumes that the model can be manipulated, misunderstand context or produce an unexpected output.

The application should remain secure anyway.

That is the real objective of LLM security testing.

Frequently Asked Questions About LLM Security Testing

What is LLM security testing?

LLM security testing is the adversarial assessment of an AI-powered application to determine whether attackers can manipulate model behavior, expose sensitive information, cross authorization boundaries or abuse connected tools and agents. A complete assessment evaluates both the model and the surrounding application architecture.

Is LLM penetration testing the same as prompt injection testing?

No. Prompt injection is one important area, but an LLM penetration test should also examine sensitive data exposure, RAG authorization, tool permissions, agent autonomy, system integrations, backend access controls and downstream handling of model output.

What are the main LLM security risks?

OWASP’s current LLM application risk set includes categories such as Prompt Injection, Sensitive Information Disclosure, Excessive Agency, System Prompt Leakage, Vector and Embedding Weaknesses and Unbounded Consumption, among others.

Can automated tools perform an LLM penetration test?

Automation can provide useful coverage and identify candidate behaviors, but application-specific impact often requires manual security reasoning. Authorization, business logic, data sensitivity and attack chaining depend heavily on product context.

How do you test a RAG application for security?

Testing should examine retrieval authorization, tenant isolation, document provenance, untrusted content, indirect prompt manipulation and whether unauthorized documents can enter model context. Vector similarity should not replace access-control decisions.

How do you test an AI agent?

AI agent testing should evaluate instruction boundaries, available tools, agent permissions, autonomous actions, backend authorization and whether untrusted content can alter the agent’s objectives. The assessment should also verify whether high-impact actions require independent validation.

Should system prompts be treated as secrets?

Not as a primary security control. OWASP’s current guidance on System Prompt Leakage emphasizes that prompts should not be relied on as secure secret storage and that sensitive security controls should exist outside the LLM.

When should LLM security testing be performed?

Testing is most valuable before production launch and after material changes to AI capability, data access, tool access, RAG architecture or agent autonomy. Systems that can access confidential data or perform privileged actions should receive particularly careful assessment.