AI applications become significantly more powerful when a model can do more than generate text. Tool use and function calling allow an LLM or AI agent to query databases, call APIs, retrieve files, send messages, create objects and trigger application logic. In MCP, for example, tools are explicitly defined as functions that models can actively call, including operations that can write to databases, invoke external APIs, modify files or trigger other logic.

That capability is what turns an AI assistant into an operational system.

It is also what turns a model mistake into a potential security incident.

A manipulated language model may generate an incorrect answer without significantly affecting the surrounding application. A manipulated model connected to a privileged account-management function, production API or external messaging tool can produce a very different outcome.

This is why AI tool security and function-calling security must be treated as application-security problems rather than prompt-engineering problems.

The model can propose an action. The surrounding system must decide whether that action is actually allowed.

OWASP identifies this distinction through its Excessive Agency category. It notes that LLM-enabled systems often receive the ability to call functions or interact with other systems, and that damaging actions can result when those systems provide excessive functionality, permissions or autonomy.

The most effective security architecture therefore assumes that model-selected tool calls can occasionally be wrong, manipulated or contextually inappropriate — and remains secure anyway.

Why Tool-Enabled LLMs Require Stronger Security Controls

A text-generating model has a relatively narrow direct effect on its environment. It receives information and returns information.

Tool-enabled systems introduce action.

An AI agent might receive a request to help manage an account. It could first retrieve account information, then inspect a billing system, update a database record and send an external confirmation.

Each action may occur through a different tool.

The user sees one conversational interface.

Behind it, the application may have executed several privileged operations.

This means the real security boundary is no longer the chat box. It stretches from the original user identity through model reasoning, tool selection, parameter generation, backend authorization and finally the system that carries out the operation.

Anthropic’s work on trustworthy agents makes a similar distinction between the model, the tools it can use and the environment those tools can reach. It notes that even a capable model can still become unsafe when the surrounding harness is poorly configured, tools are overly permissive or the operating environment exposes too much access.

For penetration testers, this changes the question from:

“Can we manipulate the model?”

to:

“What can a manipulated model cause the application to do?”

How Function Calling Changes the Threat Model

Function calling gives a model a structured path into application functionality.

Instead of returning only natural language, the model can select a predefined operation and provide arguments that the application then uses to execute that operation.

This structure is useful because tool inputs can be schema-defined and validated. MCP, for example, defines tools through structured interfaces with typed inputs and outputs and uses JSON Schema for tool definitions.

But structured parameters do not make the model trusted.

The model still decides which function it wants to call and may generate parameters based on untrusted user input, retrieved content or prior tool results.

A syntactically valid request can therefore remain completely unauthorized.

An AI agent may produce a perfectly valid customer identifier.

That does not mean the current user is allowed to access that customer.

It may generate a valid destination email address.

That does not establish permission to send confidential information there.

It may request a valid account deletion operation.

That does not establish that deletion should happen.

This is one of the most important principles in AI application security:

schema validity is not authorization.

Common AI Tool Abuse Scenarios

Tool abuse rarely requires a completely broken integration. More often, the system works as designed but gives model reasoning too much influence over security-sensitive operations.

OWASP describes excessive agency as a problem caused by excessive tool functionality, excessive downstream permissions or excessive autonomy. Its examples include read-only use cases implemented with tools that can also modify or delete data, integrations using unnecessarily privileged identities, and high-impact actions executed without independent confirmation.

The practical security scenarios therefore tend to appear at the boundary between what the model requests and what the backend actually permits.

Unauthorized Data Access

Consider an AI assistant connected to a customer database.

A legitimate workflow requires the assistant to retrieve information about the authenticated user’s account.

The tool, however, connects to the database using an identity capable of reading every customer record.

The model is effectively sitting between limited users and an unlimited data-access capability.

If prompt manipulation, a business-logic mistake or ordinary model error causes the agent to request another customer’s data, the database may return it because the tool itself is authorized.

OWASP explicitly warns about this pattern: an extension operating on behalf of an individual user can become excessively privileged when it accesses downstream systems through a generic high-privilege identity.

The correct fix is not merely to instruct the LLM to request only the current user’s records.

The backend must enforce the user’s authorization.

Privilege Escalation Through Tools

AI applications frequently separate users into different roles.

Administrators can perform actions ordinary users cannot.

That distinction should remain true when the same capabilities are accessed through an AI interface.

A dangerous architecture allows an ordinary user to communicate with an AI agent that then uses a privileged backend identity to execute operations.

The user cannot call the administrative API directly.

The agent can.

If the AI layer becomes the only thing preventing the user from reaching that function, authorization has effectively been delegated to model reasoning.

OWASP’s mitigation guidance specifically recommends executing tool operations in the relevant user’s security context and applying minimum privileges rather than relying on generic high-privilege access.

The security architecture should therefore ask:

Who is the human user?

What role does that user have?

Which tool is being requested?

Which permissions should apply to that request?

Those questions should be answered by deterministic software.

Dangerous Parameter Manipulation

Even when the correct function is selected, its arguments can become dangerous.

A model may request a file operation with a path it derived from user input.

It may construct an external destination from retrieved content.

It may generate an object identifier, URL, command argument or query condition.

Structured schemas can verify that a parameter is a string, number or date, but they do not determine whether the value is safe in the application’s business context. MCP documentation similarly separates schema-defined tool interfaces from the broader control mechanisms needed around tool execution.

The receiving application therefore needs normal security validation.

A file path may require confinement to an approved directory.

A customer ID may need ownership validation.

An external URL may need destination restrictions.

An account operation may require role checks.

The AI should not become a bypass around controls that would be mandatory in a conventional API.

Chained Tool Actions

The most interesting agent vulnerabilities frequently emerge when several individually legitimate tools are combined.

A read tool may be safe.

A messaging tool may also be legitimate.

Together, they can potentially create a data-exfiltration path.

A search capability may be low risk.

A modification capability may be appropriate for certain workflows.

An attacker who influences the agent into using the first tool to locate a target and the second tool to change it has created an attack chain.

This is why security teams should map capability combinations, not simply review tools one at a time.

The MCP project has increasingly emphasized this issue in its work on tool-risk annotations, noting that the security implications of tools often depend on combinations and context rather than on a single descriptive property.

Never Treat Model Output as Authorization

This is the central rule of secure AI function calling.

The language model should not be the final authority deciding whether a sensitive action may occur.

A prompt can influence the model.

Retrieved content can influence the model.

Tool responses can influence subsequent model reasoning.

Model output can also simply be wrong.

OWASP therefore recommends implementing authorization in downstream systems rather than allowing the LLM to decide whether an operation is permitted. Its guidance describes this as complete mediation: requests made through extensions should still be checked against security policies.

Imagine an AI system with a function called deleteAccount.

The model can determine that the user’s request appears to involve account deletion.

That is useful interpretation.

Before the backend performs the action, however, it should still determine which account is being deleted, whether the authenticated identity controls that account and whether any additional confirmation or policy check is required.

The tool call is a request.

It is not proof of authorization.

Function Calling Should Preserve the Original User Identity

Many AI architectures lose the user somewhere in the middle of the workflow.

The frontend authenticates the user.

The AI service receives the request.

The model selects a function.

The backend executes the function using a service credential.

If no explicit user authorization context survives this chain, every operation may eventually execute with the privileges of the AI service rather than the privileges of the person requesting it.

This is especially dangerous in multi-user SaaS products.

A model cannot safely infer which objects belong to which users if the backend itself exposes everything.

The downstream tool should receive enough identity context to enforce the intended permission model.

That may involve delegated user authorization, scoped credentials, explicit role information or another secure mechanism appropriate to the application.

The exact implementation can vary.

The security property should not:

User A’s AI session should not acquire User B’s privileges simply because both requests pass through the same model.

Validate Tool Inputs at the Backend

AI-generated parameters should be treated as untrusted input.

This remains true even when the model is operated by the application provider and even when parameters conform perfectly to the tool schema.

The backend needs to validate meaning, not only format.

Suppose an AI tool accepts an invoice_id.

JSON Schema can confirm that it is a valid string.

The application must still determine whether the current user is permitted to view or modify that invoice.

Suppose another tool accepts a recipient.

The application may need to determine whether external recipients are allowed for the current workflow.

Suppose an agent can request a database search.

The tool should expose a constrained operation rather than giving the model the ability to construct arbitrary backend queries.

OWASP recommends avoiding open-ended tools where more specific capabilities can accomplish the same task. It gives the example that a narrowly defined file-writing extension is safer than exposing a general shell capability when the application only needs to write a file.

Specific tools reduce ambiguity.

They also reduce blast radius.

Narrow Functions Are Safer Than Generic Execution Tools

Generic tools are attractive because they allow an agent to handle many tasks without developers creating separate integrations.

They also create enormous security surfaces.

A general shell tool can potentially execute many commands.

A generic HTTP requester can reach many destinations.

A broad database interface can perform many operations.

A specialized tool such as getCurrentUserInvoices exposes a much smaller capability.

This is not merely an AI-specific idea.

It is capability security.

The narrower the primitive, the easier it is to reason about what a compromised or confused caller can accomplish.

OWASP explicitly recommends minimizing extension functionality and avoiding open-ended extensions where more granular alternatives are possible.

For AI engineering, that means tool design should optimize for security clarity, not only model convenience.

Separate Read Tools From Write Tools

One of the most practical improvements is separating information retrieval from state-changing actions.

An AI assistant may need to inspect account information frequently but update it only occasionally.

Those operations should not necessarily share the same tool or credential.

Read access can often be pre-authorized more broadly.

Write operations can require stronger validation.

Destructive operations can require stronger controls again.

This creates a natural risk hierarchy.

Anthropic uses a similar concept in its current agent-control model, where users can decide that certain tool operations are always allowed while others require approval or are blocked entirely.

The exact interface is product-specific, but the security principle is general:

different levels of impact should receive different levels of trust.

Tool Descriptions Are Not Security Policies

AI agents often receive descriptive information explaining what tools do.

This metadata helps the model choose between available capabilities.

It should never become the only source of truth for security.

The MCP project explicitly describes tool annotations as hints, not enforceable contracts. Its documentation states that clients should treat annotations as untrusted unless they come from a trusted server, and current annotations may indicate whether a tool appears read-only, destructive, idempotent or connected to the open world.

This distinction is crucial.

A tool can claim to be read-only.

The backend implementation could still modify state.

A tool can claim to be safe.

Its output could still contain adversarial content.

A description can help with user experience and risk classification.

Only actual controls determine what the tool can do.

Tool Results Are Also Untrusted Input

Security attention often focuses on what enters a tool.

The output matters too.

An AI agent can call a legitimate external tool and receive content controlled by somebody else.

Search results may contain attacker-controlled pages.

An inbox tool may retrieve a malicious email.

A support system may return user-generated tickets.

A remote MCP server may return content from an external environment.

That information can then enter model context and influence the next decision.

Anthropic describes prompt injections as malicious instructions hidden in content that an agent processes and warns that more open environments and broader tool sets increase both entry points and potential consequences.

This creates a circular threat model:

User input influences the model.

The model calls a tool.

The tool returns untrusted content.

That content influences the model.

The model calls another tool.

Security teams therefore need to treat tool output as another input channel.

Prompt Injection Becomes More Dangerous When Tools Are Available

Prompt injection does not automatically mean system compromise.

Impact depends on what sits behind the model.

An injected instruction against a model with no tools may primarily influence generated text.

The same manipulation against an agent with access to internal data, communications and state-changing functions can create a much larger attack surface.

OWASP explicitly lists direct and indirect prompt injection among the triggers that can lead to Excessive Agency when agents possess excessive capability.

Anthropic reaches a similar conclusion in its agent-security research: the more tools and environment access an agent has, the more an attacker can potentially do if manipulation succeeds.

This is why prompt injection security and AI tool security should be assessed together.

Model defenses reduce the likelihood of successful manipulation.

Tool controls reduce the consequences when manipulation succeeds.

Both matter.

Design Tool Permissions Around Least Privilege

An agent should not receive every available capability simply because those capabilities might be useful eventually.

Tool access should match the current product purpose.

If an assistant only needs to read email, the associated integration should not automatically provide permission to send or delete messages.

If a product only needs to retrieve catalog information, the AI integration should not receive database modification rights.

If a developer agent only needs repository read access for a particular task, production deployment authority should remain separate.

OWASP recommends limiting both the number of available extensions and the privileges those extensions have on downstream systems.

Least privilege also improves incident containment.

A model can only misuse capabilities it actually possesses.

The easiest dangerous tool call to secure is the one the model can never make.

Reduce the Number of Tools Available Per Task

AI systems sometimes expose every possible integration to one general-purpose agent.

That approach maximizes flexibility but creates unnecessary complexity.

The agent must distinguish between many capabilities.

Prompt injection gains more potential targets.

Tool chaining becomes harder to reason about.

Security review must consider a much larger set of combinations.

A safer architecture can make tools available dynamically according to the current workflow.

A document-research task receives research capabilities.

An account-support task receives limited account-support tools.

Administrative capabilities appear only inside workflows where they are genuinely required and appropriately authorized.

MCP supports models discovering available tools, and its documentation notes that applications can also give users control over which tools are available during specific interactions.

Reducing the tool set simplifies both model behavior and attack-surface analysis.

High-Impact Functions Should Require Stronger Confirmation

Not every tool invocation needs human approval.

Requiring confirmation for every harmless lookup can create enough friction that users stop paying attention.

The important distinction is impact.

Reading a public document may be safe to pre-authorize.

Sending an external message may deserve confirmation.

Deleting important records deserves stronger control.

Changing access rights or affecting production infrastructure may justify an entirely separate approval mechanism.

MCP’s current server concepts explicitly support user oversight patterns such as tool-availability controls, execution approval dialogs, pre-approval for safe operations and activity logs. Anthropic likewise distinguishes between actions that can be always allowed, require approval or remain blocked.

The goal is not constant interruption.

The goal is meaningful intervention before meaningful consequences.

Human Approval Must Contain Enough Context

An approval button is weak if the user cannot understand what is being approved.

A confirmation interface should communicate the material effect of the requested operation.

If the agent intends to send information externally, the destination matters.

If it will modify an object, the affected object matters.

If it will perform a destructive operation, the destructive nature matters.

If several tool calls form a larger workflow, users may need visibility into the overall intention rather than being forced to approve a confusing sequence of isolated actions.

Anthropic notes that repeated per-action permission prompts can create friction and describes plan-level review as one way to move human oversight from individual steps toward the broader strategy of an agent workflow.

For security teams, the important principle is that confirmation should support an informed decision.

Secure Function Calling Needs Output Validation Too

The security boundary continues after the tool executes.

Tool output may be rendered in a browser, inserted into another prompt, passed to another function or used to generate subsequent application actions.

If the application treats model-generated or tool-generated information as automatically trustworthy, one compromised stage can propagate into another.

OWASP distinguishes Excessive Agency from Improper Output Handling, which concerns insufficient scrutiny, sanitization or validation of model-generated outputs before other systems consume them.

This is especially important for agentic systems.

An AI-generated value that becomes a backend instruction should receive normal validation.

An AI-generated page fragment rendered into an application should receive appropriate output handling.

An AI-generated query should not become unrestricted database execution.

The model should not function as a sanitizer.

Log Tool Execution, Not Only Chat Messages

Security monitoring for AI applications needs to extend beyond conversation logs.

If an agent performs a sensitive operation, incident responders need to understand how that operation occurred.

Which user initiated the workflow?

Which model or agent requested the function?

Which tool was selected?

What authorization context applied?

Which target object was affected?

Was human approval required?

What was the result?

MCP guidance includes activity logs among the mechanisms applications can use to maintain user oversight of tool execution. OWASP similarly recommends logging and monitoring LLM extension activity and downstream actions as part of limiting the consequences of excessive agency.

That does not mean storing unlimited sensitive context.

AI logs should themselves follow data-minimization and access-control principles.

The purpose is accountability, not indiscriminate retention.

Rate Limits Can Reduce the Blast Radius

Authorization prevents unauthorized operations.

Rate limiting serves a different purpose.

It can reduce how rapidly a compromised workflow produces damage.

An agent that can perform one problematic action is concerning.

An agent capable of repeating it thousands of times automatically can create much larger impact.

OWASP specifically recommends rate limiting as one mechanism for reducing the amount of damage that undesirable tool actions can cause before monitoring or intervention detects the issue.

Rate limits can be applied according to user, operation, tool or risk level.

They should not substitute for authorization.

They provide another independent layer.

Avoid Giving Agents Raw Secrets

An AI tool may need to interact with a system that requires credentials.

That does not mean the model needs to see those credentials.

Secrets should ideally remain within the backend integration responsible for using them.

The agent requests an authorized operation.

The application performs it using protected credentials.

The model receives only the result necessary for its task.

This helps avoid turning model context or tool definitions into a secret-storage mechanism.

OWASP’s System Prompt Leakage guidance gives the concrete example of credentials embedded in a model’s system prompt being exposed and subsequently reused by an attacker.

The safest secret presented to an LLM is generally the one it never receives.

AI Tool Security Must Consider the Open World

Some tools operate entirely within a controlled environment.

Others reach external services, websites, users or organizations.

That difference matters because an open-world tool can bring attacker-controlled content back into the AI workflow.

The MCP project’s tool-risk work uses openWorldHint specifically to describe tools that interact beyond a closed environment and argues that external tool content should be treated as potentially untrusted.

This creates an important architectural distinction.

An internal read-only database lookup has one threat model.

An unrestricted web-fetching function has another.

An external messaging capability has another again.

Tool classification should therefore consider both what the function can change and where the function can reach.

Secure AI Tool Design Should Be Explicit

A strong AI tool usually has a narrow, understandable purpose.

The function name is clear.

Inputs are constrained.

Authorization is enforced independently.

The backend executes under appropriate permissions.

Potentially dangerous operations are separated from routine ones.

Outputs are treated according to their trust level.

High-impact actions receive additional review.

Tool activity is auditable.

None of these requirements depend on the model behaving perfectly.

That is exactly why they work.

The language model can remain flexible while the capability boundary stays predictable.

How to Penetration Test AI Tool Integrations

Security testing should begin by mapping every capability available to the model.

Researchers should understand what each tool can access, which identity it uses and whether the operation changes application state.

From there, the assessment follows likely attack paths.

Can a low-privilege user influence selection of a privileged function?

Can one user’s identity access another user’s objects through the AI layer?

Can indirect prompt injection inside retrieved content influence subsequent tool calls?

Can tool parameters escape their intended business boundary?

Can multiple legitimate functions be chained into an unauthorized outcome?

Does the backend independently reject operations the current user cannot perform?

Can high-impact functions execute without meaningful confirmation?

The strongest LLM security testing does not stop after demonstrating that a model requested a dangerous operation.

It determines whether the security architecture allows that operation to succeed.

Test With Multiple User Roles

Role-based testing is essential for AI tool security.

An ordinary user and administrator should not acquire equivalent effective authority simply because both communicate through the same agent.

Researchers should compare tool availability and tool execution across test identities.

If the model requests an administrative action for a low-privilege account, the backend should reject it.

This test is particularly important when integrations rely on generic service identities.

The AI orchestration layer may accidentally flatten permissions that the conventional application otherwise enforces correctly.

Test Tool Parameters Separately From Tool Selection

Security testing should distinguish between two questions.

Can the model call the wrong tool?

And can the correct tool be used against the wrong object?

A product may correctly restrict a user to an account-management tool while still allowing that tool to manipulate accounts belonging to other users.

That is an object-authorization problem rather than a tool-discovery problem.

Both need testing.

This is one reason conventional API penetration-testing skills remain essential in AI security.

Tool calling creates another path to the API.

It does not change the need for object-level authorization.

Test Indirect Influence Through Tool Results

An agent may first call a read-only tool and receive malicious content.

That result may then influence a later write-capable operation.

This sequence should be part of the threat model for tools that process user-generated or external information.

The security tester should ask whether information returned from a low-trust source can cross into a higher-trust action.

That is particularly important for email, web, RAG and external MCP integrations.

A read-only tool can therefore contribute to a serious exploit chain even if it cannot change anything by itself.

Test Tool Combinations

AI tool security is inherently compositional.

A file reader plus an external uploader.

A customer search plus an account modifier.

A repository reader plus a code-execution environment.

A billing lookup plus a refund function.

Each pair creates a different set of possible workflows.

The goal of penetration testing is not to abuse those capabilities unnecessarily.

It is to determine whether an attacker can cause the agent to combine capabilities in a way that violates the application’s intended security model.

This is where AI Agent Security and Excessive Agency become directly relevant.

Validate Impact Without Performing Unnecessary Damage

A controlled security assessment should prove the boundary with minimum necessary impact.

If an unauthorized update can be demonstrated using a test object, there is no need to modify production customer data.

If the agent appears capable of sending external information, a controlled destination can establish the finding.

If a destructive function lacks authorization, the tester does not need to perform real destruction when safer evidence can prove the flaw.

The purpose is understanding exploitability.

Not maximizing consequences.

That distinction is central to ShabuShabu’s testing methodology and Security Crash Test approach.

Remediation Should Change Capability, Not Only Model Behavior

AI tool vulnerabilities are often “fixed” by updating the system prompt.

A developer tells the model more strongly not to use a certain function in a dangerous way.

The original test is repeated.

The model now refuses.

This may improve behavior.

It does not necessarily fix the vulnerability.

If the tool still has unnecessary privileges, the architecture remains dependent on model obedience.

A stronger remediation changes the capability boundary.

Remove an unnecessary tool.

Split a broad tool into narrower functions.

Reduce backend permissions.

Preserve user authorization.

Add object-level access checks.

Require confirmation before high-impact actions.

Constrain parameter ranges.

The model can still receive behavioral instructions.

They become defense in depth rather than the security foundation.

Retesting Should Verify the Security Property

Function-calling vulnerabilities should be retested at the architecture level.

If the original issue allowed a low-privilege user to manipulate another user’s record, the important question is whether that unauthorized modification remains technically possible.

It does not matter whether the exact original wording still works.

If a different prompt causes the same backend action, the vulnerability remains.

Likewise, if prompt injection previously caused an agent to send information externally, remediation should be validated against the underlying permission or approval boundary rather than one known prompt sequence.

Durable security controls survive new wording.

How to Build a Safer AI Tool Architecture

A secure tool-enabled AI system can be summarized by one relationship:

The model chooses from permitted possibilities. It does not define what is permitted.

The application defines which tools exist.

Identity determines which tools and objects are available.

Tool permissions limit downstream capability.

Input validation constrains parameters.

Authorization verifies each sensitive operation.

High-impact actions receive stronger approval.

External results remain untrusted.

Monitoring connects actions back to the user and agent that caused them.

This design allows models to remain flexible without giving them unrestricted authority.

Function Calling Is Safe Only When the Backend Remains in Control

The long-term direction of AI products is clear: models will increasingly interact with software rather than simply describe what users should do.

That makes tool security foundational.

The dangerous architecture is one where a model decides an action is appropriate and the application blindly executes it.

The safer architecture treats the model as an intelligent but untrusted decision component operating inside controlled boundaries.

It may suggest.

It may plan.

It may choose tools.

But authentication, authorization, capability limits and business rules remain outside the model.

OWASP’s guidance on Excessive Agency reduces the principle to familiar security engineering: minimize functionality, minimize permissions, execute operations in the proper user context, require approval where appropriate and enforce authorization in downstream systems.

That is the foundation of secure function calling.

The AI can become more capable without automatically becoming more privileged.

Frequently Asked Questions About AI Tool Security

What is AI function-calling security?

AI function-calling security is the practice of controlling how LLMs select and invoke application functions so manipulated or incorrect model behavior cannot cause unauthorized access or unsafe actions. It includes tool permissions, backend authorization, input validation, user identity, approval controls and monitoring.

Are AI tool calls safe because they use structured parameters?

No. Structured schemas can validate the format of inputs, but they do not establish authorization. A syntactically valid user ID, filename or account identifier can still refer to something the current user should not access. MCP uses schema-defined tools, but its broader security model still includes user control and permission mechanisms.

Should an LLM decide whether a tool call is authorized?

No. OWASP recommends enforcing authorization in downstream systems rather than relying on the LLM to decide whether an action is allowed.

Why are generic AI tools risky?

Generic tools expose a broad range of possible behavior. A narrowly defined operation provides a smaller capability boundary and is easier to authorize and audit. OWASP specifically recommends avoiding open-ended extensions when a more granular function can perform the required task.

Should every AI tool require human approval?

Not necessarily. Approval should be proportional to potential impact. Low-risk read operations may be pre-approved, while sensitive write or destructive actions can require explicit confirmation. Current MCP and agent-control guidance supports differentiated permissions rather than treating every operation identically.

Can prompt injection abuse function calling?

Yes. Prompt injection can alter model reasoning and therefore influence which tool a model attempts to call. The actual impact depends on whether the downstream tool independently enforces authorization and operates with appropriately limited permissions.

How should AI tools handle user permissions?

Operations should execute within the appropriate user security context wherever practical, and downstream applications should independently enforce access control. The agent should not receive broader effective privileges simply because it uses a shared backend service.

What should an AI tool penetration test cover?

Testing should examine tool discovery, user roles, function authorization, parameter manipulation, object-level access, indirect prompt injection, tool output, chained actions, excessive permissions, human approval and the downstream impact of tool execution.