Machine learning systems are increasingly responsible for decisions that used to be handled entirely by deterministic software. Models classify information, detect fraud, identify malicious activity, recommend actions, generate content and increasingly interact with software through AI agents. As these systems become more important, attackers have an incentive to manipulate not only the applications surrounding the models but the learning systems themselves.

This field is known as Adversarial Machine Learning, or AML.

NIST’s current Adversarial Machine Learning taxonomy, NIST AI 100-2e2025, organizes AML around machine-learning methods, lifecycle stages, attacker goals, capabilities and knowledge. For predictive AI, NIST identifies evasion, poisoning and privacy attacks as major categories; for generative AI, the taxonomy additionally includes misuse attacks.

That distinction is useful because AI security is considerably broader than prompt injection. An attacker may manipulate an input presented to a deployed model, corrupt information used to build or update the model, attempt to extract information about sensitive data, or intentionally use a generative model in ways its designers did not intend.

Understanding these different attack classes helps security teams identify which part of the AI lifecycle is actually under attack.

What Is Adversarial Machine Learning?

Adversarial Machine Learning studies how attackers can deliberately influence, manipulate or extract information from machine-learning systems.

Unlike ordinary software exploitation, an AML attack does not always require a traditional programming vulnerability such as broken access control, memory corruption or an exposed API.

The attacker may instead exploit how a model learns or makes decisions.

A machine-learning model receives information, processes that information according to patterns learned from data and produces an output. If an attacker understands enough about that process, they may try to manipulate the relationship between the input and the output.

NIST’s taxonomy approaches this systematically by considering not only attack type but also the stage of the AI lifecycle, attacker objectives, attacker knowledge and available capabilities.

That framework matters because two attacks that produce similar incorrect outputs may represent completely different security problems.

One attacker may manipulate the model during inference.

Another may have corrupted the data used to train it months earlier.

A third may only be interested in determining whether sensitive information was contained in the training dataset.

The observable result can look similar while the underlying attack path is completely different.

Why Traditional Cybersecurity Is Not Enough for Machine Learning

Traditional cybersecurity remains essential for AI systems.

Models still run on servers. APIs still require authentication. Cloud infrastructure still needs access control. Applications can still contain ordinary software vulnerabilities.

But machine learning introduces another layer of behavior.

A traditional application usually executes explicitly programmed rules. A machine-learning model derives behavior from training data and learned parameters.

That means the data itself becomes part of the effective security surface.

NIST’s AML report is intended to provide a common vocabulary for managing precisely these risks and describes attacks across different learning methods and stages of an AI system’s lifecycle.

For a penetration tester, this changes the threat model.

The question is no longer only:

“Can the application be compromised?”

It may also be:

“Can the model’s behavior be deliberately changed without compromising the surrounding application?”

The AI Lifecycle Determines the Attack Surface

One of the most useful ways to understand adversarial machine learning is by considering when the attacker interacts with the system.

Some attacks happen before or during model development.

Others occur after the model has been deployed.

That difference can determine both impact and remediation.

If an attacker manipulates inference-time input, the model itself may remain unchanged. Removing the malicious input may restore normal behavior.

If an attacker poisons training data, the resulting model may have learned the compromised behavior. The vulnerability can persist even after the attacker’s original access disappears.

Privacy attacks follow another path entirely. Rather than altering model behavior, they attempt to learn information from the model or its outputs.

NIST’s taxonomy explicitly incorporates these lifecycle stages and attacker objectives rather than treating AML as one homogeneous category.

Evasion Attacks: Manipulating the Model at Inference Time

An evasion attack attempts to influence a machine-learning system when the model is already deployed.

The attacker manipulates the input in a way designed to produce an incorrect or attacker-preferred output without necessarily modifying the model itself. NIST includes evasion among the major attack classes affecting both predictive and generative AI systems.

The central idea is straightforward.

The model learned a decision boundary from its training data.

The attacker searches for an input that crosses that boundary in an unexpected way.

Manipulating Inputs Without Changing the Model

Imagine a machine-learning system designed to classify incoming content.

Under normal conditions, the system correctly identifies most inputs.

An adversary attempts to change characteristics of a particular input so the model interprets it differently while the attacker’s underlying objective remains unchanged.

The software around the model can operate exactly as intended.

The model executes successfully.

The attacker succeeds because the model makes the wrong decision.

This illustrates an important difference between AML and conventional application exploitation.

A traditional penetration tester may search for a way to bypass an authentication function.

An AML researcher may instead investigate whether a model relied upon by that function can be induced to classify malicious input incorrectly.

The security boundary exists in the model’s decision behavior.

Evasion Is Not Only an Image-Classification Problem

Adversarial examples became widely associated with computer vision, but the underlying concept is broader.

NIST’s current AML taxonomy covers attacks across multiple machine-learning methods and data modalities rather than restricting adversarial behavior to one model type.

In practical security engineering, evasion may matter anywhere a model’s inference influences a meaningful decision.

Fraud detection.

Spam filtering.

Malware classification.

Anomaly detection.

Content moderation.

Identity-related models.

Generative AI applications.

The exact manipulation technique depends heavily on the system.

The security principle is consistent: if the application assumes the model’s output is always trustworthy, successful evasion can become a route around the broader security control.

Model Confidence Does Not Prove Security

One particularly dangerous assumption is that a highly confident model output must be reliable.

Machine-learning confidence reflects the model’s internal prediction process.

It does not prove that the input is legitimate.

A carefully manipulated input can still produce a confident incorrect decision.

For security teams, confidence should therefore not replace independent safeguards when the resulting decision has high impact.

If a machine-learning classifier influences a security-sensitive operation, organizations should consider what happens when that classifier is wrong.

This is the same defense-in-depth principle used throughout conventional cybersecurity.

Security should not collapse because one detection component makes one incorrect decision.

Poisoning Attacks: Manipulating What the Model Learns

If evasion targets a deployed model’s decision, poisoning attacks target the information involved in learning or adapting model behavior.

NIST includes poisoning as a major category affecting predictive and generative AI.

This changes the threat model considerably.

The attacker is no longer merely trying to fool the system once.

The objective may be to influence future behavior by affecting the data or learning process from which the system derives its decisions.

Why Training Data Is a Security Asset

Modern machine-learning systems can depend on enormous datasets.

Security discussions often focus on protecting the finished model while treating the data pipeline primarily as an engineering problem.

That is dangerous.

If an attacker can influence training, fine-tuning or other learning data, they may influence the behavior of the model produced from that information.

The data pipeline therefore becomes comparable to a software supply chain.

Organizations need to understand where data originates, who can modify it and which validation or governance processes exist before it influences a production system.

NIST’s AML taxonomy explicitly includes lifecycle considerations because attacks can occur at stages other than runtime inference.

Data Poisoning Can Be Broad or Targeted

A poisoning attack does not necessarily attempt to destroy overall model accuracy.

A crude attack might broadly degrade performance.

A more subtle attacker may want the model to behave incorrectly only under particular conditions while continuing to appear normal during ordinary evaluation.

The second case is much more difficult operationally because standard quality metrics may remain acceptable.

Security teams therefore need to think beyond average model performance.

A model can perform extremely well across a test dataset while still containing behavior that matters under a narrow attacker-controlled condition.

This is why adversarial validation focuses on threat scenarios rather than only aggregate benchmarks.

RAG Knowledge Poisoning Extends the Same Idea

Retrieval-Augmented Generation introduces a related form of information integrity risk.

The underlying foundation model may remain unchanged, but an attacker can attempt to influence information retrieved into model context.

From the user’s perspective, the AI appears knowledgeable because it is connected to an external knowledge base.

From a security perspective, that knowledge base is part of the decision pipeline.

If malicious or misleading content enters it, later model responses can be affected.

This links classic AML concepts with the RAG security problems discussed earlier in this series.

The implementation differs from ordinary model-training poisoning, but the security lesson is similar:

AI behavior depends on information supply chains, and attackers may target those supply chains.

Poisoning Makes Data Provenance Important

One of the strongest protections against poisoning is understanding where information comes from.

Organizations should know which datasets are authoritative, which external sources are incorporated, who can modify internal training data and how updates are reviewed.

This becomes more difficult as AI development pipelines become larger.

Training data may come from several organizations.

Fine-tuning information may be created internally.

Feedback may be collected automatically from users.

RAG systems may ingest documents continuously.

Each input source represents a different trust level.

Security engineering should preserve those differences rather than collapsing all available information into one undifferentiated model-development pipeline.

Privacy Attacks: Learning What the Model Knows

Not every adversarial machine-learning attack attempts to make the model wrong.

Some attackers want information.

NIST includes privacy attacks among the primary AML categories for both predictive and generative AI.

This category changes the objective entirely.

Instead of asking:

“How can I alter this prediction?”

the attacker asks:

“What can this model reveal about the data or system behind it?”

This can matter when models were trained, fine-tuned or connected to sensitive information.

Machine Learning Can Become an Information Side Channel

Organizations sometimes assume that because users cannot directly access a training dataset, the information inside it is automatically protected.

Machine-learning research has shown that model interactions can sometimes reveal information about underlying data or model behavior, which is why privacy breach is explicitly included in NIST’s AML terminology and taxonomy.

For security teams, the important idea is that the model itself may become an information interface.

The attacker does not necessarily need direct database credentials.

They interact with the model and analyze what it reveals.

The feasibility and impact depend heavily on the architecture, model and attacker access.

Privacy risk should therefore be threat-modeled rather than assumed.

Training Data and Runtime Data Are Different Privacy Problems

AI security discussions often combine all forms of information disclosure under one label.

It is useful to separate them.

One problem involves information associated with model training.

Another involves information supplied to the model at runtime through prompts, RAG or connected tools.

The second is often closer to traditional application authorization.

For example, if an LLM assistant retrieves another customer’s records and exposes them, the fundamental weakness may be broken data isolation rather than a sophisticated adversarial ML privacy attack.

This distinction helps remediation.

Training-data privacy may require changes to model development or data practices.

Runtime cross-user leakage may require fixing authorization in the application.

Calling both “AI data leakage” without identifying the layer can lead engineering teams toward the wrong solution.

Privacy Testing Should Be Proportionate

Adversarial privacy testing can involve sensitive information by definition.

That makes controlled scope especially important.

A security assessment should seek enough evidence to establish whether a confidentiality boundary fails without extracting unnecessary real data.

The same principle applies in broader penetration testing.

If a controlled test record demonstrates unauthorized access, there is rarely defensive value in collecting a large volume of unrelated customer information.

For ShabuShabu, this fits the same impact-validation model used in LLM security testing: demonstrate the security property that failed, document the impact and avoid unnecessary exposure.

Misuse Attacks: When Generative AI Works as Intended for the Wrong Objective

Generative AI introduces another important category: misuse.

NIST’s 2025 AML taxonomy lists misuse attacks specifically for generative AI alongside evasion, poisoning and privacy attacks.

Misuse is conceptually different from many traditional security attacks.

The attacker may not need to break the model.

The model may work exactly as designed.

The problem is that the capability is intentionally used toward an undesirable or harmful objective.

This distinction becomes essential as generative systems grow more capable.

Misuse Is a Capability-Governance Problem

Traditional cybersecurity often focuses on whether a user can do something they are not authorized to do.

Misuse can occur even when the user technically has access to the model.

The security challenge shifts toward how capabilities are governed.

Who can use the system?

Which functionality is available?

What data can the model access?

What tools can an agent invoke?

What monitoring or abuse controls exist?

What level of activity should trigger additional scrutiny?

This is one reason AI security cannot be reduced to model alignment or vulnerability scanning.

Some risks exist because powerful capabilities are available, not because the implementation contains a conventional software defect.

Misuse and Excessive Agency Can Interact

Misuse becomes particularly important when generative AI is connected to operational tools.

A text-only model may assist an attacker with information.

An agentic system may be capable of taking actions.

The risk therefore depends on both model capability and application authority.

This creates a direct link between adversarial ML misuse and the Excessive Agency problem discussed earlier.

If the application gives the AI unnecessary tools or privileges, malicious use can become more consequential.

Security teams should therefore ask two separate questions.

Can the user manipulate or misuse the model?

And what can the surrounding architecture allow that user-controlled interaction to reach?

Adversarial ML vs Prompt Injection

Prompt injection receives enormous attention because LLM applications are highly visible.

But prompt injection is only one part of the broader adversarial AI landscape.

Prompt injection typically involves adversarial instructions influencing an LLM application’s runtime behavior.

Evasion is a broader inference-time concept that can apply across machine-learning systems.

Poisoning attacks target learning or knowledge inputs.

Privacy attacks attempt to extract or infer protected information.

Misuse focuses on intentionally applying generative capabilities toward unwanted objectives.

NIST’s taxonomy is valuable precisely because it provides a broader vocabulary instead of treating every adversarial AI behavior as “prompt hacking.”

For cybersecurity teams, using the correct category helps identify which control needs to change.

Adversarial ML vs Traditional Vulnerability Exploitation

Traditional vulnerability exploitation usually targets implementation flaws.

AML often targets model behavior.

That difference affects reproduction.

A conventional API authorization vulnerability may be deterministic. The same unauthorized request repeatedly returns another user’s object.

An adversarial ML result may depend on model version, input variation, system state or probabilistic behavior.

This makes validation methodology important.

The researcher should establish the security property being violated rather than overfocusing on one exact input.

If an evasion finding depends on a fragile one-off condition that cannot be reproduced meaningfully, its risk may differ from a robust attack affecting a critical decision pipeline.

Attacker Knowledge Changes the Threat Model

NIST’s AML framework includes attacker knowledge as part of its conceptual hierarchy.

This is important because attackers may have very different levels of visibility into the model.

An internal researcher may know the architecture and parameters.

An external attacker may interact only through an API.

Some attackers may know the training procedure.

Others may infer behavior only from repeated queries.

The amount of available knowledge changes both feasibility and attack design.

A realistic security assessment should therefore model the attacker the organization actually expects rather than assuming either perfect knowledge or complete ignorance.

Attacker Capability Matters Just as Much

Knowledge alone does not determine risk.

Capabilities matter too.

Can the attacker submit one model input or millions?

Can they influence training data?

Can they modify a RAG document?

Do they have authenticated access?

Can they observe confidence values or only final decisions?

Can they call the system repeatedly?

Can they influence feedback used for future learning?

NIST’s taxonomy explicitly incorporates attacker capabilities for this reason.

Security testing should mirror realistic access.

Otherwise, organizations may either underestimate attacks that normal users can perform or overstate scenarios requiring unrealistic control over the system.

Adversarial Machine Learning Is a Threat-Modeling Discipline

A useful AML assessment should start with architecture.

What kind of model is deployed?

Where does the data come from?

Can external users influence training or feedback?

Which decisions depend on the model?

What sensitive information could be inferred?

Does the model connect to tools?

What happens when the model is wrong?

This architecture-first approach prevents testing from becoming a collection of disconnected adversarial examples.

The objective is not simply to demonstrate that machine learning can fail.

Every complex model can fail under some conditions.

The objective is to determine whether an attacker can intentionally drive the system toward a security-relevant failure.

How Security Teams Should Approach Evasion Testing

Evasion testing should begin by identifying the decisions that matter.

A model output that affects only a cosmetic interface feature has a different security impact from one controlling fraud detection, access decisions or sensitive workflow automation.

Researchers should then test whether attacker-controlled inputs can systematically influence those decisions while remaining inside the authorized test environment.

The result should be evaluated according to practical impact.

Does the attack reliably change the security-sensitive decision?

What attacker access is required?

Are additional application controls present?

Can the system detect abnormal behavior?

The goal is not maximum model confusion.

The goal is measuring security robustness under adversarial input.

How Security Teams Should Approach Poisoning Risk

Poisoning requires looking earlier in the lifecycle.

Which users or systems can contribute training information?

Which datasets are externally sourced?

Are model updates automatic?

Can feedback loops influence future behavior?

Can knowledge repositories be modified by low-trust identities?

If the answer to these questions is unknown, the organization may not fully understand its AI supply chain.

Security testing may therefore include data-governance review alongside technical testing.

This is one area where conventional penetration testing and AI security governance intersect.

Compromising the account that controls a training-data pipeline may be a conventional cybersecurity attack.

Using that access to alter future model behavior becomes an AML problem.

How Security Teams Should Approach Privacy Testing

Privacy testing should identify what sensitive information may be associated with the model or its runtime environment.

The threat model should distinguish model-training information from application context, RAG data and tool responses.

Researchers can then evaluate whether unauthorized users can infer or retrieve information beyond their intended access.

For enterprise LLM applications, ordinary access control often remains one of the strongest privacy protections.

If unauthorized data never enters the user’s model context, many runtime disclosure paths disappear.

This is why adversarial privacy testing should be integrated with AI data leakage testing rather than conducted as a completely separate exercise.

How Security Teams Should Approach Misuse Risk

Misuse risk begins with capability analysis.

What can the generative system actually produce or perform?

Who receives access?

Are different user populations given different capability levels?

What external systems are connected?

What level of activity is visible to security teams?

Not every misuse risk should be solved inside the model.

Access controls, rate limits, tool restrictions, user verification, anomaly monitoring and environment isolation can all reduce consequences depending on the product architecture.

This is consistent with a broader security principle repeated throughout this research series:

do not force one probabilistic model to carry the entire security architecture.

No Single Mitigation Solves Adversarial Machine Learning

NIST’s current AML report discusses mitigations while explicitly recognizing limitations in existing mitigation techniques.

This is an important point for security teams.

A defense may reduce one known attack while leaving another pathway available.

A model hardened against one form of evasion may still face poisoning risk.

Strong training-data governance does not automatically solve runtime privacy leakage.

Prompt-injection filtering does not prevent an overprivileged AI tool from executing an unauthorized operation.

AML should therefore use defense in depth just like traditional cybersecurity.

Different attack surfaces need different controls.

AI Security Testing Should Measure Impact, Not Novelty

Adversarial ML research can produce technically fascinating failures.

A security assessment needs to go one step further.

Does the failure matter to the actual product?

A strange classifier output may be academically interesting.

A repeatable method for bypassing a security-critical detector has operational impact.

A generative model producing unusual text may be low severity.

A manipulated AI agent using a privileged tool can be high severity.

Security teams should therefore prioritize findings according to realistic exploitability, attacker capability and business consequence.

This is where AML testing becomes part of practical cybersecurity rather than simply model evaluation.

Retesting Adversarial ML Findings Requires Care

Remediation should address the security property that failed.

If a model was vulnerable to a particular evasion input, blocking that exact sample may not establish robustness.

If poisoned knowledge influenced an AI workflow, deleting one malicious document may not fix weak publishing controls.

If privacy exposure resulted from broad RAG authorization, modifying the prompt does not remove the data-access problem.

The strongest retest asks whether the original attacker capability can still produce the original security outcome through reasonable variations.

This principle is especially important in probabilistic systems because the exact output may naturally change between runs.

Adversarial Machine Learning and Penetration Testing Are Converging

AI systems are increasingly becoming ordinary components of production software.

That means AML and penetration testing can no longer remain completely separate disciplines.

A penetration tester may discover that an AI classifier controls access to a workflow.

An AML researcher may discover an exploitable model behavior but need API and authorization expertise to determine its impact.

An LLM red team may manipulate an agent but need application-security knowledge to understand whether the resulting tool call is actually permitted.

The attack path crosses disciplines.

For modern products, the strongest AI and LLM security testing therefore combines model-aware adversarial testing with traditional application-security reasoning.

The Future of AML Is Broader Than Model Robustness

The 2025 NIST taxonomy covers predictive and generative AI, different learning methods, multiple data modalities and several distinct attacker objectives, showing how adversarial AI security has expanded beyond its early association with narrowly crafted model inputs.

As AI becomes more agentic, security teams will increasingly need to consider the entire system around the model.

Data integrity.

Model behavior.

Retrieval.

Privacy.

Tools.

Permissions.

Autonomy.

The question will no longer simply be whether the model is robust.

It will be whether the AI-enabled application remains secure when one part of the learning system is deliberately manipulated.

That is the practical meaning of adversarial machine learning for cybersecurity.

Frequently Asked Questions About Adversarial Machine Learning

What is adversarial machine learning?

Adversarial machine learning is the study and management of attacks in which adversaries deliberately manipulate or exploit machine-learning systems. NIST AI 100-2e2025 provides a taxonomy based on ML methods, attack lifecycle stages, attacker goals, capabilities and knowledge.

What are the main adversarial machine learning attacks?

NIST’s current taxonomy identifies evasion, poisoning and privacy attacks for predictive AI. For generative AI, it includes evasion, poisoning, privacy and misuse attacks.

What is an evasion attack in machine learning?

An evasion attack targets a deployed model by manipulating inference-time inputs in an attempt to produce an attacker-preferred or incorrect model decision without necessarily modifying the underlying model.

What is a poisoning attack?

A poisoning attack attempts to influence model behavior by manipulating information involved in the learning process or another trusted AI information source. Its effects can persist because the compromised information influences future model behavior.

What is an AI privacy attack?

A privacy attack attempts to obtain information about sensitive data, model behavior or information associated with an AI system rather than primarily changing the model’s prediction. Privacy attacks are included in NIST’s AML taxonomy for both predictive and generative AI.

What is a generative AI misuse attack?

Misuse involves intentionally using generative AI capabilities toward an unwanted or harmful objective rather than necessarily exploiting a conventional software vulnerability. NIST includes misuse as a specific attack category for generative AI.

Is prompt injection an adversarial machine learning attack?

Prompt injection fits within the broader adversarial AI landscape, but adversarial machine learning is much wider. AML also includes evasion, poisoning, privacy and misuse threats across predictive and generative AI systems.

How is adversarial ML different from normal penetration testing?

Traditional penetration testing primarily looks for exploitable security weaknesses in software, infrastructure and business logic. AML additionally examines whether attackers can deliberately manipulate model behavior, learning processes or information exposure. Modern AI penetration tests increasingly need both perspectives.