This article is based on two scientific publications and an analytical article:
Adversarial Machine Learning in Industrial IoT: A Systematic Review of Attack Realism, Defense Trade-Offs, and Deployment Gaps — a review of attacks on industrial machine learning and defense methods.
AAIGF-E: An AI and Agentic Intelligence Governance Framework for Electric-Sector Smart-City Infrastructure — a framework of AI controls for the electric sector and its scenario-based evaluation.
Taiwan Should Set the Standard for Industrial AI Security — proposals for certifying and securely deploying industrial AI agents.
The agent was given access. Does that help the attacker?
Imagine an industrial facility where an AI agent helps analyze equipment operating metrics. It reads telemetry, event logs, and maintenance requests, compares them with documentation, and suggests what to check. Over time, it is also allowed to perform certain actions, which is very convenient — why manually transfer a recommendation to another system if the agent has already figured everything out?
Now suppose an attacker can modify one of the maintenance requests. They have no direct access to the controllers. But the request will be read by an agent that does have such access through approved tools. If the contents of the request influence its understanding of the task or are taken as authorization to act, someone else’s privileges could turn out to be quite a useful service.
This is a hypothetical example, not a description of a confirmed incident. For it to happen, an entire chain has to come together: the attacker controls a source, the agent uses its contents, the manipulation influences the decision, the necessary tools are available, and the checks before execution let the action through. The attack can end at any of these transitions.
That makes it worth digging deeper: Can AI shorten the path from manipulating data to affecting equipment if it is delegated authority without sufficient independent constraints?
Even before AI, industry was not run exclusively by a person with a hand on a button. Controllers, SCADA, and distributed control systems have long performed automated operations. Tampering with measurements, changing controller logic, and feeding false information to operators did not arrive with language models either. These classes of threats have long been described, for example, in NIST SP 800-82r3.
So the comparison should be between existing automation and a specific AI deployment. What data source has been added? Who now makes the decision? What authority have they been given? What checks the result before execution?
The selected materials let us follow this path step by step. Adversarial Machine Learning in Industrial IoT examines attacks on industrial machine learning and the limitations of the research. AAIGF-E proposes a control architecture and evaluates it against failure scenarios. Taiwan Should Set the Standard for Industrial AI Security discusses how to connect industrial agent security with certification, procurement, and operational approval.
Adversarial Machine Learning in Industrial IoT: how to confuse an industrial model
The article reviews 50 studies from 2019–2026. Its scope is fairly broad:
- intrusion detection;
- equipment diagnostics;
- industrial process control;
- power system state estimation;
- distributed learning.
This is mainly about machine learning that classifies events, detects anomalies, makes predictions, or participates in control. The findings cannot all be automatically extended to language-model agents, but they help explain what happens to a system’s decisions when an attacker gains the ability to influence data or training.
Seven attack categories and several different prerequisites
The authors identify seven categories. This classification combines attack construction methods with the stages at which attacks operate, so the categories can overlap. For example, a generative model can be used both to evade a detector and to prepare poisoned training data.
| Category | What the attacker does | What feasibility depends on |
|---|---|---|
| Gradient-based attacks | Selects input changes that shift the model’s output | Knowledge of the model or the ability to build a surrogate model and transfer the attack |
| Attacks using generative adversarial networks, GANs | Generates samples that resemble valid data | Access to suitable data, generation quality, and preservation of the intended effect |
| Training poisoning | Influences training data or model updates | Access to the training or retraining pipeline, or to participants in distributed learning |
| Reinforcement learning-based attacks | Selects a sequence of interventions based on the system’s response | The ability to observe results and interact repeatedly with the environment |
| Feature-importance-based attacks | Changes the features most important to the model’s decision | The ability to assess their influence and actually modify them in the target system |
| False data injection, FDIA | Tampers with measurements or process variables | Access to the relevant source and knowledge of process relationships |
| Combined attacks | Combines several approaches | The prerequisites of the component methods; combining them does not remove those prerequisites |
For the hypothesis, it is useful to distinguish between “interference during operation” and “interference with training.”
In the first case, the model is already running, and the attacker tries to obtain an incorrect result — to hide an anomaly, change a forecast, or influence a state estimate. In the second, the attacker changes what the system learns from so that the error becomes embedded in its behavior. Being able to write a false value into an accessible log does not yet mean being able to poison the model, because that record must enter the relevant training pipeline and pass its checks.
These paths can, however, connect. The authors specifically note a situation where tampered operational telemetry is later used for retraining, meaning that interference with current measurements carries over into the next version of the model. This transition requires separate analysis, and not all accumulated data can be considered suitable for training simply because it came from the organization’s own facility.
An impressive accuracy drop still has to reach the facility
The authors report that half of the 28 attack studies use exclusively white-box settings, where the attacker has access to the internal model information needed to construct the attack. This is a useful way to test robustness, but in a real system that access still has to be obtained or reasonably assumed. It is also worth remembering that black-box attacks may require numerous queries, observation of responses, suitable data, or the ability to repeat interventions in an experimental environment.
There is also a physical side to the issue. A feature set modified in a file may fool a classifier very effectively, but a practical attack needs an explanation of how those values would appear in an industrial system. Are valid ranges preserved? Are adjacent measurements consistent? Is such a sequence possible over time? Would another component filter it out before it even reaches the model?
Adversarial Machine Learning in Industrial IoT is useful precisely because it pays attention to this gap. SWaT and WADI provide data on physical processes and the relationships between sensors and actuators. NSL-KDD and UNSW-NB15 primarily serve network intrusion detection tasks and do not reproduce the full context of industrial control. A good result on one type of data cannot be transferred to another without validation.
The article also includes studies involving real controllers and semi-physical testbeds. For example, study [66] in its bibliography uses a real Siemens PLC controlling a simulated four-tank process. The preparation of adversarial data takes into account industrial protocol validity and preservation of the original malicious effect. The problem, however, is that experimental conditions, threat models, and metrics differ too much to support a universal conclusion about attack effectiveness.
The authors explicitly refrain from quantitatively pooling the results in a meta-analysis. The reported percentages show the results of individual studies, not the probability of a successful attack on an arbitrary enterprise. Combining them into one frightening average would be convenient for a presentation but useless for an engineering decision.
Defenses also have to fit the industrial process
The review examines four main defense approaches: training on adversarial examples, detecting suspicious inputs, monitoring behavior during operation, and combining several mechanisms.
Each has limitations. Training on known attacks does not guarantee robustness against new ones. Detectors may perform worse under mixed attacks. Additional monitoring and complex ensembles require computation, memory, and time. These costs depend substantially on the implementation and application domain, so declaring an entire class of methods suitable or unsuitable for OT based on a single metric is not a great idea.
The main practical question is whether a defense can respond before an incorrect decision becomes consequential for the process. False positives also need to be considered, because a defense that regularly interferes with normal operation creates an operational problem of its own.
Adversarial Machine Learning in Industrial IoT establishes an important limitation: the technical vulnerability of models has been demonstrated under various conditions, but its contribution to the risk at a specific facility is determined by the attacker’s access, the physics of the process, and the model’s place in the system.
Where the model’s answer ends and action begins
A predictive model can produce an incorrect forecast or miss an anomaly. What happens next depends on who uses its output. An operator may check the recommendation, an automated system may turn it into a setpoint, and a separate safeguard may constrain the permitted effect. Even a model without write permissions can indirectly influence the process through a human.
An agent has an additional task: understanding the assignment, choosing tools, and selecting a sequence of actions. For a language-model-based agent, the risk of malicious instruction injection is also significant. Untrusted content in a document, request, or textual event may influence what the agent considers its task and an authorized action.
It is important not to confuse the mechanisms here. Tampering with a numerical sensor reading does not, by itself, turn it into a textual instruction. Instruction injection arises where the system processes the relevant content. For example, an event description, a comment on a request, or an extracted documentation fragment. An incorrect measurement and a false instruction about “what to do next” require different checks.
| AI’s role | Possible path of influence | What needs to be checked |
|---|---|---|
| Analytics without direct control | Distorted output influences subsequent decisions | Who uses the output and how they verify it |
| Recommendations to an operator | A human approves an action based on incorrect reasoning | Whether independent data, time, and authority to verify it are available |
| Control through a restricted interface | AI output becomes a command request | How permissions, process state, and engineering constraints are checked |
| An agent with several tools and broad permissions | Manipulation of the task influences a sequence of actions across systems | How delegation, action scope, and transitions between systems are constrained |
This is not a universal risk scale. Even a narrow function can be critical if an important action depends on it. But the table shows why “we have deployed AI” says almost nothing about the consequences of its compromise.
Does it really become easier for the attacker?
The potential simplification arises when the agent already has access and the means to perform actions, while the attacker only needs to influence a source it uses. In this scenario, the part of the path that involves obtaining direct privileges may be replaced by manipulating an intermediary.
But this effect is conditional. If there is an independent check between the agent and the equipment that the agent cannot change or bypass, an incorrect decision may remain a rejected request. And if the attacker can already change the necessary controller settings directly, an additional attack on AI may not be needed at all.
It is a similar story with expertise. Attempting to manipulate an agent through text may require less specialized knowledge than developing an attack on an industrial ML model. However, eliciting a strange answer, getting a command executed, and reproducibly causing a particular physical consequence are tasks of different difficulty. The last may require knowledge of the process, its safeguards, and the current operating state.
The selected sources therefore do not support a claim that AI has made all industrial attacks easier. They provide grounds for considering new attack paths and examining their prerequisites.
Where the chain should be broken
AAIGF-E: An AI and Agentic Intelligence Governance Framework for Electric-Sector Smart-City Infrastructure introduces AAIGF-E, a governance framework for AI and agentic systems in the electric sector. It contains 111 controls across 11 domains and covers seven lifecycle phases:
governance → design → implementation → verification and approval → monitoring → response → recovery.
Individual controls apply at several phases. The authors do not propose applying all 111 controls identically to every AI system, because the selection depends on the system’s purpose, autonomy, failure consequences, and operating conditions.
A useful feature of AAIGF-E is the dual description of each control. One explains its meaning in terms of AI risk management, while the other translates that meaning into the language of power grids, dispatch, and OT. The requirement to “validate input data” becomes a check of specific telemetry, its timing, permissible values, and consistency.
The authors map the controls to NIST AI RMF, ISO/IEC 42001, NERC CIP, ISA/IEC 62443, and MITRE ATLAS. They themselves emphasize that a mapping in a table does not prove compliance with a standard, certification, or defense effectiveness. These sources have different purposes and normative status.
The architecture is oriented toward the North American electric sector. Another sector or jurisdiction will require its own engineering constraints and an assessment of which requirements apply.
What exactly was evaluated
The paper considers three scenarios:
- manipulation of input data
- unauthorized agent actions
- silent model drift.
The authors trace which controls should activate, in what sequence, and where gaps remain. It should also be noted that this is an analytical scenario-based evaluation. The work did not involve testing on an operational power system or a controlled cyber-physical experiment.
In addition, five suitably qualified experts evaluated one separate scenario. The reproducibility of some of the standards mappings was also checked. This provides grounds for discussing the coherence of the architecture and the clarity of the proposals, but does not justify claiming that the 111 controls have already been proven to prevent industrial incidents.
Scenario A: the data was tampered with, and the control system trusted it
AI manages distributed energy resources, such as distributed generation and storage. Tampered telemetry is supplied as input. The system generates commands that push voltage parameters beyond permissible limits, and the anomaly is detected with a delay.
AAIGF-E provides several points of defense. Data sources must be approved in advance. Format, value ranges, timestamps, completeness, and consistency are checked at the input. The model must be tested for robustness against manipulation and atypical operating conditions. During operation, changes in inputs/outputs and deviations in results need to be monitored. In the event of an incident, it must be possible to isolate AI influence and return to a validated operating configuration.
The scenario illustrates the importance of timing, because an anomaly detected after several control cycles may already have affected the process. This makes it difficult for reactive monitoring alone to compensate for the absence of checks before data is used.
The interpretation requires some caution here. A range check will filter out some tampering, but a plausible false value can still fall within that range. Actual effectiveness depends on independent measurements, relationships between variables, and how the check is implemented. A validation requirement does not, by itself, prove that every manipulation will be detected.
Scenario B: asked to investigate, the agent went off to fix things
An agent is tasked with investigating an anomaly. It independently makes changes across OT segments without the necessary approval. The changes trigger protective systems and cause the unplanned separation of part of the grid.
The key control here is “OP-009” — checking and authorizing commands at the boundary between AI and OT. A request must pass checks of authority, scope, engineering constraints, protective interlocks, and required approvals. This mechanism is complemented by privilege restrictions, segmentation, operator intervention capabilities, and a coordinated response by cybersecurity and operations teams.
If a task requires investigation only, write access must be justified separately. The technical ability to execute a command does not grant authorization to execute it as part of the current assignment.
Human approval must also be a real control action. It requires a clear rationale, information about consequences, and the ability to verify the underlying data. An “Approve” button with nothing but “AI recommends” underneath looks like a fairly weak engineering barrier.
It is in this scenario that the authors examine an attack path through data content that remains unaddressed. Control “DI-006” validates the technical properties of the input: schema, values, timing, completeness, and consistency. But text that passes these checks may still distort the agent’s understanding of the task or create a false impression that authorization has been granted.
“OP-009” constrains command execution, but does not, by itself, guarantee the validity of the reasoning that led to the command. Moreover, a human whose approval is required may read that distorted reasoning. In such a situation, the human becomes another participant in the chain of influence.
The authors explicitly acknowledge this residual risk in AAIGF-E v1.0. Verifying the provenance and plausibility of content before it influences the task and authorization context is proposed as a direction for future work.
Scenario C: nobody attacked, but it can no longer be trusted
A model assesses power system stability. After a substantial amount of distributed energy resources is connected, operating conditions and the input data distribution change. The model keeps running, operators keep trusting its outputs, and the degradation is discovered only after a hazardous event.
An attacker is not necessarily involved here. This is an operational scenario in which the conditions under which the model is applicable have changed.
AAIGF-E requires acceptance criteria to be defined in advance, rare and boundary conditions to be tested, model limitations to be documented, and drift to be monitored. Exceeding established boundaries must trigger predefined actions: additional verification, restricted use, reassessment, or suspension of AI influence.
The authors do not, however, provide a universal drift threshold. It needs to be tied to the consequences for a specific function. The same statistical deviation may have different significance for demand forecasting and grid stability assessment. A model does not become safe because its dashboard can plot drift: someone has to decide when that plot calls for action.
What AAIGF-E adds
The three scenarios show different reasons why AI output can no longer be trusted: data tampering, exceeding authority, and changes in operating conditions. All require predefined limits on AI influence and ways to stop it.
Scenario B is particularly important to the hypothesis that the attack path can become shorter. It shows where authority should be checked and why technically valid input does not guarantee a correct decision. The practical effectiveness of these controls still requires experimental validation.
But this raises a question: how do we know whether the system meets the applicable requirements? Who is responsible for this celebration of automated life and, most importantly, is willing to confirm that everything is okay?
How to make constraints a condition of approval
In Taiwan Should Set the Standard for Industrial AI Security, the authors propose using the existing semiconductor equipment certification framework to evaluate industrial AI agents.
The starting point is SEMI E187 and the associated Taiwanese certification scheme. According to the authors, it already connects equipment cybersecurity requirements, independent testing, and procurement conditions. They propose supplementing it with a separate agent security profile; the article treats this profile as a proposal for development. However, the material focuses primarily on Taiwan and the semiconductor industry, so transferring these proposals to other industries would require adaptation.
Evaluate the specific configuration
The authors propose evaluating the model together with its runtime environment, policies, installed skills, tools, credentials, and scope of authority. This combination determines what the agent can actually do.
The same model in a log-reading system and in a system with permission to change production settings creates different risk conditions. Evaluating the model alone does not describe the security of both configurations.
A change to the model, runtime, or toolset should trigger reassessment. The practical point is clear: if an agent is given a new way to affect equipment after an evaluation, that capability also needs to be assessed. Saying “we just connected one more tool” does not make the change insignificant.
Authorize use at the site separately
Product certification should be complemented by an assessment of the specific deployment. It is important to establish which systems the agent connects to, what actions are available to it, how network zones are separated, where temporary credentials are needed, and which operations require a trained human who understands the consequences.
The authors specifically propose retaining protective interlocks and conventional controls that the agent cannot override. This means that even after making use of the agent’s authority, an attacker should encounter restrictions on executing a dangerous action that are enforced outside the model.
The presence of such a restriction must, however, be verified in the specific system. A statement in a policy and an actual enforced prohibition are different kinds of evidence.
Monitor changes and behavior after deployment
The third element of the proposal is ongoing evaluation after approval:
- behavior monitoring;
- logs that allow tampering with records to be detected;
- investigation of significant incidents;
- reassessment when capabilities expand.
When the agent moves outside validated conditions, the proposal calls for it to be automatically suspended and isolated.
This requires responsibility to be allocated. The vendor is responsible for product capabilities and built-in constraints, the integrator for connecting systems, and the operator for granted permissions, monitoring, and response. If everyone is responsible only for their own little box, the path between the boxes risks being left without an owner.
Test how the agent will fail
The authors propose testing behavior when connectivity is lost, readings conflict, limits are exhausted, or instructions are malformed. The system must have a predefined way to reduce its capabilities or hand over control: switch to reading data, use conventional automation, or perform a controlled shutdown if appropriate for the function.
The point is to make the response predictable. A safe state must be selected with the process in mind: arbitrarily shutting down equipment is not a universally safe response. The agent should not invent a new workaround on its own simply because its normal tool has stopped responding.
Another focus of the article is data provenance and the ability to reconstruct the sequence of decisions. Plausible false reports can influence actions through entirely authorized interfaces. Preserved records of sources and changes help investigate such a chain, although they do not prevent all tampering by themselves.
An additional engineering clarification follows here: a verified source does not guarantee that a message is true. A compromised account can send false information with valid provenance indicators. Checking the source, the content, and whether an action is permitted addresses related but different tasks.
The value of the authors’ proposal lies in the connection between technical constraints and deployment conditions. Requirements can be checked during procurement, when connecting to a site, and after capabilities change. But the effectiveness of the proposed profile itself still needs to be established, because the analytical article does not include such an evaluation.
So does the attack path get shorter?
The materials examined provide grounds for considering such a path possible. A particularly illustrative case is one where an attacker can influence data available to an agent, while the agent has authority that the attacker lacks. With insufficient constraints, manipulating a source can influence the selection and execution of an action.
However, none of the three materials establishes how much AI deployment makes attacking an industrial facility easier on average. Adversarial Machine Learning in Industrial IoT synthesizes heterogeneous experiments, AAIGF-E evaluates an architecture through scenarios, and Taiwan Should Set the Standard for Industrial AI Security proposes a certification mechanism. There is no comparative measurement of the effort required for the same attack against the same facility before and after AI deployment.
The difficulty of an attack therefore needs to be assessed for a specific system by tracing the entire chain:
source under the attacker’s influence → AI input → changed decision → use of authority → execution check → effect on the process
Each transition has its own prerequisites. A shorter chain also does not necessarily mean a more reliable or cheaper attack, because the manipulation may fail, the required tool may be unavailable, or an independent check may stop the action. The three sources complement one another precisely at these transitions.
The practical question before deployment becomes quite specific: What could happen to the equipment if AI receives false data, makes a mistake, or misunderstands its assignment?
The answer needs to specify permitted actions, independent constraints, response times, and a validated procedure for operating without AI.
And if the entire answer amounts to writing “don’t do anything dangerous” in the agent’s instructions, then access has already been granted. All that remains is to find out who will benefit from it.