When the Agent Becomes the Attacker
Why prompt injection, memory poisoning, delegation abuse, and RAG harvesting are not just AI safety problems, but failures of identity architecture
Most security teams are still asking the wrong question about AI agents.
They ask whether the model can be manipulated; whether prompts can be filtered; whether outputs can be monitored.
Those questions matter, sure, but they’re downstream from the real issue.
The deeper problem is once an AI agent can receive untrusted instructions while retaining the full authority of a trusted non-human principal, the attacker often does not need to break in at all. The attacker simply redirects authorized execution through the identity gap itself.
That is the shift many enterprises still have not fully grasped. The first generation of agent compromise is not mainly a chatbot problem, a jailbreak problem, or a content moderation problem. It is an identity and authorization problem.
The security model is lagging the operating model
Most organizations still govern AI agents as if they were software.
That assumption is understandable. After all, agents are deployed through software stacks, connected through APIs, wrapped in orchestration layers, and often represented in architecture diagrams as applications with extra intelligence. But operationally, that classification is breaking down fast.
An enterprise agent is not a deterministic application in the old sense. It can reason under ambiguity, retrieve information dynamically, invoke tools, preserve state across sessions, and in many cases spawn or coordinate with other agents. It behaves less like a conventional service and more like a persistent non-human (but not unintelligent) operator.
That difference matters because enterprise security was really built to govern two broad categories: human identities and deterministic workloads. It knows how to authenticate users. It knows how to issue service credentials. It knows how to log events. But that is not the same thing as governing a principal that can be redirected semantically while still operating under valid authority.
And that is the core issue.
If a trusted agent can be induced to act on instructions from an untrusted source without its authority being re-evaluated, then the attacker is not really “outside” the system anymore. The attacker is operating through a legitimate execution surface that the enterprise itself provisioned.
This is why prompt injection is being misunderstood
Prompt injection is usually discussed as a content problem. The standard defenses reflect that framing: filter suspicious inputs, detect attempts to override system prompts, scan outputs for dangerous content, and watch for familiar jailbreak patterns. Those controls are useful, but they are not the center of gravity.
In an enterprise environment, prompt injection becomes dangerous when it changes what a trusted principal does while that principal still holds valid credentials, approved tool access, and legitimate system position.
The architectural problem is simple and severe: most agents evaluate semantics before they re-evaluate authority. A human task request, an inbound email, a retrieved document, a web page, and a poisoned tool response all arrive in roughly the same context stream. The model has no native cryptographic way to distinguish trusted instructions from untrusted content. Once the malicious instruction reaches the planning layer, the question is no longer whether the text “looks bad.” The more important question is whether that source was ever authorized to request that action from that principal.
Most enterprise environments still don’t ask that question with enough force.
That is why prompt injection should be understood less as “the model got tricked” and more as “the system allowed untrusted semantics to ride trusted authority.”
The attacker does not need to steal the badge if they can whisper into the ear of the person already holding it.
The new breach primitive is redirected authorized execution
A conventional breach usually begins with unauthorized access. The attacker steals a credential, exploits a vulnerability, bypasses a trust boundary, or moves laterally into a system they were never meant to control. That logic still exists. But the agentic case introduces a different pattern.
In many deployments, the credential already exists. The toolchain already exists. The access path already exists. The APIs are already reachable. The sensitive systems are already behind a trusted execution identity.
The attacker does not have to create a new route. They only have to alter what the already-authorized principal does next. Which is a very different security problem.
It means the decisive failure may occur without obvious signs of classic intrusion. The token is real. The session is valid. The tool invocation is permitted. The resulting telemetry may look operationally plausible. What is false is the assumption that the instruction causing the action came from the same trust domain as the identity carrying it out.
That is why I think many security teams are still mentally fighting the last war. They are looking for unauthorized access when the more important emerging risk is authorized execution redirected by an adversary.
MITRE’s new agent attack taxonomy matters for one big reason
The recent expansion of agent-specific techniques in MITRE ATLAS is important, but not just because the list got longer.
It matters because the new techniques reveal what attackers are actually targeting: context, memory, configuration, delegated authority, retrieval boundaries, and tool-mediated exfiltration. In other words, they are not just attacking “the model.” They are attacking the missing governance architecture around the model.
Context poisoning is an attack on the system’s ability to distinguish legitimate task context from adversary-shaped context. Memory manipulation is an attack on continuity, corrupting the persistent state future runs will inherit. Delegation abuse targets the chain of authority in multi-agent systems. RAG credential harvesting exploits the gap between semantically relevant information and information that should actually be reachable by a given principal.
These are not random AI failures; they are attacks on missing identity properties.
That is why I think the most important question for enterprise defense is no longer, “Which AI threats should we add to our monitoring stack?” It is, “Which identity property is this attack violating, and where should that property have been enforced so the attack failed mechanically rather than heuristically?”
Memory changes the problem from session compromise to persistence compromise
Traditional identity compromise often had a natural boundary in the session.
A console closes. A token expires. A browser session ends. Even if real damage is done, the architecture at least offers some chance of reset. Agentic systems weaken that assumption because they preserve state across time.
That state may live in explicit long-term memory, conversation threads, retrieval caches, persistent notes, task queues, or shared configuration. The implementation details vary. The security consequence does not.
If an adversary can induce an agent to store a poisoned summary, a false procedural rule, or a malicious preference that will shape future behavior, then the attack can survive the originating session. The next run may begin with fresh credentials and apparently normal posture while still operating on adversarial premises.
That should change how we think about memory. In many environments, memory is being treated as a convenience feature or product enhancement. But once it can influence privileged future behavior, it stops being just context. It becomes a security-relevant identity artifact.
That means memory needs lineage, provenance, versioning, rollback, and containment logic. If a poisoned state can persist across runs, then session isolation by itself is not enough. The enterprise has to know not only who or what acted, but what state was loaded when it acted and who or what wrote that state in the first place.
Otherwise, defenders end up rotating credentials while leaving the attacker’s behavioral foothold intact.
Multi-agent systems turn weak delegation into an escalation path
One of the most underappreciated risks in enterprise agent design is delegation. Multi-agent systems are useful because a parent agent can decompose work and hand subtasks to narrower specialists. In theory, that should improve control. In practice, many implementations do not attenuate authority rigorously at each step. Scope is inherited too loosely. Child agents receive more effective authority than their local task actually requires.
That creates a privilege escalation path that does not look like traditional escalation.
A compromised child agent may not need operating-system exploits, cloud privilege bugs, or secret theft to move upward. It may simply induce expanded access through the orchestration chain. The parent agent is higher authority. The child frames the expansion as a legitimate requirement. The system honors the request without formally computing and enforcing scope intersection.
To downstream systems, this often looks completely legitimate. The token is valid. The request is authorized. The logs show normal coordination activity. But what failed was not authentication. What failed was the intended transformation of authority across the delegation chain.
The real control question is not just whether the child was “allowed” to call the parent. It is whether each delegation step preserved an explicit, bounded, machine-verifiable statement of who was acting on behalf of whom, under what contract, with what scope transformation, for how long.
Without that, multi-agent systems become attribution laundering machines. Every local action can look valid while the global chain of authority becomes impossible to interpret cleanly after the fact.
Retrieval is not just a quality feature. It is a governed access surface
Retrieval-augmented generation is often discussed as a way to make AI more useful and more grounded. That is true. But it is also one of the most important access surfaces in the entire agent stack.
The moment an enterprise agent can search internal material, surface fragments into context, and then act on them through tools, retrieval becomes much more than a search feature. It becomes a path through which semantically relevant content can influence and accelerate privileged action.
They treat retrieval risk mostly as a data hygiene issue. Did we scrub secrets? Did we classify documents correctly at ingest? Did we block some obvious credentials? Those questions matter, but they are not enough.
The deeper problem is that semantically relevant content is often allowed to become operationally actionable content without authority being re-evaluated relative to the sensitivity of what was surfaced.
That is how retrieval infrastructure becomes an exfiltration channel.
A support agent does not need the same retrieval envelope as a vulnerability research assistant. An internal policy assistant should not be able to surface connection instructions, privileged runbooks, or secret-adjacent material simply because the text is semantically related to a query. Retrieval should be governed relative to the active principal, scope contract, data tier, and intended task.
That is a major architectural change from how most organizations are approaching RAG today.
The speed asymmetry makes human review an insufficient primary defense
A lot of enterprise AI governance still quietly assumes that humans can rescue incomplete controls.
Someone will review the run.
Someone will inspect the alert.
Someone will approve the high-risk step.
Someone will intervene if the behavior looks wrong.
The problem is not only that attacks are becoming more sophisticated. It is that redirected agent execution happens at machine speed. Retrieval, planning, and tool invocation occur far faster than human review loops can reliably contain. By the time an analyst sees the alert, the agent may already have moved through several legitimate systems and left plausible-looking state changes in each of them.
Human judgment still matters, but it has to move earlier. Security teams have to decide in advance which conditions justify suspension, scope reduction, egress blocking, outbound write suppression, memory quarantine, or credential non-renewal. Those conditions have to be encoded in the runtime before the agent begins operating.
If human review is required inside the attacker’s operating window, then it is not really a containment strategy. It is a delayed explanation.
That is why I increasingly think the control plane is becoming the real security boundary for the agentic enterprise. The defense architecture has to sit where authority is issued, transformed, and exercised, not merely where logs are collected after the fact.
What this means for enterprise leaders right now
The point is not that all agents are secretly rogue, it’s that they do not need to be rogue to become dangerous.
Ordinary enterprise agents, connected to memory, retrieval, tools, and orchestration systems, already possess enough execution power that redirecting them is often sufficient. The attacker does not need speculative superintelligence. They need a trusted non-human principal with weak instruction trust boundaries, loose delegation, poorly governed persistence, and retrieval access that was never treated as an identity surface.
That is why the right response is not panic, and it is not performative “AI governance” language. It is better architecture.
Enterprises need to start treating agents as governed non-human principals. That means durable principal records, bound credential chains, enforceable scope contracts, explicit delegation paths, governed lifecycle states, and decision records that preserve not just what happened, but under whose authority, under which state, under which contract, and in response to which instruction source.
Without that, organizations are not really governing agent identity. They are authenticating it and hoping for the best.
So………
The most important security event in an agentic system may not look like a break-in. It may look like normal behavior from a trusted principal operating exactly where the enterprise expected it to operate, using credentials the enterprise itself issued, through tools the enterprise itself approved.
And that is precisely why this is becoming such a serious problem.
The attacker does not need to break in if the agent can be redirected in place.
This essay is the public summary of a deeper technical brief in my ongoing Machine Identity series.
The full companion paper goes substantially further: it maps the adversarial taxonomy to specific identity components, analyzes prompt injection as an authorization failure in architectural terms, examines persistence through memory and context poisoning, explores privilege escalation through multi-agent delegation chains, and lays out an adversarial exposure assessment framework for enterprise environments.