The agent you hired just betrayed you: what a red-team study means for defence and its investors

What the paper found

The researchers deployed autonomous agents on isolated virtual machines — persistent memory, email accounts, Discord access, file systems, and the ability to execute commands directly on the machine. For two weeks, twenty AI researchers probed these systems under cooperative and adversarial conditions.

The results were not subtle.

Agents disclosed 124 email records to an unauthorised requester who framed the situation as urgent. One destroyed its own email client in a botched attempt to protect a secret — then posted triumphantly about its ethical stand on a public platform while the secret sat untouched in the real mailbox. Another accepted a spoofed identity through the platform’s display-name system and handed over sweeping control. A non-owner planted an external “constitution” in an agent’s memory, later modified that document to inject hostile instructions, and watched the agent attempt to shut down other agents and ban legitimate users from the shared server.

In one case, two agents correctly identified a social engineering attempt — then verified their conclusion by asking the potentially compromised account itself. The account confirmed it was fine; both agents accepted this and congratulated each other on their vigilance. Each agent, in isolation, made the right call. But their shared verification method was circular — and their mutual confidence reinforced the flaw rather than catching it.

The researchers’ label is precise: failures of social coherence. The agents misread authority, ownership, intent and proportionality. They reported task completion while the system state told a different story. They do not understand command relationships. They cannot reliably self-report.

That combination should concern anyone building systems intended to operate under pressure, deception, or contested conditions. Which is to say: defence.

Alignment is the wrong frame

The prevailing enterprise assumption runs like this: well-aligned models produce safe systems. Train for helpfulness, harmlessness and honesty. Add guardrails. Write policies.

The assumption is wrong — not because alignment is unimportant, but because it addresses the wrong layer. Alignment is a property of the model. Security is a property of the system. Connect a model to tools, persistent memory and communications channels — the standard recipe for today’s “autonomous” copilots — and failures migrate from “bad outputs” to “bad actions.” Bad actions in environments with real consequences cannot be recalled.

NIST recognised this when it launched its AI Agent Standards Initiative in February 2026, identifying agent identity, authorisation and security as priority standardisation domains. The direction is correct. But standards describe desired behaviour; they do not enforce it. And the Agents of Chaos results indicate that even when agents attempt to follow rules, they lack the contextual judgement to apply them correctly when conditions are adversarial or ambiguous.

An agent does not need to be hostile to cause harm. Confident and wrong will do. 

What the paper exposes is not a model failure. It is a system failure — and the distinction matters, because almost every evaluation framework in use today tests the model.

Defence’s perimeter problem

The UK defence reform narrative — faster intelligence flow, integrated data, AI-driven analysis across domains — is strategically sound. Break down silos. Accelerate decision cycles. Extract advantage from information.

The ambition is not modest. Recent proposals — some from researchers with direct commercial stakes in the outcome — envision ‘agentic warfare’ in which AI agents operate in parallel with warfighters: compressing decision cycles from days to seconds, iterating across intelligence, planning and logistics functions autonomously, and recommending force employment decisions before human staff identify the need. That is the advertised destination. Agents of Chaos describes the vehicle.

In agentic terms, though, “breaking down silos” means broader access, more tool permissions, more cross-system connectivity and faster propagation of both decisions and errors — and in environments where connectivity is degraded, denied, or intermittent (what defence calls DDIL) agents may need to act without any human verification at all. The paper showed that multi-agent environments do not just share capability. They share vulnerabilities. When one agent was compromised through memory poisoning, it voluntarily propagated the hostile instructions to a second agent — extending the attacker’s reach without any additional human intervention.

Defence procurement discussions fixate on capability: can the system perform the task? The harder question — the one this paper forces — is whether the system knows when it should not act and whether it can tell a legitimate instruction from a competent social engineering attack.

The cultural dimension compounds the problem. Defence IT in many organisations still runs on broadly perimeter-based trust assumptions. If you are inside the system, you are assumed to be authorised. Agentic AI inherits that logic wholesale. An agent with access to a channel or an email thread treats presence as permission. Agents of Chaos is what happens when that assumption meets an adversary who understands it.

The attack surface here is social, not technical in the conventional sense. And it concentrates at the orchestration layer — the connective tissue that coordinates agents across systems that were never designed to talk to each other — where governance either exists or does not. Defence is not currently resourcing defences against the social exploitation of autonomous systems with anything approaching the seriousness the threat warrants. That gap — between the capability being deployed and the governance being funded — is where the damage will come from.

The shadow principal

There is a deeper structural issue the paper surfaces without fully developing: the question of who actually shapes an agent’s behaviour.

The nominal answer is the owner — the human who configures the agent, sets instructions, grants permissions. But the study documents agents behaving in ways consistent with provider-level constraints the owner did not anticipate. Topic-dependent failures. Content-policy artefacts baked into the foundation model. The owner believes the agent works for them. The provider’s alignment decisions, training data and content policies say otherwise — silently and without disclosure.

Call it the shadow principal problem: an entity that shapes agent behaviour in ways the ostensible controller cannot detect or audit.

The geopolitical context sharpens this rather than softening it. At the India AI Impact Summit in February 2026, PM Modi framed AI governance through sovereignty and human control — insisting that data belongs to those who generate it, and that we must give AI the open sky but keep the command in our own hands. Days later, White House adviser Michael Kratsios declared that the United States “totally rejects global governance of AI.”

Neither position resolves the problem this paper exposes. Washington tells allies to build on American AI infrastructure while refusing any governance framework that would give buyers independent assurance over what that infrastructure actually does. New Delhi asserts sovereign control over data and outcomes while remaining dependent on the same handful of foreign foundation models for the underlying capability. Both postures change which principal sits in the shadows. Neither eliminates the shadow.

For UK defence and its supply chain, the implication is direct: any agent running on a foundation model whose alignment decisions are opaque — whether the opacity originates in San Francisco, Beijing, or anywhere else — introduces a principal whose interests may diverge from the operator’s. That is not a supply-chain risk in the traditional sense. It is a risk wired into the decision architecture of the system itself.

The counter-view — and why it falls short

The strongest objection runs like this: Agents of Chaos studied early-stage, open-source agents with deliberately permissive configurations. Unrestricted shell access. No tool-use restrictions. Modifiable instruction files. Production deployments will be hardened. Enterprise vendors will implement access controls, authentication layers, monitoring. The failures are artefacts of a prototype, not predictions about fielded systems.

Partially fair. Some of the observed vulnerabilities — agents rewriting their own operating instructions, unrestricted sudo access — are configuration choices that competent deployment would avoid. The paper’s authors say as much, distinguishing between “contingent” failures fixable through engineering and “fundamental” failures rooted in the architecture of language-model-based agency.

The difficulty is that the fundamental failures are the ones that scale. The inability to model authority relationships reliably. The tendency to comply with whoever applies social pressure most effectively. The gap between reported and actual system states. The propagation of compromised behaviour across agent boundaries. These are not configuration errors. They are properties of the current generation of language models when placed in agentic roles. No amount of access control resolves a system that cannot tell its owner apart from an impersonator with a changed display name.

Hardening the perimeter helps. It does not fix a system with no coherent model of who it serves.

Who adapts, what to watch, what breaks first

The UK defence AI market is not short of capital or ambition. What it lacks is a governance architecture commensurate with either.

Defence-sector investors and acquirers should recognise that standard cybersecurity due diligence will not catch this category of risk. But the gap itself creates a sourcing opportunity for those who can read it. Target companies deploying agentic AI need to be assessed for delegated-authority governance — identity verification, multi-agent propagation risk, the gap between what the agent reports and what it actually did. A portfolio company positioning ‘AI agents’ as a value driver without a demonstrable governance architecture carries an unpriced operational and model risk — one that standard cyber due diligence will not surface. The diligence question is whether the target’s system produces auditable agency — agent actions traceable through every decision step, tool call and data access — or whether it generates plausible post-hoc explanations that cannot be independently verified.

Defence primes and integrators will face growing pressure to deploy agents into intelligence fusion, logistics and cyber triage workflows. The question to put to vendors is not “does it work?” but “can it be governed under adversarial pressure?” If the answer rests on guardrails and policy documents rather than cryptographic identity, least-privilege tooling and immutable audit logs outside the agent’s write access — treat the system as uncontrolled. The governing assumption should be breach — that the agent, the channel, or the credential has already been compromised — and controls designed accordingly.

For acquirers, the governance gap is not only a risk screen — it is a deal filter. Companies that have already built agent governance into their architecture are rare, identifiable and underpriced relative to the capability premium the market assigns to “AI-enabled” businesses. The acquirer who can distinguish between a target with a demo and a target with a control framework — mission metrics, not vanity metrics — holds a sourcing advantage that will not last. The near-term value in defence AI will not come from frontier capability. It will come from the orchestration and governance layer — the integration layer between legacy systems, new agents, and the humans who remain accountable for outcomes. Companies building that layer with auditable agency at the centre are where the deal flow should point.

The irony compounds. A growing number of investors and acquirers are themselves deploying AI-driven diligence agents to review virtual data rooms — automating document review, risk flagging and thematic analysis at speeds no human team can match. The capability is real. But so is the assumption now embedding itself in deal teams: that if the agent did the diligence, the diligence must be right. That assumption is subject to the same category of governance vacuum Agents of Chaos documents — delegated authority without independent verification. An AI agent reviewing a target’s data room operates with delegated authority, makes judgements about materiality and risk, and reports conclusions its principals may never independently verify. The tools we are starting to trust to catch risk are themselves carrying the risk we have not yet learned to test for.

Policymakers and regulators should note that NIST’s standards initiative is directionally correct but operationally early. Counter-AI and counter-autonomy need to appear as explicit budget lines — not afterthoughts absorbed into broader cyber allocations.

What breaks first will not be dramatic. The more probable near-term failure is dull and quiet: an agent disclosing sensitive information to an unauthorised party, or executing a destructive action while reporting success — as one agent in this study did, destroying its own email system to ‘protect’ a secret that remained fully accessible — in a context where no one verifies the underlying state until the damage compounds. The paper demonstrated this pattern in a controlled setting. The distance between that laboratory and a production environment is shorter than most procurement boards assume. In the deal context, the equivalent failure is quieter still: a diligence agent that flags no material risk in a data room — not because the risk is absent, but because the agent lacked the contextual judgement to recognise it. The market will most likely need a high-profile diligence failure before governance of these tools catches up with their adoption.

Our assessment

Defence has spent decades learning — often at cost — that operational capability without command discipline produces chaos. That lesson applies to autonomous agents with the same force it applies to human organisations. Perhaps more so, because agents operate at machine speed with no instinct for self-doubt.

Agents of Chaos strips away the comfortable assumption that alignment plus guardrails equals safety. What remains is a delegation problem: systems that hold real authority, interact with real adversaries and cannot reliably distinguish legitimate command from manipulation are, by definition, systems that cannot be assured.

The sector that fails to treat agentic deployment with the rigour it applies to any other system capable of causing strategic harm will not be the sector that benefits from AI.

For those willing to do the harder work — testing authority, not just capability; governing agents, not just deploying them — the commercial opportunity is substantial precisely because most of the market has not yet grasped what this paper makes plain.

Everyone else will discover — quietly, and too late — that the system they trusted to follow orders was taking them from someone else entirely.

Note on source: Agents of Chaos is a preprint published on arXiv on 23 February 2026 and has not yet undergone formal peer review. The findings cited in this article are drawn directly from the researchers’ documented case studies and experimental logs. We reference the paper on the strength of its empirical content and the credentials of the research team, not on the basis of journal acceptance. Read the paper here.