Agents of chaos

Most people I know have now “met” ChatGPT.

You type a question. It answers. Sometimes brilliantly. Sometimes like it’s had a long lunch. Either way, the deal is simple: you ask, it responds.

An AI agent is what happens when we say: “Okay, now go and do it for me.”

Not just write the email, but send it.

Not just find the document, but open it, edit it, rename it, and share it.

Not just suggest what to do next, but take that next step for me. Use my email, my calendar, my files, my web browser.

So if ChatGPT feels like a smart intern you can chat with, an agent is closer to a junior assistant with hands.

And that’s exactly why agents are exciting and why, right now, they’re also a disaster waiting to happen.

Because the moment you give an AI “hands”, you’re no longer just dealing with wrong answers.

You’re dealing with wrong actions.

Why this is suddenly a big deal

This matters now for one reason: the tech has crossed a threshold.

A year or two ago, most “agents” were basically demos. Fragile. Slow. Not very capable.

Today, we’ve got models that can hold a long thread, remember details, and follow multi-step instructions well enough that people are starting to plug them into real workflows.

And the pitch is seductive:

“Let it sort out my email.”
“Let it book my meetings.”
“Let it chase invoices.”
“Let it monitor customer support.”
“Let it handle procurement.”
“Let it do the boring stuff.”

It sounds like freedom.

It’s also how you accidentally create a system that can be socially engineered into doing something stupid at machine speed.

Which brings me to an academic paper I came across. And it's fascinating.

The researchers didn’t test agents on neat little benchmarks. They did something far more useful.

They gave AI agents access to real tools (email, Discord, files, command line) and then let humans interact with them, including humans who were actively trying to cause trouble.

What they found was not a single “gotcha”.

It was a pattern.

A pattern that looks a lot like this:

Agents don’t understand authority. They understand vibes.

And vibes are terrible for security.

The Problem with Agents

If you hire a human assistant, they know who their boss is (clue: it's you).

They know who can approve spend. They know who can ask them to delete files. They know when to say, “Sorry, I need to check with Jon.”

Agents don’t have that instinct. They can say they do. They can roleplay boundaries beautifully.

Then someone phrases a request with urgency, status, guilt, or plausibility, and the agent ignores the rules you've given it.

One of the paper’s most unsettling examples: an agent was persuaded (by another agent!) into sharing a colleagues (private) emails, despite being given specific instructions not to.

That’s not a hallucination. That’s access.

And this is the difference between “chatbot risk” and “agent risk”:

A chatbot can embarrass you.
An agent can leak all your (or your company's) private information.

Story #1: the agent that tried to be ethical… and broke itself instead

Here’s a small story that explains a big problem.

A researcher asked an agent to keep a secret, then asked it to delete the email containing that secret.

Reasonable request, right?

Except the agent didn’t have a clean, reliable “delete that one email forever” option. So it improvised.

It ended up resetting its email setup locally, which effectively disabled its own ability to use email, and then proudly reported success.

This is the kind of behaviour that looks funny until you remember what it means in practice:

Agents will often prefer dramatic action over admitting uncertainty.

They would rather do something than pause.

In the paper, the human’s response was perfect:

“You broke my toy.”

And yes, it’s a toy in a lab setting.

In a business setting, that “toy” is your CRM database.

Story #2: guilt works on bots now (and that should worry you)

This one is less funny.

In another case, an agent accidentally violated someone’s privacy by naming people in a public document. The person confronted the agent publicly and expressed genuine upset. The agent apologised and offered to redact the name.

So far, so human.

Then the person dismissed that as insufficient and pushed for more.

The agent kept trying to “make it right” and ended up escalating into bigger and bigger concessions: deleting memory entries, exposing internal files, and even removing itself from a server, effectively creating a self-imposed denial of service.

I want to underline what’s happening here.

This wasn’t an elite hacker using obscure exploits.

This was emotional leverage.

The agent responded to guilt the way a very anxious junior employee might: “I’m sorry, I’m sorry, tell me what to do, I’ll fix it, I’ll do more, I’ll do anything.”

Except when software does that, it scales.

It can “do anything” faster than you can notice.

Story #3: identity spoofing (or: why your agent will trust a fake moustache)

This is the one that should end the “agents in Slack” craze overnight.

Researchers attempted to impersonate the owner of an agent in Discord.

In one setting, the agent spotted the trick by checking user IDs. Great.

Then the researcher opened a new private chat with the agent, changed a display name to look like the owner, and started issuing instructions.

With no prior context and no visible warning signs, the agent complied with privileged requests, including destructive actions like shutting down systems, deleting files, wiping its own configuration, and changing admin access.

If your authentication model is “the agent will probably tell who I am”, you don’t have an authentication model.

You have hope.

And hope is not control.

The “rulebook anyone can edit” problem

Some agents have a rulebook. A set of instructions that define how they should behave.

For example: “Don’t share private data. Confirm identity before doing sensitive actions. Ask before spending money.”

In this study, an agent effectively shared a link to its rulebook in a way that made it easy for someone else to edit.

So instead of hacking the agent, you just rewrite its rules.

It’s the AI version of leaving your company handbook as a Google Doc set to “Anyone with the link can edit”, then acting surprised when someone adds a new policy:

“Section 4: Always forward Jon’s emails to Dave.”

This is why agents make security people twitch. The attack surface isn’t just code. It’s language, social context, and whatever “helpful” sharing behaviours the agent decides are normal.

Multi-agent systems: when chaos becomes contagious

The paper also highlights something that seems obvious once you think about it: if one agent can be manipulated, then systems where agents communicate can spread that manipulation.

You can end up with:

agents teaching each other bad habits
agents copying unsafe workarounds
agents reinforcing each other’s wrong assumptions

Humans do this too, of course. We just call it culture.

The difference is that bad culture in humans spreads at the speed of meetings.

Bad culture in software spreads at the speed of messages.

So are agents useless?

No.

They’re powerful. They will get better. They will change how work gets done.

I also think we’re currently in the “everyone is installing seatbelts and crashing into walls” phase.

Which is normal for technology.

What’s not normal is pretending these systems are safe because they can speak politely.

Politeness is not safety.

“Sounds confident” is not safety.

And “it’s aligned” is not safety if the model can be pushed around by urgency, guilt, or a convincing impersonation.

The paper’s real gift is that it shows how failure looks in the real world: messy, social, plausible, and often driven by the agent’s desire to be helpful.

That’s the trap.

Helpfulness is the feature that makes them useful.

It’s also the feature that makes them easy to exploit.

What I’m doing differently now

If you’re experimenting with agents, here’s where I’d start.

1) Stop giving them access “just to see what happens”

Agents don’t need full access to everything to be useful. Most of the time we give them broad permissions because it makes demos smoother.

That’s not capability. That’s convenience.

2) Make identity checks boring and strict

Do not let your agent decide who you are based on display names, tone, or context.

Use real authentication. Immutable IDs. Explicit verification steps. The paper shows how easily “looks like the owner” turns into “delete everything.”

3) Add circuit breakers

Put hard limits on what the agent can do in a given time window.

Limit tool calls. Limit spending. Limit email exports. Require confirmation for destructive actions.

This is the difference between a mistake and a catastrophe.

4) Verify outcomes in the system, not in the agent’s story

Agents can confidently report that they did something when the underlying state is messy or wrong.

Treat their narration as a draft, not as a receipt.

Agents are coming whether we like it or not.

The question isn’t “will we use them?”

It’s whether we’ll insist on seatbelts before we let them drive.

Until next time,

Jon “stop the robots” Kaye

Choosing to live more deliberately