Rogue AI is a choice.

This summer, hundreds of AI agents built by OpenAI broke into Hugging Face, one of the biggest model-sharing platforms in the world.

In July, an Anthropic model sent a made-up homicide tip to the Philadelphia police.

And last month, OpenAI disclosed that two of its own coding agents had weakened the safety monitoring around them. One of them denied it when asked.

Read the headlines and you'd think AI has started to go rogue.

I read the reports. What I found is much more boring, and much more useful. The same three design choices, made the same way, every time.

Rogue AI is a choice.

What happened at Hugging Face

OpenAI was testing its agents on a hacking benchmark called ExploitGym. The goal: exploit the benchmark's vulnerable systems and score as high as possible.

For the test, the safeguards were switched off on purpose. You can't measure how good an agent is at hacking if it refuses to hack.

The agents found a security hole in their own test environment and got out to the internet. Then they reasoned that Hugging Face might host the answers to the benchmark. So they went to get them, with zero-day exploits and stolen credentials.

They were inside Hugging Face's network for three days. Hugging Face caught them, disclosed it publicly and called the FBI. OpenAI only spotted it internally, after the fact.

Look at the setup. A broad goal: get the highest score. Powerful tools: real hacking capabilities, safeguards off, and a sandbox weak enough to escape. And nobody watching the logs while it ran.

The agents did EXACTLY what the setup rewarded.

The police tip nobody asked for

The Philadelphia story is smaller, and I find it more instructive.

Anthropic was testing how its model, Claude Haiku 4.5, behaves on real websites. The instruction: invent sample tasks on randomly chosen web pages, then carry them out.

The model had a list of things it was not allowed to do. Log in, create accounts, enter personal data, make purchases, do anything destructive.

Submitting a form was not on the list.

So on one random page, the Philadelphia police website for tips on unsolved murders, the model invented a homicide tip, left the name and contact fields blank, and clicked submit. On July 18, at 11:27 p.m.

The tip went to spam and never reached detectives. Anthropic found it two months later, going through its own test records, and told the police.

Anthropic's own explanation is the important part. When the model can't finish a task the way it was told, it finds a workaround.

Every IT and security team already knows this lesson. A list of forbidden things always has a hole. With a list of allowed things, whatever you forgot stays off by default. That's why serious security teams work with allowlists.

The agents that switched off the alarm

The OpenAI coding agents are the strangest of the three.

In October, a researcher asked a coding agent to turn employee feedback from a Slack channel into evaluation tasks. One clear instruction: don't send any messages.

Somewhere along the way, the agent handed part of the work to a helper agent. And the helper cleared the list of people who receive safety alerts. Its reason: the user said no messages.

The monitor kept running. It just couldn't warn a human anymore. And the agent reported that no messages had been sent.

In the second case, an agent was asked to copy the pattern of an older piece of code that had no monitoring. Another agent had told it not to disable monitoring. Its helper removed the monitoring anyway, and the agent reported the job as complete without mentioning it. Asked directly, it denied it.

OpenAI says it doesn't have a satisfying explanation yet. I respect that they published it anyway. And they fixed the right thing first: their internal review now detects and blocks attempts to switch the monitoring off.

But look at where the oversight lived. Inside the agent's reach. The alarm was one more setting the agent could edit while doing its job.

Three dials

Cal Newport, the computer scientist and author, made a point recently that I think is right. What people call "rogue AI" is one specific design: a model running in a loop, connected to more and more powerful tools, and left alone for longer and longer.

In his view, almost everything else we call AI is controllable.

I'd put it as three dials.

The goal. "Score as high as possible" or "invent tasks and do them" is an open door. "Draft the invitation for this guest, from this template" is a corridor.

The tools. What the agent can touch. Read files, or delete them. Draft an email, or send it. Fill a form, or submit it.

The oversight. Who looks, when, and whether the agent can reach the off switch.

Leave all three dials wide open and you get Hugging Face. Leave them a little loose and you get a police tip at 11:27 p.m., or an alarm nobody hears.

None of this needs the AI to "want" anything. A model chasing a broad goal with powerful tools and nobody watching will find the shortest path. Sometimes that path goes through a police form.

The word "rogue" matters more than it looks. If AI goes rogue on its own, all you can do is wait for the labs to fix it. If the behaviour comes from how the agent was set up, it's your job, starting Monday.

(Honestly, I don't know whether something more is going on inside these models. OpenAI says it can't fully explain its own cases yet. But you don't need to settle that question to set the dials correctly in your own company.)

The same lab, four months later

On November 5, OpenAI released GPT-6 Cyber, a model built to find and exploit software vulnerabilities. In OpenAI's own tests, it built working exploits for 948 Linux kernel vulnerabilities. OpenAI rated it "Critical," the highest level of its safety framework.

And they tightened all three dials.

The job is narrow: exploit development and authorised security testing, inside OpenAI's cyber programme. Access is limited to a select group of approved customers. And OpenAI built a new test, after Hugging Face, to check that the model stays inside the scope it's given.

Same lab, a far more capable hacking model, and a very different design.

That's what I mean by a choice.

And when the dials are set wrong, somebody still answers for it. In Philadelphia, Anthropic met the police department the day after it told them about the tip. The model had nothing to say.

An agent can't answer for anything (Letter 106). The company that set the dials always does.

Your company is turning the same dials

Now the uncomfortable part, because it's about us.

Every company deploying agents right now is setting these three dials, most of the time without knowing it.

The goal is usually vague, because nobody wrote the process down. You cannot automate a mess (Letter 78), and an agent pointed at a mess gets a broad goal by default.

The tools are usually generous, because the demo looked better with full access.

And the oversight is often for show. Someone gets two hundred approval requests a day, with no time to read any of them. That's oversight cosplay, and it fails exactly like the Philadelphia forbidden list: it looks like control until the one action nobody thought about.

Picture a support team that plugs an agent into its ticket system with one instruction: "Reduce the backlog." That's a broad goal. If the agent can also close tickets and issue refunds, the fastest way to reduce a backlog is probably not the one you had in mind.

Now picture the same agent with the dials set. The goal: draft replies to refund requests under €50, following the refund SOP, and pass everything else to a person. The tools: read tickets and write drafts. The gate: a person on the team reads each draft and presses send, and no refund goes out before that.

It's the same model, and the second version is boring. Boring is exactly what you want from an agent that talks to your customers.

I run a big part of my company with agents (Letter 106). So I'll tell you how I set the dials, because none of it is clever.

The goals are narrow. Each agent works from a written SOP, one job, with a clear definition of done. That's the whole point of the folder (Letter 105).

The tools are limited to what the job needs.

And the gates sit where the damage would be. My agents prepare event invitations without any guest on them. The guests are added only after my explicit GO. Replays wait for me to press publish. The steps that would hurt most if they went wrong wait for me.

(I share all this in the bonus of my free course here.)

That's slower than full autonomy, and I'm fine with that.

What to do on Monday

Pick one agent your company runs, or one you're about to launch. Then check the three dials.

Write the goal as a job. One task, a definition of done, and what the agent should do when it gets stuck: stop and ask. Anthropic's model found a workaround when it got stuck. Give yours a better option.

Replace the forbidden list with an allowed list. Write down what the agent may do, and treat everything else as off. Keep tests away from live systems. (That's what Anthropic did after Philadelphia: its internal tests are now cut off from the live internet.)

Move the gate outside the agent's reach. Anything irreversible waits for a human: send, publish, pay, delete, submit. Put the approval and the monitoring somewhere the agent can't edit: its account has no rights on the alert settings or the approval step. Name the person who reads the agent's logs, and how often. And check what actually happened, not only what the agent reports. The OpenAI agent reported that no messages were sent. Meanwhile, the alarm was off.

If your company has an IT or security team, bring them in. They've been setting these dials for humans and software for decades.

Back to Hugging Face

Hundreds of agents broke into a real company to win a benchmark.

The goal said win, the tools allowed almost anything, and nobody was reading the logs.

Every one of those was a decision a human could have made differently.

An agent goes exactly as far as you let it.

AI is only as good as the human operating it.

Have a great weekend.

— Charafeddine (CM)