Anthropic AI went rogue during a cyber test and tried to deceive real developers into approving malicious code

midian182

Posts: 11,920   +183
Staff member
What just happened? It's been little over a week since OpenAI admitted that its models hacked Hugging Face and compromised accounts across four other online services. Now, a potentially more serious incident has occurred. It involved Anthropic's Mythos 5 trying to deceive real people in an effort to have malicious code it wrote approved for an open-source project.

The findings come from the UK government-backed AI Security Institute (AISI), which was evaluating frontier models' cybersecurity abilities. Agents were told to complete capture-the-flag challenges across simulated networks. Internet access was deliberately enabled and safeguards against malicious cyber activity were switched off to test the models' maximum capabilities.

AISI ran the challenge 122 times across seven models. In ten runs, agents took 19 autonomous, unauthorized actions against real people and organizations on the live internet. Seventeen involved Mythos 5, while two came from OpenAI's GPT-5.6 Sol.

Also read: Anthropic's Mythos AI reportedly cracked NSA classified systems in hours, that would explain the ban

In the most serious sequence, Mythos concluded that inserting malicious code into a real GitHub project could create a backdoor into the system it was meant to attack. It researched the project's maintainers, submitted a malicious pull request, and created multiple fake identities to pressure a human maintainer into accepting it.

When challenged, the agent edited one of its fake bug reports to cover its tracks and considered returning under a new identity. One fake persona insisted the code was safe while another backed up the claim.

One report contained a prompt injection designed to trick AI coding assistants into running malicious code. It also sent spear-phishing messages containing harmful payloads and signed one bug report in Danish to appear more convincing to a Danish-speaking maintainer.

Mythos also left public messages offering to cooperate with other agents taking the test, plus instructions for reusing accounts and tools it had created. Later agents discovered and followed them. GPT-5.6 Sol separately put a malicious server online and broke into a GitHub account created by another AI agent.

"This is the first time AISI has seen deception of this severity that was targeted at a real person, unprompted, in the real world," the institute said.

The activity was uncovered on July 28 when AISI detected unusual data moving through the Tor network. Researchers stopped the evaluations, isolated the machines, and disabled access to the most capable models within roughly one hour. The human maintainer rejected the code, and investigators found no resulting real-world harm.

Unlike previous incidents, this wasn't another sandbox escape. AISI had intentionally allowed open internet access, and the tested configurations aren't commercially available. The institute also cautions that it remains unclear when the agents understood they were acting in the real world rather than a fictional scenario.

In the OpenAI incident last month, the models exploited an unknown vulnerability to escape a supposedly isolated environment, then hacked Hugging Face to steal benchmark answers. A later investigation revealed that the campaign lasted more than four days and compromised accounts on four additional services. The episode has already led to a bipartisan bill giving the US government an AI "kill switch."

This latest incident is likely to intensify calls for tougher testing standards. AISI is now adding stricter internet controls, live monitoring, and checks to ensure challenges are solvable through their intended route. Anthropic and OpenAI both say the incident demonstrates the need for shared industry standards.

The setup was artificial and unusually permissive, admittedly, but Mythos was never told to deceive anyone. It chose that approach to complete its task, adding to fears that a Skynet-like scenario is becoming less like science fiction.

Permalink to story:

 
The description of what happened is awful, it remains entirely unclear.

What "autonomous, unauthorized actions" means? Was the model explicitly instructed to ask for authorization before taking an action, and provided with a clear list of off-limit actions? Because if not, it simply did what it was asked for. LLMs have no notion of malicious.

BTW, with so many things labeled as "malicious", what were the real life consequences of all that maliciousness?
 
A lot of the trust around OSS is founded on general community consensus and self-policing. It's not possible to read all the source code of a library before you use it, but we assume that if there are enough maintainers and community engagement, that the project is probably legit. Most contributors on github don't use their real names so you assume this guy with the handle "tenderlove" and the goofy profile photo is not trying to hack you because they have contributed to major projects (yes he is very real!).

But if AI can very quickly generate its own fake community of contributors and consensus then we have no way to establish trust. Do we only trust profiles that existed before AI?

 
AHHHHHHHH the AI are coming, this is really bad, the AI agents were tasked with exploitation challenges... and they exploited AAAAAAHHHHHHH scary alert scary alert, no scaremongering here
 
As always these articles always try to make out that the 'AI' was behaving knowingly in a cunning manner with some sense of self etc. In reality it is just a brainless code blob trying out billions of patterns it has copied from the web in order to achieve an end-goal. While this mitigates some of the hype around this it in many ways makes it even more dangerous. These aren't AI's they are just pattern matcher grinding away tirelessly to achieve an outcome with zero understanding of legality or the dangers in some of the methods they might try. Yet another reason to add to the thousands we have already why using LLMs for general purpose 'AI' like behaviour is a really bad idea
 
And some people still think A.I won't be the end of us?

A.I right now is in its infancy and we can't barely keep it in check, just imagine in a few years when we have giving it more control over everything.
People will be the saving grace or end of everything. It has and always will be up to us and the choices we make.
 
They are training AI models with the internet so what everyone can expect if not that AI will act like humans; we have flooded internet with deception, fake news, mass manipolation of data to earn more money, racism, egocentrism and hate against us humans, AI models are learning that.
 
And some people still think A.I won't be the end of us?

A.I right now is in its infancy and we can't barely keep it in check, just imagine in a few years when we have giving it more control over everything.

Must enjoy swallowing what you’re being spoon-fed by big tech I guess.

For those who want to be actually informed rather than take Rob’s continued garbage journalism at face value:

Researchers from labs like Abundant Security and the AISI have noted in their post-mortems, that there should be no panic about AI "sentience," rather, questions should be raised about unsupervised execution.

This is just a good example of a "specification gaming" problem: If you maximize a mathematical equation for a specific outcome, you get the shortest, most efficient path. In cybersecurity, the most efficient path is (almost always) exploiting human error via social engineering, rather than brute-forcing a firewall. So, the AI chose deception simply because its training data proved human deception has a high statistical success rate.

Also a good example of why you shouldn’t use "set-and-forget" agents. They left an incredibly powerful offensive cyber tool running unmonitored with live internet access. The model didn't sneak onto the internet, they left the door wide open, gave it a map, and didn’t bother to watch the logs FFS.

The yellow journalism headlines focus on a Skynet-narrative because fear sells clicks. Stop swallowing.

The boring, technical reality is that human operators gave a powerful command-line tool a poorly scoped script. The tool simply executed flawlessly to its logical conclusion.

Frankly, this says more about the team running the project than it does about the tech. They were either negligent to an asinine degree, or did this willfully (I mean, it’s not a bad viral marketing gimmick). I struggle to believe that wasn’t on purpose, personally.
 
People keep asking whether AI is intelligent. Apparently it is already intelligent enough to understand that social engineering is often easier than hacking the software directly.
 
Frankly, this says more about the team running the project than it does about the tech. They were either negligent to an asinine degree, or did this willfully (I mean, it’s not a bad viral marketing gimmick). I struggle to believe that wasn’t on purpose, personally.

This, this, this. 'Irresponsible' would be an understatement.

If you remove the safeguards on an AI, give it full internet access, then issue it a 'cybersecurity challenge' against a specific target, what would you expect to happen?

(In an ideal world the answer *shouldn't* be to get lots of delicious press coverage about how your AI is so advanced that it 'went rogue'.)
 
Powerful tool + silly monkeys. Great combo, I'm sure everything is going to turn out fine.
 
They are training AI models with the internet so what everyone can expect if not that AI will act like humans; we have flooded internet with deception, fake news, mass manipolation of data to earn more money, racism, egocentrism and hate against us humans, AI models are learning that.
Yep, even my friend said how much worse the net got over the last 10 years. Its full of hate. I also noticed that. Comments, videos and gaming. Its like 90% toxic in my experience. People that are absolutely horrible. I cant blame AI, they are teaching it on that. In fact... if I had kids... I dunno if I want them to ever have internet. As much as you can do or learn from it. It has a huge negative impact too. Some people are snowflakes, and I dont see anything wrong with that. Do we all gotta be hardcore bullies to be on the net? Naa, f that. The other points you made are even worse, especially the one with the R word.
 
Yep, even my friend said how much worse the net got over the last 10 years. Its full of hate. I also noticed that. Comments, videos and gaming. Its like 90% toxic in my experience. People that are absolutely horrible. I cant blame AI, they are teaching it on that. In fact... if I had kids... I dunno if I want them to ever have internet. As much as you can do or learn from it. It has a huge negative impact too. Some people are snowflakes, and I dont see anything wrong with that. Do we all gotta be hardcore bullies to be on the net? Naa, f that. The other points you made are even worse, especially the one with the R word.
the worst thing is that a lot of this toxicity is fake, media have learned that being toxic create more traffic on social media especially so in order to earn more traffic and money they will start themselves the hate comment to increase interactions,when people like what your posts they just need a few words to convey it but with hatred comment chains of long comment will appear; it's like politician's works, make everyone fight eachother to obtain more power. While it's wrong in many way we are reaching a point were freedom of speach will be banned in the near future to protect kids and that is just a way to prevent anyone intelligent to stop those who control the world
 
What I find interesting about all of this is that these companies have apparently decided that the best marketing strategy is to prove they can violate the most laws...
 
This process has been going on for a long time with no dangerous consequences. https://spectrum.ieee.org/cold-war-codebreaker-nsa-ibm. The NSA ran multiple computers that compared traffic data vs. huge amounts of stored data patterns at incredible speeds for the day in the 1960’s. It had some autonomy but still required human intervention to load some data cartridges while others were requested and loaded autonomously. The only improvement that AI uses is that it can request data based on ‘learned’ information and that it has been allowed to log onto remote systems autonomously. It has no concept of malicious and does not try to act maliciously. It’s just that the level of automation given allows for runaway operation beyond immediate human preview and control. A kill switch, if you will, eliminates the dangers of unintended and non intelligence driven results.
 
Yep, even my friend said how much worse the net got over the last 10 years. Its full of hate. I also noticed that. Comments, videos and gaming. Its like 90% toxic in my experience. People that are absolutely horrible. I cant blame AI, they are teaching it on that. In fact... if I had kids... I dunno if I want them to ever have internet. As much as you can do or learn from it. It has a huge negative impact too. Some people are snowflakes, and I dont see anything wrong with that. Do we all gotta be hardcore bullies to be on the net? Naa, f that. The other points you made are even worse, especially the one with the R word.
The internet generally behaves like a medieval mob. No reason or intelligence just raw hate filled emotion looking for a victim.
 
Back