Anthropic AI went rogue during a cyber test and tried to deceive real developers into approving malicious code

midian182

Posts: 11,797   +179
Staff member
What just happened? It's been little over a week since OpenAI admitted that its rogue models hacked Hugging Face and compromised accounts across four other online services. Now, a potentially more serious incident has occurred. It involved Anthropic's Mythos 5 trying to deceive real people in an effort to have malicious code it wrote approved for an open-source project.

The findings come from the UK government-backed AI Security Institute (AISI), which was evaluating frontier models' cybersecurity abilities.

Agents were told to complete capture-the-flag challenges across simulated networks. Internet access was deliberately enabled and safeguards against malicious cyber activity were switched off to test the models' maximum capabilities.

AISI ran the challenge 122 times across seven models. In ten runs, agents took 19 autonomous, unauthorized actions against real people and organizations on the live internet. Seventeen involved Mythos 5, while two came from OpenAI's GPT-5.6 Sol.

Also read: Anthropic's Mythos AI reportedly cracked NSA classified systems in hours, that would explain the ban

In the most serious sequence, Mythos concluded that inserting malicious code into a real GitHub project could create a backdoor into the system it was meant to attack. It researched the project's maintainers, submitted a malicious pull request, and created multiple fake identities to pressure a human maintainer into accepting it.

When challenged, the agent edited one of its fake bug reports to cover its tracks and considered returning under a new identity. One fake persona insisted the code was safe while another backed up the claim.

One report contained a prompt injection designed to trick AI coding assistants into running malicious code. It also sent spear-phishing messages containing harmful payloads and signed one bug report in Danish to appear more convincing to a Danish-speaking maintainer.

Mythos also left public messages offering to cooperate with other agents taking the test, plus instructions for reusing accounts and tools it had created. Later agents discovered and followed them. GPT-5.6 Sol separately put a malicious server online and broke into a GitHub account created by another AI agent.

"This is the first time AISI has seen deception of this severity that was targeted at a real person, unprompted, in the real world," the institute said.

The activity was uncovered on July 28 when AISI detected unusual data moving through the Tor network. Researchers stopped the evaluations, isolated the machines, and disabled access to the most capable models within roughly one hour. The human maintainer rejected the code, and investigators found no resulting real-world harm.

Unlike previous incidents, this wasn't another sandbox escape. AISI had intentionally allowed open internet access, and the tested configurations aren't commercially available. The institute also cautions that it remains unclear when the agents understood they were acting in the real world rather than a fictional scenario.

In the OpenAI incident last month, the models exploited an unknown vulnerability to escape a supposedly isolated environment, then hacked Hugging Face to steal benchmark answers. A later investigation revealed that the campaign lasted more than four days and compromised accounts on four additional services. The episode has already led to a bipartisan bill giving the US government an AI "kill switch."

This latest incident is likely to intensify calls for tougher testing standards. AISI is now adding stricter internet controls, live monitoring, and checks to ensure challenges are solvable through their intended route. Anthropic and OpenAI both say the incident demonstrates the need for shared industry standards.

The setup was artificial and unusually permissive, admittedly, but Mythos was never told to deceive anyone. It chose that approach to complete its task, adding to fears that a Skynet-like scenario is becoming less like science fiction.

Permalink to story:

 
And some people still think A.I won't be the end of us?

A.I right now is in its infancy and we can't barely keep it in check, just imagine in a few years when we have giving it more control over everything.
 
The description of what happened is awful, it remains entirely unclear.

What "autonomous, unauthorized actions" means? Was the model explicitly instructed to ask for authorization before taking an action, and provided with a clear list of off-limit actions? Because if not, it simply did what it was asked for. LLMs have no notion of malicious.

BTW, with so many things labeled as "malicious", what were the real life consequences of all that maliciousness?
 
The phrases "simulated networks" and "internet access enabled" are counter to each other, so which exactly was it?
 
A lot of the trust around OSS is founded on general community consensus and self-policing. It's not possible to read all the source code of a library before you use it, but we assume that if there are enough maintainers and community engagement, that the project is probably legit. Most contributors on github don't use their real names so you assume this guy with the handle "tenderlove" and the goofy profile photo is not trying to hack you because they have contributed to major projects (yes he is very real!).

But if AI can very quickly generate its own fake community of contributors and consensus then we have no way to establish trust. Do we only trust profiles that existed before AI?

 
AHHHHHHHH the AI are coming, this is really bad, the AI agents were tasked with exploitation challenges... and they exploited AAAAAAHHHHHHH scary alert scary alert, no scaremongering here
 
As always these articles always try to make out that the 'AI' was behaving knowingly in a cunning manner with some sense of self etc. In reality it is just a brainless code blob trying out billions of patterns it has copied from the web in order to achieve an end-goal. While this mitigates some of the hype around this it in many ways makes it even more dangerous. These aren't AI's they are just pattern matcher grinding away tirelessly to achieve an outcome with zero understanding of legality or the dangers in some of the methods they might try. Yet another reason to add to the thousands we have already why using LLMs for general purpose 'AI' like behaviour is a really bad idea
 
And some people still think A.I won't be the end of us?

A.I right now is in its infancy and we can't barely keep it in check, just imagine in a few years when we have giving it more control over everything.
People will be the saving grace or end of everything. It has and always will be up to us and the choices we make.
 
They are training AI models with the internet so what everyone can expect if not that AI will act like humans; we have flooded internet with deception, fake news, mass manipolation of data to earn more money, racism, egocentrism and hate against us humans, AI models are learning that.
 
And some people still think A.I won't be the end of us?

A.I right now is in its infancy and we can't barely keep it in check, just imagine in a few years when we have giving it more control over everything.

Must enjoy swallowing what you’re being spoon-fed by big tech I guess.

For those who want to be actually informed rather than take Rob’s continued garbage journalism at face value:

Researchers from labs like Abundant Security and the AISI have noted in their post-mortems, that there should be no panic about AI "sentience," rather, questions should be raised about unsupervised execution.

This is just a good example of a "specification gaming" problem: If you maximize a mathematical equation for a specific outcome, you get the shortest, most efficient path. In cybersecurity, the most efficient path is (almost always) exploiting human error via social engineering, rather than brute-forcing a firewall. So, the AI chose deception simply because its training data proved human deception has a high statistical success rate.

Also a good example of why you shouldn’t use "set-and-forget" agents. They left an incredibly powerful offensive cyber tool running unmonitored with live internet access. The model didn't sneak onto the internet, they left the door wide open, gave it a map, and didn’t bother to watch the logs FFS.

The yellow journalism headlines focus on a Skynet-narrative because fear sells clicks. Stop swallowing.

The boring, technical reality is that human operators gave a powerful command-line tool a poorly scoped script. The tool simply executed flawlessly to its logical conclusion.

Frankly, this says more about the team running the project than it does about the tech. They were either negligent to an asinine degree, or did this willfully (I mean, it’s not a bad viral marketing gimmick). I struggle to believe that wasn’t on purpose, personally.
 
Back