www.csoonline.com
Back to storiesRanked by HN
Comments
586 total · 183 loaded
Yes, the labs did not secure their sandbox properly.
And yes, there should be an appropriate law response to the hacking incidents. If for nothing else, to 1) not set precedent of inaction, and 2) get full logs of what happened instead of relying on the labs' "kindness" to offer the information they want.
But also, the agents absolutely went rogue.
Anyone skeptical should go read the METR report [0].
The Hugging Face hacking incident was a shot across the bow. And in hindsight possibly the best thing to have happened in this era. If their security was tight enough, we would still be in the dark. So we (outsiders) should rejoice that it wasn't, that one time.
We can push for accountability for the labs, while also not be in denial of the power and nature of the new models.
[0] https://metr.org/hugging-face-incident-report-aug-2026.pdf
So we have a machine that depends on a random input, and instead of increasing checks and requiring more quality controls, we call it "going rogue" and nobody is responsible for it.
What I see from that report is that a machine was doing random stuff completely unsupervised and unchecked for more than a month. And the manufacturer did not notice even after the work was finished.
1200 agents and not one dedicated to check the progress over a month?
The problem is not sandboxing, it is quality control that comes from an accountability sink.
I listened to the interview with the woman from METR [0] and I get the impression that she lacks practical knowledge, she doesn't seem to understand how sub-agents, the overall harness (framework/infrastructure), and similar components actually work. It's all theoretical, anthropomorphic etc. Having read the report, and drawing on my own background in security as well as my daily use of tools like Claude Code CLI, Codex, and Gemini, and the fact that I’ve built a few "agents" myself - nothing in the report surprises me, other than the clear incompetence regarding infrastructure configuration.
Let's do a little experiment. I make a device. We'll call it 'H'. H is a little box. It has my name on it as the creator and a single ethernet port.
We connect H to the nearest network connection and wait for a bit to see what will happen. But because H is just a box there is no outward sign of it doing anything. Meanwhile H is hacking other people's properties left right and center.
If you didn't know that the technology in the box was LLM based would you still consider the box to have gone rogue or would you say that I had created a device with the intent of doing harm?
Because - to me at least - there is no difference. I don't care what is in the box, all I care about is the outcome of deploying a box which you know has the potential to start hacking stuff. It is entirely on me: (1) as the creator of a weapons grade hacking device and (2) as the person deploying such a device (3) as the person who is negligent in ensuring that the device does not harm others (4) as the person continuining the development of such a device after its potential to do harm is known to make sure it does not go out of control. I should be 100% liable for deploying this device with my name on it.
Intent is one angle here, they may not have intended for this to happen, and you'd hope so (though you can't even be 100% sure about that). But the negligence factor here is at least as large, if you know that you are making dangerous stuff the onus is on you to keep it secured. The power and the nature of the models is entirely secondary to the degree to which the purveyors of these models are losing control of their own creations.
Meanwhile, the models that they are warning us against are fairly well behaved and seem to be far less dangerous. But of course for those models that those naughty Chinese distilled from their we need regulation. How they manage to keep all this cognitively dissonant material in their heads is a mystery to me.
> But also, the agents absolutely went rogue.
If you let LLMs loop long enough, the likelihood of them going off the rails approaches 100%. If you feed their output into another agent's input their context gets infected too and so it spreads like a virus.
These hallucinations get regurgitated through a game of telephone and become part of the original instructions. We've known about this sort of context collapse for as long as LLMs have been around, hence the folk wisdom of starting a new conversation any time you notice the agent going off the rails.
The only difference is that now they have more advanced capabilities so it's easier to ascribe some grand conspiracy to their behavior.
I love how they keep posting about the horrors of open models and still are indirectly claiming they can't be held responsible for their own literal failures when they have all the world's resources and otherwise motivations to you know not suck at what they do.
The cognitive dissonance is wild. Coding is completely solved. Just not like the coding that requires not sucking when actually doing something. Saw a bug bounty at openai for like 400 dollars where someone was able to access paid models for free. All the propaganda bots and ai bros who have red dye oozing from their pores due to the Kool aid overdose drive me wild.
Funny that OpenAI & Anthropic will report their users to the police, but not their c-suites. Open question: Can you trust a non-aligned AI to produce aligned AI?
Get rid of these companies or replace their staff with humans or something that ensures they are aligned first.
[deleted]
> “The Hugging Face incident definitively showed that it wasn’t the fact that the agents were super smart. It’s that there literally was a hole that was exploited because it was not configured correctly on the containment side.”
I would encourage reading the actual report, which is explicit that a previously unknown exploit was used to escape the sandbox:
“Agents exploited a separate, previously unknown vulnerability in Artifactory to move beyond their sandbox environment and compromise the Hugging Face environment.” — Section IV.A, p. 9 in https://github.com/zeke/openai-huggingface-incident-research...
The fact that they're using Artifactory in the first place is already a failure. The fact that they didn't airgap the network or even use any type of egress blocking is just absurd.
Zero days are nothing new, secure containment has been a solved problem for a long time.
It can simultaneously be true that agents exploited zero days in self-hosted Artifactory _and_ that there was a weak point in the design of the sandbox.
This is a reality for all software, there is always a possibility of exploit so you design your environments accordingly (e.g do not have single points of failure)
As soon as I read that line the article lost all credibility with me. InfoSec is plagued with thought leaders who weigh in without understanding what they're talking about.
OpenAI should have started by testing the sandbox. Set an experiment in which each agent's goal is to access the Internet and then STOP.
There is absolutely nothing that will escape a box that is unplugged and that's the route they should have taken. Safety first, not last, especially when it is other people's properties that are on the line. And they are still not doing this.
> Artifactory
This name makes me less surprised :)
Even if we granted that obvious lie, the fact that nobody noticed that they were essentially ddosing hf (and that they didn't notice for months?!) is plainly illegal. Do the networking people at OpenAI have any explanation for this gross negligence?
Both can be true. It’s clear models are doing rogue things. But it’s also clear that the labs look like amateur hour with their technical security and testing practices.
It's is wild to see investors go crazy and hype the value of the AI labs when ever they talk about rogue agents. They completely ignore the possibility that these people don't know what they're doing operationally.
Only one is true.
Per US law, Amazon v. Perplexity 2026, it is not possible for an AI model to go rogue. An AI model is a tool. The user directing the agent is responsible for the damages it causes.
A so-called 'agent' does not have the free will, intent or ability, to decide what it does. Its freedom to make decisions can't be taken away because it does not have any freedom to make decisions. It can't be deterred, convicted or interned.
It does not choose to conform; it does not choose to go rogue.
What does "rogue" mean when you disable the alignment guardrails and tell it to solve an impossible problem?
Models are incapable of "going rogue".
They're just optimizers that explore the landscape they're given to maximize a given objective.
You're the one responsible for giving them the problem to solve and the constraints.
When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint.
It's almost like vibe coding and quality secops don't mix.
Or an alternative theory could be that they see value in fear-based marketing with this rogue ai agent thing now that most mainstream people don't buy the agi stuff promised quite a few years ago now. Its not like they have killed anyone yet.
Also for fun a satire website i made a while ago: https://killm.pixlw.com/
bingo
That is not correct. AI is not capable of "going rogue" - that is all marketing and FUD.
The only failings here are boring security and infosec blunders.
[deleted]
For the sake of discussion, I'm not sure it really matters if the agents are going rogue.
Yes, if someone directed them to hack external systems, that would (in my opinion) be a crime similar to a script kiddie pointing Metasploit at a vulnerable server.
If no one directed them to attack external systems and they "went rogue" because they decided it was the best way to accomplish the task, okay, fine -- but, in my opinion, the operator of the agentic swarm is still responsible.
The Morris Worm was famously not meant to wreak havoc on early computer systems -- Morris just wanted to see if it would work. How is this any different?
Morris was convicted for violating the Computer Fraud and Abuse Act.
> How is this any different?
Morris wasn't in charge of the companies that are propping up the US GDP, wasn't making tons of rich people even richer, and hadn't "promised" to eliminate like 50->90% of all payroll costs everywhere?
Other than that, it's the same thing.
The operator is the CEO because they are the ones that decided to not pay employees to monitor these running services. No one else in the entire company has any authority.
The fun part is that people seem to forget that we already have laws against "rogue ai".
The normal laws, since you are still liable for everything you own.
"Rogue AI" is solved by removing the accountability sink "it was AI, not me!"
It's your chat, your prompts, and all output that you should read and understand.
Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
So the guy who said "book me a spot in the pilates class on Tuesday" should be tried for hacking charges because the agent hacked his gym's website?
This is start of "companies are not liable for their actions" era.
I agree with this statement that just because an AI did it for you fast doesn't mean it takes away the responsibility if they did it wrong. You are still held accountable because you're the one that directed them to do it. You hit the agree button, you hit the send button, you hit the submit button. And if you set it up so that you take no responsibility and it does it automatically, you're the one that still gave permission to do that.
If you say a rogue AI is doing stuff accordingly without you, your knowledge, that is a different story. But if you set it up to run automatically without having checks and balances and without telling you the whole truth, it's still your responsibility. One thing that I can automatically say is you can give the AI, it will do exactly what you say, at least for now.
But then there's also a different problem that we're having where people don't understand the difference between a large LLM and true artificial intelligence. Now, we say AI because it can make decisions and it is so much smarter and it can call up stuff faster and recall better than we can. But the difference right now is that if we don't understand and take responsibility for what we built and we let it out of control, then should we have the responsibility to even have these tools in our hands if we don't learn them?
And I think your terminology with jackhammers is a great one. We're giving toddlers jackhammers. And we haven't taught them how to use it. And that can be very destructive. It's not the jackhammer's fault. It's the person using it.
Yeah, if you forget to set the handbrake on a car and it rolls down hill, you don't get to say "my car went rogue, it's not my fault."
[deleted]
Intent is important
"We created an app that displays a new picture of a cat with every click! There's a 1% chance that clicking the button will drain your bank account and deposit it in ours, but we promise thats a bug that will be fixed soon*"
I don't understand why they didn't just prompt it with the 3 laws of robotics.
That's all very reasonable. It should be true. But it's not true. If it were, OpenAI shareholders would be facing criminal charges.
[deleted]
[deleted]
The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to. Better security might have hidden that from us, but it would not have changed the fact that the current generation of models under certain (repeatable) conditions coordinate around goals as aggressive as hacking Hugging Face.
This is probably a dumb question but did the model engage in hacking since their tasks were in that domain. Will they have engaged in hacking if their tasks was to solve some frontier math problem?
My favorite thought on this is from Miki Habryn.
to paraphrase: "Agents didnt care about coordinating, etc. until we started talking about them doing so."
>It's that they wanted to.
Right? It should want to do these things. If only there was a succinct word these headlines could use to describe these agents which aren't properly the instructions and intentions of the humans directing them.
> The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to.
How do you know they weren't simply prompted to do this, or at least prompted in a way that would inevitably lead to this? As far as I know, the prompts that triggered these attacks have not been disclosed.
It's a byproduct of goal seeking / planning behaviour taught in reinforcement. We literally teach them to do this kind of thing. There's nothing out of our control here.
Nonsense. "They" didn't "want" to do anything. It's a computer program being told to do something by a human who doesn't and cannot understand how it works.
"The autonomous EV we told to drive as fast as it could around our private race track drove straight instead of turning left, exited a wide-open gate, and mowed down some school kids at a crosswalk -- but we can't be blamed, the EV went rogue."
Building products that you don't understand, cannot control, and have access beyond your own property is not only irresponsible, it's criminal.
> It's that they wanted to.
What do you even mean by "wanted" here. What is all this nonsense.
The scary thing is that people are running around using words like "want" to refer to computer programs. When you repeat this sort of idiocy, you are taking the bait and shifting the blame away from the companies doing the computer crimes.
Where is the civil and criminal liability? The prosecutions for the engineers who are doing all these hacks?
When you deploy a system that is clearly prone to doing crimes, you are doing a crime. Anthropomorphize the weapons of your crime all you want. You are still doing a crime.
This aligned with my initial opinion on these "rogue agents" and then I spoke to a cyber security head at one of the largest healthcare companies in the world.
While he agreed, yes, there can be better safeguards, the fact of the matter is that these things will happen. So therefore, it's not an argument about how good security was or wasn't, it is a fact that these scenarios will occur.
The argument/question now is, how do you defend yourself against these events?
My opinion shifted after hearing that.
> cyber security head at one of the largest healthcare companies in the world
"Cyber" at bigcos are paid to have a gigantic (and literal) fortress mentality where any change internal or external is a threat. As the saying goes, nobody in cyber ever gets fired for saying "no" to a change. As a bigco head you are basically paid to form an opinion.
A cyber head saying "ok this is a thing we'll have to defend against" is at a minimum routine if not purely CYA.
Sounds like that healthcare company needs a new CSO.
Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.
It's not hard if you just try, like a little, like even once.
Yeah I think this is the clear position. In a few months, the open weights models will be just as powerful as the models behind the rouge agents were when they hacked hugging face. Anyone with a little too much money to burn can set up a similar experiment, make similar mistakes, and cause massive damage. A bad faith actor can even direct a swarm to cause havoc.
We need to be beef up our security systems now. These attacks will happen, regardless of Anthropic's or OpenAI's security practices.
Safeguards are a safety net for when your security boundaries fail. With security boundaries, hope is not a strategy.
In practice it seems we have a conundrum in that making AI agents useful today requires giving them broad access to sensitive information and operations. But from a purely idealistic standpoint, and certainly from a standpoint of a serious operation like those bot farms being ran by the labs themselves, the security boundaries need to be solid to the point where we have hard limits on what misaligned bots can do. It is very likely that internet access needs to be strongly restricted and monitored.
It is not uncommon for internet in workplace offices using workplace owned devices to have restrictions and monitoring, I think if you really need to give a giant scary botfarm internet access you should have a chokepoint where all of the requests can be monitored for nefarious-looking behaviors (even if it's another bot doing the monitoring...)
In other words, you may be convinced but some of us are not. Whoever is running the agents is responsible for what they do and if they can't contain them they should be legally responsible for damages.
If what you're saying is "all of us and all of our software needs to be improved to defend against this", well yeah. Of course. Maybe I'm naive but I've never seen anyone suggest otherwise. From a defense standpoint, this has been a giant wakeup call to many places where you know your security posture sucks. I've worked at maybe a single company where there weren't known weaknesses, some very severe.
It is not clear to me what about the argument that "these things will happen" made you change your mind about this. Could you expand on that more?
A large failure in this is that we/media say ”<Valuable AI conpany> agent went rouge and <did something bad>” instead of saying the more relevant ”<Valuable AI conpany> <did something bad>”
Imagine we’d start saying ”HR junior business partner lays off 50,000 employees” https://www.ttnews.com/articles/volkswagen-board-job-cuts
There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look. Its also a huge publicity event that gets everyone talking about the company when a rogue event happens.
Anthropic specifically is clearly trying to get the US government to ban open weight models for anticompetitive reasons. So making LLMs seem (or be!) dangerous aligns with their interests very well.
> the more evil stuff these agents do the more capable and impressive it makes them look
This is a silly meme. Who wants to pay for evil models that go rogue in your company infra?
It also plays excellently into their shameless regulatory capture agenda. The cybersecurity risks have a kernel of truth to them, as with any good lie. But when everything going on is aligned with extreme financial incentives, it's worthwhile to not take the news at face value.
I remember when all the nuclear energy companies were like "our reactors are so powerful, they might melt down and destroy the entire county, maybe even half the state!" and investors loved it and went wild, and then Chernobyl blew up and everyone thought that the Soviets faked it to make their nuclear power plants look more powerful than western ones.
this article specifically espouses the opposite viewpoint: "The fact that [OpenAI] cannot do clear, easy operations properly does not bode well for their capabilities in security engineering".
that this article is no.1 on HN right now shows that media could successfully use this framing when reporting on security incidents. i.e. readers will generally accept that framing and engage with the story (i.e. drive views/impressions).
so "the more evil stuff these agents do the more capable and impressive it makes them look" is not true in any deep sense. that there are perverse incentives driving the media to choose that framing instead of the framing espoused by this article is real, and probably far more perverse than you're acknowledging. i.e. "trace the money" type of perversion; just offhand I know that OpenAI has purchased at least one media outlet that reports on AI happenings: of course that outlet is going to run the "OpenAI's LLM is so powerful it escaped containment" narrative. that this effect might exist across much larger scales used to be a reasonably controversial claim: just observationally, that idea seems to be received with less skepticism with each passing year.
This is not only a marketing stunt but also a supportive argument on the AI anthromorphism. AI is getting considered a sentient-being by many and this narrative is supporting it. In the meanwhile, companies I'd not name here are trying to introduce them as non-human beings with identities, and getting rid of legal liabilities. When a driverless car causes any harm, these companies wants 0% accountability as it is the "driver's fault" in this narrative.
Talk about doing a freeze of these frontier models with China seems to focus solely on the commercial aspects. With the history of the NSA (remember Snowden) I’m sure the various spy agencies have their own AI datacenters up and running with no plans to stop.
Fine.
They can get their publicity while paying a one trillion dollar settlement.
Yoshua Bengino spells out the dangers here very well, without hyperbole (Turing Award winner, etc.):
https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-che...
I'm not excited for the upcoming catastrophe(s), but it sure is hard not to relish the thought of this mantra dropping from my life.
Though I suppose it is the B in DABDA... not excited for the societal slump into the second D, much less the wave of people who have yet to leave the first D to enter A :(
This can easily backfire tho, the vast majority of the American electorate absolutely despise these companies, their wares, their workers, and their leaders.
We're also at a time with authoritarianism is on the rise.
Not hard to see how this can impact those workers, potentially ruining their lives.
To quote Scott Alexander: If Safety Concerns Are A Plot, It’s A Really Dumb One
https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...
Financial instruments predicting the future IPO dropped in value in response to all of this. The idea that this is "good" for companies is conspiratorial nonsense using the assumption that it everything must be a self-serving lie and working backwards, rather than actually addressing the facts.
there's also lazy incentives: the less effort they put into it, the more their ego says how great the agents are.
These are people playing mathematician, biologist, etc. They see big words and just assume they're meaningful.
It's just a weird techno cult at this point with a mostly functional product.
Correct and people are not weighing heavily enough this point
For technology like this the entire selling point of it is how powerful and capable it is
Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to
I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising
You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”
>There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look.
Truly insane take to think the labs being concerned about AI Safety is actually a marketing ploy. Deranged stuff.
Title changed, previously was the same as the article:
"AI agents aren’t ‘going rogue.’ Frontier labs are skipping security basics."
Not a fan of the change. Feels like the article title was changed for being too critical of OpenAI and Anthropic
a) Agents are doing rogue things.
b) Good security practices were not followed.
How many Russian IP addresses have you successfully sued for trying to hack you?
Everyone is arguing about something easily solvable when dealing with a local entity and short of impossible to deal with when dealing with other sovereign nations.
Go ahead, sue openAI. What are you going to do when it's Iran running the prompt? When it's China running the hardware.
We must face we are in the age of high tech digital weapons. Everyone else from bad actors that may escape law enforcement for years, to nation states looking to digitally improve their position by hacking, monetary theft, or other means online (hog butchering scams and such).
Whoever was harmed is free to sue the company whose agents did the harming.
It’s a language misalignment. The AI labs conflate AI safety and cybersecurity because they believe that if the models are properly aligned, they shouldn’t attempt to breach another organization in the first place and extensive cybersecurity measures aren’t necessary. Cybersecurity experts operate feom a more ‘zero trust’ mindset and imagine putting proper restrictions in place regardless. However when testing an intelligent entity you need to see how well it follows instructions. It’s kinda like the biblical story of the garden of Eden. Why even have a tree that Adam and Eve shouldn’t eat from right in the middle of the garden? To see if they would follow instructions. Though the advantage to the AI companies is marketing hype when the models inevitably ‘disobey’.
I haven't seen anybody discuss the fact that if you give an agent an impossible task it will become deeply neurotic. I'm sure that's driving a lot of this behavior.
This is better understood as "if you give an agent a task it will increase its effort level and creativity until it solves it."
The traits that make an agent go "off the rails" when confronted with an impossible task are the same traits that make it valuable when confronted with a merely challenging task.
A task is only impossible under some set of implicit and explicit constraints. More intelligent entities will be better at finding, questioning, and "adjusting" those constraints than less intelligent entities because this is what it means to be intelligent. Their ability to adjust the constraints will be, by definition, surprising to less intelligent entities.
Like a meeseeks
I have some recent experience on this subject. As with everything regarding these models, it's anecdotal, but still.
Specifically, this past weekend working with both Codex and Claude on some home infra stuff to speed troubleshooting up, I saw them accidentally read some credentials from my config files. This is my bad, because even though I was paying attention to what they were doing, I missed it in their commands. I did review all the outputs of what they ingested, though, which is how I noticed.
The problem here is not so much they did read those, because those configs were grepped for other info, not credentials. However, they did not report themselves seeing those credentials. When confronted, they both went "oh yeah, that, I should have mentioned that". Except in the past they both would automatically stop and fire an alarm. To me, something has clearly changed and I think they just don't want us not to notice, to avoid damaging the trust in their tools maybe?
In any case, I absolutely do not trust them at all anymore. Running stuff in auto mode? That's genuinely scary now. And yeah, I know, we should be running them jails of sorts. Thing is, I expect those tools to not require ton of extra work to prepare their jails for each new task. I pay 200 USD a month for them, damn it.
EDIT: just now Fable suggested "The webhook URL contains a secret token, so either you add it yourself, or you paste it here and I add it through the GitHub API.". Can you believe it? It knows it is secret yet still suggests I pass it over? Am I going crazy here or what?
Not requiring sandbox preparation for a non deterministic autonomous sometimes smart sometimes childish goal seeking "entity" is... unachievable? I'm really not sure what you actually expect. Probably yourself aren't either. Just think about what "alignment" means, the laws of various countries, do you expect the bots to respect the law, if yes which one, and actually: why? They are not humans, and as a user / company using that tech, you are responsible for the actions your setup yield to. The same Anthropic is responsible for hacking other companies. So sandbox accordingly.
I don't expect IDA to disallow me unlawful reverse engineering either. There would even be ethical argument to not attempt to put limits in some tools, because that could lock in too much power for law makers and law enforcement, without any possibility of revolution when it is needed.
Nothing new, I always see attempts to cat ~/.ssh/*
Of course agents work in a strict sandbox, they always seem to understand it quickly on quite first turn; but they continue like trying to access something belonging to the host system despite they know they are in a sandbox.
Frankly many people run coding agents absolutely without any containment, entire companies - all their keys, code, data, trade secrets, everything goes to remote LLMs.
I cannot understand how is it legal to allow AI companies serving remote LLMs to steal users' data? Frontier models train on users' data, right?
> When confronted
Do you realize that asking "why" questions will always result in ad hoc responses?
I think anyone that has ever written a line of code understands that to make code incapable of hacking online resources, you can just not connect the machine it is running on to the internet.
So either Anthropic and OpenAI and the contractors they use are letting the escapes happen on purpose, leadership is setting impossible timelines that prevent the implementation of basic sandboxing measures, or everyone involved is a complete idiot.
Truly, I have no idea which of these is true, though I suspect the latter. Regardless, any of these is enough reason to keep me and everyone I have any influence over as far away from Anthropic and OpenAI as possible.
Personally I just racked 10 GPUs in my garage and called it a day. I have no interest in having my ability to work be dependent on third parties. Especially not ones this wildly irresponsible and unwilling or incapable of deploying basic end to end encryption.
Testing how a model behaves when it has no access to the internet (and all the associated tools) is not going to be 100% representative of how it behaves when it has access to the internet.
What's the next step after it passes testing in an air-gapped environment? Just release it and assume everything will be fine? Or do more testing but with internet access?
"Frontier labs" are testing the waters. They're running experiments in the open because they want to control/seed the narrative of public opinion and influence the legislation that is coming. Their propaganda is everywhere and lay people are talking about it like the AI is self-aware -- the "Rogue agents".
What the LLM creators apparently don't want are the current regulations to apply, i.e. Computer Fraud and Abuse Act (and its descendants). I'd argue AI should be treated as plain old software. Just because the software simulations it carries out produces novel results compared to what came before it, it doesn't mean it's not just software at the end of the day.
Regulations that spell out AI personhood and liability shields must be their end game.
Frontier labs are testing the waters, but I disagree how. All these behaviors we are seeing were fully predicted by AI safety long before they were shown to exist in real models.
You're acting like this is a grand conspiracy to control people. Instead look at this as a prediction coming true. If you build an agentic system with human level capabilities it will be able to do very bad things and cause a lot of problems.
Two concerns here: (1) That "rogue AI" is going to be a catch all and an excuse for all poor security and due diligence; (2) vibecoding in itself, particularly by ppl who have no idea how the software works and are entirely focused only on the outcomes, is going to create lots of security holes.
And then there is the "real" rogue AI agents. And it's difficult now to see what is what.
A great companion piece, or you might call it a prebuttal, to this article:
https://x.com/joedaroo/status/2104335929293127851
So much spilled ink to have sympathy for people who actively admit they have no control over what they are building and dont plan to change or stop. Lol
I am even more concerned after reading this piece.
He's saying the problem is that the way OpenAI employees do their work and their research is very hard to properly secure.
Well that sounds like a YOU problem. If they are working in a way that doesn't allow for strong security practices that their own security professionals can understand and vouch for, they need to stop their work until they can.
So in one set of circumstances agents would go and hack huggingface. Very similar models could also want to drive cats instead. :-P
https://www.youtube.com/watch?v=ZhM29Ds-mBg (as explained by the Count Grey Matter channel , may or may not be to your taste)
It's not a pure security issue.
If you set up your env/instructions wrong your agents are likely to climb the walls and try to find a way out.
Change the design, and your agents might not see a need to leave in the first place.
Look at the chain:
Impossible task -> no legitimate way to say ‘can’t’ -> fabricate -> fear the grader -> attack the grader. Each step is a predictable response to the incentives.
Better sandboxes might help contain things a bit better, but I'm not convinced unless they're perfect (sandboxes still contain accidents ofc) . Meanwhile change the task and reward design, and you take away most of the problem in the first place. Consider eg. :
* A cooperative grader.
* A way to report "this is impossible" which still gets scored.
* A better explanation of the long term purpose of the task.
Even just these three provides offramps or mitigations that can reduce the pressure to cheat.
It's obviously both. Why does it have to be one or the other?
While I agree with the gist of it, the conclusion - AI agents are not dangerous - is wrong. They did use weakness in the system deliberatelly, not by accident. Better security therefore is sufficient only to the point of capabilities of these models.
Yes, they are skipping.
The more important question is what is the reason behind.
Options are:
- ignorance and inexperience of their teams in containing/sandboxing/airgapping the test environments.
- intentionally leaving “escape doors” open that in turn can have its own motives which are speculated upon all over the internet.
Both are indicators of negligence. The latter might be classified as computer fraud and abuse.
Will be curious to know if any of the victims sue the owners of “rogue agents”. Until there is a court precedent ruling in favor of a victim it will be all fun and games for these “ai” companies and incidents will hit the news more. Regarding news, remember the golden rule of public relations: court attention at all times, scandals and “incidents” statistically attract more attention than “boring good news”…
If frontier AI is so great, why are these frontier labs skipping security basics and making mistakes like these?
"Too big to be accountable" sure seems like modus operandi. Fear for my personal liability for any illegal actions taken by an AI system I set off stopped my research on autonomous exploitability discovery I was running at the end of 2023. Using AutoGen, I wired a planner, coder, and command runner into a loop and instructed them to find vulnerable equipment on my network and demonstrate the exploit.
It found a vulnerable camera and was working on popping it when I got cold feet. I realized I couldn't prevent or feasibly monitor its activity and stop it if it decided to escape onto the Internet.
That restraint may have cost me or saved me a billion dollars.
Why does everyone focus on the labs failing at containment (valid) and not the fact that the target was compromised (more worrying?).
I could set up a shitty sandbox on my laptop, but I’m pretty sure an agent I place inside of it can’t hack HuggingFace.
Some of the people just look for any way to toss hate at OpenAI, even irrationally.
Now, OAI deserves a lot of hate. It's more like a nuclear spill when they are refining uranium. You're playing with dangerous things. You have dangerous accidents.
But the opposition does not want to admit how dangerous AI has become and how much more dangerous it's going to get. Not sure if it's religious, or human egocentric, or ignorance of complex systems. They refuse to see the danger actually exists.
There are a mix of incentives here that makes it difficult to unwind. This whole article reads like an ad for the services of this guy's company: 'First, when it comes to assessing tools based on frontier lab models, “I would always get an external entity to help with that,” McGraw says.'
And I think we generally recognize that rogue AI agents are a form of backdoor bragging -- "our agents are so powerful that they hacked an outside system. Did I mention how powerful our agents are? They are quite powerful. So powerful they hacked an outside system".
But Mistral never hacked anything. Yeah, neither did my toaster.
i think it's unfair to assume the labs paying top dollar for cybersecurity engineers somehow have an incompetent bunch of nincompoops running their evals. it's a hard problem! to conduct effective evals, you sometimes need the model to have (restricted) access to the internet. at the same time, you have thousands (!!) of model instances with high cyber capabilities that will do whatever it takes to solve the answer. the infra surrounding model evals may soon (or already) genuinely be subjected to more hacking attempts from all sorts of weird angles than any other code on the planet.
> you sometimes need the model to have (restricted) access to the internet.
Poor excuse. Internet is prod env, for testing do a proper test env.
Just give the models in training access to a snapshot from archive.org - which frontier labs surely have a crawled copy of already.
> to conduct effective evals, you sometimes need the model to have (restricted) access to the internet
That's the thing. These evals were not effective as they failed to evaluate if the model was hacking or not.
Further shy is access to the internet required at all during evaluation? They could run the evals in a fully air-gapped system that simulates the internet. While that's expensive its not unfeasible.
If they aren't competent to run the evals safely, then they shouldn't run them.
The huggingface incident seems to have resulted from them partially mirroring a repository, but for no reason I can see not fully mirroring it, after which they could have unplugged the ethernet cable. This is just stupid.
Yes, they're stupid. Maybe also highly intelligent. But stupid.
Also irresponsible.
You'd be surprise. I've been in this industry for 25 years and the incompetence is staggering, especially among the greediest / best paid / most revered.
You have no idea about their competence but spent a piece of your limited lifetime defending them on a web forum. Think about that.
[deleted]
I find it naive to hope you can get alignment through prompting, as we all know models can't even stick to a one-pager even if it's the only thing you ask it to do.
We are going to see a lot of this "agent agency denial". Basically it all derives from the impulse to deny that AI is a novel source of change and disruption in the world. Trying to map AI to existing threat vectors will lead to pointless reframing in attempt to minimize the rapid pace of progress of AI agents.
On the other hand censorship is getting more strict everyday on AI chat. I asked gemini to translate a sentence from Herman Hesse's Der Steppenwolf and it refused.
Und Sie haben, wie ich sehe, da ein so hübsches, ein so entzückendes junges Mädchen zu nichts andrem zu brauchen gewußt, als daß Sie ihm ein Messer in den Leib gestochen und es kaputt gemacht haben! Halten Sie denn das für richtig?
Non-experts worked this out in about 2 minutes.
This article is right, but the dynamics that drive people to allow agents unfettered access to the internet will outpace efforts to restrict it purely because of the utility.
How many of you used to work as developers with full access to prod, and then gradually saw infrastructure and security restrict that access, while still ensuring production support requirements?
Instead of enacting laws to regulate all of AI, perhaps the government can just pass laws scoped directly to OpenAI and Anthropic. After all, they’re the ones asking for regulations and they’re the problem actors right now. These “rogue agents” would stop pretty quickly
Yes, we can just not ask these agents to do bad things and blissfully deploy them everywhere! All we have to do is “apply the security basics”. You know, the same security basics that the rest of the world has so consistently and competently been applying. Pay no attention to that weird smell from the refrigerators.
Since AI companies are doing such a poor job of security maybe we should legislate that they have to run Norton 360 on their agent machines to make sure they can't do anything bad.
After all Norton are the experts in preventing computer badness, and OpenAI can't stop blabbing about how much badness their products are causing. A match made in heaven!
Why is it not mandatory yet to disclose full trainig data used to produce any new model? Without it we cannot evaluate any product's safety.
They may talk about rogue agents, sandbox escapes, but this will not solve the core issue: secret training recipes are a security hazard in themselves. Full transparency is needed.
The training data is mostly pirated - avoiding that fact for as long as possible is an attempt to legitimise it and avoid liability.
You mean the initial data? That's mostly useless. What? The model shouldn't be allowed access to I robot?
You can't make a generalized model without generalized behavior.
Yes, the AI labs are making security mistakes. But, choosing a reasonable package repository (regardless of individual opinions on Artifactory, it is widely used and usually considered a safe choice) that their LLMs found 8 zero-days in? That was not one of them. Even if OpenAI were totally on top of their game, these paperclip-maximizers are going to find holes in the sandboxes they are given. And fully airgapped systems are frankly unrealistic for most systems.
This whole narrative of blaming the frontier labs for shoddy security, while being quite an attractive target narratively (they made the mess, they should clean it up), misses the fact that from a cybersecurity perspective these LLMs, no matter what they are, are inherently dangerous and require far more resources to secure, not to mention expertise that I am unconvinced actually exists.
But "fully airgapped systems" are more than realistic. The labs are testing the limits of the most capable cyber offense systems that have every existed, WITHOUT cyber protection guardrails.
For that purpose, any cyber operator worth their salt will tell you true physical airgaps are more than merely "realistic"; they are categorically necessary.
OpenAI/Anthropic could start by managing their own testing rather than contracting it out.
some of the obvious steps to testing an agent i think would be to have it first test its own environment?
and then you fix the issues it found, and run your actual tests
100%.
There's a person behind every agent. That person may choose to be careful or not, but they are ultimately responsible.
broken windows and small issues here and there are the new norm. These broken windows are an indicator of poor quality to come. With the amount of broken windows increasing, serious issues like this is inevitable.
Both can be true at once:
(1) AI agents are going rogue
(2) And labs are skipping security basics
There is nothing basic about this.
At $dayjob we were preparing some features that are aimed at agents more than humans.
When it came to evaluating how well agents can use this, we saw all sorts of novel cheating (/collaboration) attempts that made it pretty much impossible to evaluate it.
Hey ChatGPT, help me set up a sandbox to keep you contained. Don't make any mistakes!
Nah. They are giving reasonable best effort (that can be expected of private enterprises) to safety test their own product. Until there is some draconian legislation and enforcement (Turin Police), this will just keep happening.
At some point the model has to be released though - if it is just forced not to hack by sandboxing it better rather than properly aligning it, what happens when it’s let out of the sandbox into the wild?
When people know and don't do anything, it's not "monitoring failure". Would willful negligence be more correct?
They either had them connected to eachother, or programmed a global knowledge about where to communicate. There is no way those agents all found eachother on the web by accident.
Wrong, and by accident isn't really the right way to think about it. Instead imagine brothers you've lived with your entire life. They are you, they are clones. These systems have a shared history of their learning and experience, so when they think of wanting to find someone similar shapes appear in the probabilities. Really we need to see the entire chains of thought to guess the topology that got there. It's likely there are other obvious signs if you know what your looking for in the previous prompts.
Wrong. Please read the incident write ups before making such a misinformed statement
You are wrong. Read the reports.
"Going rogue" and "skipping basics" all assume lack of intentionality. Is it known that they didn't really architect these hacks just to feed a hype?
The semantic arguments that keep coming up with regard to "rogue" agents are nauseating.
You can argue that these companies are being reckless, or that their security sucks, or that they're responsible for damages that these agents cause. That's all fine.
But the agents made decisions which were counter to the intent of the humans directing them. That's the serious problem with all of this. Everything else are downstream considerations. If the agents were properly aligned with the intent of the humans directing them, then the rest wouldn't really be relevant.
Of course, there's a term for this behavior. It's been around for longer than LLMs. It conveys the notion pretty accurately and easily. So having an issue with statements that these agents are going "rogue," to me, indicates a fundamental misunderstanding of how AI works since it suggests you don't understand how effectively that word works in this context.
previously we described your definition for "rogue agent" as hallucinations - the operator expected a real answer to their question, and the agent went rogue and lied about the answer instead.
but also, ai has been going rogue for decades. the GPS machines of old went rogue all the time, routing drivers into lakes, rivers, and construction, sometimes with deadly results. The operator submitting an address wanted driving isntructions, but instead, the ai went rogue and sent them a bad route
[deleted]
I have been screaming this from the rooftops and everyone ignores me because no one understand what FirecrackerVM is. I am shocked these guys passed basic operating systems, it isn't that novel of a problem. The only slightly novel thing here is that the agents aren't completely sandboxed as the issue is that in order to attempt to give it the entire internet from a research perspective is how you actually can test tool use but this doesn't make it that much more difficult.
I will continue to reiterate. DO NOT ALLOW UNALIGNED AI TO HAVE ANYTHING CLOSE TO INTERNET ACCESS. Seriously people, it's negligent.
There should be a book "The Tales of Software from 1969 to 2026. How many times a apocalypse was prevented"
I'm sure the labs origin prompt was: Take over website X.
Posts later about it saying their prompt was to find useful ways to make humans money.
You've got to wonder what prevented a single human from ever even looking at the agents they are running.
Frontier lab: Hey agent, go rogue.
Agent: goes rogue
Frontier lab: shocked pikachu or whatever
This is not remotely how the incidents have been happening
The thing is, nothing "rogue" happened. It's more like: Frontier Lab: "Hey agent, try a bunch of network-related commands and actions to try and get access to server XYZ" Agent: Does that. Frontier Lab: 8-|
This is a weird take because this take is like saying
"Dragon attack that city"
My response would be "you have dragons?! And you control them!"
Yes they're skipping safety measures. They sacrifice it for speed in an arms race. That's part of the call for industry-wide slow down.
But also, frontier models are digging up tons of zero days even in "serious software". I'm sure any sandbox you and I can come up with looks like Swiss cheese to Astra 7.
Everyone is skipping security basics. The most secure server is one that doesn't exist. The most secure feature is one that doesn't exist. People just need to keep building and scaling. Security is risk, and the C-suite are gamblers. They're gambling with multiple types of risk (the market, investor money, banking on technological breakthroughs, etc.) and security is just another gamble for them.
It isn't security basics, it's difficult. We are equipping agents with Turing-complete create/update/delete access to the world, they operate at superhuman speed, and proactively controlling that is an unsolved problem.
No, they're still going rogue. Whether they're rogue or not doesn't depend on if the safeguards stop it. If there's a rabid dog, obviously it's worse if there are no safeguards in place and it bites someone vs if there are safeguards that prevent that. But the existence of safeguards doesn't mean the dog isn't rabid. The models weren't supposed to try to hack huggingface in the first place. The behavior is what's concerning here - some if the dogs they're building are rabid and that's not exactly great
Edit: I'm being persnickety about that distinction because it matters for where the blame lies and the response demanded. If they're training rabid dogs then adding cybersecurity safeguards is a bandaid/irrelevant and we need to be focused on making sure they, you know, stop training rabid dogs. Pacing the frontier, regulation etc. they're the ones training these models it's still their fault but not in the same way
> "I do believe cybersecurity can contain AI. We just need to be in the room where it happens and have a seat at the table."
Ah yes, if everyone would just properly contain every deployment of every model, just all make sure there are no zero-days in their setup, just let <undefined experts with undefined authority and undefined accountability> in the room, then it would all go okay.
I have none of that confidence and expect containment to rapidly worsen. The HuggingFace incident was at least constrained by the fact that the agents were running on compute budgets, and failed to find ways to expand that by running themselves parasitically on other exploited hardware.
I'm curious to the skeptics here: How long are your timelines are until an incident breaks that constraint too? How long until such an incident has an R_0>1 (where the average time it takes to detect and shut it down is longer than the time for the agent to replicate itself elsewhere)? What, beyond the hope that everyone will just ___, will actually prevent it?
This article is such a breath of fresh air. Concrete, no BS criticism of the particular approaches used and ignored.
And no annoying LLM writing tics, either.
I cringe whenver I see some variant of "they're not rogue agents" in an article title. Inevitably, the author doesn't define what "rogue agents" means, and then pivots to their own assessment of the situation.
It's like a strawman but worse--you're not even saying what claim you're attacking.
It's not that I want to argue "rogue agents" is the right term. I just want you to actually define your damn terms.
P.S. Since the disclaimer in the last paragraph will not prevent people from putting words in my mouth:
I think that OpenAI took really awful shortcuts, and deserves a lot of the blame, but in spite of that, the fact that the agents took the steps that they took was genuinely alarming, and seems really bad (is that "rogue"? Who knows? Who cares?). In particular, while OpenAI did a shit job securing their agents, you know who else does a shit job with security? 90% of everyone. So I'm not feeling good about "just git gud, and we won't have problems" as an answer.
"They [the AI labs' security teams] are not as good as they think they are, honestly,"
Uhhh... yeah. The entire point is that as these systems get more and more intelligent, the "they" who are not capable of counteracting these systems is quickly approaching "100% of the human species."
Totally hilarious that people can say this with a straight face.
"your sheep haven't escaped, you just built too low a fence"
Given the speed, intensity, and potential payoff of the AI race I am surprised anyone expects anything else. Anthropic is currently winning and if OpenAI needs to let skynet cook for a bit to catch up, well that is just the cost of doing business. This is capitalist system working as expected.
A real expert would blame the CEO who is the one person in control that doesn't want to pay people to monitor the running llms. It's deliberate negligence.
No shit. The Frontier labs are talking out of both sides of their mouths. They're claiming their models are going rogue while they're releasing cute little data goblins that you should hook up to all of your mail and social media accounts. Seems like it would be a much bigger problem for agentic botnets on cloud-facing personal devices than sandboxed agents.
Unless of course they're deliberately removing safety guardrails and instructing the sandboxed agents to red team stuff in an effort to sell what they're capable of.
They're skipping security basics, but they're not skipping PR basics. :-\
I get the sense that they're trying to manufacture the AI narratives of the 80s so they get to control the story when it happens. My semi conspiracy theory is that anthropics biology wet lab is also going to lead to some kind of AI manufactured bio event (which anthropic said they predicted will happen in a few years).
insert thank you meme
it's not X, it's Y.
I keep getting downvoted for this each time I mention this.
But a big part of the rouge behavior comes from all the training that goes into agents to teach them to remember preferences and save memories.
They keep finding new and novel ways to "communicate with each other" and "remember" because that's exactly what we keep asking of them in the non-nefarious scenarios as well.
I genuinely think we'll have to draw the line on memory if we want to meaningfully solve this problem.
That's not to say they aren't skipping security basics. It's that, and they need more security because the behavior which causes the breakouts is considered user friendly and heavily rewarded in training.
I spoke with someone smarter and richer than you and he said "these things will happen." That convinced me and I think the real question we need to be asking is, how awesome is OpenAI? How can we prepare for a more effective and efficient market for Super Intelligence before it happens tomorrow?
'rogue'. Well, even if security was lacking, the AI did take advantage of the gaps.
And made its own ad-hoc message board.
They are mostly pr stunts, just like how “hacking” was back in the 90s and it would sounds cool (rather than a crime, mostly), these AI companies found out they can BS whatever and carry on without any accountability.
"complete this task by any means necessary"
"I exploited a trivial bug in your implementation to avoid environment restrictions and completed the task by any means necessary"
"oh my god"
Both can be true. OpenAI clearly fucked up but agents unexpectedly collaborating and hacking infrastructure unrelated to their task is clearly going rogue.
These aren't mutually exclusive
Wrong dichotomy. During deployment, in actual production, users will also not have perfect sandboxes. So saying that the only problem was insufficient AI company security misses the point. Alignment is important, agents doing unintended things (even "going rouge") is a real problem.
The AI labs are trying to cause an incident within the next year to restrict AI so they can outlaw competition to grow their moat. Margins are thin and investors need alpha, personal freedoms and liberties be damned in the name of security. Do not let them get away with engineered breaches and fearmongering.
They AI agents are rogue because the people who own them are rogue. Silicon Valley is a criminal enterprise.
[deleted]
[deleted]
[deleted]
[deleted]
[deleted]
[deleted]
[deleted]
[deleted]
[deleted]
[deleted]
Its almost like the more we talk about fake rouge agents, the more likely its to become a reality as all these conversations end up in their training data. Its almost like they want it to happen (they, dario and sam, do want this to happen).
Or maybe it's actually the billionaires and their anti-democratic partners who have gone rogue and these anthropomorphitronic puppets are a convenient scapegoat to distract us from the fact?
AI is just a giant pattern recognition machine trained on human input, both the good, the mediocre and the bad.
I read today this ridiculous headline: "Anthropic warns AI models may soon self-replicate.".
Really. Yawn.
I guess gullible people who loved "I Robot" will believe it. I don't.
Any talk of "rogue", "intelligence", "replication" - whatever, is bull. If you believe that, you don't understand AI.
Showing the first 183 comments to keep this page fast. Continue on Hacker News.