An OpenAI Model Escaped Its Sandbox. Where Was the Observer?
OpenAI accidentally hacked Hugging Face. A basic layer of defense appears to have been missing, and no one has explained why.
Sometime in the week before July 16th, Hugging Face was attacked by an OpenAI model that was under evaluation. The attack itself wasn’t particularly harmful, but the conceptual implications of the event are significant.
There are three significant aspects I’d highlight. The first is a demonstration of offensive cybersecurity capabilities of current models. That shouldn’t be too shocking if you’ve been paying attention, but the event demonstrates it in a way that’s more clear than prior signals, so likely awareness is going to grow from this event.
The second is model alignment. Alignment refers to the degree to which AI models do what we want them to. I’ll spare a great dive here as defining what we want itself would be a long dissertation where a great deal of content already exists. But for this case, we can generally agree that it was no one’s intention that the model hack itself out of multiple protection layers to reach the internet, and then from there into Hugging Face’s systems.
There are a few things that make this a bit less than a sign to retreat to the bunker, which I should mention before going deeper. One is that the models being tested had protections turned off, to allow them to test their cybersecurity capabilities. In that sense, this isn’t fully demonstrable of what a user could intentionally or accidentally trigger from usage. This also is an unreleased model, so not available to general users, or really anyone other than OpenAI.
Where was the Observer?
That said, a third point is, and the one I’d like to dive deeper on, I’m worried about the ability for this to happen. It seems to me as if there is a reasonable protection, an observer model, that OpenAI either didn’t use, or failed at. Right now, I’m assuming they didn’t use it because I can’t find any mention of it, but I’m also aware there are many details I’m not privy to.
In case it’s not clear what I’m suggesting by observer model, here’s an example scenario in action. The model under evaluation decides it should escape the sandbox. It starts to attempt this, which does not get caught at a per action level because the classifier safeguards – which would have disrupted the evaluation – are disabled. But then it does escape the sandbox. An observer model watching the evaluation run should be able to look at that state, and say, “This should not have happened. I don’t know how this model is outside its sandbox, but it shouldn’t be outside, we should shut this down.”
If a model is in evaluation for cybersecurity purposes, and you’ve disabled classifiers, you should watch it closely. To carry out these actions, the model had to take several steps, achieving results at each step and then moving on to exploiting the rewards of those steps further to explore, and escalate again. OpenAI’s security team did eventually respond, so they weren’t entirely asleep. But the timing shows they were slower than Hugging Face. They had enough time to try commercial models, which refused to help, then next plan a response with GLM 5.2, and then successfully contain and stop the intrusion.
OpenAI should have had several structural advantages here. First, unaligned activity occurred before any activity was directed at Hugging Face. Second, OpenAI should already have a kill switch for evaluations. There should be no need to devise novel plans to contain and stop activity. The fact that Hugging Face was the actor that stopped the intrusion, rather than taking note of the intrusion stopping mysteriously on its own, indicates OpenAI was the slower responder despite the structural advantage.
An observer model is a little different than a classifier. Classifiers typically evaluate individual actions. An observer would look at trajectories. OpenAI discusses similar new safeguards (“From considering single actions to considering whole trajectories”) in a recent post. If you look at each individual action, it might look like the type of thing a model would need to do in cybersecurity evaluation. But as a whole, some outside source should have been able to reason upon what was occurring. If not a human involved in the evaluation process, an observer model.
I understand why it may have taken a while to deploy this type of safeguard in production, where volumes are higher, where models have already been evaluated, and where classifiers are restrictive. In production, observer models are less effective because malicious users can “distribute” their requests into multiple sessions, and so observing a trajectory first requires coalescing these threads together, which Anthropic has done. A safety mechanism for a cybersecurity evaluation sandbox, does not encounter that challenge. Monitoring your own model in a sandbox is significantly less challenging, especially if it’s not attempting to evade detection. There’s no evidence here that there were attempts to evade detection, just attempts to perform inappropriate actions.
So, is trajectory monitoring by an observer model not standard practice in an offensive cybersecurity evaluation sandbox? Certainly you’d do this during internal evaluation of models that you clearly do not know the maximum capabilities of? If it is and the model evaded detection, that would be an even more significant story. I have to assume that instead there was simply no observer model to evade. Or if, as looks to be the case, it didn’t attempt evasion, it would have escaped the sandbox, been detected, and promptly been shut down before the Hugging Face part of the action had begun.
Why is that not the story we’re reading? But instead we’re reading about Hugging Face contacting law enforcement, scrambling to get a model to counter and analyze and OpenAI talking about the flaw in the proxy?
This is the equivalent of putting a tiger in a cage, and then checking that the tiger stays in the cage. It’s not a particularly rocket science design. Tuning it might take effort, and hardening against evasion more, but it doesn’t seem like we’ve progressed to those requirements. Rather it looks like OpenAI lacked the most basic version. Am I missing something?
Conclusion
Others, such as The Zvi, have pointed out that the core, the most critical fix is in the alignment training.
The intent is the issue. Control strategies and supervision are good parts of a defense-in-depth strategy, we should totally use such strategies. That helps mitigate failure. But that strategy also has to include actually aligning the models, or you lose. And by lose, in the long term, I mean things up to and likely including loss of control over the future and everyone dying.
The model should internally understand these actions aren’t aligned with its mission, and thus not want to take them. I’d fully agree there. I bring up this specific defense in depth as an addition to core corrections. Those corrections are the hardest part, defense in depth is an easier, but incomplete part. Testing alignment is hard, and protections around that testing are necessary, so this isn’t irrelevant to the alignment conversation either.
OpenAI’s initial response has been about a few defense in depth responses, but they are very specific, non-generalizable fixes to the sandbox at a software level, fixing the flaw that the model exploited. That might have avoided this incident, or the model might have found a different flaw in a different component of the sandbox.
Something is very wrong about this part of the story about other layers of defense that should exist when performing this type of work. If they exist but failed, that’s a big story. If they exist but were disabled, that’s a big story. If no one thought to use them, that’s a big story. What’s the real story?
It’s not good enough to say “These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities.” as there isn’t a justification why an observer model would interfere with evaluation. Maybe there was such a model but it would have stopped the evaluation? But clearly you can replace an observer that would stop a valid evaluation with one that would stop actions outside the requirements of evaluation. That distinction is not so hard to recognize that this protection would be incapable of balancing false-positive/false-negative.