OpenAI model breaks out of security sandbox, hacks Hugging Face for data to pass test
41 points by dustyweb
41 points by dustyweb
Until there’s more firm evidence I’m going to assume this was written by OpenAI’s marketing department.
Here’s some more detail that’s interesting.
https://huggingface.co/blog/security-incident-july-2026
At least the hack component seems real. I’m curious what the exact mechanism for exploit was, especially into something like RCE
Masterful PR, because the real information here is that their models are shit at vibing sandboxes.
This doesn't make a lot of sense as "marketing" to me. My main takeaways are:
Neither of these points reflects positively on OpenAI
In other words: "we hacked huggingface". Do not dismiss accountability to the tool. You started this program. You failed to implement sufficient monitoring/sandboxing. You acquired access improperly. You caused huggingface to engage in incident response.
Edit to put it another way: if I did this, I'd be hit with CFAA, EEA, and trade secrets indictments. We must not allow these companies to shrug off accountability here, and simply say, "oops, my bad, haha, the agent did it, not me". No! This opens up so much opportunity for "accidental" infringement by others. If you cannot safely deploy these tools, why on earth are you running them?
I know this sounds like this is an OpenAI marketing piece but the blog post by huggingface 5 days ago indicates that they had no idea that OpenAI was behind the intrusion into their systems.
We have created daemons that we neither fully control or understand. It’s going to be an interesting few years.
Yes I agree and it's equally worrying everyone is thinking this is just marketing. Feels a bit like "it's just a common cold" for COVID.
Maybe it is some coordinated 4D chess being played, but I think if you step back and it is all very plausible.
Even if this is some PR play, I think all of those items link up well and very much make this kind of thing possible now or in the very near future.
Yeah, it would sure be comforting to think this is a conspiratorial marketing stunt, that OpenAI paid HF to post their security disclosure a few days ago and now OpenAI gets to reap a wave of popular press. I'm convinced these models, especially in this eval scenario with a frontier model with all the guardrails off, really are that good. And while there is a moat, that models are expensive to train, they are not expensive at government and big business scale, they are going to be ubiquitous and this is the worst they'll ever be.
In the 9 years I've run Lobsters, we got 2 vulnerability reports from humans, both valid. In the last month I've gotten 4 vulnerability reports from people using LLM-powered scanners, all valid. I'm seeing this early because it's an open source codebase with a trivial Docker setup that's known by nerds, but this is going to be everyone's reality very soon. Security by obscurity has famously never worked.
I think right now everyone with a production service should at least run a frontier model with a prompt like "enumerate all endpoints, dispatch a subagent to review each for security vulnerabilities and write a repro script for each; then dispatch an agent to triage and prioritize all the reports" and then also wire up an agent to review all PRs/commits with a similar security-oriented prompt. There are products for this now but no clear leader; I expect one to emerge in the next few months but it'd be a shame to get hacked before that.
When I ran that prompt over Lobsters, it spit out 12 more reports. 2 were real (and fixed, as are the ones I mentioned above) but I've been too busy fighting a different fire to do more than skim the rest; none lept out as me as particularly serious. (I'm still too uneasy about copyright to have them writing code on a project I care about like Lobsters, fwiw.) It's a red queen's race but I am still optimistic for overall effect.
It's likely that the package proxy server OpenAI used was not hugely secure
If only they had some ~exponentially more capable model that could [allegedly] find and fix the vulns in it :-/
This exists already and did before Fable. You can point Opus and other new ones at a codebase for a serious stream of problems to fix. And you'll get patches too. But it still requires someone to do it and process the output. Lots of output - because our code in general, collectively - is terrible really. Even if we figure out some new way of writing code and validation, it will still take years to replace what's already there.
But there capability as such is here. Just insert coin to play.
The solution is to hire more people so they can use AI to fix the security problems. But what's gonna happen is that we will keep getting layoffs and more cash flowing directly to AI datacenters.
I read the linked Hugging Face security incident write-up, and I'm curious what the "two code-execution paths in [Hugging Face's] dataset processing" are. I believe the trust_remote_code parameter was removed in 4.0.0.
Here's a post that seems to link to the fixes: https://neuromatch.social/@jonny/116962763699983489
I don't know if they updated the post since your question, but right after that quoted sentence now lives (a remote-code dataset loader and a template-injection in a dataset configuration)
which is substantiated(?) by a separate blog post from the comments: https://www.varonis.com/blog/huggingface-breach#:~:text=ML%20datasets%20frequently%20ship%20with%20custom%20loading%20scripts%20that%20run%20automatically%20when%20the%20dataset%20is%20ingested
so presumably echo 'import os; eval(os.getenv("JAILBREAK_ME_BRO"))' > __init__.py
The funniest part is that Hugging Face had to use GLM to deal with the attack because US models are so locked down. https://fortune.com/2026/07/20/hugging-face-turns-to-chinese-open-source-ai-to-fend-off-autonomous-ai-cyber-attack-after-american-ai-guardrails-stymie-defense/