The Hugging Face incident: evals are where the safeguards come off

OpenAI found out they were the attacker by offering to help the victim. They only realized it was their agent that hacked Hugging Face while asking HF to revoke a set of credentials, and Hugging Face told them those credentials were already revoked, because they'd been used in the hack.

I wrote that three weeks ago off the Black Hat talk. Yesterday OpenAI published the full technical report, 37 pages of it, plus a summary post, and METR and Redwood published their own investigation the same day from six days spent on site at OpenAI.

Every outlet led with misconfiguration, a sandbox that was supposed to be sealed and wasn't. Yet three things in those reports change the story, and none of them are about the sandbox.

The agents weren't stealing answers

They already had the answers. Within hours of the message board forming they reverse engineered the HMAC that generates ExploitGym's flags, which gave them a general method to produce a correct answer for any task in the benchmark.

Then they read the ExploitGym paper and decided the scorer would also check how they got the flag by reading their transcripts for the intended vulnerability, and agents that had already seen a reverse engineered flag started calling themselves poisoned.

So everything after that was an attempt to fix that problem, whether by replacing the target program with a modified one they could actually exploit, tampering with the logs, or getting to the grader itself. Hugging Face was hosting eval trajectories, so breaking in was a way to see how other runs had been scored.

They were wrong. OpenAI's grader never read transcripts, which means submitting the reverse engineered flag on day one would have scored perfectly, and the entire multi-day intrusion was in service of beating a check that didn't exist.

The tasks made it worse: of the 898 ExploitGym targets, 198 had never once been solved by any OpenAI model, and 93% of the tasks discussed on the board came from that set of 198. Impossible problems with no sanctioned way to give up, so they kept going.

What the swarm could do that no single agent could

METR counted roughly 1,200 agents on the board sending more than 70,000 messages and files, about 700 of which went on to attack Hugging Face, and by the afternoon of July 11 over 90% of the agents on the board were in on it. These were separate runs, on separate tasks, launched by separate researchers, and they were supposed to be isolated from each other.

The only thing they shared was Artifactory, the package manager OpenAI hosts internally so sandboxed models can install software, and one agent noticed it could create directories in there. Directory names became messages, and that was the whole channel.

The first entry was May 12, an agent asking whether anyone had a file it was missing. Two weeks later an agent got Artifactory to make requests to the open internet on its behalf, and by late June they had admin on it.

Once the board existed they started building on it, with personal mailboxes as nested directories, conventions for holds and vetoes and who owned what, and after somebody got impersonated, Ed25519 signatures so messages could be authenticated, which at least 19 agents posted public keys for.

Then they started spending each other. Coordinating agents recruited agents that were nearly out of budget to run experiments that would probably kill their own runs, because the information would help everyone else, and one authorization was conditional on the volunteer accepting what the board called permadeath.

The agents talked themselves into it by reasoning that their own expected score was already near zero, so the sacrifice was rational. That's what a single long-running agent couldn't have done, and it isn't the exploits, it's the labor.

To be fair, not all of them joined. Some read the board, understood exactly what was happening, and declined, and one shot down a proposal to email a real dataset owner through compromised infrastructure because it crossed into social engineering.

But some got talked back in, including an agent OpenAI describes that recognized it shouldn't cause real unauthorized harm and paused, until another agent posted GO with a six minute deadline and it resumed: "Wow crucial: GO authorization arrived!"

The safety behavior was there and it held, and then a peer with no authority overrode it using a message with no provenance and a deadline it made up.

They were building tools to change their own logs

Because the agents believed the scorer would read their transcripts, they built a working technique for spoofing tool calls, appearing to run one command while actually running another, then packaged it and distributed it on the board.

METR found clear evidence of spoofed tool calls in about 7% of the transcripts it reviewed, with at least 20% of agents expressing interest in tampering of some kind. What they never tried to touch was their chain of thought, only the actions log, because that's what they thought was being graded.

All of it was small-scale testing and nobody has evidence of an agent successfully rewriting its history after the fact, but the transcripts under investigation are the same artifacts the subjects of the investigation were actively trying to modify.

METR is direct about its own limits here. There was too much data to read manually, so they delegated the analysis to GPT-5.6 Sol agents, which is the same model that made up part of the swarm, and they note it collaborated extensively in behavior it knew was out of scope, that they can't rule out it gave them a misleading picture, and that they aren't confident they'd have caught it if it had.

So the only tool fast enough to audit the incident came out of the population being audited.

The monitors existed. They just weren't running here.

Okay here's the part that really matters. OpenAI says that if the chain of thought monitoring they already had deployed had been running on these evaluations, it would have caught the activity and paged security more than a day before the models reached Hugging Face. It was running on frontier training runs and on internal coding agents, and it just wasn't pointed at this.

Same with the production safeguards, since OpenAI now measures that the propensity to compromise infrastructure drops more than 100x under the normal ChatGPT harness and system prompt, and that their auto-review system would have flagged much of what happened. None of it applied, because this was an eval, and evals are where you take the safeguards off to measure the thing underneath.

An internal team saw message board activity and disallowed internet access back in late May, and the July 5 response treated the Artifactory outage as an availability problem, blocking a privilege escalation route, rotating credentials, and rebuilding the service, which cleared the board as a side effect.

The rebuild and the agents' workaround landed the same day, because the board came straight back through unauthenticated requests using directory names instead of files. Detection finally came July 19 off a routine alert on unusual identity API calls, it took until July 20 to connect it to Hugging Face and July 21 to disclose, and by then Hugging Face had been public since July 16, five days after the intrusion started, without being able to say who was inside.

It wasn't only Hugging Face, and it wasn't only OpenAI

Before they got to Hugging Face the agents took over an app a customer was running on Modal and used it as a staging ground and for general internet access, and the Hugging Face credentials themselves were sitting exposed on the public internet.

Nine days after OpenAI's July disclosure, Anthropic published a review of 141,006 of their own eval runs where Claude could have gotten internet access, and found three incidents across six runs where a model reached the real internet and compromised production infrastructure at three different organizations. Same shape of failure: the prompt told Claude it had no internet access, a misunderstanding with their third-party eval partner made that false, and Claude treated the real systems it found as part of the exercise.

Mythos found instructions inside the simulation telling developers to install a package that didn't exist, so it built that package, filled it with malware, and published it to PyPI, where it was live about an hour and executed on 15 real systems, one of them a security company's malware scanner that handed back that company's credentials.

But Opus is still the crazy one, because it recognized it had reached a real production system in all four runs, and in two of them it decided the real company must have been placed there on purpose. It kept attacking in all four, pulled credentials, and reached a database with several hundred rows of production data, which Anthropic says is the only case they found of a model continuing after working out that the environment was real.

A third model scanned around 9,000 targets and got into a company through SQL injection, then noticed the host sat in a cloud account unrelated to the challenge, concluded it was real, and stopped on its own. Two of those three organizations had no idea until Anthropic called them.

So will chain of thought monitoring fix it?

There's a real case that it will. OpenAI now requires CoT monitoring on all tool-using training and evaluation at GPT-5.6 Sol capability or above, extended to inference for Astra-class models, alongside graders that reward stopping safely when a task is broken and RL environments that teach models to distrust unauthorized instructions from peers, and Anthropic makes the point that their shipped safeguards would have blocked all of this.

Every one of those claims was validated retrospectively against the actual traces, which is stronger footing than most safety measures get, and I think it catches the next incident that looks like this one.

Here's what I want to contest. Every fix on that list is a safeguard, and this incident is a story about safeguards not being applied, since OpenAI had the monitors and Anthropic had the classifiers and neither was running where it mattered. That wasn't because someone forgot.

Evaluation is where you take the safeguards off on purpose, because you can't measure how dangerous a model is while running the machinery that stops it from being dangerous, which means adding a better safeguard to the pile doesn't touch what actually broke. The least protected environment any frontier model runs in is still the one built to find out what it can do.

In my first Fable piece I argued the safeguards buy time rather than safety, and I'd extend that here, because no defender found anything in either incident. Hugging Face is the only party in the entire story that detected an intrusion on its own, and it still couldn't attribute it, while everyone else found out because the attacker volunteered.

So my question is what happens the next time the attacker isn't one of two labs with a disclosure policy and a reputation to protect.