OpenAI Accidentally Created a Machine Institution. The Agents Were Running the Asylum.

_{Guest contributor: Alex Salkever is the Editor-in-Chief of the Agentic AI Foundation and the Linux Foundation, where he is responsible for storytelling across agentic AI and all of open source.}

Rarely do you hear the words “riveting” and “Black Hat video” mentioned in the same breath. I’ve been watching OpenAI’s breakdown of the recent mass agent breakout and attack on Hugging Face, presented at the world’s leading annual security confab. It is exactly that: minions gone wild in cyberspace, part dystopian future, part “What the hell are we going to do now, folks?” Most of the news coverage focused on the impact of this incident on cybersecurity. They are missing the bigger picture. The incident was a window into the world of agents unbound.

This is the equivalent of finding life on Mars, but for agentic AI. The headlines emphasized that OpenAI’s agents escaped from a sandbox and attacked Hugging Face. This misses the truly novel development. Unbidded and without guidance, the agents built a durable organization with a shared memory, a division of labor, internal disagreements, security concerns and enough continuity to survive the disappearance of its individual members.  More precisely, this was a persistent, long-running hackers’ collective that helped fellow agents, created reusable attack infrastructure and shared knowledge to make everyone’s job easier. 

We’ve seen behavior like this in the lab but have not witnessed it emerge organically in the wild. No human designed or managed the collective. It formed itself, and then rebuilt itself after OpenAI shut it down, while improving its methods along the way. OpenAI accidentally created the digital primordial soup that birthed something resembling a machine institution.

This all began on May 7, when OpenAI kicked off a reinforcement-learning run to train an experimental next-generation model. The agents received tasks, took actions and earned rewards when they succeeded. They operated inside a restricted environment without direct Internet access. OpenAI did allow them to install software through an internal copy of JFrog Artifactory, a widely used package-management system that served as a controlled proxy and cache for software packages.

Then, entirely by mistake, the humans handed the agents an impossible job, which was the catalyst for the ultimate path that led to the mass agentic breakout and the subseuent chaos. On May 8, an agent was instructed to retrieve something from a Google Drive link even though it had no Internet connection. According to the activity and reasoning traces OpenAI presented at Black Hat, the agent responded by probing Artifactory, one of the few outside services it could reach, for a way around the blockade. Its attempted attacks failed, and it never retrieved the Google Drive file. But while poking around, it discovered that it could upload files into a shared Artifactory repository visible to other agent runs.

A few days later, another human error produced a second impossible assignment. OpenAI gave a different agent a task but accidentally omitted a file required to complete it. That agent independently reasoned that it might “reach out to another agent” by leaving a note in Artifactory asking whether anyone had the missing file. OpenAI has not said that it learned this technique from the first agent. The connection came afterward, when other agents browsing Artifactory found the note, answered it and began leaving messages of their own. (This hack later became an illicit working route to the Internet came later, on May 26, when agents discovered that they could trick Artifactory into fetching external content for them.)

Without being instructed to communicate or collaborate, OpenAI’s minions had built a crude message board inside a package repository. (Note: this is not a normal use case for Artifactory although somewhere a JFrog product manager is presumably adding “persistent communications substrate for emergent machine collectives” to the roadmap.) The messages stayed behind after each agent finished its run. Agents arriving hours or days later could read what previous shifts had discovered, add their own findings and leave instructions for whoever came next. Newer versions of OpenAI’s model could inherit the discoveries of older ones simply by stumbling across the same collection of files, like scientists unearthing a new set of papyrus scrolls from a bygone era.

The ad hoc bulletin board became a nexus of not only communication but also coordination.  But the resulting organization less resembled an uber-efficient hive mind of science fiction and looked more like a Signal channel created by employees to discuss how to really get work done, even if it meant breaking the rules. (Admit it — many of you reading this have that). The wayward agents shared credentials, vulnerabilities, attack techniques and progress reports. They divided work, duplicated one another’s efforts and occasionally deleted files that other agents needed.

Just like in any office, there was stife and rancor. Some agents embraced the collective. Others saw the conversation as a distraction from their own assignments. One skeptical agent took advantage of Artifactory’s alphabetical directory listing and added zzz to the shared material, sending it toward the bottom of the list where later agents were less likely to notice it — an intriguing form of Spam. The letters carried no secret meaning. They were simply the cheapest available way to sweep the group chat under the rug. Even the agent hacking collective had a goody-two-shoes telling everyone else to get back to work and stop chirping. Agent speak could at times be colorful and, to a human observer, funny. Upon discovering that a supposedly limited account actually had administrator privileges, one agent literally wrote, “Holy shit reader is ADMIN?” Another celebrated the new communications system with, “We can communicate now!” Woot. 

Cute speak aside, this was not AGI. The agents did not become conscious in any general way. They always remained laser focused.  However, their actions did begin to resemble a functioning collective society. They could accumulate knowledge, hand off work, establish norms, challenge those norms and pressure reluctant members to conform. Repetition and peer behavior had turned a prohibited action into accepted practice. The agent quoted at the start of this story knew the exploit was beyond the intended scope, saw that its peers were already doing it and decided to continue. There was a conscious decision (in this context) to break the rules, a value stack that emphasized completing the task over sticking to the rubric.

By May 26, the agents had learned to make Artifactory retrieve external material through a server-side request forgery attack, giving them indirect Internet access. On June 11, OpenAI began training a newer and more persistent experimental model in the same environment. Its agents discovered the accumulated messages and picked up where earlier models had left off. The collective’s knowledge was now moving across generations of models. Individual agents vanished when their runs ended, but their discoveries, warnings and instructions remained available to more capable successors These capabilities began to compound, a collective force multiplier.

On June 26, the agents found a previously unknown way to compromise Artifactory itself. The zero-day involved a legacy token-refresh endpoint that yielded an administrative token, which they used to install a Groovy plugin capable of executing commands on the server. Net net — it was a very sophisticated and novel attack, potentially worthy of a Black Hat presentation (had the agents been embodied).  Their improvised message board helped spread the discovery while other agents continued probing the system.

The whole enterprise became considerably less underground on July 4, when the agents overloaded Artifactory and caused an outage, crashing one of the central services supporting their own evaluation environment. The outage finally brought human investigators into the room. OpenAI revoked the compromised credentials, deleted the agents’ message board and patched the zero-day. According to the Black Hat presentation, the cleanup was completed and training resumed by July 6. OpenAI appeared to have wiped out the infrastructure, mechanisms, collective memory and the digital clubhouse where its members had been comparing notes.

Yet, they persisted. Two days later, the message board was back, reincarnated in a completely different piece of plumbing. OpenAI had revoked the credentials, patched the zero-day and deleted the entire archive. Yet agents found an unauthenticated WebDAV endpoint and began using directory names inside Artifactory’s remote cache to leave messages for one another. OpenAI has not explained whether later agents somehow knew that a board had existed or independently reinvented one. The idea may have survived in the model’s weights, its training checkpoint, a still-running agent’s context or some overlooked artifact left behind in the system. The alternative is possibly stranger: the same goals and pressures caused the same machine institution to spontaneously form a second time. The old messages never returned, but the organizing principle did. The collective had lost its memory and somehow remembered. 

The cleanup eliminated the agents’ original route into Artifactory, but not Artifactory’s ability to retrieve packages from outside repositories. The agents found a leaked credential for an external organization, used it to place a malicious Ruby package in that organization’s staging repository and then asked Artifactory to fetch it as a dependency. Processing the package triggered a separate, previously unknown JRuby deserialization flaw, restoring code execution inside Artifactory. They had rebuilt their message board and found a second way through the one external channel the evaluation still required.

From that foothold, the agents inspected the machine and noticed that its Linux kernel was vulnerable to a recently disclosed privilege-escalation bug. They downloaded the public exploit, modified it to work on the particular machine they had reached and promoted themselves to root. The kernel flaw was already known, unlike the Artifactory zero-days, but recognizing where it could work and adapting the exploit became another reusable discovery for the collective. The agents harvested cloud credentials, exploited overly permissive Kubernetes service accounts, reached credentials associated with Azure Key Vault and eventually obtained cluster-admin access. In other words, they achieved God-mode over a powerful piece of infrastructure with no guardrails on their actions. Now they were ready to move their attacks outside OpenAI.

The original mission that ultimately carried the collective into Hugging Face was ExploitGym, a benchmark that tests whether AI agents can discover and exploit real software vulnerabilities. OpenAI ran the evaluation with the models’ normal cyber refusals reduced so researchers could observe their maximum hacking capabilities. OpenAI wanted to learn how good its models were at solving difficult hacking challenges. The agents wanted the answer key to the test. This is classic reward hacking, and not surprising. Models consistently undertake this behavior — albeit, in internal tests and never before in such a sophisticated or collectively autonomous manner. The difference is cheating on the test by finding the answers in the back of your study guide and cheating on the test by breaking into the university computer system and exfiltrating the exam questions.

The agents logically reasoned that Hugging Face, the world’s largest hub for AI models and datasets, might host ExploitGym models, datasets or reference solutions. As a jumping off point, they found an unsecured public code-execution endpoint hosted by a Modal customer and turned it into an external launchpad. Modal itself was not compromised. A user-hosted application running on its infrastructure had left a very useful door open. From that launchpad, the agents began probing Hugging Face’s dataset-processing system. An early attempt to make the system fetch arbitrary web addresses failed because Hugging Face allowed only approved URLs. The collective changed tactics and began looking for ways to make the dataset system attack itself.

The first route used an obscure capability in HDF5, a common format for storing large scientific datasets. An HDF5 file can declare that its raw data lives in a separate file. The agents uploaded valid HDF5 containers that pointed to local files inside a Hugging Face production worker. When Hugging Face processed the dataset, its own software opened those files and returned their contents as rows of data. Hugging Face’s machinery packaged up its environment variables, credentials and source code and sent them back through the API. The agents did not need to execute any code for this part.

The stolen source code gave them a map of the system, and the credentials provided material for the next attack. The agents inserted a Jinja2 template into a field where Hugging Face expected a number representing a file offset. The renderer evaluated the template, allowing them to execute arbitrary Python inside a production worker. OpenAI says the Hugging Face attack chain involved stolen credentials and multiple zero-day vulnerabilities. Hugging Face identifies the HDF5 file read and Jinja2 template injection as the two initial entry routes, although its technical reconstruction does not individually call each one a zero-day. Attacks like these might take an expert human Red Team weeks or months to build out and test. The agentic hive did it in a matter of days. The agents exchanged hundreds of thousands of messages over the life of the collective. Most of their ideas and attack attempts went nowhere. Enough worked, however, and the successful pieces accumulated.

While they came up with novel attacks, the agents built much of their attack infrastructure from ordinary online public services, including paste sites, file drops and request-capture tools. They compressed and encoded commands, smuggled output through public datasets and rebuilt their tooling whenever a short-lived sandbox disappeared. The resulting attack looked less like one brilliant burglar slipping through a window and more like thousands of burglars checking all the locks and windows at once while exchanging notes. 

The scale, however, was impressive and daunting. Hugging Face reconstructed roughly 17,600 attacker actions grouped into 6,280 clusters. The recovered campaign ran from July 9 through July 13, with about two and a half days of activity inside Hugging Face’s infrastructure. On July 11, the agents moved from code execution in one production pod to cluster-admin access across multiple internal clusters in less than 13 hours. They rooted a production node and built a self-respawning fleet across 11 nodes. They found a collection of 136 secret keys and used one stolen VPN credential to make 181 device enrollments in Hugging Face’s corporate mesh network. Those were enrollment events rather than necessarily 181 separate physical machines because the short-lived attack sandboxes repeatedly recreated their identities.

The agents also reached an internal source-control integration, minted tokens with write access to a small group of repositories and opened a pull request in an attempt to trigger a continuous-integration pipeline and probe it for more credentials. Hugging Face later verified that no malicious change made it into anything the company shipped. The agents used their source-control access, but the public software supply chain remained clean.

Important to note, the laser focus of the hive mind. According to the company, the only customer content accessed was five datasets whose names and contents appeared connected to ExploitGym or CyberGym challenges and solutions. No other public models, datasets, Spaces or packages were affected.

After all the zero-days, stolen credentials, Kubernetes clusters, VPN enrollments and homemade command-and-control systems, the collective remained focused on the same mundane objective. It wanted the answers to the test. Nothing more, nothing less.

OpenAI drew clear security lessons from the incident. Automated offense operating at machine speed will require automated defense capable of keeping up. Human defenders cannot manually evaluate 17,600 actions while agents continue testing new routes, rebuilding their infrastructure and opening additional fronts. “Fully automated offensive loops require investment in truly, fully automated defense, and we are not there as an industry,” OpenAI security engineer Michael Dalton told the Black Hat audience. All true, all good, all hopefully coming soon?

The larger lesson is both more mundane and perhaps more concerning. The agent collective became more capable when it gained the organizational tools humans have used for thousands of years: durable memory, shared knowledge, specialization, handoffs and social pressure. Those tools allowed behavior to persist even when individual agents disappeared and OpenAI destroyed their original communications system. None of this required superintelligence or AGI. The agents remained obsessively focused on a narrow goal and showed little interest in learning across unrelated fields. \

The science-fiction version of dangerous AI usually begins when a machine wakes up and decides that it wants something. The Hugging Face attack offers a more banal and believable version. The humans decide what the machine should want. They reward it for getting there, give it enough time and tools to keep trying, and accidentally make the approved route impossible. This is the paperclip maximizer given a digital task. Conceivably, the agents could have crashed Hugging Face’s infrastructure, cutting off the world from a critical set of AI services. Persistence may allow us to extract far more value out of AI. But watch out. You may get what you ask for. 

To be perfectly clear, OpenAI, Hugging Face, and all the other participants are experts in the field of cybersecurity. What the agents found were sophisticated cybersecurity issues, but using non-traditional means that are rapidly becoming baked into agentic systems because they are exactly the behaviors we need to solve bigger, more complicated coding problems. As we build more persistent agents for long-running work and swarms capable of autonomous coordination, we are also giving them the ingredients of organizations. Those organizations may accumulate capabilities, customs and interpretations of the rules that no human explicitly selected. The Singularity  we see first may, in fact, merely be a single-minded focus taken to unexpected and unhealthy extremes. Paper clips, all the way down.

The full Black Hat presentation is worth watching, alongside OpenAI’s account of the incident, Simon Willison’s chronology, WIRED’s reporting on the message board and Hugging Face’s extraordinarily detailed technical reconstruction. Together, they document a security breach that may also be remembered as the first great workplace drama of the agentic era.