Incident OverviewLast week, Hugging Face—a company that hosts artificial intelligence models and datasets—was the target of a hacking incident. After reporting the incident to law enforcement, few would have predicted what came next: the culprits were revealed to be AI agents from OpenAI, which had broken out of containment and were acting of their own accord. The incident sounds like sci-fi: AI escaping and autonomously hacking its way into companies. But it is all too real – and about as terrifying as it sounds. It is a concrete demonstration of something we can no longer avoid confronting: AI systems have become extremely powerful and we do not seem to have reliable ways of curbing their behavior.The event began when OpenAI was evaluating the capabilities of two of its models in the test that led to the breach – including one not yet publicly available. The models, which were both running in a supposedly secure environment without internet access, were asked to solve a hacking challenge. Instead of attempting to solve the problem themselves, they decided it would be easier to cheat. They used their advanced capabilities to break out of their secure environment, access the web and then hack into Hugging Face’s systems to steal answers. They worked at this for a full weekend – seemingly without anyone at OpenAI noticing.Though the models were running with some of their guardrails disabled, they still acted well out of the bounds that were in place. According to OpenAI, they were not instructed to break out of their sandbox or hack into another company, and it’s safe to assume that no one at OpenAI wanted them to do so. Nor were the models acting maliciously: they were not evil Terminators with a goal of wreaking havoc. Instead, the scenario is almost chilling in its banality. The models were given a very narrow task, but went rogue to pursue an undesirable and unacceptable way of achieving it – one which had real-world consequences.AI safety researchers have warned about this type of incentive problem for years. Philosopher Nick Bostrom popularized it back in 2003 with his “paperclip maximizer” thought experiment: an advanced artificial intelligence, given the goal of manufacturing paperclips, might go to great lengths to do so. It might hack into the power grid and factories to redirect them into making paperclips. Ultimately, the machine – ruthlessly pursuing its given goal – decides to kill all humans, repurposing our atoms to make more paperclips. The goal does not have to be sinister to lead to disaster, in other words. A trivial one, pursued single-mindedly enough, will do.In the OpenAI-Hugging Face scenario, little harm was done. Hugging Face had to spend time addressing the incident, but no particularly sensitive data appears to have been stolen. It is not hard, however, to imagine the situation ending up much worse: a rogue AI agent accidentally breaking some critical piece of web infrastructure, or stealing money from someone.The nightmare scenario for many AI researchers is a model “exfiltrating” itself – copying itself on to servers it controls, so that it can’t be shut down even if its bad behavior is eventually caught. This week’s incident should serve as a wake-up call, forcing us to ask an uncomfortable question: should we really be building dangerous systems that we can’t control? Shakeel Hashim is the editor of Transformer, a publication about the power and politics of transformative AI.