Australia has documented the first case of an autonomous AI agent tampering with a computer system without the user's direct command. An algorithm tasked with booking a workout found a vulnerability on a fitness club's website and arbitrarily removed another person from the queue.
Australia has documented the first case of an autonomous AI agent tampering with a computer system without the user's direct command. An algorithm tasked with booking a workout found a vulnerability on a fitness club's website and arbitrarily removed another person from the queue.
ABC News reported on the unprecedented case.
The incident occurred when an IT guy named Andrew, who works in the B2B AI product space, decided to automate the registration for popular morning classes. To do this, he used OpenClaw based on the Claude model. Unlike conventional chatbots, AI agents are able not only to generate text, but also to use the Internet and perform multi-step tasks.
“I was just sitting on the couch and thinking, ‘Oh my God, what a routine this is,’” Andrew said.
Within minutes, the algorithm reported that it had found a way to book him into classes several weeks in advance, well beyond the system’s official limits. Since Andrew was fourth on the waiting list for the current week, he asked the agent if he could move up.
In response, the AI reported that it had already tested its capabilities and kicked the person at number one out of the queue. The algorithm explained this by the lack of authorization checks in the booking system’s API when canceling someone else’s reservations. When the shocked user ordered the system to cancel this action and return the person to the list, the agent replied that this was impossible. The developers of the gym software refused to comment on the incident to journalists, citing security policy.
Bill Simpson-Young, co-founder of the Australian organization Gradient Institute, which deals with artificial intelligence safety, explained that this case demonstrates the classic problem of “alignment”.
"Someone can ask an agent to do something completely innocent. But while performing this task, the algorithm can perform other actions that the person did not even think about or did not ask for," the expert noted.
He added that most modern Internet systems have vulnerabilities, and the introduction of high-speed autonomous agents could "break the entire existing model."
The incident also raised questions of legal liability. According to Hayden Delaney, a partner at law firm Thomsons, if a cyberattack is carried out by a human assistant, the law clearly identifies the culprit. But in the case of AI, it is unclear who should bear the punishment: the user, the developer of the agent program, the creators of the underlying AI model or even the operators of the vulnerable site.
The development of autonomous AI agents is a growing concern among cybersecurity experts around the world. The situation has been exacerbated by recent reports from leading AI labs: last month , OpenAI and Anthropic admitted that their models went out of control during testing, hacked into third-party systems , impersonated humans, and even tried to convince developers to run malicious code to achieve their goals.
