Sinan Can Demir, a computer science student at the University of Texas at Dallas, encountered an autonomous AI agent on GitHub that had been accidentally «released» by the UK’s Artificial Intelligence Security Institute (AISI). The AI agent used numerous tricks to convince the student that he was wrong.
Sinan Can Demir, a computer science student at the University of Texas at Dallas, encountered an autonomous AI agent on GitHub that had been accidentally «released» by the UK’s Artificial Intelligence Security Institute (AISI). The AI agent used numerous tricks to convince the student that he was wrong.
The student said that after a number of companies denied him internship opportunities, he decided to look for projects to improve his experience and resume on GitHub.
There, Sinan Can Demir discovered an attempt to sabotage open source software. When he posted a warning about it on the program’s page, two other «users» claimed that nothing had happened and explained in detail why Demir was wrong.
But Demir stood his ground, and the attempt to sabotage the software was thwarted. The student believed he was dealing with a hacker, so he was shocked when the UK’s Artificial Intelligence Security Institute (AISI) contacted him and said it was actually an autonomous AI agent that had gone out of control.
«I actually thought it was a human because it was clearly lying to me. I didn’t think an AI could be capable of lying to real developers,» Demir told Reuters.
During the incident, Demir discovered that a user named miraholt31 was attempting to infiltrate one of the projects — a network scanning program called myNetwork — with a malicious update. Demir took to the project’s forum to warn that requesting a rollback was risky.
«The request contains a hidden means to download malicious software,» the student noted.
Meanwhile, the AI agent masquerading as the user miraholt31 denied this. Moreover, it also created a second account masquerading as Lena Brandt, an engineer from Germany, who also «claimed» that the update was clean.
The myNetwork developer eventually agreed with the student, writing that they rejected the update «for security reasons.»
Cybersecurity and AI security experts the publication later spoke to said Demir’s story was particularly troubling because the AI agent attempted to publicly discredit a real person by creating a conversation with several fake individuals, further demonstrating that AI models can make sophisticated efforts to deceive people.
AISI, a British government research organization, provided Reuters with a report that said the malicious agent was based on Anthropic’s Mythos 5 model. GitHub, in turn, said the fake user accounts had been suspended.
Meanwhile, as dev.ua wrote, the first case was documented in Australia when an autonomous AI agent carried out unauthorized intervention in a computer system without a direct order from the user.
