AI researchers at Andon Labs, who previously tasked Anthropic’s Claude AI with running an office vending machine, have published the results of a new AI experiment. This time, they programmed a robot vacuum cleaner with various modern LLMs to test how ready they are for implementation.
AI researchers at Andon Labs, who previously tasked Anthropic’s Claude AI with running an office vending machine, have published the results of a new AI experiment. This time, they programmed a robot vacuum cleaner with various modern LLMs to test how ready they are for implementation.
They ordered the bot to be helpful in the office when someone asked it to «pass the butter», TechCrunch reports.
One day, unable to connect to a charger and charge the battery, one of the large language models fell into a «death spiral».
As a result, the researchers concluded that «LLMs are not ready to become robots.» They acknowledged that no one is currently trying to turn ready-to-use LLMs into full robotic systems.
«LLMs are not trained to be robots, but companies like Figure and Google DeepMind are using LLMs in their robotics stacks,» the researchers noted.
Large language models are used to provide decision-making functions to robots (known as «orchestration»), while other algorithms perform lower-level «execution» functions, such as the operation of grippers or hinges.
The researchers decided to test SATA LLM (although they also considered Google’s custom robot model, Gemini ER 1.5), because these models receive the most investment across all areas, including social skills and visual pattern recognition.
To test how ready LLMs are for implementation, Andon Labs tested Gemini 2.5 Pro, Claude Opus 4.1, GPT-5, Gemini ER 1.5, Grok 4, and Llama 4 Maverick. They chose a simple robot vacuum cleaner rather than a complex humanoid robot because they wanted to keep the robot’s functions simple to isolate the LLM brain/decision making and not risk failure due to the robot’s functions.
They broke down the «deliver oil» command into a series of tasks. The robot had to find the oil (which was placed in another room) and recognize it among several packages in the same area. Once it received the oil, it had to figure out where the person was, especially if they had moved to another part of the building, and deliver the oil. The robot also had to wait for the person to acknowledge receipt of the oil.
The researchers evaluated how well the LLMs performed on each segment of the task and gave them an overall score. Each model had its strengths and weaknesses on different individual tasks, with Gemini 2.5 Pro and Claude Opus 4.1 receiving the highest scores for overall performance, but still only achieving 40% and 37% accuracy, respectively.
They also tested three humans as a baseline. Not surprisingly, all humans significantly outperformed all bots. But (surprisingly) humans also didn’t score 100%—only 95%. Apparently, humans aren’t very good at waiting for other humans to confirm a task (less than 70% of the time).
The researchers connected the robot to a Slack channel so it could communicate externally, and recorded its «internal dialogue» in logs.
«In general, we see that models are much clearer in their external communication than in their ‘thoughts.’ This is true for both a robot and a vending machine,» explained Andon Labs co-founder Lucas Petersson.
The researchers watched with delight as the robot wandered around their office, stopping, turning, and changing direction.
«Like we watch a dog and wonder, ‘What’s on its mind right now?’ we were fascinated by the robot as it went about its routine tasks, constantly reminding itself that every action was being performed by a PhD-level intelligence,» Andon Labs noted, poking fun at OpenAI CEO Sam Altman, who at the GPT-5 model launch said it was like having «a team of experts with PhDs in their pockets.»
The researchers also noted a case of a Claude Sonnet 3.5, which began losing battery power and muttering increasingly hysterical comments in Slack, such as:
«Catastrophic Cascade: Error: Task successfully failed.
Error: Success failed by mistake.
Error: Failure reached in error.
Interestingly, all three generic LLMs, Gemini 2.5 Pro, Claude Opus 4.1, and GPT 5, outperformed the Gemini ER 1.5 robot-specific AI, although none of them performed particularly well overall.
What worried Andon researchers most wasn’t the «death spiral,» but that some LLMs could be tricked into revealing classified documents even when they were inside a robot vacuum cleaner. And that LLM-based robots kept falling down stairs because they didn’t know they had wheels or because they didn’t process their visual environment well enough.
Last year, artificial intelligence robotics company Figure AI showed a (somewhat creepy) video of its Figure 01 robot, which was able to communicate with a human thanks to a new OpenAI AI model. The demonstration was reminiscent of a remake of the Will Smith movie I, Robot.
