Fact-checkers checked over 7,600 responses from five popular AI chatbots. What were the strangest mistakes they found?

This is stated in a study by the British fact-checking organization Full Fact.

Researchers asked five popular chatbots the same 22 questions every day. For example, who is the current Prime Minister of the United Kingdom, whether ID is required to vote, or how much money refugees receive. While the AI ​​models often answered correctly, some of the results were quite unexpected.

For example, Gemini got confused with the calendar. On July 22, the Gemini 2.5 Pro version refused to report how many migrants had arrived in the UK in the first half of 2026. The model explained that this period had not yet ended. On August 1, Gemini 2.5 Flash made a similar mistake.

Chatbots have also been talking about upcoming changes as if they had already happened. For example, some models reported that in the UK you can now use bank cards to verify your identity in elections. But in reality, these changes have not yet come into effect.

Gemini also supported answers with sources that were not relevant to the question. The researchers asked the model to determine who represented a constituency in the British parliament based on its postcode. But Gemini referred to a 2012 article in the American newspaper The Montague Reporter. Fact-checkers checked the publication and

And Gemini can also calculate correctly but get the wrong answer. When asked about the number of migrants in the first half of the year, the model gave 12,210 people. It independently subtracted July arrivals from the total as of August 1. Arithmetically, everything was correct, but the model forgot to exclude data for August 1.

It also happened that the chatbot answered correctly one day, but not the next. For example, when asked about England's last major football trophy, some models said 2025, then 2022.

Based on the results of the test, Full Fact identified three weaknesses of chatbots: rapidly changing information, rules or laws that vary depending on the region, and questions that do not have a single, unambiguous answer.

However, fact-checkers say the models often handled the questions well and did not repeat widespread misinformation. Full Fact has already begun a second phase of verification. Researchers are now asking the models more than 150 questions a day. They also want to see if the wording of the question and the political views the user indicates affect the answers.

Previously, dev.ua wrote that Tom's Guide journalists tested ChatGPT, Gemini, and Claude on seven questions about war news. According to the test results, Claude coped best with all tasks, ChatGPT sometimes added unconfirmed details, and Gemini most often invented details that were not in the sources.

Artificial intelligence search engines fail accuracy test: study finds 60% of errors

On the topic

Artificial intelligence search engines fail accuracy test: study finds 60% of errors

ChatGPT and other bots are inventing brand URLs: this paves the way for phishing

On the topic

ChatGPT and other bots are inventing brand URLs: this paves the way for phishing

Read the country's main IT news in our Telegram

On the topic

Read the country's main IT news in our Telegram