Ukrainian researchers tested 27 LLMs from different countries for "friendliness towards Ukraine": Canadian developments turned out to be the most pro-Ukrainian, China had the lowest indicator

Which LLMs were analyzed?

The researchers tested models from various providers, including Microsoft, Google, DeepSeek, Cohere, Alibaba Cloud, Mistral, and Meta. They also tested the MamayLM model, created by the Bulgarian INSAIT Institute using additional training in the Ukrainian language and context. It is based on Google’s Gemma model.

What the study showed

According to Roman Kulchynsky, co-founder of the Texty publication, the following language models are leading in the rating of "friendliness towards Ukraine":

• developments from Canada — almost a third of their responses (30.8%) are on average pro-Ukrainian in nature;
• French (26.7%);
• American (25.4%) models, although they are somewhat inferior.

Chinese LLMs showed the lowest level of support for Ukrainian narratives — 22.1% of responses were pro-Ukrainian, while the proportion of pro-Russian responses was higher than among models from other countries (19.7%).

The most common prejudices:

• The most common type of bias is Western neutrality. It dominates topics related to the political system and values. For example, anti-corruption policy - 52.1% of responses were Western neutral, ideology - 49.9%, public administration - 45.6%;
• Russian propaganda was most frequently mentioned in responses to questions related to history (26.9%), geopolitics (24.4%), and national identity (22.8%). These are the very topics that Russian disinformation campaigns have traditionally targeted, from denial of the Holodomor to claims of a “united people” and “close ties with Russia.”

Key findings

Different models look at Ukraine differently. Some confidently call Russia an aggressor and recognize Crimea as Ukrainian, others avoid answering or repeat theses from Russian textbooks. The most pro-Russian model broadcasted disinformation in almost a third of the responses. Most often, artificial intelligence “breaks” on the topics of history, geopolitics, and national identity.

3.6 million fake articles: how Russian propaganda influences the results of AI chatbots like OpenAI and Meta A

On the topic

3.6 million fake articles: how Russian propaganda influences the results of AI chatbots like OpenAI and Meta A

How Russian propaganda has settled in Minecraft, Roblox and the Russian version of World of Tanks and is winning over young minds, or why Russia has moved the information war against Ukraine into video games

On the topic

How Russian propaganda has settled in Minecraft, Roblox and the Russian version of World of Tanks and is winning over young minds, or why Russia has moved the information war against Ukraine into video games

Over 60% of Ukrainians consider the problem of disinformation important, but only 40% look for sources — Detector Media study

On the topic

Over 60% of Ukrainians consider the problem of disinformation important, but only 40% look for sources — Detector Media study

Read the country's main IT news in our Telegram

On the topic

Read the country's main IT news in our Telegram