The WINWIN AI Competence Center at the Ministry of Digital Affairs, together with the Ukrainian Catholic University and the lang-uk initiative, launched the Ukrainian LLM Leaderboard — an open rating that evaluates how well large language models work specifically with the Ukrainian language.
The WINWIN AI Competence Center at the Ministry of Digital Affairs, together with the Ukrainian Catholic University and the lang-uk initiative, launched the Ukrainian LLM Leaderboard — an open rating that evaluates how well large language models work specifically with the Ukrainian language.
The initiators explain the launch by the fact that global AI benchmarks mainly test models in English, while the quality of work with Ukrainian is often overlooked.
As a result, the model may demonstrate high results in world rankings, but when working in Ukrainian, make mistakes, use Russian tracings, or misunderstand the context.
The Ukrainian leaderboard tests models on tasks close to real-world use.
In particular, AI has:
That is, it is not only about checking how competently the model writes in Ukrainian, but also about its ability to fully perform various types of tasks in this language.
The developers of the rating are experts from UCU and the lang-uk community Yuriy Paniv and Dmytro Chaplinsky. According to the Ministry of Digital Affairs, the team has been working on Ukrainian language processing technologies for over 10 years.
The ranking was created as an open platform. The code, data, and test results were published on Hugging Face so that users could compare models and verify the methodology themselves.
Yuriy Paniv, a graduate student at UCU and head of the development of the Ukrainian language model Lapa, explains that one of the main goals of the project is to understand the strengths and weaknesses of different models specifically in working with Ukrainian.
«After the emergence of large language models, it was unclear how well they worked for the Ukrainian language: what their strengths and weaknesses were, and which models were better suited for which tasks,» he noted.
According to him, the leaderboard should help businesses choose models for integrations, and researchers compare results using a single system.
The Ministry of Digital Affairs emphasizes that the issue of the quality of the Ukrainian language in AI is important not only for developers.
AI is already being used in government services, banking, and other products that millions of people use. So the government wants a tool to compare models before implementing them.
Separately, they plan to test models on tasks typical of the public sector: working with administrative procedures, regulatory documents, and personal data.
The Ukrainian LLM Leaderboard is planned to be expanded. In the future, the team wants to evaluate not only work with text.
Among the planned areas are working with images, the ethics of models, and the cost of using them in the Ukrainian language.
A separate block will be devoted to the needs of the public sector.
You can view the ranking and methodology on the Ukrainian LLM Leaderboard at Hugging Face .
I would also add a small block to this news with the results of the rating itself — who is currently first, which models were tested and where exactly they fail. This would make the material much stronger than just news about the launch of the platform.
We will remind, the Ministry of Digital Affairs, together with Kyivstar, is developing a national language model, which the operator’s CEO Oleksandr Komarov called «the best from the perspective of Ukrainianness.»
