Yuriy Paniv, one of the developers of the Ukrainian national AI model Lapa LLM, announced the creation of a leaderboard containing measurements of how well large language models work with the Ukrainian language.
Yuriy Paniv, one of the developers of the Ukrainian national AI model Lapa LLM, announced the creation of a leaderboard containing measurements of how well large language models work with the Ukrainian language.
It was created together by the LLM development team for the Ukrainian language MamayLM. «We measured all practical open language models that a user can run on their computer. In addition, the measurement project is completely open, other researchers can reproduce the measurements themselves for transparency,» Paniv said.
The leaderboard displays the performance indicators of language models on Ukrainian language tests used during the development of Lapa LLM. It evaluates models according to various criteria and tasks in NLP (natural language processing) in Ukrainian. The following skills are evaluated:
Currently, the leaderboard features such LLMs as Lapa LLM, MamayLM, Qwen, Llama, and Gemma. The top three in terms of average score include Ukrainian models Lapa LLM and MamayLM, as well as Gemma from Google.
In the near future, the leaderboard developers plan to add measurements of commercial language models from OpenAI, Anthropic, Google, DeepSeek, compare how models work with image processing containing Ukrainian text, and add measurements of model ethics. They also want to upload full benchmark logs (2 GB for each model), add a comparison of tokenizer efficiency, and increase the number of parameters.
The leaderboard can be viewed on Hugging Face, and its code is available on GitHub .
