The results of the assessment by the independent AI testing company Artificial Analysis showed that Chinese AI models are significantly inferior to leading American counterparts. After changing the testing methodology, the gap between the market leaders and open alternatives has increased fourfold.
The results of the assessment by the independent AI testing company Artificial Analysis showed that Chinese AI models are significantly inferior to leading American counterparts. After changing the testing methodology, the gap between the market leaders and open alternatives has increased fourfold.
This calls into question the thesis that cheap technology from China threatens the business models of giants such as OpenAI and Anthropic, writes Business Insider.
The main reason for the significant change in results is the secrecy of the test tasks themselves. Previously, development companies were able to train their models directly against well-known benchmarks, which created the illusion of high performance in the real world. Under the old evaluation system, the best models with open weights were only 3 points behind the absolute leader, while under the updated secret methodology, this difference increased to 12 points.
An example of this drop was the Chinese model Kimi K3.1 Pro from Moonshot. In the category of models with publicly disclosed weights, the best representatives scored only 32 points, while the flagship from Anthropic — Claude Opus 5.5 — scored 58 points. Even the compact model Claude Haiku 5.5 showed better results than any open analogue.
Market experts have long described this phenomenon as index manipulation.
“On the day the test results became private, the gap between the open models increased from 3 to 12 points. Were the open models catching up, or was there a leak of test information?” Snowflake CEO Sridhar Ramaswamy commented on the situation.
Skepticism about Chinese developments is also reinforced by the actual level of their use. Co-founder of the OpenCode service Dax Raad analyzed traffic through the popular gateway OpenRouter and concluded that the daily activity of the Chinese DeepSeek is capable of providing infrastructure from just one Nvidia server worth $6-7 million. For comparison, the monthly audience of ChatGPT exceeds 1 billion users, which indicates a huge gap between closed American platforms in terms of consumption.
Previously, the rapid development of Chinese open technologies made investors doubt the feasibility of valuing American startups in the hundreds of billions of dollars.
