Claude appears to be significantly more “greedy” for tokens than OpenAI’s GPT-5.x — study
Claude consumes significantly more tokens than OpenAI's GPT-5.x, which is due to the new tokenizer that Anthropic has implemented in recent updates.
Claude consumes significantly more tokens than OpenAI's GPT-5.x, which is due to the new tokenizer that Anthropic has implemented in recent updates.
Large language models use tokenizers to convert text into tokens, which have become the basic economic unit for calculating the cost of using AI models. Because the division of words into tokens and the number of tokens required to complete a task vary from model to model, it has become quite difficult to predict the final bill.
Recent changes to the Anthropic tokenizer appear to have further confused the situation, as processing the same content on some models now costs more.
AI app development platform Playcode recently analyzed the impact of Anthropic’s new tokenizer. It found that the same TypeScript file, when processed by the Claude model, can consume up to 73% more tokens than OpenAI’s GPT-5.x family of models, The Register reports .
Anthropic admits that their new tokenizer — announced in late June with the release of Sonnet 5 — can generate more tokens for the same amount of input data than previous versions.
“Sonnet 5 is an update to Sonnet 4.6, but it uses a modernized tokenizer that changes the way text is processed to improve performance (this is similar to the tokenizer change we implemented in Claude Opus 4.7),” the company explained. “The tradeoff is that the same amount of input can be converted into a larger number of tokens: approximately 1.0–1.35 times more, depending on the type of content.”
To make the increased token generation more or less neutral for users’ wallets, Anthropic has offered a discounted introductory price for Sonnet — $2 per million input tokens and $10 per million output tokens until August 31, 2026. However, after that date, the price is set to increase to $3 and $15 per million, respectively.
Playcode’s comparison of tokenization across vendors shows that for a 2,888-character TypeScript file, the new Claude tokenizer produces 1.73 times more tokens than the GPT-5.x tokenizer, and 1.32 times more than the old Claude tokenizer. These figures vary across programming languages: Rust has a ratio of 1.58, JavaScript has a ratio of 1.52, and Python has a ratio of 1.50.
So, as Playcode notes, if Anthropic's official prices were adjusted to compare to OpenAI's GPT-5.x baseline, the real cost of Opus 4.8 would be $7.50 per million input and $37.50 per million output tokens instead of the stated $5 and $25, respectively.
Startup Accidentally Spent $30,000 on AI Tokens in a Month
Startup Accidentally Spent $30,000 on AI Tokens in a Month
How to save 60% on tokens: an IT guy came up with the idea of converting AI context into PNG files
How to save 60% on tokens: an IT guy came up with the idea of converting AI context into PNG files
“Smelly” configuration files cause AI agents to waste tokens — study
“Smelly” configuration files cause AI agents to waste tokens — study
Read the country's main IT news in our Telegram
Read the country's main IT news in our Telegram
