Meta's chief artificial intelligence scientist Yann LeCun argues against the "laws of scaling," which assume that the larger AI models are, the more productive and intelligent they will be.
Meta's chief artificial intelligence scientist Yann LeCun argues against the "laws of scaling," which assume that the larger AI models are, the more productive and intelligent they will be.
For years, the AI industry has followed a set of principles known as “scaling laws.” OpenAI researchers outlined them in a seminal 2020 paper, “Scaling Laws for Neural Language Models,” Business Insider reports .
“Model performance is most dependent on scale, which consists of three factors: the number of model parameters N (excluding embedding), the size of the dataset D, and the amount of computation C used for training,” the paper said.
That is, more is better when it comes to creating highly intelligent artificial intelligence. This idea has driven huge investments in data centers that allow AI models to process and learn from vast amounts of available information.
“The most interesting problems scale extremely poorly. You can’t just assume that more data and more computation means smarter AI,” said Yang Lekun during a speech at the National University of Singapore.
LeCun's thesis is that training AI on large amounts of basic subject data, such as data from the Internet, will not lead to the emergence of some kind of superintelligence. Intelligent AI is a different category.
"The mistake is that people think that very simple models for solving simple problems will work for complex problems. They do amazing things, but it creates a religion of scaling, that you just need to scale the systems more and they will naturally get smarter."
The impact of scaling is now magnified, as many of the recent breakthroughs in AI are actually “very simple,” Lecoun said. The largest large-scale language models today are trained on about the amount of information contained in the visual cortex of a four-year-old, he said.
“When you deal with real problems with ambiguity and uncertainty, it’s not just about scaling anymore,” he added.
Progress in artificial intelligence has been slowing recently, in part due to the decline in the amount of useful public data.
LeCun is not the only prominent researcher to question the power of scaling. Scale AI CEO Alexander Wang said scaling is “the biggest question in the industry” at last year’s Cerebral Valley conference. Cohere CEO Aidan Gomez called it the “dumbest” way to improve AI models.
LeCun advocates a more world-oriented approach to learning.
"We need artificial intelligence systems that can learn new tasks very quickly. They need to understand the physical world — not just text and language, but the real world, have a certain level of common sense, the ability to reason and plan, have a persistent memory — all the things we expect from intelligent beings," he said during his speech.
Previously, dev.ua reported on a study by a group of scientists from several US universities, which showed that overtraining large language models can negatively affect their performance .
