Building GPT-4 required the collaboration of hundreds of people. Now, OpenAI claims it can rebuild GPT-4 with just 5-10 people, thanks to the experience gained from building its latest model, GPT-4.5.
Building GPT-4 required the collaboration of hundreds of people. Now, OpenAI claims it can rebuild GPT-4 with just 5-10 people, thanks to the experience gained from building its latest model, GPT-4.5.
On Friday's OpenAI podcast, Sam Altman asked three key engineers behind GPT-4.5: What is the smallest OpenAI team that could rework GPT-4 from scratch today?
At the same time, Altman noted that almost all of the company's efforts went into creating GPT-4, which involved hundreds of people, Business Insider writes .
Alex Paino, who led the machine learning training for GPT-4.5, said that retraining GPT-4 will now likely require only 5 to 10 people.
“We trained GPT-4o, a GPT-4-caliber model that we retrained using a lot of the same material as the GPT-4.5 research program,” Paino said. “It actually requires a lot fewer people to run the actual run.”
Daniel Selsam, an OpenAI researcher who works on data efficiency and algorithms, agreed that rebuilding GPT-4 will now be much easier.
"Just knowing that someone else has done something makes it a lot easier," he said. "I feel like the very fact that something is possible is a huge cheat code."
In February, OpenAI released GPT-4.5, claiming it was the company’s largest and most powerful model to date. It was designed to be “10 times smarter” than GPT-4, which was released in March 2023, according to Paino.
“We are 10 times better than what we did before with these GPT pre-trainings,” Paino said.
As for what's needed for the next 10- or 100-fold leap in scale, Selsam said it's data efficiency.
GPT models process information very efficiently, but there is a “ceiling that limits the depth of understanding they can extract from the data,” he said. “At a certain point, as the computations keep growing and growing, the data grows much more slowly and it becomes a bottleneck.”
Going beyond that, he said, would require “some algorithmic innovation” to extract more value from the same amount of data.
