The high-profile lawsuit against Meta has revealed a wealth of internal company documents. One document in particular has caught the attention of some AI researchers: it details a method for improving Llama’s AI models.
The high-profile lawsuit against Meta has revealed a wealth of internal company documents. One document in particular has caught the attention of some AI researchers: it details a method for improving Llama’s AI models.
These court documents describe how Meta researchers used a process called ablation to determine what data helped improve Llama's AI models, Business Insider writes .
Ablation is a medical technique that purposefully destroys tissue to improve brain function. In AI, it involves removing parts of a system to study how those components affect performance.
In Meta’s ablation experiments, the company replaced some of the AI training data with pirated books from the giant LibGen database. The company then retrained its Llama model to see how that affected the results.
We wrote about the Russian pirate library LibGen earlier. Meta tried to prevent publicity by filing a lawsuit about the fact that the company used LibGen to train AI. The case concerned copyright infringement, the so-called “Kadrey v. Meta”. In addition, it later turned out that Meta may have trained its AI models on unpublished books .
In one experiment, Meta added science and technology books as well as fiction books to the training data. In a second experiment, it added only fiction books.
In both experiments, Llama's performance improved noticeably in industry benchmarks, according to an internal Meta document (p. 18-19).
This suggests that Meta has the ability to assign meaning to specific learning data,” says Nick Vincent, an associate professor in the School of Computer Science at Simon Fraser University.
For example, one Meta engineer on LinkedIn mentions performing over 100 ablations during the development of Llama 4 and previous versions of the company’s large AI models.
Meta does not publish the results of these experiments, and other AI companies also keep this information secret, Vincent said.
One possible reason: If tech giants tell the world exactly what training data helped their AI models, the creators of that information will want to get paid—and they can calculate how much money they're owed.
The release of the results of ablation experiments could also affect the serious copyright lawsuits raging in the tech industry — a good example is this case, "Kadrey v. Meta."
In such cases, tech giants and AI startups argue that “training” machines based on material published online is not copyright infringement. And such internal documents that determine the value of certain content can work against them.
Meta’s ablation experiments focus on this first stage of learning, which uses mountains of data to help models understand the world. For example, to teach a machine to recognize a llama, you need to show it as many pictures of llamas and alpacas as possible so that it can tell the difference between the two animals.
Meta’s first ablation experiment showed that adding science, technology, and fiction books to the training data improved Llama’s performance by 4.5% on the industry benchmark BooIQ. Adding fiction books alone resulted in a 6% improvement.
BoolQ is a test of 15,942 yes/no questions that AI models must answer. The more questions they answer, the better their performance. A 5% improvement is equivalent to answering almost 800 additional questions correctly.
According to an internal Meta document, the performance gain from these ablation experiments was 5.5% on another test known as SIQA.
Peter Henderson, an associate professor of computer science at Princeton, posted several diagrams from the court document on Twitter that demonstrate these achievements.
Lots of internal Llama 2 data mix ablations revealed as part of discovery in the ongoing copyright litigation. Link below. pic.twitter.com/7YeRyYSEWV
— Peter Henderson (@PeterHndrsn) January 15, 2025
While a performance gain of around 5% seems small, in the AI race, any advantage is important.
“It’s actually a lot, because it’s very difficult to get every extra point on AI tests,” said Bill Gross, CEO of ProRata, a startup that tries to compensate creators for their contributions to AI.
