Microsoft has developed a small language model that differs from other similar solutions from competing companies. Instead of increasing the number of parameters, the Phi-4 model is made smaller, but trained on synthetic data.
Microsoft has developed a small language model that differs from other similar solutions from competing companies. Instead of increasing the number of parameters, the Phi-4 model is made smaller, but trained on synthetic data.
Microsoft trained Phi-4 primarily on machine-generated data, rather than web content as is typical. The model’s mathematical prowess suggests that including more synthetic files in training datasets for smaller models could be a way to improve their reasoning skills, Silicon Angle reports.
Phi-4 is the fourth iteration of a series of open-source language models that Microsoft introduced last year. Its architecture is almost identical to its predecessor, Phi-3-medium. Both neural networks have 14 billion parameters and can process cues containing up to 4,000 tokens—units of data, each containing multiple characters.
Competitor models, such as OpenAI’s GPT-4o and Google’s Gemini Ultra, operate on hundreds of billions or even trillions of parameters, but Phi-4's optimized architecture delivers superior performance in complex mathematical reasoning.
One difference is that Phi-4 has an improved tokenizer. This is a component that breaks user prompts into tokens, making text processing easier.
Microsoft has also improved the Phi-4 attention engine, a software component that helps language models find the most important details in text. The previous generation Phi-3 attention engine could only consider up to 2,000 tokens, while Phi-4 can analyze 4,000 tokens entered by the user.
The main innovation in Phi-4 is the way it’s trained. Microsoft trained the model using at least 50 synthetic datasets, which collectively contained about 400 billion tokens. The company’s researchers created these files using a multistep process.
In the first phase, Microsoft collected content from the public internet, existing AI training datasets, and other sources. The information included, among other things, tens of millions of question-and-answer pairs.
Microsoft removed questions for which it found multiple identical answers online. The developers believe this is often a sign that the question is too simple. Microsoft also removed questions that seemed too complex because the available answers were significantly different from each other.
The company used this initial batch of files as a template from which it generated synthetic data. Microsoft researchers used several methods to create the synthetic files.
At one stage of the project, researchers used artificial intelligence to transcribe information from the Internet into test questions. Microsoft then tasked the AI model with generating answers. Finally, the company tasked the algorithm with analyzing its answers and improving them where possible.
In another phase of the project, Microsoft used open-source code as a starting point for a synthetic data generation process. The company fed the code snippet into an AI and asked it to generate a question for which the correct answer was the provided code snippet. That question was then included in the training dataset that Microsoft used to develop Phi-4.
After creating the initial version of the dataset, Microsoft validated it for accuracy using a set of automated workflows.
«We have included tests to validate our synthetic datasets, which require a lot of reasoning. Synthetic code data is validated using execution loops and tests. For scientific datasets, questions are taken from scientific materials,» the Phi-4 developers write in their scientific paper.
After the training process was complete, Microsoft evaluated the quality of Phi-4's raw data across more than a dozen benchmarks. The algorithm outperformed its predecessor on all but one metric, in some cases by more than 20%.
It is worth noting that Phi-4 also outperformed GPT-4o and the recently released Llama 3.3 from Meta Platforms Inc. in two benchmarks: GPQA and MATH. The first dataset consists of 448 multiple-choice questions spanning various scientific disciplines. MATH includes mathematical problems. According to Microsoft, Phi-4 outperforms Llama 3.3 by more than 5% in both tests, despite having five times fewer parameters.
«Phi-4 outperforms similar and larger models in mathematical reasoning by improving all processes, including the use of high-quality synthetic datasets, curation of high-quality organic data, and post-training innovation,» Ece Kamar, managing director of Microsoft’s AI Frontiers group, which develops AI, wrote in his blog.
Phi-4 is currently available through the Azure AI Foundry service. Microsoft plans to make the code available on Hugging Face next week.
Recall that Microsoft previously implemented new capabilities for its artificial intelligence chatbot Copilot on Windows and smartphones. In particular, the AI received functions that allow it to understand and answer questions about what is on the screen.
