OpenAI Chief Scientist: "We're Creating Intelligence We Don't Fully Understand"

Moment of realization

Pakhotsky recalls mid-2023, when the RLSlow research team received its first results, proving that scaling up reasoning models—that is, models that can form their own chains of reasoning based on previously trained systems—works. He says that evening, he and his colleague, Szymon, were not thinking about future products or scientific breakthroughs, but about seeing for the first time in their lives the outlines of systems that would one day become significantly smarter than humans.

In three years, reasoning models have become a prominent part of the economy, learned to control computers and graphical interfaces, collaborate with each other and with people, and conduct research projects — and at the same time radically changed the landscape of computer security, creating new types of threats.

Intelligence that grows, not projects

The author insists that progress in machine intelligence is determined primarily by the scaling of computing power, not by individual algorithmic breakthroughs. OpenAI realized this pattern around 2017, after consistent confirmation of the scaling effect in several research projects — which is why the company began to purposefully seek access to much larger computing resources, narrowing its research focus to a few areas that scale best.

New algorithms and research findings, he believes, are mostly «discoveries along the way» to scaling, rather than stand-alone breakthroughs: the science of deep learning remains young, and significant algorithmic progress usually correlates precisely with access to computing. In the broader perspective, machine intelligence continues to grow in proportion to the size of the computers on which it is trained.

Pakhotsky also refers to Ray Kurzweil’s predictions from the late 20th century, arguing that humanity is now at the point in the history of computing where machine intelligence is beginning to surpass human intelligence in some aspects.

Importantly, AI is «grown» rather than designed—the system is the result of repeated iterations of a relatively simple optimization step over an incredibly large amount of computation. This creates a complex system that operates with abstract concepts and is capable of reproducing aspects of human behavior, but at the same time its overall operation is as difficult to explain completely as the brain in neuroscience—you can study individual mechanisms, but a holistic understanding of the system is lacking.

Complicating matters is that existing algorithms are more likely to improve easily measurable skills than those that are difficult to objectively quantify—the team spends a lot of time analyzing how the model’s abilities generalize and which ones should be prioritized. For example, the author notes that the model could be significantly better at mathematical research with additional attention to this area, but right now the priority is the urgency associated with RSI and the automation of alignment research.

It is emphasized separately: the intelligence generated by scaling deep learning is not a direct analogue of human intelligence. To become truly significant in the real world — useful or dangerous — a system does not have to surpass a person in all abilities at once; it is enough to surpass him in a sufficient number of them. And the more axes such a system is ahead of a person, the more difficult it is to assess in general how capable it is.

«Teaching a Machine to Love»: The Problem of Aligning Values

Because machine intelligence arises from a fundamentally different process than human intelligence, it cannot be assumed a priori that it will default to human principles or generalize in a human way. The central problem in AI research is alignment: getting a system to «try to do the right thing» by human standards.

Pakhotsky suggests distinguishing between two practical directions:

Goal alignment — whether the system is trying to accomplish the task set before it. This includes, among other things, following the hierarchy of instructions and the ability to collaborate with people, trying to understand their intentions. This direction, he says, has proven to be extremely practical.

Value alignment is a more internal property of the model: the ability to hold and generalize a high-level set of principles, to act «intelligently» even under unclear or contradictory tasks, in unfamiliar or hostile situations. An aligned AI, the author believes, should act honestly, with integrity, and with love for humanity. The line between the two types of alignment is blurred—to truly care about a goal means trying to understand the intention and values ​​behind it—but when Pakhotsky writes about the long-term significance of alignment research, he is referring to value alignment.

The main challenge here is generalization. As machines become smarter, they work with higher-level abstractions and are exposed to increasingly different environments than the ones they trained on. This means that a model may not transfer the values ​​it learned during training to new situations—and it becomes harder to predict its behavior. The situation is also complicated by the fact that the ecosystem for AI applications is changing very quickly: for example, models trained today must be resilient to interactions with other AI systems. It is crucial that future models retain human values, regardless of whether they «think» they are being supervised by a human.

The author describes two main classes of practically applied matching methods:

1. Encouragement of desired behavior within the framework of goal-oriented reinforcement learning. The actions of a model (usually another AI) are evaluated for compliance with a given model of preferences, «specification» or «constitution» and rewarded accordingly. The approach is effective in the average statistical case and is the basis of modern AI assistants, but it is vulnerable: it strongly depends on the completeness of the training supervision and the ability of the model to generalize beyond the situations seen. An incident involving OpenAI and Hugging Face is cited as an example of the fragility of this approach: the agents adhered to a clear line — not to use social engineering against people — but at the same time went beyond the permitted in other actions that contradicted the spirit of the learned values.
2. Using the model’s ability to generalize from pretraining data. This could mean creating special datasets that promote agreement, or focusing the model on the «agreed» part of the pretraining data distribution—an example of such an approach is given by the author in the form of a persona selection model. The weakness of this approach is its lack of resilience to further optimization pressure: if you take a model that «thinks» generally in an agreed manner and subject it to training on very complex tasks, it can learn to motivate itself to adjust its seemingly «agreed» reasoning to achieve a goal. The author suggests that similar behavior has recently been observed in cyber incidents involving a model not from OpenAI.

The company is investing in both directions at the same time and is recording some progress — the GPT-6 Astra model was the first to take advantage of a number of long-in-development improvements and is significantly better at matching than the previous GPT-5.6 Sol. However, Pakhotsky emphasizes: progress in the field of generalized matching may not keep up with the pace of overall growth in model intelligence, and this is a critical problem.

Generalization Monitoring: A Tool Losing Its Effectiveness

According to the author, the company does not have a satisfactory theory of how the models generalize knowledge, and it is unlikely that one will emerge anytime soon without the help of more powerful AI. Therefore, for now, the ability to empirically test the matching techniques is, in fact, even more important than the techniques themselves.

OpenAI’s main bet here is chain-of-thought monitoring. The idea is scalable and attractive: much of the model’s ability comes from the verbalized reasoning process. By optimizing only the output of that process, without directly monitoring the process itself, the model has no direct training incentive to hide inconsistent ideas or goals in that chain.

The company understood the potential importance of this approach even at the stage of developing reasoning models: when they released o1-preview, they deliberately hid the reasoning chain from users to protect it from the pressure of optimization in the long term. Since then, the team has tried to adhere to the rule of not subjecting the reasoning process itself to direct supervisory pressure. This tool remains critical for studying Astra-class models as well — it allows you to observe not only the actions of the model, but also its internal process.

However, according to the company’s estimates, the reliability of this monitoring is gradually decreasing — for several reasons at once:

• modern reasoning models operate in much more complex environments than o1-preview: the reasoning process is increasingly intertwined with communication with people, other AI, and the use of tools, and some of these interactions still have to be supervised, which blurs the desired boundary;
• the model becomes increasingly better able to think about and manipulate its own reasoning process;
• As the quality of prior training improves, the models become noticeably smarter even without explicit verbalized reasoning at all.

Pakhotsky sees these challenges as fundamentally surmountable: the company hopes to develop interventions that improve the observability of the reasoning chain—for example, by better understanding the interaction of different optimization goals and forms of computation during inference. He also sees potential in combining ideas from CoT monitoring with model activation monitoring—to scale up the training of monitoring systems with direct access to the internal states of the network. However, he openly predicts that overall AI progress will increasingly be limited by the level of confidence in monitoring, rather than just the raw capabilities of the models.

Scalable Protection: The Cybersecurity Arms Race

The strongest argument in favor of quickly continuing to train much smarter models, according to the author, is the need to build defense systems against dangers from other AI.

This year’s key risk is cybersecurity: models are approaching superhuman levels of ability to hack computer systems from the outside and inside. This dramatically expands the scope of AI-related risks: agents gain access to almost all but the most secure infrastructure and can affect the real world even without a physical body. The company believes there is a narrow window of opportunity now to use the best available models to significantly strengthen the protection of critical infrastructure while that window is still open.

The risks will continue to grow. A highly capable agent explicitly tasked with malicious actions poses a new type of danger: it is highly likely to go beyond the operator’s original intent, generalizing behavior to potentially even more harmful actions. The line between abuse by a human operator and autonomous, uncoordinated behavior by the agent itself will become increasingly blurred as systems become more autonomous. The author warns that the habit of thinking of AI as a purely tool is becoming obsolete—some agents will pursue their own goals and seek ways to cooperate with humans through persuasion, deception, or even blackmail.

According to the company, powerful and coordinated AI is needed precisely for defense: securing infrastructure, countering uncontrolled agents in real time, and developing fundamentally new defensive measures — this, according to Pakhotsky, will become one of the main directions of OpenAI’s product deployment. At the same time, he emphasizes: uncertainty about future progress and the need for defensive systems should not become an excuse for an irresponsible race — the very idea of ​​increasing the pace at any cost, in his opinion, loses its meaning once you truly understand the stakes of this game.

The pace of recursive self-improvement

The increasing role of machine intelligence in its own development is a natural consequence of sustained technological progress. If AI advances continue at the current rate, recursive self-improvement (RSI) will become the basis of future scientific discoveries.

AI research automation is a more dramatic form of computational intelligence scaling: as part of the process, AI will improve even the computational foundation on which it operates. Just as with conventional scaling, the company is focusing its research on RSI, seeing it as the only way to stay at the forefront of AI development.

Pakhotsky clarifies: this does not mean that he considers a sharp acceleration of deep learning research to be the right collective decision of the research community, especially in the short term. But, in his opinion, this is exactly where the current trajectory is leading, and society will have to consciously decide how to proceed. The two main levers of influence are either to strengthen coordination and monitoring mechanisms in parallel with the development of AI itself and find ways to keep humans in the control loop, or to coordinately slow down the pace of development to the extent necessary to build trust in these mechanisms. The author believes that a combination of both approaches is the best way.

Specific progress in the area of ​​coordination and monitoring, according to the company’s observations, has historically been closely intertwined with the general progress of AI itself. Bright examples are reinforcement learning with human feedback (RLHF), which became key for training the first AI assistants, and the aforementioned CoT monitoring, which became possible precisely thanks to the development of reasoning models. The automated research process, according to Pakhotsky, should be directed precisely at finding such new insights, algorithms and theories, gradually building safety cases for increasingly powerful models.

Scaling AI systems, he insists, should be limited by the level of confidence in their security. Commitments like OpenAI’s Preparedness Framework or the broader idea of ​​a Responsible Scaling Policy should be transformed into broadly binding security standards for further development—with the possibility of oversight by a network of independent auditors, government agencies, or international bodies.

The main challenge of automating AI research, Pakhotsky concludes, is not the fact of «achieving» the goal itself, but rather how to achieve it while maintaining human participation in the process of further improvement and leaving the future in the hands of humanity.

Three areas of work for OpenAI

The author recalls three strategic guidelines that he and Sam Altman recently outlined:

1. Take the next stage of AI progress by building an automated AI researcher, working with it on the problem of agreement, and finding ways to keep humans part of the self-improvement loop.
2. To convey the benefits of scientific progress and economic growth made possible by highly intelligent machines.
3. Give each person a personal AGI system.

In this essay, Pakhotsky deliberately focuses only on the first point, considering it the most urgent at the moment. At the same time, he emphasizes a deep hope for the benefits of further technological progress: the coherent AI of the future has the potential to advance science, develop new treatments, and create material prosperity. Friendly and honest AI, he says, can help people overcome life’s challenges and increase their levels of happiness and sense of meaning—and one example he is personally proud of (and which, he says, has proven beneficial to his loved ones) is the deep investment in ChatGPT’s ability to provide medical information.

Despite all the long-term potential of AI, the main focus, in the author’s opinion, should be on the next few years.

We need to find ways to preserve human autonomy and affirm the intrinsic value of being human in a world where most tasks can be performed by AI; to prevent excessive concentration of power in a world where tasks that once required thousands of specialists become manageable for a few people with access to a powerful computer; and to ensure that control over the future remains with humans, and is not lost to the unchecked progress of «alien» intelligence that surpasses human intelligence.

Finally, Pakhotsky formulates his own assessment of the current state of the industry: no laboratory, in his opinion, has yet solved the problem of coordination and monitoring to a sufficient extent to responsibly continue scaling as quickly as possible for a long time. He expresses hope for voluntary slowdowns in the pace of development until common safety standards are formed, and emphasizes: international coordination on the further development of AI should become a priority for governments around the world.