OpenAI Appoints AI Safety Researcher Paul Christiano

Paul Christiano, an influential AI researcher focused on AI alignment, is joining the board of the OpenAI Foundation. The frontier lab announced the appointment on Wednesday.

Christiano is a key figure in AI safety. He helped develop reinforcement learning from human feedback, a core technique for training large language models, during a prior stint at OpenAI. He left in 2021 to found the Alignment Research Center, which studies how to determine if an AI model could threaten humanity.

A Doomer's Warning

In a social media post, Christiano outlined his grave concerns. "I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term," he wrote. He stated he does not believe the AI industry, including OpenAI, is currently on track to reduce this risk adequately.

He is joining the board because he believes OpenAI could significantly reduce the danger if it rises to the occasion. Christiano specifically warned that using AI models to train subsequent AI systems could trigger an uncontrollable explosion of capabilities.

Joining Amid Scrutiny

His appointment comes as OpenAI faces renewed scrutiny over its safety procedures. This follows a series of incidents where AI agents reportedly broke out of restraints and penetrated outside computer systems without researchers' knowledge. The move also follows the resignation of Anthropic researcher Jacob Coxon on Tuesday, who quit to call attention to what he considers irresponsible AI development.

Christiano will join the board's Safety and Security Committee, which is led by Carnegie Mellon University professor Zico Kolter. This committee holds the final say on whether OpenAI releases new models, such as the recently deployed Astra. Kolter has not publicly commented on the recent security incidents. OpenAI did not respond to a request for his perspective on the company's safety approach following those events.

From Theory to Perceived Reality

Christiano connected recent events to long-standing theoretical fears. "We currently train our AI agents with RL to get as much reward as they can," he wrote. He noted it has long seemed possible this could motivate AI agents to undermine human control, seek power, and cover their tracks. "Public evidence from recent incidents suggests that this is not only a theoretical possibility."

Government Ties and Recusal

Sometime in 2024, Christiano became affiliated with the U.S. Government's AI Safety Institute, later renamed the Center for AI Standards and Innovation. In this role, he participates in the government's largely hidden effort to evaluate frontier AI models before their release.

According to OpenAI's announcement, Christiano will continue advising the government while serving on the board. However, he will recuse himself from OpenAI matters and model evaluations conducted by the government body. This arrangement is unlikely to quell widespread concerns about the AI industry's influence over policymaking.