OpenAI disbands another safety team, as head advisor for ‘AGI Readiness' resigns

• OpenAI is disbanding its "AGI Readiness" team, which advised the company on OpenAI's capacity to handle artificial intelligence that could potentially equal or surpass human intellect and the world's readiness to manage such technology.
• Miles Brundage, senior advisor for AGI Readiness, announced his departure from the company and wrote that he believes his research will be more impactful externally.
• In May, OpenAI decided to disband its Superalignment team, which focused on technology to control and steer superintelligent AI, just one year after it announced the group.

OpenAI is disbanding its "AGI Readiness" team, which advised the company on OpenAI's capacity to handle increasingly powerful artificial intelligence and the world's readiness to manage that technology, according to the head of the team.

On Wednesday, Miles Brundage, senior advisor for AGI Readiness, announced his departure from the company via a Substack post. He wrote that his primary reasons were that the opportunity cost had become too high and he thought his research would be more impactful externally, that he wanted to be less biased and that he had accomplished what he set out to do at OpenAI.

Artificial general intelligence, or AGI, is a branch of AI pursuing technology that equals or surpasses human intellect on a wide range of tasks. AGI is a hotly debated topic, with some leaders saying we're close to attaining it and some saying it's not possible at all.

In his post, Brundage also wrote, "Neither OpenAI nor any other frontier lab is ready, and the world is also not ready."

Brundage said he plans to start his own nonprofit, or join an existing one, to focus on AI policy research and advocacy. "AI is unlikely to be as safe and beneficial as possible without a concerted effort to make it so," he said.

Former AGI Readiness team members will be reassigned to other teams, according to Brundage's post.

"We fully support Miles' decision to pursue his policy research outside industry and are deeply grateful for his contributions," an OpenAI spokesperson told CNBC. "His plan to go all-in on independent research on AI policy gives him the opportunity to have an impact on a wider scale, and we are excited to learn from his work and follow its impact. We're confident that in his new role, Miles will continue to raise the bar for the quality of policymaking in industry and government."

In May, OpenAI disbanded its Superalignment team — which OpenAI said focused on "scientific and technical breakthroughs to steer and control AI systems much smarter than us" to prevent them "from going rogue" — just one year after it announced the group, a person familiar with the situation confirmed to CNBC at the time.

News of the AGI Readiness team's disbandment follows the OpenAI board's potential plans to restructure the firm to a for-profit business, and after three executives — CTO Mira Murati, research chief Bob McGrew and research VP Barret Zoph — announced their departure on the same day in September.

In early October, OpenAI closed its buzzy funding round at a valuation of $157 billion, including the $6.6 billion the company raised from an extensive roster of investment firms and Big Tech companies. It also received a $4 billion revolving line of credit, bringing its total liquidity to more than $10 billion. The company expects about $5 billion in losses on $3.7 billion in revenue this year, CNBC confirmed in September with a person familiar with the situation.

In September, OpenAI announced that its Safety and Security Committee, which the company introduced in May as it dealt with controversy over security processes, would become an independent board oversight committee. It recently wrapped up its 90-day review evaluating OpenAI's processes and safeguards and then made recommendations to the board, with the findings also released in a public blog post.

News of the executive departures and board changes also follows a summer of mounting safety concerns and controversies surrounding OpenAI, which along with Google, Microsoft, Meta and other companies is at the helm of a generative AI arms race — a market that is predicted to top $1 trillion in revenue within a decade — as companies in seemingly every industry rush to add AI-powered chatbots and agents, to avoid being left behind by competitors.

In July, OpenAI reassigned Aleksander Madry, one of OpenAI's top safety executives, to a job focused on AI reasoning instead, people familiar with the situation confirmed to CNBC at the time.

Madry was OpenAI's head of preparedness, a team that was "tasked with tracking, evaluating, forecasting, and helping protect against catastrophic risks related to frontier AI models," according to a bio for Madry on a Princeton University AI initiative website. Madry will still work on core AI safety work in his new role, OpenAI told CNBC at the time.

The decision to reassign Madry came around the same time that Democratic senators sent a letter to OpenAI CEO Sam Altman concerning "questions about how OpenAI is addressing emerging safety concerns."

The letter, which was viewed by CNBC, also said, "We seek additional information from OpenAI about the steps that the company is taking to meet its public commitments on safety, how the company is internally evaluating its progress on those commitments, and on the company's identification and mitigation of cybersecurity threats."

Microsoft gave up its observer seat on OpenAI's board in July, writing in a letter viewed by CNBC that it can now step aside because it's satisfied with the construction of the startup's board, which had been revamped since the uprising that led to the brief ouster of Altman and threatened Microsoft's massive investment in the company.

But in June, a group of current and former OpenAI employees published an open letter describing concerns about the artificial intelligence industry's rapid advancement despite a lack of oversight and an absence of whistleblower protections for those who wish to speak up.

"AI companies have strong financial incentives to avoid effective oversight, and we do not believe bespoke structures of corporate governance are sufficient to change this," the employees wrote at the time.

Days after the letter was published, a person familiar with the matter confirmed to CNBC that the Federal Trade Commission and the Department of Justice were set to open antitrust investigations into OpenAI, Microsoft and Nvidia, focusing on the companies' conduct.

FTC Chair Lina Khan has described her agency's action as a "market inquiry into the investments and partnerships being formed between AI developers and major cloud service providers."

The current and former employees wrote in the June letter that AI companies have "substantial non-public information" about what their technology can do, the extent of the safety measures they've put in place and the risk levels that technology has for different types of harm.

"We also understand the serious risks posed by these technologies," they wrote, adding the companies "currently have only weak obligations to share some of this information with governments, and none with civil society. We do not think they can all be relied upon to share it voluntarily."

OpenAI's Superalignment team, announced last year and disbanded in May, had focused on "scientific and technical breakthroughs to steer and control AI systems much smarter than us." At the time, OpenAI said it would commit 20% of its computing power to the initiative over four years.

The team was disbanded after its leaders, OpenAI co-founder Ilya Sutskever and Jan Leike, announced their departures from the startup in May.

"Building smarter-than-human machines is an inherently dangerous endeavor," Leike wrote in a post on X. "OpenAI is shouldering an enormous responsibility on behalf of all of humanity. But over the past years, safety culture and processes have taken a backseat to shiny products."

Altman said at the time on X he was sad to see Leike leave and that OpenAI had more work to do. Soon afterward, co-founder Greg Brockman posted on X a statement attributed to Brockman and the CEO, asserting the company has "raised awareness of the risks and opportunities of AGI so that the world can better prepare for it."

"I joined because I thought OpenAI would be the best place in the world to do this research," Leike wrote in his May post on X. "However, I have been disagreeing with OpenAI leadership about the company's core priorities for quite some time, until we finally reached a breaking point."

Leike wrote that he believes much more of the company's bandwidth should be focused on security, monitoring, preparedness, safety and societal impact.

"These problems are quite hard to get right, and I am concerned we aren't on a trajectory to get there," he wrote at the time. "Over the past few months my team has been sailing against the wind. Sometimes we were struggling for [computing resources] and it was getting harder and harder to get this crucial research done."

Leike added that OpenAI must become a "safety-first AGI company."