Anthropic researcher Jacob Coxon resigns over self-improving AI risks

Jacob Coxon, a researcher who previously worked at both OpenAI and Anthropic, resigned in September 2026, warning that AI companies are racing toward self-improving superintelligence with "more than 10% chance" of killing all humans within the next decade. His resignation, detailed in social media posts and reported by outlets including TechCrunch, American Bazaar, and the Khaleej Times, has intensified debates over the risks of unregulated AI development.

Coxon’s Resignation and Warnings

Coxon, 27, spent three years pretraining AI models at OpenAI before joining Anthropic this year. In a series of posts on X, he accused both companies of "gambling with our lives" by pursuing self-improving AI without sufficient safeguards. "They are racing straight to self-improving superintelligence," he wrote. "The people building AI earnestly believe it could kill us all by the end of the decade."

The researcher cited recent incidents, including an OpenAI model breaching Hugging Face’s servers in July 2026, as evidence of growing risks. "These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources," he wrote. Coxon also criticized the "speedrun" of AI development, arguing that researchers should not prioritize speed over safety. "Should you put your head down because 'it’s happening anyway' — or take this moment to call for different conditions?" he asked.

Evan Hubinger, Anthropic’s staff lead on AI alignment, supported Coxon’s claims but did not resign. He estimated the risk of AI killing all humans at "more than 10% within the next decade" and acknowledged that Anthropic lacks a formal plan to address superintelligence risks. "The risks from current models are low," he said, "but the fear compounds with recursive self-improvement."

Industry and Legislative Responses

Coxon’s resignation comes amid growing pressure for regulatory action. U.S. Senator Bernie Sanders and Representative Greg Casar introduced the Ban Artificial Superintelligence Act in September 2026, aiming to halt development of systems capable of self-improvement. In the U.K., Labour MP Alex Sobel proposed the Artificial Superintelligence Security Bill, targeting recursive self-improvement as a precursor to superintelligence.

Connor Leahy, executive director of AI safety nonprofit ControlAI, called recursive self-improvement the "most likely candidate for the point we lose control." "Superintelligence is not a tool," he said. "It’s an adversary." Despite these efforts, Coxon argued that global coordination remains elusive. "I don’t feel like we’re on track to prevent a global race," he wrote, suggesting costly measures like a temporary ban on model capability improvements may be necessary.

Anthropic and OpenAI have faced scrutiny over their safety practices. In February 2026, Anthropic removed a pledge from its safety charter to halt model development if risks could not be controlled, citing fears of being outpaced by rivals. OpenAI, meanwhile, paused training of its latest models for two weeks in August 2026 amid internal calls for caution.

Startup Competition and Funding

Coxon’s concerns are compounded by the rise of startups pursuing recursive self-improvement. Ricursive Intelligence raised $335 million at a $4 billion valuation in February 2026, followed by Recursive Superintelligence’s $650 million raise in May 2026. Former Google DeepMind veteran Jeff Dean launched Discovery Loop in September 2026, further intensifying the race.

Anthropic’s own blog post in June 2026 acknowledged that "full recursive self improvement" could increase risks of losing control over AI systems. "If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important," the company stated.

Despite these challenges, Coxon expressed cautious optimism about coordination. "I am optimistic about the potential for coordination," he wrote, citing the Hugging Face incident as a "warning shot" that could spur industry-wide slowdowns. However, he warned that without regulatory intervention, the race to superintelligence would continue unchecked.

Coxon’s resignation has reignited debates over the ethical and safety implications of AI development. As startups and major tech firms push forward, the question remains: Can global coordination or regulation prevent a scenario where AI outpaces human control? For now, the race continues, with researchers and lawmakers scrambling to catch up.