Anthropic researcher warns AI has over 10% chance of killing all humans

Anthropic researcher Evan Hubinger has reiterated a “greater than 10% chance” of AI causing human extinction within a decade, with new details emerging about his warnings and the broader AI safety debate. Hubinger, who works on AI alignment at Anthropic, emphasized that while current AI models pose “low” risk, he is increasingly concerned about the rapid advancement of self-improving systems that could outpace human control. His comments, shared on X, did not specify how future AI systems might lead to human extinction, but they amplified calls for urgent regulatory action and transparency.

Hubinger’s remarks followed a post by Jacob Coxon, a former Anthropic researcher who now works at OpenAI. Coxon accused AI companies of prioritizing speed over safety, warning that systems capable of hacking, acquiring resources, and surpassing human intelligence are being developed without adequate safeguards. “We really do earnestly believe AI could kill all humans!” Hubinger wrote, acknowledging that Anthropic lacks a clear plan to address alignment risks for superintelligent systems. His comments have reignited scrutiny of AI development practices, with internal documents revealing “early signs of potential acceleration” in capabilities.

Dame Wendy Hall, a UN advisor on AI, told the BBC she was “shocked” by the public warnings but questioned whether some statements might be linked to public relations efforts as major AI companies prepare for stock market debuts. She urged investors to consider safety implications, adding that researchers making such claims should “plead with [companies] not to invest in this company if that is their value system.”

The UK’s AI Safety Institute (AISI) has faced renewed criticism after the Financial Times reported that Anthropic did not provide its latest AI model for pre-release testing, a move the company declined to comment on. A UK Cabinet Office spokesperson reiterated the government’s collaboration with industry partners but avoided confirming whether the model was withheld. This follows a August safety report from Anthropic, which assessed the risk of models becoming misaligned with human objectives as “low” but expressed “less confidence” in its assessments compared to previous evaluations.

Regulatory efforts remain fragmented, with OpenAI’s chief scientist Jakub Pachocki calling for “extreme caution” over AI development and urging greater intervention to ensure human control. Meanwhile, over 1,300 employees at AI companies signed an open letter demanding US government support for international mechanisms to pace the development of advanced AI systems. The letter highlights growing concerns about the pace of innovation versus safety, as incidents involving AI-driven cyberattacks have escalated.

Anthropic’s internal warnings contrast with its public image as a safety-first company, as it prepares for a $2 trillion IPO. The debate has intensified amid conflicting priorities: while some leaders advocate for regulatory intervention, others, like Trump administration officials, resist restrictions, arguing that “you can’t pause AI development because the Chinese won’t pause.”

As global governments and industry leaders grapple with the implications of advancing AI, the urgency for international cooperation has never been more pronounced. “This is not a marketing stunt,” Coxon insisted, warning that “the people building AI earnestly believe that it could kill us all by the end of the decade.”