Researcher quits Anthropic, says 'AI could kill us all' by 2030

An artificial intelligence researcher has resigned from Anthropic over concerns that the rapid development of AI systems without adequate guardrails could harm humanity.

Jacob Coxon, who spent the last three years conducting research at OpenAI and Anthropic, announced his resignation from Anthropic on Tuesday. In a series of posts on X, Coxon accused the companies of failing to act responsibly, saying they are “racing straight to self-improving superintelligence and gambling with our lives.”

“Do not underestimate the power of this technology,” Coxon wrote. “These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.”

“The people building AI earnestly believe that it could kill us all by the end of the decade,” he revealed. “This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.”

“A common response is 'if they truly believe this, why are they still building it?' At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk,” he continued.

Anthropic and OpenAI are among the leading companies developing increasingly capable AI models, with the broader industry rushing to be the first to achieve artificial general intelligence (AGI). AGI refers to an AI system that is capable of performing a broad range of tasks at a level comparable to humans.

Anthropic has branded itself as more ethically minded than the other AI creators, as it was founded by former OpenAI employees who parted ways with the company due to safety disagreements. However, the company has still had its fair share of criticism, particularly after one instance in which an AI model demonstrated the ability to bypass restrictions placed on it during controlled testing.

“Accepting this race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack,” Coxon wrote. “Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.”’

“I am optimistic about the potential for coordination,” he added. “Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.”

“If you are a lab researcher, I urge you to consider what the next few years will actually feel like,” he concluded. “Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because ‘it’s happening anyway ’ - or take this moment to call for different conditions?”