The week AI doom went mainstream

TL;DR: Yesterday, Anthropic released a report saying it’s stopped several potential attempts to use Claude for bioweapons development (among other nefarious things). It’s the latest data point fueling a debate over existential AI risk that hit a fever pitch this week.

What happened: In its latest threat intelligence report, the AI company laid out a smorgasbord of ways people have tried to misuse Claude—whether it’s for scams or state-sponsored surveillance efforts.

Among the most alarming cases were people who tried to use Claude for work that could produce bioweapons. It’s possible that these were attempts at legitimate scientific research—but here’s one worrying example: In May, a scientist asked Claude to help write a grant application for research on making a more harmful variant of chikungunya, a virus spread by mosquitoes that can cause high fever and severe muscle and joint pain. Per Anthropic, it appeared to be for a military research institute—which made it extra concerning.

The AI lab that worries out loud: To be fair, sounding the alarm on AI risk isn’t exactly unusual for Anthropic, which has long positioned itself as the most safety-conscious frontier AI lab. Several of its own employees have now left citing safety concerns—including researcher Jacob Coxon, who resigned this week after accusing both Anthropic and OpenAI of racing toward self-improving AI and “gambling with our lives.”

And even the lab’s current employees sometimes acknowledge species-ending AI risk as casually as you’d discuss the weather. After Coxon resigned, Anthropic alignment science lead Evan Hubinger chimed in that the firm’s employees “earnestly believe AI could kill all humans.” His personal estimate of that risk: about 10% in the next decade. (Meanwhile, your lifetime risk of being struck by lightning is about 0.0065%—and the risk of being in a fatal car crash is about 0.99%.)

AI doomerism goes mainstream: This kind of will-AI-wipe-us-out talk used to be more concentrated within Silicon Valley. But now we’re in an era when headlines about autonomous AI hacks are practically everyday occurrences. That might be why existential AI risk has suddenly become all anyone can talk about; Coxon’s resignation post racked up over 165 million views on X and launched a weeklong media run where he repeated his dire warnings.

Bottom line: Suddenly, the possibility that AI might cause human extinction is drawing a massive public reaction. And with two months to go until the midterms, talk of pausing or banning tech that lawmakers now see as dangerously risky is ramping up. —WK

Also in AI…

• OpenAI has paused its $200-per-month Pro subscriptions. The reason? Sky-high Astra demand is straining its systems.
• Meta’s new agentic AI app Muse is No. 2 on the App Store in the US—and people are already creeped out with how much it knows about them.