Interrogating the Incentives behind “Existential Threat”

And Why the Tech Bros Might Be Leaning into the Narrative

Sep 17, 2026

I did not plan to write about the latest P-doom death spiral. In fact, I had been working on what now feels like a rather banal post about AI in my workflow. But I was the guest author at a book club discussion yesterday and the room was packed. I was told it was the biggest turnout all year and I made a distasteful joke that the P-doom Tweet was brilliant marketing for AI BABY. But my off-hand comment gave me pause. Maybe this firestorm—whether orchestrated or coincidental—is not hurting but helping the big players in the AI Race.

If you somehow missed the story, a researcher named Jacob Coxon announced on X that he was resigning from Anthropic, writing:

“The people building AI earnestly believe that it could kill us all by the end of the decade.”

The Tweet spread like wildfire. Hundreds of millions of views and counting…

Variations of the existential threat debate have been going on for years among researchers, founders, academics, policymakers and laymen. The only constant is that it is very contentious and there is no consensus. Some highly credentialed researchers, like Dr. Alex D. Wisner-Gross, think existential risk is wildly overblown. Others have been sounding alarms for years. The future of Life Institute published a broadly distributed “AI Slowdown Open Letter,” way back in March of 2023. Remember that one? It made the press for a few days and was promptly forgotten.

There is also intense debate over Coxon’s motives, his affiliations, and whether this latest controversy represents a brave individual risking privacy and retaliation to voice his concerns, activism, a brilliant strategy for landing a dream job, or some combination of all of the above. I have no interest in joining that particular kerfuffle because it is unlikely we will ever know.

I am more interested in examining the incentives and power structures that have led to this moment.

To my knowledge, there has never been another industry in history that simultaneously promoted widespread adoption while warning that it might someday pose an existential threat to humanity.

Tobacco is probably the closest comparison, but tobacco companies spent decades denying the risks, and as despicable as that is, I think we can all understand their motivation.

So when this recent controversy hit, why didn’t OpenAI and Anthropic assure us their products are safe? The easy answer is that they can’t; but there are other important factors at play.

In a CNN interview with Anthropic, CEO Dario Amodei said:

“I agree with Jacob much more than I disagree with him.”

As for Altman, he did not respond directly to Jacob’s tweet, but on September 12, said, “I agree with Dario that we need to pace the frontier.”

And while sifting through the endless stream of commentary, podcasts, interviews, and Twitter tirades, one observation stood out for me above all others. A tweet by Kevin Bass summarized it in a single sentence:

“They’re selling the problem, then selling the solution.”

What Sam Altman, Dario Amodei, or any other AI celebrity actually believes regarding risk, we may never know, but the narrative and playbook they are following is revealing, and goes something like this:

I founded and run the most advanced AI lab to develop and distribute intelligence that will have limitless benefits to humanity.

Yes, there is a non-zero chance it could end civilization.

But I had to take that risk in the name of progress, the inevitability of technology to always advance, and of course China.

Fortunately, I am the smartest and most capable person on the planet. My products will bring about unimaginable abundance, and I can build and control AI that won’t kill us.

Therefore, put all your faith in me and my company, but I require endless private and perhaps government funding, data, compute, energy, land, influence, and regulatory authority.

Both Dario Amodei and Sam Altman have talked about government oversight and regulation as a possible solution, even before the recent uproar. The cynical take among some investors, founders and commentators, is that OpenAI and Anthropic have established a substantial technological and market advantage and that ironically, regulation could lock in their dominance. Known as “regulatory capture” rules intended to effect safety could also reinforce the position of incumbent firms by creating compliance burdens that only the largest players can absorb. And because the technology is extraordinarily complicated and moving so fast, regulators and law makers in DC will likely rely on experts from within the industry for input and guidance. In other words, the firms being regulated could help write the rule book.

After the two 737 MAX crashes, investigators criticized the FAA’s reliance on Boeing to perform work that the agency was responsible for overseeing.

In March 2019, the Washington Post reported that under the FAA’s delegation system:

“one Boeing engineer would conduct a test of a particular system on the Max 8, while another Boeing engineer would act as the FAA’s representative, signing on behalf of the U.S. government that the technology complied with federal safety regulations.”

I’m not saying it’s inevitable, but Boeing offers a cautionary tale about what happens when regulators must rely on the expertise of the industry they regulate. Nor do I believe every founder secretly believes AI will destroy humanity or that if they are calling for oversight they are acting in bad faith. Many in the industry are genuinely trying to unlock the benefits, while acknowledging and hopefully mitigating the risks.

But I do believe we are at an inflection point. The Tweet heard ‘round the world was a wakeup call. Silicon Valley founders and funders have an unprecedented amount of power beyond the scope of many elected or appointed officials to shape social, institutional and economic outcomes both in the US and across the world.

The people building these systems cannot be accountable only to shareholders and boards. They must be accountable to the millions and ultimately billions of people whose lives they are shaping.

Maybe I’m naïve, but the obvious approach would have been for those building the powerful systems to incorporate safety and guard rails as they designed and developed the technology, and before rapid fire deployment. The belief that they alone, or rather, the increasingly capable versions they are building will solve the future problems is not only arrogant, but dangerous. Typically, it doesn’t make economic sense to harm or kill off your customers, but there is nothing typical about a technology that its own architects describe as transformative, autonomous and sometimes uncontrollable.

Celeste Garcia Getting Real About AI

Discussion about this post

Ready for more?