Thou shalt not scale

TL;DR: Yesterday, Pope Leo XIV dropped a 42,300-word moral caution on AI, warning that it’s being built by a tiny technocratic elite with little oversight and a winner-takes-all mindset. It’s probably not the blessing the tech industry was hoping for after months of meeting with the Vatican in hopes of influencing the pope’s AI letter. But even an Anthropic co-founder who joined the pope yesterday conceded that the industry can’t be trusted to govern itself.

What happened: Leo’s first encyclical—a formal papal letter on the most pressing moral and social questions of the day—underscores the idea that we’re in a tech upheaval on the scale of the Industrial Revolution (a framing that’s drawn jeers from college grads recently). The pope’s core message: AI could deepen inequality because too much power has been placed in the hands of the few who are developing and governing it.

Highlights from the treatise, which is a whopping 235 pages in booklet form, include:

• A comparison of AI to the Tower of Babel—the biblical parable about humanity's failed project to build a tower that reaches heaven—as an example of how unchecked ambition can ultimately lead to division and dehumanization.
• The pope’s belief that AI shouldn’t be confused with human intelligence because models “do not feel joy or pain, do not mature through relationships,” and have no moral conscience. (Ironically, at least one famous atheist firmly disagrees.)
• A call to "disarm" AI from its competitive mentality, both in its commercial aims and its military uses.
• A Lord of the Rings quote from the wise wizard Gandalf about doing what small amount of good we can within our lifetimes, even if we can’t fix the whole world.

Mixed reactions: Predictably, the tech world and political class can’t agree on whether the pope is an AI doomer. Blake Scholl, CEO of a firm developing a supersonic airliner, outright called it a “bad take.” Former Trump AI czar and tech investor David Sacks gave a lukewarm nod to the pope’s concerns about AI “domination” over humans—but then argued that giving governments sweeping power over AI in the name of safety could enable Orwellian censorship and surveillance.

Bottom line: The pope is calling on Silicon Valley to put human well-being first when building AI, even slowing its development if the potential risks demand it. Whether anyone in the AI industry has a come-to-Jesus moment and actually pumps the brakes is another question. —WK