Adolescence to Maturity

Dario Amodei has changed his tune. Or at least the key.

In January of 2026, Amodei's essay titled The Adolescence of Technology broke down why the Anthropic CEO was concerned about the future of AI. At that time, his concerns ranged from AI agents replacing all human labor to AI-developed bioweapons. While bioweapons ranked at the top of Amodei's list, he spent more time in the essay discussing the risk presented by non-democratic governments and bad actors abusing a strategic advantage granted by a powerful AI system they alone possessed. As such, Anthropic's guidance was to race faster in America in the hopes that we would solve alignment ("alignment" being the industry term used to describe the problem of ensuring AI systems that are smarter than us don't also immediately want us gone) AND the home team would prevail over any foreign bad actors (implied here is that we should be rooting for a homegrown AI competitor, as surely an American superintelligence will be more trustworthy than a Chinese or Russian one).

A new piece from Anthropic titled When AI builds itself seems to challenge those assumptions. The essay states that experts within Anthropic believe AI-led recursive improvement (this being a point at which AI research and development is mostly or entirely driven by AI) could be possible in the near future, and that could present a real problem. An intelligence explosion that has very little human oversight could have catastrophic effects well before Claude becomes Skynet. A business with access to digital labor that is faster, cheaper, and more skilled than most humans in the same field could potentially overwhelm rivals or enable layoffs on an unprecedented scale. If a talented manager has access to a small army of superhuman developers for the cost of a summer intern, they might decide that the 3rd-party payroll software their company uses is a mediocre product and design an alternative. The same applies to every other technology within a workflow or tool chain. A company whose business was once very limited in scope (for a random example, imagine an online bookseller), might rapidly expand into a vertically-integrated monolith, buying out or undercutting not only their competition, but also every other adjacent business until they control a dangerous amount of global capital and logistics.

Anthropic seems to be pointing toward something I've been saying behind closed doors for some time: we are not ready. But this time, rather than insanely recommending we hit the gas, they suggest we try to find a way to finally install some brakes on this bus that the frontier AI developers forced all the denizens of this planet to board. Surprisingly, I find their proposed solution to be a good one. Many liken frontier AI developers to the global teams of scientists racing to develop nuclear weapons around the end of WWII, and so their idea is to solve this problem in a similar way: build a coalition. Get buy-in from the world's foremost AI developers and the governments who fund them, put in place monitoring and reporting systems in AI research labs that allow researchers and safety officers to keep an eye on each other, and if at all possible, slow things down. Give safety-minded researchers time to solve problems like alignment, labor takeover, disinformation spread, and whatever other hellscape-inducing phenomena that are likely to emerge from hyperintelligent AI systems, and maybe our species will survive long enough to see the real benefits of those systems.

This is not an easy row to hoe. As with nuclear disarmament and non-proliferation efforts (or really any negotiation), there will be parties who will advocate for their primary benefit to the detriment of those across the table. A company lagging behind frontier models might argue that they should be allowed to catch up with the leaders. An AI leader might argue that they should receive the bulk of the coalition's resources as they stand the greatest chance of solving dangerous problems and achieving AI smarter than humans. Add in the decidedly fraught global political climate, and it's not hard to imagine a world where Anthropic's coalition building efforts go nowhere. But considering where we've been headed in AI development for the past decade, where salespeople are encouraging everyone to adopt as quickly as possible because AI will solve every problem everywhere, this is a refreshing change of tone. If Anthropic means what they say and convince others to play along, there's finally a chance for AI to become a technology that is consistent, controlled, trustworthy, and mature.