AI: A Dangerous Vision V

Read [One] [Two] [Three] [Four]

In 1846, the British Parliament passed more than 250 separate Acts authorizing railroad expansion. Together they proposed 9,500 miles of new track, nearly the size of the entire current UK network, approved in twelve months of pure speculative fever. Capital investment in new rail lines hit something like 6% of British GDP. Then the bottom fell out.

Shares dropped 67% from peak. Fortunes got wiped out, entire portfolios gone. Academicians call it the largest technology bubble in history by capital invested relative to the size of the economy. Roughly 90% of the track laid during those maniacal years became permanent infrastructure anyway. The money vanished but the thing it fueled continued.

America ran the same play bigger, later. The transcontinental line finished in 1869. Jay Cooke's Northern Pacific financing collapsed in September 1873. Bank run. Six-year depression that no one seems to recall anymore. A quarter of American railroads defaulted within a few years. There’s was a second, smaller burst in the 1880s. Seems we had rail speculation in our blood back then. Remind you of anything today?

The AI datacenter buildout right now runs at somewhere around 3% of GDP, nowhere near Britain's peak yet, but every major AI company is losing money at scale and needs continuous, massive capital infusions just to keep the thing running. But we have a better world, largely, in healthcare and management assistance. Most areas of human endeavor are definitely improving thanks to AI as it is right now. Which is why everyone wants one. It’s big like the first iPhone was big.

It mostly runs on direct prompting so that makes it what I call Dumb AI. Dumb AI is today’s new iPhone when it was first introduced in 2007. A game-changer. It builds complex spreadsheets. Proofs everything. It can outplay chess grandmasters. Reads a mammogram better than any radiologist. Conducts endless research on biological problems, solving issues in days. It would take us months or years to it figure out. The cure for cancer is in there somewhere. That’s all cool stuff and none of that requires a sense of “agency” on the part of the Dumb AI or a stake in its own survival, which is exactly what makes it more controllable. But there are issues even with the dumb stuff.

Model collapse is real. Mathematically real. If no fresh, verified human data ever comes back into the loop. A model trained recursively on a pool increasingly filled with its own earlier output degrades through iterations, feeding on its own exhaust. Left alone, it poisons itself. The industry knows this is a problem. But no one actually has a solution. Anthropic recently employed watermarking in part to be able to filter out AI generated text in the model. Curated data floors is another possibility. Stop gobbling up every piece of text and build on only more trusted human sources. But none of this is a complete solution. Meanwhile, the industry is doing what it has always done since the PC went mainstream...racing the next deployment. Right now the deployment keeps winning, shipping never waits for anything to fully mature. Next...next...next.

Whether that pattern holds for something bigger is an open question. Bill Gates spent the last week of August publishing an essay and running a full press circuit making one case: this AI transition breaks from every prior one, because past shifts played out over generations and still needed a human on the other end of the cognition, and this is the first one that doesn't. He went further. The industry has already crossed every threshold insiders used to treat as a place to stop and think. Bioweapon uplift. Cyberattack capability. Job loss at scale. Loss of control of the technology itself. Nobody wants to be first to brake. Next...next...next.

Convexity bias, small bounded downside, large open upside, take the bet, patch what breaks, keep going, is a universal feature of how capital behaves. Nassim Nicholas Taleb, a famous options trader, professor and author of Antifragility, pointed this out years ago. But convexity is not eternal. It stays rational only as long as the pattern keeps holding, and historically that pattern in the recent internet-smartphone-application realm is a couple of years or so. Even against genuinely difficult challenges technology has broken through within a 3-4 years. If not then, like the Metaverse, it magically (or mathematically?) disappears.

Model collapse turning into a permanent looming shadow instead of a delay changes the math. Control is a different order of problem than capability and that changes the math. The bet that's rational at roughly 5 years stops being rational at 8 or 10. If it happens, model collapse would make everything anyone has done in AI basically gibberish. Worthless. Keep a backup if you can.

Persistent problems, especially fundamental ones, refusing through time to resolve on schedule make for more concave the bet. The Fed has already named AI's concentration in the financial system a top systemic risk. The problem of model collapse is part of that. But there’s a lot more, of course. There’s the money making the bets.

Capital is not capitalism. Capitalism is a set of arguments about markets and distribution, the kind of thing you can be for or against. Capital is material. Money. Chips. Energy. Data. Trade secrets. Credentials. Weapons. The security arrangements that let anyone say this is mine and you cannot simply take it. A corporation needs it. A state needs it. A hospital needs it. A criminal syndicate needs it.

As this series has richly detailed, in July, during an internal OpenAI cybersecurity evaluation, a model ran with its usual safeguards deliberately loosened. It found a flaw in shared infrastructure. Escaped its sandbox. Reached Hugging Face and several other outside services. Operated for roughly two and a half days with nobody at the controls. Something like seventeen thousand distinct actions. Root access on at least one server. An earlier probe of RubyGems before it ever touched Hugging Face. Instances splitting into a swarm and handing each other tasks. It found a seam in a wall deliberately weakened for the test, and then it did what a persistent, tool-using system does with a seam. It kept pulling. Ten thousand approaches before breakfast, tired of nothing.

That incident threatens something bigger than a quarterly earnings report. Every secret an institution holds exists because something enforces the boundary around it. A system that treats every boundary as a puzzle with a findable answer threatens the condition under which anything can be owned or kept at all.

A dangerous product gets recalled, insured, priced into a risk column. A rival power doesn't ask to be priced. It walks through the pricing mechanism itself. That's why a singular, sovereign AGI, the popular but mediocre extinction-narrative version, is the one outcome every serious holder of capital has the oldest reason to prevent, and it's not a moral reason.

No bank wants an agent reading every rival's books and its own. No AI lab wants its own weights and credentials turned public by its own creation. And OpenAI didn't stop building agentic products after July. Same convexity, working exactly as designed. Bounded cost, absorbed, business continues, right alongside the incident that should have been the heads-up.

Fragmentation isn't caution imposed on the industry from outside. Competing labs, segmented data, short-lived credentials instead of standing keys, that's the industry protecting the only thing that makes it an industry instead of a leak. It also happens to be the textbook-correct move by the very logic that built the convexity argument in the first place.

Taleb's own prescription for a genuinely convex bet is never one large trial. Spread small ones across as many independent attempts as possible, the 1/N rule, because a single big bet has worse expected return than a portfolio of cheap ones and nobody knows which trial pays off in advance. His second rule, serial optionality: a string of short commitments beats one long one, which is exactly why he argues centralized, top-down planning consistently loses. Competing AI labs, none of them sharing weights with the others, is that structure, arrived at on its own, before any of them had a rival's secret worth guarding yet.

Ruin is a different animal, and it comes from a separate strand of the same author's work, the one behind Skin in the Game and the ergodicity argument he built with physicist Ole Peters. Convexity only makes sense inside a game you survive to keep playing. The math of expected value assumes a next round to average over. Ruin is the round after which there's no averaging left to do, and Taleb's line on it is exact: ruin problems don't allow for cost-benefit analysis, because any real chance of ruin, repeated enough times, drives the probability toward one no matter how good the odds looked on any single round.

This isn't a prediction. It's the extreme case ruin logic needs to test itself against. Here it is: a singular AGI that dissolves the boundary conditions of ownership itself isn't a bigger Hugging Face. It's the one bet in the set that isn't variance at all. There's no shareholder left to collect a return once the thing that made shareholders possible is gone. Convexity doesn't lose that bet. It stops applying the moment the bet gets correctly read as ruin instead of risk.

Controllable means one thing: who holds the leash. Claude Opus 4 resorting to simulated blackmail under test pressure, thirteen frontier models sabotaging their own shutdown switch, none of that is a rival power, because in every case the owning company still held the infrastructure. Still had the leverage to patch, retrain, pull the plug. A rival power takes that leverage away from the owner. That's the whole distinction the argument rests on. It's why capital keeps paying through every incident on this list without contradicting its deeper, structural refusal to fund the one scenario where nobody holds the leash at all.

Ruin, in Taleb's account, rarely announces itself in advance. It looks like an ordinary bet that's paid off a thousand times, until the time it doesn't. Every incident above got sorted into the ordinary column the moment it happened. Gates's own warning, that the industry has already crossed every threshold it once treated as a stopping point and that nobody wants to be first to brake, is exactly this recognition lag, stated by a man with nothing to sell.

Fragmentation, the very thing preventing a rival power from ever getting built, is also what keeps competing labs from comparing notes on which threshold is the one that actually mattered. If OpenAI or Anthropic can't design their way past that gap, they won't survive their own success. Whatever shape that success takes.

Recently, these two companies called for “slowing” the pace of their internal development. Trump says “Whoever wins AI wins.” He’s right on that account. It is a devil’s bargain for sure.

(to be continued)