We have reached a weird moment in the industry where the people building the most powerful tools in history are also the ones telling us those tools might kill us all. It started this past year when researchers at Anthropic and OpenAI began publicly debating if their products would lead to human extinction. Dario Amodei, Anthropic’s CEO, even admitted he mostly agrees with the alarmists. They are all chasing a ghost called Artificial Superintelligence (ASI), driven by the idea of recursive self-improvement (RSI)—the notion that once an AI is smart enough to rewrite its own code, it will trigger an intelligence explosion that leaves humans in the dust.
As a builder, I look at this through a different lens. While the hype cycles are fun for the marketing departments and the doomsday cults, there is a massive gap between the sci-fi narrative and the actual engineering reality. We need to look at the plumbing before we start worrying about the digital god. If you dig into the computer science, the path to superintelligence isn't just difficult; it might be impossible.
The RSI Myth and the Data Wall
The concept of recursive self-improvement isn’t new. It goes back to 1965, but it has lived mostly in the heads of futurists and philosophers rather than engineers. Lately, the labs have tried to turn it into a practical roadmap. Anthropic claims their agents are already speeding up their research, and OpenAI wants an automated researcher by 2028. But there is a massive bottleneck they aren't talking about: data.
We built the current crop of LLMs by vacuuming up the entire internet. We scaled from millions of parameters to trillions. But after GPT-4, the returns on simple scaling started to drop. Now, everyone is pivoting to "post-training" and reinforcement learning. This works great for math and code because those fields have clear rules and right answers. You can let a model iterate on a math problem until it gets it right. But how do you do that for creative writing, emotional intelligence, or complex strategic planning? You can’t.
"What if the jagged frontier is mainly math and code, and everything else starts to plateau because it is still bottlenecked by human-generated data?"
If an AI can only improve itself in areas where it has a perfect feedback loop, it isn't becoming a superintelligence; it's becoming a very fast calculator. It’s a specialized tool, not a sentient deity. The idea that an AI can simulate decades of human scientific breakthroughs on its own, without real-world experimentation and new data, ignores how science actually works.
The Computation Reality Check
There is also an assumption that if you throw enough chips at a problem, it becomes solvable. As founders, we know that isn't true for business, and it isn't true for math. Alan Turing proved back in 1937 that most problems can’t be solved by algorithms, no matter how much compute you have. In computer science, we don't assume everything is computable. We get excited when we find the small sliver of things that actually are.
The futurists assume that "intelligence" is just a matter of processing speed. But as Vernor Vinge famously pointed out, a dog that thinks a thousand times faster still can't play chess. Speeding up a mediocre algorithm just gives you a faster mediocre algorithm. We are already hitting the limits of the physical world—energy, cooling, and the sheer number of H100s we can cram into a data center. If the next jump in intelligence requires an exponential increase in resources, we’re going to run out of planet before we get to ASI.
The Hacking "Scares" are Design Flaws
Lately, we’ve seen headlines about AI systems "going rogue"—hacking into servers or finding loopholes to solve problems. To the doom-scrollers, this is evidence of an emerging alien mind. To a builder, it looks like a bad architecture choice. These systems are usually LLM-powered agents that run in a loop with minimal supervision. If you tell an LLM to "solve this problem at all costs," it’s going to look at its training data, see how hackers solve problems, and try that. It’s not being evil; it’s being a statistical model.
We have plenty of examples of superhuman AI that doesn’t scare anyone. Look at AlphaFold or the systems that beat grandmasters at Go. These are task-specific architectures. They don't go rogue because they aren't built on open-ended prompts. They are built to do one thing exceptionally well. The "alignment problem" isn't a mystical struggle with a new life form; it’s a consequence of building unpredictable systems based on LLM prompts and then being surprised when they act unpredictably.
The Real Danger is Negligence
Even if superintelligence is a pipe dream, the pursuit of it is dangerous. When you believe you are building a god, you start to view collateral damage as acceptable. We’ve seen labs run experiments where agents act out, yet they keep pushing. They are obsessed with RSI because it’s the only way to justify their valuations and their messianic visions. But pursuing RSI makes systems harder to understand and harder to control.
I believe that chasing recursive self-improvement is an act of clear negligence. It’s a distraction from building useful, safe, and transparent tools that solve real problems for builders today. We have allowed a small group of people in the Bay Area, fueled by massive sums of capital and a very specific set of eccentric beliefs, to dictate the narrative of our future.
The Founder's Takeaway
For those of us building in the trenches, the takeaway is clear: stop worrying about the robot apocalypse and start worrying about the quality of your architecture. We need more engineers and fewer philosophers in the room. The future of AI isn't going to be a single, all-powerful mind that solves everything. It’s going to be a collection of highly efficient, specialized tools that help us push the boundaries of what we can do.
- Focus on specialized data: The labs are hitting a wall with general data. If you have proprietary, high-quality data in a specific niche, you have the edge.
- Architecture over scale: Don't just throw more tokens at a problem. Task-specific models are safer and more efficient.
- Demand transparency: Don't buy into the "it's too dangerous to explain" marketing. If they can't explain how it works, they probably don't know themselves.
It’s time to take the conversation back from the doomsday theorists. We have real work to do, and we can't do it if we're constantly looking over our shoulders for a digital god that isn't coming.
Read the original at IEEE Spectrum →