STKR News0 of 3 free this monthThe era of massive model training is giving way to the era of inference, shifting the hardware bottleneck from raw compute to memory and creating a wild new landscape for builders.
Originally on IEEE Spectrum →Adrian Boysel
Contributor
Sep 15, 2026
5 min read
Photo illustration / STKR News
For the last few years, the narrative in AI has been dominated by a single word: training. If you wanted to build something impressive, you needed more parameters, more compute, and a bigger cluster. We watched Large Language Models (LLMs) go from million-parameter toys to trillion-parameter giants. We saw benchmarks like GPT-3’s 43.9% accuracy jump to nearly 90% in just a few iterations. But as a founder, I’ve noticed the conversation changing. Training is becoming background noise; the real war has moved to inference.
It makes sense. Training a model is like teaching a student; inference is that student actually going to work. Now that these models are finally useful, everyone wants them to work 24/7. Whether it’s agentic AI running autonomous loops or reasoning models using "chain of thought" to double-check their own homework, the demand for output is exploding. We aren't just asking for a paragraph anymore; we're asking for thousands of tokens of deep reasoning. This shift is exposing a massive flaw in our current hardware stack: our chips were built for the wrong fight.
If you’ve ever wondered why your local Llama instance feels sluggish despite having a decent GPU, it usually isn't a lack of processing power. It’s memory bandwidth. In the training phase, GPUs are king because they can crunch numbers in parallel at an incredible scale. But inference is a different beast. It’s autoregressive, meaning the model has to look at everything it just wrote to decide what to write next. This creates a massive data traffic jam.
Current high-end hardware, like Nvidia’s H100, often sits idle for 50% to 80% of the time during inference. The processor is ready to work, but it’s waiting for the "weights" to be fetched from memory. It’s like having a Ferrari engine attached to a fuel pump that only delivers a drop a second. For builders, this means the cost of running an AI application isn't just about how "smart" the model is, but how efficiently that model can move data back and forth from its memory cache.
Because the stakes are so high, we’re seeing some truly bizarre alliances and acquisitions. Nvidia recently spent $20 billion to acquire talent and tech from Groq, a startup that focuses on Language Processing Units (LPUs). Amazon is splitting its workloads, using its own Trainium chips for the easy stuff and partnering with Cerebras—which makes chips the size of a dinner plate—for the heavy lifting. Even Anthropic is reportedly paying SpaceXAI over a billion dollars a month for compute. This isn't just growth; it’s a frantic scramble for efficiency.
Two main schools of thought are emerging on how to fix the memory problem. The first is to shrink the distance between the brain and the memory. Startups like d-Matrix are literally stacking accelerators on top of DRAM dies, moving data micrometers instead of millimeters. The second approach, led by firms like Majestic Labs, is to rethink the interface entirely, allowing memory to sit further away but connected by high-speed copper links that can handle terabytes of data across a rack. Both are trying to move away from expensive High Bandwidth Memory (HBM) toward cheaper, more plentiful commodity DRAM.
We can't just throw silicon at the problem, though. The software side is getting smarter through quantization—basically shrinking the precision of the numbers the model uses. Think of it as changing a high-definition image to a high-quality JPEG. You lose a little bit of detail, but the file size drops dramatically. Nvidia and others are pushing 4-bit formats that allow models to run three times faster with less than a 1% drop in accuracy. For founders, this is the low-hanging fruit: if you can quantize effectively, you can cut your inference bill in half overnight.
Then there are the radicals. Startups like Etched are building ASICs (Application-Specific Integrated Circuits) that hardwire the transformer architecture directly into the silicon. Their "Sohu" chip is designed to do one thing and one thing only: run transformers. It can allegedly pump out 500,000 tokens per second. The risk? If the industry moves away from transformers to a new architecture next year, those chips become very expensive paperweights. It’s a high-stakes bet on the future of AI architecture.
If you're building in this space, you need to stop thinking about models as static objects and start thinking about them as fluid resource consumers. The "winner" of the inference war won't just be the company with the best model, but the one that can deliver that model at the lowest cost per token. We are moving toward a "systems approach" where different chips handle different parts of the request—one chip to read your prompt, another to generate the response.
The demand for AI is currently insatiable, and while fears of a bubble exist, the shift toward persistent, 24/7 agentic workloads suggests we are only at the beginning of the inference explosion.
The transition from training to inference is the most important shift in AI since the release of ChatGPT. For founders, this means the competitive moat is no longer just your dataset or your fine-tuning; it’s your ability to navigate a complex, rapidly evolving hardware ecosystem. The bottleneck has moved from the "brain" of the AI to its "nervous system"—the memory and data paths that keep it alive. Pay attention to the chips, because they are currently dictating the limits of what you can build.
Read the original at IEEE Spectrum →
A six-minute morning dispatch on the markets and the technology shaping them.
Free. No spam. Unsubscribe anytime.
Earn $STKR for published stories on markets, protocols, and culture.
RegulationComments
24 reader responses