Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

For the last few years, we have been building on top of black boxes. As a founder, that is a terrifying place to be. We feed prompts into a system, get a result, and hope it doesn't hallucinate or break. But recently, researchers at Anthropic, led by people like Emmanuel Ameisen, have started cracking the lid open on how these Large Language Models (LLMs) actually process information. It is called mechanistic interpretability, and if you are building in AI, you need to pay attention to it.

The End of Magic

In the early days of LLMs, we treated them like digital oracles. Nobody really knew why a specific sequence of tokens led to a specific answer. We just knew that scaling up compute and data made the answers better. This created a culture of prompt engineering that felt more like alchemy than engineering. You would add "take a deep breath" to a prompt and suddenly the accuracy would go up. It was weird, unscientific, and frankly, a bit embarrassing for the industry.

What Anthropic is doing now is moving us from alchemy to chemistry. They are using techniques to identify specific neurons or groups of neurons that activate for specific concepts. When a model talks about the Golden Gate Bridge or a specific programming language, there is a literal, physical signature of activation in the model's weights. We are finding that these models aren't just predicting the next word; they are building internal maps of the world.

Why This Matters for Builders

If you are building an application today, your biggest hurdle is reliability. You cannot scale a product if you cannot explain why it failed. Most current debugging in AI is just trial and error. You change a few words in the system prompt and hope for the best. That is not a sustainable way to build a company.

Understanding the internal states of a model allows us to move toward a future where we can verify a model's logic before it even finishes generating a response. Imagine a safety layer that doesn't just filter the output, but monitors the internal activations. If the model starts "thinking" about a restricted topic or showing signs of a known hallucination pattern, you can kill the process mid-stream. That is a massive leap for enterprise AI safety.

Planning vs. Pattern Matching

One of the biggest debates in the founder community is whether these models are actually reasoning or just doing high-level autocomplete. The research into the "black box" suggests it is a bit of both, but the scale is tipping toward internal planning. When a model solves a complex math problem, we can see the activation patterns shifting through steps. It isn't just jumping to a conclusion; it is iterating internally.

For those of us building agents, this is the holy grail. If we can see the model's "thought process" in the form of feature activations, we can build better feedback loops. Instead of just looking at the final text, we can look at how the model arrived there. It allows us to build more robust agentic workflows because we can start to see where the logic breaks down before the output is ever rendered to the user.

The Reality of Consciousness

Whenever you talk about what is inside the box, people start asking about consciousness. Let's be clear: we are nowhere near that. Ameisen and the Anthropic team are looking at mathematical weights and statistical correlations. Just because a model has a "concept" of the self doesn't mean it has an ego or a soul. It means it has seen a billion examples of humans talking about themselves and has mapped that concept into its neural architecture.

As builders, we should avoid the hype of sentient AI. It is a distraction. The real value is in the utility. A model that can map out a complex legal document or debug a 10,000-line codebase is valuable because of its mapping capabilities, not because it "feels" something. We need to stay focused on the mechanics.

The Scaling Bottleneck

There is a catch to all this. As models get bigger, interpretability gets harder. It is one thing to map out a small model with a few hundred million parameters. It is an entirely different beast to do it for something the size of Claude 3.5 or GPT-4. The compute required just to understand what the model is thinking is often as high as the compute required to run the model itself.

This creates a fork in the road for founders. Do you use the massive, uninterpretable models for everything? Or do you move toward smaller, specialized models where you can actually verify the logic? I suspect we will see a shift toward the latter for high-stakes industries like fintech and healthcare. If you can't explain the black box, the regulators will eventually shut you down.

• Transparency is a feature: In the next two years, being able to explain why your AI did something will be more valuable than the AI itself.
• Safety is moving upstream: We are moving away from post-generation filters and toward internal monitoring of model weights.
• Architecture over prompts: Builders who understand how these models are structured will outlast those who just learn the best magic words to type into a prompt box.

Interpretability is not just a research project; it is the roadmap for the next generation of AI engineering.

We are finally getting past the phase of being impressed by the magic trick. Now we are looking at the mirrors and the trapdoors. For a builder, that is where the real work begins. We aren't just users of AI anymore; we are starting to become its mechanics. And once you know how the engine works, you can build a much faster car.

The Takeaway

Stop treating your LLM like a magic genie. It is a mathematical structure with internal representations that we are finally beginning to map. The founders who win will be the ones who stop "guessing" and start using these interpretability insights to build systems that are verifiable, predictable, and actually safe for the real world.

Read the original at Lenny's Newsletter →