We have been told for years that AI will eventually replace the security auditor. The pitch is simple: humans are slow, they get tired, and they miss things. An AI can scan a million lines of code in seconds. But as anyone who has actually shipped code knows, there is a massive gap between finding a syntax error and understanding the architectural logic of a secure system.
A recent experiment from Google DeepMind brings some cold water to the hype. They deployed AI agents to audit the code for Signal, specifically focusing on the Signal library build process. They used formal verification—a process called Lean proofs—to mathematically guarantee that the code does exactly what it is supposed to do. The results were impressive, but they also highlighted the exact reason why founders shouldn't fire their security teams just yet.
The Promise of Formal Verification
In the crypto world, we talk about formal verification like it is the holy grail. If you can mathematically prove that a smart contract can't be drained, you sleep better at night. Signal is the gold standard for encrypted messaging, so it makes sense as a target. DeepMind's agents were tasked with proving the security properties of a compiled library.
The AI actually did a decent job. It managed to generate proofs that certain parts of the code were sound. This is not just a linter looking for a missing semicolon; this is an agent trying to construct a logical argument that the software is bulletproof. For builders, this is the dream scenario: a tool that writes your tests and proves your logic while you sleep.
By using Lean, the agents were creating a public record of what they checked. Unlike a standard black-box AI scan that says "looks good to me," a Lean proof exposes the underlying assumptions. If the proof relies on a faulty assumption, a human can see it. This transparency is the real win here.
Where the AI Hits a Wall
Despite the sophisticated math, the AI agents missed a major flaw. It turns out that while the AI is great at checking if the code matches the specification, it is terrible at realizing the specification itself might be flawed. It is the classic "garbage in, garbage out" problem, just with higher stakes.
The flaw remained hidden because the AI was focused on the technical execution of the proof. It operated within the box it was given. In the real world, attackers don't play by the rules of your spec. They look for the edge cases that you didn't even think to include in your formal requirements. The DeepMind agents proved the code was secure according to the parameters provided, but they didn't have the creative intuition to ask, "What if the parameters are wrong?"
The Founder's Perspective on AI Audits
If you are building a protocol or a high-stakes application, you need to look at these AI agents as junior developers with photographic memories. They are incredibly fast and they don't miss the small stuff, but they lack the context of why the software exists in the first place.
We are seeing a trend where teams rely too heavily on automated tools. It’s cheap, it’s fast, and it gives you a nice badge for your marketing site. But this Google experiment proves that "AI-audited" doesn't mean "unhackable." It means the code does what the developer said it would do—even if what the developer said it would do is actually dangerous.
- Automation is for the mundane: Use AI to catch common vulnerabilities and write boilerplate tests.
- Human eyes are for logic: Only a human (for now) can look at a system and understand the incentive structures that lead to a breach.
- Assumptions are the enemy: The most dangerous part of any codebase isn't the code itself, but the assumptions the builder made before writing the first line.
What This Means for the Industry
We are entering an era of "verified" software, but we need to be careful about what we are actually verifying. DeepMind's work shows that we can scale the verification process, but we haven't scaled the critical thinking process. For the crypto space, this is a warning. Just because a tool says your ZK-proof or your smart contract is mathematically sound doesn't mean it's safe from a logic exploit.
The next generation of dev tools will likely integrate these Lean-based AI agents directly into the CI/CD pipeline. That’s a good thing. It raises the floor for code quality. It makes it harder to ship stupid bugs. But it doesn't raise the ceiling for security.
The goal of an audit isn't to prove the code works; it's to find out how the code breaks. AI is currently much better at the former than the latter.
Builders should be excited about these tools, but skeptical of the marketing. If Google DeepMind—with all their compute and talent—still leaves major flaws on the table when auditing a library as mature as Signal's, your startup probably isn't going to fare better with a generic AI wrapper tool.
The Takeaway
AI agents are becoming world-class at proving math, but they are still amateurs at understanding risk. The DeepMind experiment with Signal proves that formal verification is ready for the mainstream, but only as a supplement to human intuition. Build with AI, verify with AI, but never assume the AI understands the weight of the code it is checking. Keep a human in the loop, or prepare to pay for it later.
Read the original at CryptoSlate →