We talk a lot about the protective power of artificial intelligence in the crypto stack. Usually, it is in the context of catching bad code or spotting sandwich attacks before they hit. But recently, Coinbase highlighted a real-world scenario where their internal AI models actually did what they were supposed to do: they stopped a massive scam in its tracks. Or at least, they tried to.
A user was in the process of sending $100,000 to what turned out to be a fraudulent destination. The automated systems flagged it, the warnings popped up, and the transaction was effectively stalled. On paper, this is a win for the industry. It is the kind of story that executives use to justify massive R&D budgets for safety tools. But the fallout of this story reveals the massive, gaping hole that AI cannot fill: the human element.
The Psychology of the Bypass
After the AI successfully blocked the $100,000 transfer, the user did not stop and think about why the system was screaming at them. Instead, they got on the phone with a human support agent. They did not try to push the full six figures through again; they lowered the stakes. They asked to send just $5,000. When the agent questioned the legitimacy of the transaction, the user lied. They claimed they knew the recipient and that the money was for a legitimate purpose.
The agent, faced with a insistent customer and a much smaller dollar amount, bypassed the protocols and let the transaction through. It was a scam. The $5,000 was gone. The AI did its job perfectly, but the social engineering was more effective than the code. As builders, we have to look at this and realize that our security models are often built on the assumption that users are rational actors who want to be protected. They aren't.
Why AI is Only a Speed Bump
AI is incredible at pattern recognition. It can scan millions of wallet addresses, identify clusters of suspicious activity, and calculate the risk score of a transaction in milliseconds. In this specific case, Coinbase’s models reportedly blocked nearly $5 million in total scam attempts over a short period. That is a massive achievement that probably saved dozens of people from total financial ruin.
However, AI is essentially a gatekeeper that can be talked into opening the door if the person holding the key is persistent enough. In the startup world, we often focus on the "happy path"—the journey where the user follows our UI prompts and heeds our warnings. We spend less time on the "desperation path," where a user is being actively manipulated by a third party on the other end of a phone line or a Telegram chat.
- Automation fatigue: When users see too many warnings, they stop reading them.
- The authority gap: Users often trust a voice on the phone more than a prompt on a screen.
- Salami slicing: Scammers know that $100k triggers red flags, so they coached the victim to ask for a smaller, "safer" amount to bypass the manual review threshold.
What This Means for Founders
If you are building a consumer-facing crypto app, you cannot rely on a "safety layer" to be your final defense. If your security relies on the user being honest or smart, your security is broken. We have to start building systems that recognize the signs of social engineering, not just the signs of on-chain fraud.
This might mean implementing mandatory cool-off periods that cannot be bypassed by support staff, or using AI to analyze the sentiment and urgency of the user's support tickets. If a user is suddenly frantic and trying to move money to a new address while ignoring every warning, that is a behavioral pattern that needs its own set of rules.
The hard truth for founders is that the better your AI gets at stopping scams, the more aggressive scammers will get at manipulating your users into lying to you.
The Support Agent Problem
We also need to talk about the burden we place on support staff. In the Coinbase example, an agent had to make a judgment call. That is a terrible position to put a low-level or mid-level employee in. If the AI says "this is a scam," but the customer is yelling that it isn't, the agent is caught between a protocol and a customer satisfaction metric.
We need to build tools that empower agents with more than just a "yes/no" button. They need context. They need to see why the AI flagged the transaction. If the system could have shown the agent that the recipient address was linked to ten other reported scams in the last hour, that agent might have had the confidence to say no to the $5,000 request.
Building for the Real World
The takeaway here isn't that AI is useless. Far from it. Without that AI, $4.7 million would have vanished into the void. The takeaway is that AI is an advisory tool, not a cure-all. We are still in an era where the most sophisticated hacking is not happening at the smart contract level, but at the psychological level.
As we move toward more autonomous agents and AI-driven wallets, this problem is going to scale. Imagine an AI wallet that is told by its owner to send money to a scammer. If the wallet tries to block it, and the owner insists, what does the wallet do? If it obeys, it fails its mission to protect. If it refuses, it fails its mission to be a tool for the user.
We are entering a phase where "Code is Law" is being challenged by "The User is Always Right." In crypto, the user is often wrong, and they are often being lied to. Our job as builders is to find the middle ground where the tech can protect the user from themselves without becoming a digital prison.
The Bottom Line
Coinbase's AI did its job, but the human workflow failed. Stop building safety features that assume your users are thinking clearly. Build for the user who is being pressured, lied to, and coached by a professional thief. If your system can't handle a simple lie, it doesn't matter how smart your AI is.
Read the original at CryptoSlate →