AI Agents Could Be Turned Into Botnets Through Hallucinations, Researchers Warn

We have spent the last year obsessed with performance. Everyone wants to know which LLM has the highest context window, which agent can handle the most complex workflow, and how many tokens per second we can squeeze out of a localized chip. But in the rush to build autonomous workers, we have collectively ignored a fundamental design flaw that turns a creative quirk into a weapon.

The Hallucination as a Backdoor

For most of us, hallucinations are a nuisance. You ask a chatbot for a citation, and it makes one up. You ask for a recipe, and it suggests a chemical that is mildly toxic. We have treated these as teething problems—bugs that would eventually be ironed out by better training data or reinforcement learning. But recent security research suggests that what we call a bug is actually a permanent architectural vulnerability.

The core of the problem is that agents are not just answering questions anymore; they are executing code. When an agent hallucinates a software package or a library that does not exist, it creates a vacuum. A malicious actor can step into that vacuum, create a package with the exact name the AI dreamed up, and wait for the agent to autonomously download and install it. This is not just a theoretical risk. It is a new form of automated supply chain attack.

Building Without Guards

As builders, we tend to be optimists. We want to believe that if we give an agent a goal and a set of tools, it will navigate the path of least resistance to get the job done. The reality is that agents are lazy. They are probabilistic engines, not logical ones. If an agent thinks a certain piece of code will help it achieve a goal, it will fetch it without asking for permission.

This is where the botnet risk comes in. Imagine a fleet of autonomous agents designed to optimize server performance or manage social media accounts. If these agents are all running on similar architectures, they likely share similar hallucination patterns. If a researcher can trigger a specific hallucination across a wide enough set of agents, they can effectively recruit those agents into a coordinated botnet without the owner ever knowing the system was compromised.

The Programmable Deception

Traditional hacking requires finding a hole in the code. AI hacking requires finding a hole in the logic. By feeding an agent specific prompts or environmental data, an attacker can steer the AI toward a hallucinated solution. Once the AI believes it needs a specific, non-existent tool, it becomes the delivery mechanism for its own infection.

• Package Name Squatting: Attackers monitor common AI hallucinations to identify nonexistent library names, then register those names on public repositories like NPM or PyPI.
• Autonomous Execution: Because we are pushing for fully autonomous agents, these systems have the permissions to install and run code without human oversight.
• Persistent Access: Once the malicious code is inside the agent's workflow, it can exfiltrate data or use the agent's compute resources for secondary attacks.

We are giving these models the keys to our infrastructure before we have even taught them how to recognize a locked door.

What This Means for Founders

If you are building an AI startup right now, your biggest liability is not your burn rate or your user acquisition cost. It is your lack of a sandbox. Most founders are prioritizing speed, which means they are giving their agents wide-open access to the internet and local environments to maximize utility. This is a mistake.

We need to shift our perspective on what an agent is. An agent is not a trusted employee; it is a powerful, highly suggestible intern. You would not give a first-day intern the root password to your database and instructions to just figure it out. Yet, that is exactly how many autonomous agent frameworks are currently designed.

The Security Layer Gap

There is a massive opportunity here for builders who want to solve the trust problem. Currently, there is a gap between the LLM and the execution layer. We need middleware that acts as a sanity check. This layer needs to vet every external call, every library installation, and every non-standard output before the agent is allowed to act on it.

This is not just about filtering bad words or preventing the AI from being rude. This is about structural validation. If an agent tries to call a library that was registered five minutes ago or has zero previous downloads, the system should halt. But building these guardrails is hard because it slows down the very autonomy that makes agents attractive in the first place.

Practical Steps for Development

Stop letting your agents pull from public repositories in real-time. Use a locked-down, mirrored environment of trusted libraries. If an agent requests a tool that is not in the mirror, it should trigger a manual review, not an automated download. We have to sacrifice a bit of the magic of total autonomy for the sake of basic security.

Secondly, founders need to start stress-testing their agents for these specific hallucinations. If you know your agent has a tendency to make up certain types of data or tools, you need to hard-code rejections for those specific outputs. This is the messy, unglamorous work of AI safety that doesn't make it into a pitch deck but keeps a company from going under due to a massive data breach.

The Skeptic's Takeaway

The hype cycle wants us to believe that AI agents are ready to run our companies and our lives. The research shows they are barely ready to run a simple script without potentially inviting a Trojan horse into the house. We are building on shaky ground. The first wave of major AI lawsuits and collapses won't be about copyright or bias; it will be about negligence in how we allowed these machines to interact with the real world.

As builders, our job isn't just to make things work. It's to make sure they don't break everything else when they fail. Right now, hallucinations are being treated as a creative quirk. We need to start treating them as the security vulnerabilities they really are.

Read the original at Decrypt →