Loading prices…
STKR NewsSTKR News0 of 3 free this month
AI

Researcher plants 2m canaries into the AI supply chain. A privacy disaster surfaced.

A massive experiment involving two million digital canaries proves that AI models are leaking sensitive data at a scale most founders are currently ignoring.

Originally on Ars Technica →
AB

Adrian Boysel

Contributor

Nov 20, 2026

4 min read

Photo illustration / STKR News

We have been told for years that the data we feed into large language models is safely abstracted behind high-dimensional mathematics. The narrative is that these models learn the essence of information without memorizing the specifics. A recent experiment involving two million trackable tokens, or digital canaries, just proved that narrative is largely a fantasy.

The Canary in the Silicon Mine

Security researchers recently executed a massive stress test of the AI supply chain. They planted two million unique, trackable strings of data into various points where AI models ingest information. These were not random strings; they were designed to behave like canaries in a coal mine. If that data showed up in an AI response elsewhere, it meant the data had leaked from its supposed container.

The results are troubling for anyone building in this space. These canaries started popping up in places they never should have been. It confirms what many of us suspected: the supply chain for AI training data is essentially a sieve. Whether it is through fine-tuning, RAG pipelines, or direct model training, the walls we think exist between datasets are remarkably thin.

The Founder's Privacy Paradox

As a founder, you are caught in a difficult spot. You need the most powerful models to provide value to your users, but those models are hungry. They require data to be effective, and often, that data is sensitive. We have been operating on a system of trust, assuming that providers like OpenAI, Anthropic, and Google have robust sanitization processes.

The canary experiment suggests these processes are failing. When you build a feature that asks a user to upload a document for analysis, you are potentially contributing to a global pool of training data that might eventually leak that user's private information to a stranger six months from now. This isn't just a technical glitch; it is a fundamental architecture flaw in how we currently handle machine learning weight updates.

Why Sanitization Is Not Working

Most companies claim to scrub PII—personally identifiable information—before it touches a model. The problem is that AI is very good at connecting dots that humans can't see. Even if you remove a name and a social security number, the surrounding context can be unique enough to act as a fingerprint. This experiment showed that even highly specific, obscure data points are being retained and regurgitated.

For builders, this means the standard advice of just using an API and trusting the privacy policy is no longer sufficient. If you are building a healthcare app or a legal tech tool, you are playing with fire. You are relying on a supply chain where the manufacturers themselves don't fully understand how to stop the leakage.

The Hidden Costs of the AI Supply Chain

The AI industry has been built on a move-fast-and-break-things ethos, but privacy is the one thing you can't easily fix once it's broken. Once data is baked into a model's weights, you can't just delete it. There is no easy way to unlearn a specific fact without retraining the entire model at a cost of millions of dollars.

This creates a massive liability for startups. If a major model provider is found to be leaking your users' data, the legal fallout will likely trickle down to you. You are the one who collected the data. You are the one who chose the vendor. The canary test proves that the risk is not theoretical; it is happening at a rate that should make any CTO lose sleep.

How to Build With This Reality

If we accept that the supply chain is leaky, we have to change how we build. Relying on the goodwill of big tech isn't a strategy. Builders need to look toward local execution and data minimization as primary features, not afterthoughts.

  • Local Inference: If the data never leaves the user's device or your private VPC, it can't be leaked into a global training set.
  • Aggressive Differential Privacy: We need to stop trusting basic scrubbing and start using mathematical noise to mask data before it ever hits an external API.
  • Transparency: Be honest with users about where their data goes. The era of claiming absolute privacy while using third-party LLMs is over.

The Skeptical Takeaway

The AI gold rush has blinded a lot of smart people to basic security hygiene. We are so enamored with what these models can do that we aren't looking at what they are taking. The discovery of these two million canaries across the web is a wake-up call that the infrastructure we are building on is fundamentally insecure.

Building a successful company requires a stable foundation. Right now, the AI foundation is leaking user secrets at an alarming rate. If you are a founder, your job is to protect your customers. That might mean moving away from the most popular models in favor of ones you can actually control. The convenience of a powerful API is not worth the risk of a total privacy collapse.

Takeaway for Builders

Stop treating AI models like black boxes that keep secrets. They are more like sponges that squeeze out everything they have absorbed when the right person asks. If you wouldn't post your user's data on a public forum, you shouldn't be sending it to a model that hasn't proven it can actually isolate that data. The canary test proves they can't.


Read the original at Ars Technica →

The Brief

Stay Updated on Cutting-Edge Tech

A six-minute morning dispatch on the markets and the technology shaping them.

Free. No spam. Unsubscribe anytime.

Write for STKR

Become a Contributor

Earn $STKR for published stories on markets, protocols, and culture.

  • Earn $STKR for every published piece
  • Editorial support from the STKR desk
  • Byline visibility across the network
  • First look at the upcoming creator program
Apply to Write

Keep reading

All stories

Comments

24 reader responses