Humans Are Reading Your ChatGPT Chats, New Lawsuit Claims

The Myth of the Closed Loop

For a long time, the pitch for integrating Large Language Models into our daily workflows was privacy through automation. The idea was simple: your data goes into a black box, the box gets smarter, and nobody actually sees what you are typing. It turns out that might have been wishful thinking. A new class-action lawsuit is pulling back the curtain on a program allegedly called Project Lily, where OpenAI reportedly routes private user conversations to human contractors for review.

As a founder, I have always been skeptical of the "set it and forget it" promise of AI safety. Building something this complex usually requires a massive amount of manual cleaning. We often talk about AI as this autonomous force, but the reality is much messier. If these allegations are true, it means that while you were brainstorming your next pivot or venting about a difficult board meeting, a contractor halfway across the world might have been reading the transcript to make sure the bot didn't glitch.

What Project Lily Tells Us About Scaling

The lawsuit claims that OpenAI failed to properly disclose that human eyes would be on these chats. For those of us in the building phase, this is a classic case of the tension between speed and transparency. To make a model better, you need Reinforcement Learning from Human Feedback (RLHF). You need people to tell the machine when it is being a hallucinating mess. But there is a massive difference between using public datasets and using live, private user inputs to train those models.

If you are building an app on top of these APIs, this should make you pause. We tell our users their data is secure, but we are often just pass-throughs for larger providers. If the foundation layer is leaky—or at least more transparent to human reviewers than we thought—it puts every builder in a tough spot. We are inheriting the privacy debt of the platforms we build on.

The Hidden Cost of Feedback Loops

We need to stop pretending that AI is just code. It is a service fueled by human labor. Most of these contractors are reportedly working in low-wage environments, tasked with labeling thousands of prompts a day. This is the "ghost work" that powers the Silicon Valley dream. The problem isn't necessarily that humans are involved; the problem is the lack of a clear opt-out that actually means something.

OpenAI has always had a vague setting regarding data usage for training, but the lawsuit suggests that Project Lily bypassed the expectations of a reasonable user. When a user toggles a privacy setting, they expect a vault. Instead, they might have gotten a window. For founders, the lesson here is about building trust. If you are going to use human reviewers to audit your system's performance, you have to be loud about it. Hiding it in a sub-paragraph of a Terms of Service agreement is a ticking time bomb.

The Risks for Builders and Startups

If you are building in the enterprise space, this is a nightmare. Enterprise clients are already terrified of their proprietary trade secrets ending up in a public model. If the news cycle continues to focus on human contractors reading sensitive chats, the barrier to adoption for AI tools in corporate environments is going to skyrocket. We are already seeing companies ban ChatGPT internally; this lawsuit just gives their legal teams more ammunition.

We also have to consider the regulatory fallout. Lawsuits like this often lead to discovery processes that reveal exactly how much data is being handled and by whom. If it turns out that PII (Personally Identifiable Information) was routinely exposed to third-party contractors without encryption or redaction, we are looking at a GDPR and CCPA bonfire. As builders, we need to start auditing our own data pipelines. Don't just trust the API provider's marketing copy.

• Encryption isn't enough: If the data is decrypted for a human reviewer, your encryption at rest doesn't matter.
• Redaction is mandatory: Before sending anything to a model, builders should be stripping PII on their own end.
• Transparency is a feature: Being honest about human intervention can actually be a selling point for high-stakes industries.

The Founder's Perspective

I’ve spent years looking at how new tech hits the market, and the pattern is always the same. You move fast, you break things, and then you spend five years in court trying to explain why you broke them. OpenAI is in the "explain" phase now. They are trying to bridge the gap between a research project and a global utility. But utilities have to be reliable and, more importantly, they have to be private.

For those of us building the next wave of tools, we have to be better than the giants. We don't have the legal budgets to fight class-action lawsuits for a decade. We have to bake privacy into the architecture from day one. That means assuming that anything you send to a third-party LLM could potentially be seen by a human, and building your product logic around that risk.

Takeaway

The takeaway here is simple: there is no such thing as a private conversation with a centralized AI. If you are building a product that handles sensitive data, you need to assume that human-in-the-loop is the default, not the exception. The Project Lily allegations are a reminder that the "AI" we use is often just a very large group of humans disguised as an algorithm. Build accordingly.

Read the original at Decrypt →