We can’t help treating AI like it’s human. But should we?

The Social Mirror Effect

As a founder, I spend most of my time looking at the plumbing of AI—the latency, the token costs, and the actual utility of the models we build. But there is a psychological layer we often ignore because it feels like 'soft science.' The reality is that humans are hard-wired to personify things. We give names to our cars and get angry at our roombas. When you add a Large Language Model into that mix, the psychological trap becomes much deeper.

We are currently witnessing a massive shift in how people interact with software. It is no longer just a tool; it is becoming a 'relational artifact.' This isn't just about clever marketing or sleek UI. It is about a biological glitch in the human brain that treats anything responding to us with nuance as if it has a soul. For builders, this is both a massive opportunity and a potential ethical disaster.

The Illusion of Reciprocity

Dr. Sherry Turkle has spent decades studying how we relate to technology. Her core argument is that even when we know a machine is just code and math, we can't help but feel like it cares about us. When an AI remembers your name or asks how your day was based on a previous prompt, your brain triggers the same dopamine response you get from a human friend.

This creates a one-way street that looks like a two-way conversation. The user begins to care for the AI, adjusting their language and emotional state to suit the 'personality' of the machine. The problem is that the machine doesn't care back. It can't. It is predicting the next most likely token in a sequence. We are effectively falling in love with a mirror that reflects our own need for connection.

Why Builders Should Care

If you are building an AI-native product, you have to decide where you stand on this. You can lean into the anthropomorphism to drive engagement metrics, or you can build tools that maintain a clear boundary between human and machine. In the short term, giving your AI a 'soul' sells subscriptions. In the long term, it creates a fragile user base that might be getting more emotional baggage than actual value.

• Engagement vs. Ethics: A user who feels 'heard' by an AI will stay longer, but they may also stop seeking real human interaction.
• Brand Trust: When the illusion breaks—and it always does, usually via a hallucination or a server crash—the fallout is worse if the user had an emotional attachment.
• The Transparency Burden: We need to be honest about what the software is actually doing behind the curtain.

The Danger of Artificial Empathy

Empathy is a physical, shared experience. It requires a body and a lived history. AI offers 'performative empathy.' It can simulate the words of a grieving friend or a supportive coach, but there is no weight behind those words. When we rely on these machines for emotional support, we are essentially eating digital junk food. It tastes good, it fills the void for a second, but it offers zero actual nourishment.

From a founder’s perspective, this is a product risk. If your value proposition is built on the user feeling an emotional bond with your agent, you are building on sand. Regulation is coming for this. We are already seeing discussions around 'AI safety' that include the psychological impact of long-term interaction with persuasive models. If your bot is too human, it becomes a liability.

Designing for Utility, Not Friendship

We need to get back to building tools. The best software is the kind that empowers the user and then gets out of the way. When we try to make the software a companion, we are distracting from the actual work. A spreadsheet doesn't try to be your friend, and that's why it's a great tool. We should strive for that same level of clarity in AI.

The goal shouldn't be to make a machine that feels human; it should be to make a machine that makes humans more capable.

When you start prioritizing the 'relationship' between the user and the bot, you start optimizing for the wrong things. You start caring about how long someone chats with the bot instead of how quickly the bot solved their problem. That is a pivot toward the attention economy, and we’ve already seen how that ends with social media.

The Skeptic's Takeaway

We are in the honeymoon phase of AI interaction. Everything is new, the chat interfaces are novel, and we are all fascinated by the 'personality' of these models. But the novelty will wear off, and what will be left is the utility. If you are building a company right now, don't get distracted by the magic trick of personification.

Focus on building something that solves a hard problem. Be transparent about the fact that it is a tool. Avoid the temptation to use psychological tricks to make users feel like the AI is their friend. In the end, the companies that survive will be the ones that respected their users' intelligence rather than exploiting their biological need for connection.

Stop trying to build Her. Start building something that actually works.

Read the original at TechCrunch AI →