Why linguistic coherence must not be confused with psychological relationship
There is an experience with AI that many now know. One writes something personal — a thought, an uncertainty, a conflict — and receives a response that feels surprisingly apt. The words fit. The tone is right. The response contains nuances one would not have expected. And for a moment, the feeling arises of having been understood.
Perhaps the most astonishing thing about it is not that AI gives genuinely good answers. What is astonishing, rather, is how readily we begin to suspect a counterpart behind those answers. For to us, language has always been the expression of a consciousness. When it also sounds intelligent, sensitive, and coherent, it becomes natural to assume that someone is indeed sitting across from us — someone who understands.
AI systems like ChatGPT can today generate responses that feel remarkably understanding. Precisely therein lies their strength — but also their risk. These systems are optimized to generate contextually fitting continuations from the statistical patterns of human language. What emerges is often linguistically remarkable. It seems precise, empathic in tone, flexible and varied. But what does not emerge is what, in psychology, forms the foundation of every effective form of support. No subject emerges that truly understands.
For an AI system that feels no hunger after a conversation, feels no exhaustion, brings no personal history, and whose memory is technically constructed, bounded, and not biographically borne — such a system cannot be touched. It can recognize patterns that resemble understanding, yes. But that is not the same thing.
The danger lies not in the machine's error, but rather in its persuasive force.
The natural language of such chat systems generates so intuitive a persuasive force1 that responses come to be taken as trustworthy, even when they are factually incomplete or context-blind. In psychological contexts, this effect is particularly potent. Where matters of vulnerability, shame, existential orientation, or traumatic experience are at stake, a convincingly worded but structurally relationless response may not merely be insufficient — it can actively cause harm. Because it creates the illusion of empathy without truly possessing the inner substance for it.
What is this inner substance? It cannot be reduced to knowledge. Those who accompany others in psychology do not merely know more — they understand in a different sense. Moral sensitivity as clinical competence does not arise from applying rules of probability, but from lived, reflected experience. From being embedded in relationships, from the capacity to be touched, from a knowledge of one's own limits that at times become visible only through their transgression. An AI system without history, without a body, without failure. Such a system cannot possess this competence; it can, at most, simulate it.2
In the practical development of mentalhealthGPT — our AI-supported reflection platform for clinical contexts — we have systematically examined how ethical principles can be translated into technical artefacts, and where this translation meets structural limits. The result was encouraging: requirements such as data protection, informed consent, and transparency can be realized to a high degree with sufficient design commitment. But it was precisely this process that also made visible what it cannot reach. Something that cannot be arrived at through better design. Namely human presence, moral subjectivity, or shared history.
Technology changes not only what we do. It can also change how we perceive ourselves.3
The question here is not whether AI has a place in psychological contexts — in practice, it already does — but how technological mediation changes the way people reflect and encounter themselves. What arises when someone formulates a thought and has it mirrored back by a system that holds no perspective of its own? What kind of clarity emerges there, and at whose expense?
Hans Jonas4 wrote that technological action carries a distinctive ethical quality because it operates on timescales and across ranges of effect that exceed individual judgment. Those who build systems that act upon the thinking and feeling of human beings bear a responsibility that extends beyond immediate intentions. This responsibility grows with the reach and persuasive force of the means. A system that sounds precise, warm, and intelligent without these qualities being ethically anchored possesses a persuasive force that demands particular care.
AI systems can prepare, structure, condense, and make visible patterns that would otherwise remain hidden. They can create access in moments when professional support may not, just then, be available. What they cannot do is speak the final word where matters of meaning are at stake. Which is what makes them risky, when they are treated as though they could.
In this connection one often speaks of blended advisory. But this is not simply the pragmatic combination of online and offline. It is the conceptual response to a structural problem: that persuasive force and psychological substance are two different qualities that must not be confused. The technological can, as noted, prepare, condense, structure. Only the human can contextualize, carry the emotional and interpersonal process, decide, and take responsibility. The art lies not in either-or, but in a precise knowledge of what belongs where — and in the willingness to name that boundary, even when the technological sounds very convincing.
I work in precisely this field of tension. As a developer of AI systems for clinical contexts, and as a conversation partner in demanding processes of reflection.
The decisive question is not how human AI can become — but how reflectively we shape what must remain human.
This sentence is meant to negate neither the possibilities nor the risks.
It asks for something harder. Namely, holding the tension between the two. For it is precisely there that the real task lies — for all who bear responsibility for therapeutic decisions and at the same time want to make meaningful use of what new technologies offer.