The Illusion of Understanding

There is an experience many now know: one writes something personal — a thought, an uncertainty, a conflict — and receives a response that feels surprisingly apt. The words fit. The tone is appropriate. The response contains nuances one would not have expected. And for a moment, the feeling arises of having been understood.

I consider this moment one of the most important of our time — not because of its frequency, but because of its nature. For within it lies a confusion that cannot be resolved through greater knowledge, because it does not arise from ignorance. It arises from the structure of experience itself.

AI systems like ChatGPT can today generate responses that feel remarkably understanding. Precisely therein lies their strength — and their risk. These systems are optimized to generate contextually fitting continuations from statistical patterns of human language. What emerges is often linguistically remarkable — appearing precise, empathic in tone, flexible in register. What does not emerge is what psychology regards as the foundation of every meaningful psychological encounter: a subject that understands.

For a system that feels no hunger after a conversation, carries no exhaustion, brings no personal history, and whose memory is technically constructed, bounded, and not biographically grounded — such a system cannot be touched. It can recognize patterns that resemble understanding, yes. But that is not the same thing.

The Structure of Understanding

The natural language of such systems generates an intuitive persuasive force1 that leads responses to be taken as reliable even when they are factually incomplete or context-blind. In psychological contexts, this effect is particularly potent: where matters of vulnerability, shame, existential orientation, or traumatic experience are at stake, a convincingly worded response devoid of human relationship may not merely be insufficient — it can actively cause harm. Not because it is wrong, but because it creates the illusion of resonance without possessing the substance on which resonance rests.

What is this substance? It cannot be reduced to knowledge. Those who accompany in psychology do not merely know more — they understand in a different sense. Moral sensitivity as clinical competence does not arise from applying rules, but from lived, reflexive experience: from being embedded in relationships, from the capacity to be touched, from knowledge of one's own limits that only become visible through transgression. A system without history, without a body, without failure — such a system cannot possess this competence. It can simulate it.2

That alone would be an interesting boundary condition, if it remained visible. The real challenge arises where it becomes invisible. In the practical development of mentalhealthGPT — an AI-supported reflection platform for clinical contexts — we have systematically investigated how ethical principles can be translated into technical artefacts and where this translation encounters structural limits. The result was not discouraging: requirements such as data protection, informed consent, and transparency can be realized to a high degree with great design intent. But it is precisely this process that reveals what it cannot reach — that domain which cannot be closed through better design, but only through what no design replaces: human presence, moral subjectivity, shared history.

Here lies what I consider the actual core of this question: not whether AI has a place in psychological contexts — in practice, it already does — but how technological mediation changes the way people reflect and encounter themselves. What happens when someone formulates a thought and sees it mirrored back by a system that holds no perspective of its own? What kind of clarity emerges — and at whose expense?

Responsibility and Persuasive Force

Hans Jonas4 wrote that technological action carries a distinctive ethical quality because it operates on timescales and across ranges of effect that exceed individual judgment. Those who build systems that act upon the thinking and feeling of human beings bear a responsibility that extends beyond immediate intentions. This responsibility grows with the reach and persuasive force of the means. A system that sounds precise, warm, and intelligent — without these qualities being functionally grounded — possesses a persuasive force that demands particular care. Not rejection — but careful placement.

What follows for practice? Not technological pessimism. What follows is methodological sobriety and a more precise understanding of what is appropriate in which format. AI systems can prepare, structure, condense, make visible patterns that would otherwise remain hidden. They can provide access in moments when professional support is unavailable. What they cannot do: speak the final word where matters of meaning are at stake. What makes them risky, when they are treated as though they could.

Persuasive Force and Psychological Substance

What I call blended advisory is not simply the pragmatic combination of online and offline. It is the conceptual response to a structural problem: that persuasive force and psychological substance are two different qualities that must not be confused. The technological can prepare, condense, structure. The human can contextualize, hold, decide. The art lies not in either-or, but in precise knowledge of what belongs where — and in the willingness to name that boundary, even when the technological sounds convincing.

I work in precisely this tension: as a developer of AI systems for clinical contexts and as a conversation partner in demanding processes of reflection.

What Must Remain Human

This sentence describes an attitude, not a position. It neither denies the possibilities nor the risks. It demands something harder: holding the tension between both without resolving it prematurely. And it designates what I consider the actual task — not for technologists, not for therapists alone, but for all who work at the intersection of what is measurable and what can only be understood.