When explaining AI becomes dangerous — sometimes

In the first issue of IAtrogénique × IAtus, I asked a deliberately uncomfortable question: What if AI in healthcare were more than a support tool… but also a force that transforms — or even weakens — our cognitive abilities? I discussed cognitive debt, intellectual comfort, the shifting of responsibility, and that grey area where humans do not disappear… but gradually fade behind the machine. 👉 For anyone who would like to revisit that starting point, Newsletter #1 is here: https://www.linkedin.com/pulse/iatrog%C3%A9nique-iatus-newsletter-1-lionel-reichardt-xnjxe/ Today, I would like to take this one step further, drawing on recent scientific work that confirms these intuitions are more than philosophical ideas or campaigning positions: 👉 they are now measurable, quantifiable and documented.

And what they reveal is deeply counterintuitive.

⚠️ Questioning the dogma of transparency

For several years, a consensus seems to have taken hold: 👉 For AI to be acceptable, it must be explainable. This idea permeates: public policy (the EU AI Act), regulatory requirements (the FDA, health authorities), research into Explainable AI (XAI), and even industry messaging. • public policy (the EU AI Act), • regulatory requirements (the FDA, health authorities), • research into Explainable AI (XAI), • and even industry messaging. Article 13 of the AI Act is its clearest expression: so-called “high-risk” AI systems must be sufficiently transparent to allow appropriate use. 👉 https://artificialintelligenceact.eu/article/13/ On paper, this seems beyond reproach.

The answer, now supported by research, is: no.

🧪 An unsettling finding: the transparency paradox

A recent experimental study involving future doctors sheds decisive light on this. The protocol is rigorous, and the results are robust. The researchers tested a medical AI (realistic performance, ~73%) in different configurations: with or without explanations, with deterministic or probabilistic recommendations. • with or without explanations, • with deterministic or probabilistic recommendations. Participants had to make clinical decisions before, during and after the AI’s intervention, allowing the researchers to measure not only the final decision, but also the dynamics of trust, doubt and reasoning.

The results are striking:

✅ When AI is right, explanations improve the quality of decisions. ❌ When AI is wrong, explanations make decisions worse, sometimes more so than if AI had not intervened. • ✅ When AI is right, explanations improve the quality of decisions. • ❌ When AI is wrong, explanations make decisions worse, sometimes more so than if AI had not intervened. 👉 This is what the authors call a transparency paradox: the explanation amplifies AI’s influence, regardless of the quality of its recommendation.

🧠 Why an explanation becomes a cognitive trap

This is the crux of the problem: generative AI models produce explanations that are equally convincing, whether they are true or false. Our brains, however, work with simple heuristics: fluent reasoning, technical vocabulary, a coherent logical structure, • fluent reasoning, • technical vocabulary, • a coherent logical structure, ➡️ are interpreted as signs of reliability. The result: the explanation no longer serves to evaluate, but to persuade. Research on calibrating human–AI trust shows this very clearly: 👉 https://arxiv.org/abs/2102.09692

📉 The illusion of responsible uncertainty

We might think AI can “make up for it” by expressing its doubts: “diagnosis A: 70%, diagnosis B: 30%”. That is the idea behind probabilistic recommendations, often presented as more ethical and more responsible. But here again, research shows a perverse effect: 👉 when an explanation accompanies uncertainty, it turns that uncertainty into a narrative. The narrative reassures us. And what was meant to encourage caution becomes… a driver of overconfidence.

🩺 Clinicians, cognitive biases and algorithmic authority

Medicine has never been a purely rational exercise. The literature on diagnostic errors has reminded us of this for decades. 👉 Croskerry explains this very well in his work on cognitive biases: https://qualitysafety.bmj.com/content/22/Suppl_2/ii58 Confirmation bias, anchoring, automation… AI does not eliminate these biases. 👉 It reconfigures them. What is changing today is the emergence of a new authority: always available, statistically high-performing, linguistically convincing. • always available, • statistically high-performing, • linguistically convincing. And paradoxically, it is often experienced, cautious professionals who are capable of doubt who are most vulnerable when AI makes a mistake: their initial judgement is overwhelmed by a persuasive explanation.

🧠 Digital overload and decision fatigue

These phenomena sit within a broader context, well documented by the OECD: information overload and decision fatigue in healthcare. 👉 OECD report – Collective Action for Responsible AI in Health: https://www.oecd.org/content/dam/oecd/en/publications/reports/2024/01/collective-action-for-responsible-ai-in-health_9a65136f/f2050177-en.pdf Sometimes, this means: more noise, more dependence, less perspective. • more noise, • more dependence, • less perspective.

🧭 What this (really) changes

If we put this research in perspective alongside the reflection opened in Newsletter #1, one conclusion becomes clear: 👉 The problem is not AI. 👉 The problem is how it fits into our cognitive and organisational architectures. Explainability is not an absolute value. It is a design tool, which must be: contextualised, conditional, adaptive. • contextualised, • conditional, • adaptive.

🔚 Conclusion

AI in healthcare can save time, improve quality and reduce certain errors. But it can also produce cognitive iatrogenic effects that are silent and progressive. That is precisely why IAtrogénique × IAtus exists: to document these invisible effects, without naive fascination or ideological rejection. 📩 If these questions concern you, 📌 if you work in healthcare, innovation, regulation or digital technology, 📣 then this discussion is only just beginning. See you very soon for what comes next. — Lionel Reichardt - aka Pharmageek - IAtrogénique × IAtus Do you like this content? Help spread the word.. open up the debate.. share.. like...

HAPPY HOLIDAYS TO EVERYONE!!