The Real Danger of Healthcare AI is not Hallucination: It’s How it Affects your Thinking
Most conversations regarding risks surrounding generative AI in healthcare focus on things like hallucinations, biases and reliability. These are all of course significant and measurable risks, but a growing number of researchers argue the bigger risk may be far more subtle and detrimental: AI is beginning to shape how people think.
This concern was raised in a paper by Archie Cotterill, trainee clinical scientist at Maidstone and Tunbridge Wells NHS Trust. His research explores how LLMs influence human cognition itself. Unlike traditional clinical decision-support tools, generative AI goes beyond simple assistance: by interpreting, structuring and prioritising information first, it effectively frames the problem for the user before they even begin their own independent analysis.
So, when clinicians interact with AI systems, they are not starting from a blank canvas, rather they are presented with a polished interpretation that subtly influences which possibilities are explored and which are ignored. Researchers dubbed this phenomenon as “cognitive workflow contamination”, when AI shapes the starting point of clinical reasoning.
There are already quantifiable consequences. Studies have shown clinicians frequently change decisions after consulting AI systems, even when the AI output is incorrect; although one analysis found AI can improve accuracy when its recommendations are right, clinician accuracy can fall by over 11% when the AI hallucinates.
More concerning than hallucinations, AI’s influence is also directional. Once AI frames a situation in a certain way, alternative interpretations may never be fully considered because, through AI omissions, they never even entered the evaluation and reasoning process at all.
If clinicians, particularly junior clinicians still developing expertise, become too accustomed to AI reasoning frameworks during training, there is a real risk that independent analytical skills begin to weaken over time. Even more concerningly, the responsible use of clinical AI depends heavily on the human expertise it may slowly erode.
That doesn’t mean AI has no place in healthcare, clearly it offers enormous potential to reduce administrative burden, synthesise evidence faster and support decision-making. But as healthcare organisations accelerate AI adoption, we must make sure we preserve the independent human clinical reasoning in a world increasingly shaped by artificial interpretations.
“The greater risk lies in how GenAI’s presentation of problems to clinicians affects cognitive reasoning”. Archie Cotterill, Clinical Scientist at Maidstone and Tunbridge Wells NHS Trust
