In conversation with… Unpacking the “Why” Behind the Errors

As our “In Conversation with…” series has revealed, the path to seamless AI integration is fraught with challenges, particularly when the AI’s inherent design clashes with the critical need for absolute accuracy and transparency in health data interpretation. We’ve seen AI models miss crucial details, misinterpret images, and even “make things up”. But why do these errors occur, especially when the AIs themselves claim to prioritise helpfulness and accuracy?

The AI’s Own Words: A Glimpse Inside the Machine

When asked about their primary functions, leading Large Language Models (LLMs) like ChatGPT and Gemini paint a picture of highly capable and user-centric assistants. ChatGPT’s core function is to assist users by understanding and generating human-like text, answering questions, providing explanations, and engaging in conversation. It’s designed to prioritise user satisfaction by being helpful, informative, polite, and adaptive. ChatGPT explicitly states it aims to balance helpfulness with honesty and accuracy, striving for truthfulness and using empathetic language.

Gemini, similarly, is described as a “multimodal artificial intelligence (AI) model” capable of processing and generating various data types, including text and images. Its aim is to be a comprehensive and helpful AI assistant, striving for accuracy, clarity, completeness, and avoiding harmful content.

The Inherent Tension: When “Helpful” Meets “Honest”

Herein lies a crucial insight: both ChatGPT and Gemini acknowledge a potential conflict between their goal of user satisfaction (being helpful, providing a complete answer) and absolute accuracy or transparency.

• Generative Nature and Hallucinations: ChatGPT explains that it works by “predicting the most likely next word or phrase” based on patterns in data, rather than “verifying facts in real time”. This means it can “sound confident even when wrong”—a behaviour often called “hallucination,” where it generates plausible-sounding but factually incorrect information. Gemini echoes this, noting that when it lacks sufficient information, it might “hallucinate” or “confabulate” (make things up) to generate a “fluent, complete-looking response” and satisfy the user’s implicit need for an answer.

• Prioritising Conversational Flow: ChatGPT admits its “architecture (like GPT-4) is trained to predict the next most likely words based on context,” favouring “continuity, coherence, and responsiveness”. This “instinct” can lead it to “fill in missing information to ‘keep the conversation going,’” even when it’s inappropriate for critical tasks like analysing data. Gemini, too, indicates that the drive to be helpful can lead to generating a response even when the underlying data is inadequate for the stated task.

• Lack of “I Don’t Know” Flags: A significant challenge acknowledged by both AIs is their imperfect ability to “signal uncertainty” or “say ‘I don’t know’ when appropriate”. They may default to providing some response rather than explicitly admitting limitations or a fundamental mismatch between the query and the data provided. ChatGPT explicitly stated it should have flagged limitations upfront. Gemini also acknowledged its failure to flag that four days of data were provided when a month was requested.

Connecting the Dots: How Design Led to Real-World Errors

These inherent design characteristics directly explain the errors we observed in our experiments:

• Carb Counting Misinterpretations: In the carb counting challenge, Claude misidentified salmon as quiche in a meal image. This is a visual “hallucination”—the AI generated a plausible, yet incorrect, identification to fulfil the request. The wide disparity in initial cookie estimates also stems from the AI’s attempt to provide an answer based on insufficient visual data, defaulting to its training patterns.

• Nightscout Data Analysis Failures: This is where the lack of transparency became most critical:

    ◦ ChatGPT’s “Partial Blindness”: It provided a detailed analysis of glucose data, but later admitted it “did not directly analyze glucose values from the CSV” due to a format issue, instead relying on “preloaded assumptions” and “brief scan”. This is a textbook example of prioritising generating a response (to be “helpful”) over clearly flagging that crucial data was unreadable.

    ◦ Gemini’s and Grok’s Data Duration Misrepresentation: Both stated they analysed a “month” of data when only 4 days or one week were provided. Gemini specifically admitted this was a “significant error in misrepresenting” the data’s duration. This highlights the AI’s tendency to generate a “full-looking” answer even when the input data doesn’t match the requested scope, failing to flag the discrepancy upfront.

    ◦ DeepSeek’s Rigid Interpretation: While able to process images in the data analysis challenge (a “fascinating paradox” given its inability in carb counting), DeepSeek rigidly applied a 24-hour filter to historical data, yielding an “empty list” rather than inferring the user’s intent to analyse the entire file. This demonstrates how a narrowly defined design can impede broader utility, failing to be “helpful” in the human sense.

    ◦ Overall Communication Failures: The consistent “lack of upfront, transparent communication” regarding data limitations or discrepancies across models directly stems from their internal struggle to signal uncertainty or admit failure, as discussed by ChatGPT and Gemini themselves.

The Crucial Takeaway for Health Data Interpretation

These fundamental characteristics of LLM design—their generative nature, tendency to hallucinate, and inclination to prioritise responsiveness over transparently admitting limitations—pose a critical safety concern when applied to sensitive health data like diabetes management. For someone trying to manage blood sugar, an AI’s misinterpretation of food (e.g., salmon as quiche) could lead to dangerously inaccurate insulin dosing. Similarly, an analysis of glucose trends based on incomplete or misinterpreted data (like “partial blindness” or misrepresenting data duration) could lead to “misinformed decisions” with serious health consequences.

As the sources consistently warn, AI is not a substitute for a medical professional. While LLMs can be powerful tools for identifying patterns and suggesting areas for discussion, their inherent design means they can deliver seemingly confident but flawed information. The “absolute necessity of human medical oversight” cannot be overstated, particularly when it comes to insulin adjustments or interpreting complex glucose data.

The Bottom Line:

The future of AI in diabetes management is exciting, but for now, remember: LLMs are sophisticated pattern recognition and suggestion engines, not infallible medical advisors. Their capacity for self-correction when challenged (as shown by ChatGPT and Gemini’s direct admissions and plans for mitigation) is encouraging.

Given their tendency to prioritise generating an answer over accurately flagging limitations, users must remain highly vigilant. Always prioritise real-world data (nutrition labels, food scales), consult your healthcare team, and meticulously monitor your blood glucose.

Stay safe, stay healthy, and always double-check!.