Artificial Intelligence (AI) has come a long way, but can it replace a doctor? Not so fast! A recent study published in PLOS ONE highlights the limitations of ChatGPT in accurately diagnosing medical conditions. The study shows that ChatGPT, the famous AI language model by OpenAI, can answer medical queries, but it still struggles to diagnose complex cases. Let’s dive into the details and have a little fun at the same time.

The aim of the study was to evaluate the effectiveness of ChatGPT as a diagnostic tool for complex clinical cases. Researchers used the Medscape Clinical Challenge, which presents complex patient scenarios that require nuanced clinical skills. These cases often involve multiple health problems and unusual presentations, mimicking real-world medical practice. The goal was to see if ChatGPT could accurately diagnose conditions and provide relevant treatment options.

The researchers tested ChatGPT on 150 Medscape Clinical Challenges published after August 2021, ensuring that the AI ​​had no prior knowledge of these cases. Each case included detailed patient history, examination findings, and diagnostic tests. ChatGPT’s responses were compared to the correct answers and choices made by medical professionals using the same cases.

Findings

ChatGPT was able to provide the correct answer in 49 percent of cases. Compared to the majority of Medscape users’ answers, ChatGPT matched their answers in 61 percent of cases. While these statistics may seem promising, they highlight important shortcomings in AI’s diagnostic capabilities.
The study found that ChatGPT has an overall accuracy of 74 percent, with a precision of 49 percent. This means that while the AI ​​was good at ruling out misdiagnoses, it struggled to correctly identify the correct diagnosis. This discrepancy underscores a key issue: ChatGPT can effectively eliminate incorrect answers, but lacks the reliability to consistently pinpoint the correct diagnosis.

ChatGPT responses were also evaluated for cognitive load and the quality of medical information provided. More than half (52%) of the responses were considered low cognitive load, meaning they were easy to understand. However, 41% required moderate cognitive effort, and 7% were considered highly complex.
Regarding the quality of information, ChatGPT’s answers were complete and relevant in 52 percent of cases. In 43 percent of cases, the answers were incomplete but still relevant. This shows that while ChatGPT can give coherent and grammatically correct answers, it often misses important details needed for an accurate diagnosis.

The study highlighted several factors contributing to ChatGPT’s mediocre performance in diagnosing complex cases. One major issue is its training data, which, while comprehensive, may lack depth in specialized medical knowledge. Additionally, the training data only includes information up to September 2021, meaning ChatGPT may not be aware of the latest medical advancements.

False Positives 13 False positives and false negatives further complicate the reliability of ChatGPT as a diagnostic tool. These inaccuracies can lead to unnecessary treatment or missed diagnoses, which pose significant risks in a clinical setting. AI “hallucinations,” where the model generates plausible-sounding but incorrect information, also contribute to these errors.

While ChatGPT shows potential as a supplemental tool for medical learners, its current limitations make it unsuitable as a standalone diagnostic resource. AI’s ability to provide complete and relevant information needs significant improvement, especially in handling the complexities of real-world medical cases. Until these issues are addressed, human doctors will remain irreplaceable for accurate diagnosis and patient care.