The rapid ascension of generative artificial intelligence has sparked a global conversation about the future of professional expertise, particularly within the high-stakes corridors of modern medicine. As platforms like ChatGPT and Gemini become household names, the medical community has begun to grapple with a fundamental question: can these general-purpose "large language models" (LLMs) outperform the specialized, expert-driven systems that have quietly assisted clinicians for decades? New research conducted by computer scientists and medical professionals at Massachusetts General Hospital (MGH) suggests that while the new frontier of AI is impressive, the specialized "expert systems" of the past still maintain a critical advantage in diagnostic accuracy.
For those outside the medical informatics field, it may come as a surprise that artificial intelligence is not a newcomer to the hospital setting. Medical professionals have been utilizing various forms of AI to streamline and verify diagnoses for nearly forty years. These tools, known as diagnostic decision support systems (DDSSs), were designed specifically to act as a digital safety net for clinicians. Among the most venerable of these systems is DXplain, a platform first developed in 1984 by the Laboratory of Computer Science (LCS) at MGH. As a founding member of the Mass General Brigham healthcare system, MGH has long been at the forefront of integrating technology into patient care, and DXplain represents one of the earliest and most enduring examples of this mission.
DXplain was built upon a foundation of structured medical knowledge, relying on a massive database that includes thousands of disease profiles, clinical findings, and specific data points. Its primary function is to take a set of clinical observations and generate a ranked list of potential diagnoses, providing clinicians with a comprehensive "differential diagnosis" that helps ensure no rare or easily overlooked condition is ignored. Over the past four decades, the system has evolved, absorbing new medical research and refining its algorithms to remain a relevant tool in the modern clinical environment. However, the sudden explosion of generative AI and LLMs—which are trained on vast swaths of the internet rather than curated medical databases—prompted the investigators at MGH’s LCS to see how their legacy system measured up against the modern giants of Silicon Valley.
The study, recently published in the journal JAMA Network Open, provides a direct comparison between the veteran DXplain system and two of the most prominent LLMs available today: OpenAI’s ChatGPT and Google’s Gemini. The goal was to determine which approach—specialized medical logic or generalized linguistic intelligence—offered the most reliable support for physicians during the diagnostic process. The findings revealed a nuanced picture of the current technological landscape. While DXplain performed somewhat better across the board, the LLMs demonstrated a level of proficiency that suggests they are rapidly closing the gap, albeit with different strengths and weaknesses.
The researchers at MGH believe that the conversation should not necessarily be about which system is "better," but rather how they can be integrated to create a more robust diagnostic framework. The investigators envision a future where the rigid, evidence-based logic of a system like DXplain is paired with the fluid, conversational capabilities of an LLM. This synergy, they argue, represents the optimal way forward for clinical AI, potentially enhancing the efficacy of both systems while significantly improving patient outcomes.
The historical context of this study is essential to understanding its implications. Edward Hoffer, MD, a co-author of the study and a member of the LCS at MGH, noted that the current hype surrounding generative AI often obscures the long-standing successes of earlier models. "Amid all the interest in large language models, it’s easy to forget that the first AI systems used successfully in medicine were expert systems like DXplain," Dr. Hoffer said. These expert systems were built with a specific purpose: to serve as a repository of medical knowledge that could be accessed systematically. Unlike modern LLMs, which function as "black boxes" that predict the next likely word in a sentence, systems like DXplain are built on explicit rules and clinical relationships.
The utility of these systems lies in their ability to augment the human mind, which, despite its brilliance, is subject to fatigue and cognitive limitations. Mitchell Feldman, MD, the corresponding author of the study and also a member of MGH’s LCS, emphasized that these tools are designed to serve as a backup to the physician’s own reasoning. "These systems can enhance and expand clinicians’ diagnoses, recalling information that physicians may forget in the heat of the moment and isn’t biased by common flaws in human reasoning," Dr. Feldman explained. He pointed out that human clinicians can sometimes fall victim to "anchoring bias"—the tendency to rely too heavily on the first piece of information encountered—or simply forget a rare condition that fits a patient’s profile. An AI system, whether it is an LLM or a DDSS, does not suffer from these specific human fallibilities.
To put these systems to the test, the MGH team developed a rigorous methodology involving 36 diverse patient cases. These cases were carefully selected to span a wide range of racial, ethnic, age, and gender categories, ensuring that the AI systems were tested against a representative sample of the human population. This diversity is crucial in medical AI research, as there are ongoing concerns about how different models handle demographic-specific medical nuances. For each of the 36 cases, the investigators presented the clinical findings to DXplain, ChatGPT, and Gemini.
The testing was conducted in two distinct phases: one in which the systems were provided only with the patient’s clinical symptoms and history, and another in which lab data was included. The inclusion of lab data is a critical variable, as it often provides the definitive evidence needed to confirm or rule out a diagnosis. The results showed that all three systems benefited significantly from the addition of laboratory results. With lab data included, the systems were able to identify the correct diagnosis a majority of the time. DXplain led the pack with a 72% success rate, followed by ChatGPT at 64% and Gemini at 58%.
The gap between the specialized system and the general-purpose models became even more pronounced when lab data was withheld. In these scenarios, which mimic the early stages of a patient consultation where a doctor must rely on physical exams and patient history alone, DXplain maintained a correct diagnosis rate of 56%. In contrast, ChatGPT’s accuracy dropped to 42%, and Gemini fell to 39%. While DXplain’s lead in this category was clear, the researchers noted that the results were not statistically significant, likely due to the relatively small sample size of 36 cases. Nevertheless, the trend suggested that DXplain’s deep, structured database allowed it to make more accurate inferences from limited information than the broader, less structured knowledge bases of the LLMs.
One of the most intriguing observations made during the study was that the different systems did not always fail in the same ways. The researchers noticed that the DDSS and the LLMs often caught diseases that the others missed. A case that stumped ChatGPT might be easily identified by DXplain, and vice versa. This lack of overlap in errors is a strong indicator that the two types of AI are utilizing different "thought processes" to reach their conclusions. It is this specific finding that has led the MGH team to advocate for a hybrid approach.
The investigators have already begun preliminary work to explore how these two disparate technologies might be fused. One of the primary hurdles in using a system like DXplain in a busy clinical setting is the requirement for structured data entry. A physician must manually input specific clinical findings into the system, a process that can be time-consuming. Conversely, LLMs are exceptionally talented at processing narrative text—the natural way that doctors write their progress notes and patient histories. By using an LLM to "read" a doctor’s notes and extract the relevant clinical findings, that data could then be automatically plugged into a system like DXplain.
This synergistic model would allow the LLM to handle the linguistic and organizational tasks while the DDSS provides the rigorous, evidence-based diagnostic conclusion. Dr. Feldman believes this combination is the key to the next generation of medical support. "Now, we think combining the powerful explanatory capabilities of existing diagnostic systems with the linguistic capabilities of large language models will enable better automated diagnostic decision support and patient outcomes," he said.
As healthcare systems around the world look for ways to reduce physician burnout and improve diagnostic accuracy, the results from MGH offer a roadmap for the responsible integration of AI. Rather than discarding the "expert systems" that have served the industry for decades in favor of the newest trend, the research suggests that the most effective strategy lies in honoring the precision of the past while leveraging the accessibility and linguistic power of the present. The study serves as a reminder that in the complex field of medicine, the most advanced solution is often not a single piece of technology, but a thoughtful combination of proven methods and new innovations. For the patients at Mass General Brigham and beyond, this hybrid future promises a diagnostic process that is more thorough, less prone to human error, and more capable of handling the complexities of modern healthcare.
