The integration of artificial intelligence into the clinical environment is often discussed as a modern phenomenon, a product of the recent explosion in computing power and the advent of sophisticated neural networks. However, for the medical professionals at Massachusetts General Hospital, the use of automated systems to assist in the complex task of patient diagnosis is a practice that stretches back four decades. Long before the public consciousness was captured by the conversational fluency of generative AI, clinicians were utilizing what are known as diagnostic decision support systems (DDSSs) to navigate the intricacies of rare diseases and overlapping symptoms.
A new study led by investigators at the Massachusetts General Hospital Laboratory of Computer Science (LCS) has sought to bridge the gap between these two eras of technology. The research, recently published in JAMA Network Open, provides a rigorous comparison between the hospital’s long-standing proprietary diagnostic system, DXplain, and the current leaders of the generative AI revolution: OpenAI’s ChatGPT and Google’s Gemini. The findings suggest that while modern large language models (LLMs) possess remarkable capabilities, the structured, expert-driven logic of legacy systems still holds a significant edge in clinical accuracy.
The origins of DXplain date back to 1984, a period when the Laboratory of Computer Science at MGH—a founding member of the Mass General Brigham healthcare system—first sought to codify medical knowledge into a searchable, logical framework. Over forty years, DXplain has evolved into a massive repository of clinical intelligence, encompassing thousands of disease profiles and tens of thousands of individual clinical findings. Unlike modern LLMs, which learn patterns from vast swathes of internet data, DXplain was built as an "expert system," relying on curated data points to generate and rank potential diagnoses for use by working clinicians.
The emergence of generative AI and large language models has fundamentally shifted the landscape of medical technology. These models are prized for their ability to process natural language and provide human-like responses, making them highly accessible to healthcare providers who may not have the time to navigate complex database interfaces. Recognizing this shift, the MGH researchers aimed to determine if the newer, more versatile models could match or exceed the performance of a system specifically engineered for medical diagnostics.
In their comparative analysis, the research team utilized 36 distinct patient cases. These cases were carefully selected to represent a broad spectrum of the human experience, spanning various racial and ethnic backgrounds, age groups, and genders. This diversity was essential to ensure that the diagnostic tools were being tested against the kind of varied presentations seen in a real-world clinical setting, where demographic factors can often influence the likelihood or presentation of certain conditions.
The testing protocol was divided into two distinct phases to evaluate how each system handled different levels of information. In the first phase, the systems were provided with patient histories and physical examination findings but were denied access to laboratory data. In the second phase, the systems were given the full suite of information, including laboratory results, to see how the addition of objective data points influenced their diagnostic accuracy.
When laboratory data was included in the case descriptions, DXplain demonstrated its enduring relevance by identifying the correct diagnosis in 72% of the cases. ChatGPT followed with a 64% accuracy rate, while Gemini correctly identified the diagnosis in 58% of the cases. While the modern models performed admirably, DXplain’s specialized architecture allowed it to maintain a lead in providing the most accurate "differential diagnosis"—the list of potential conditions that could explain a patient’s symptoms.
The disparity became even more pronounced during the phase where laboratory data was withheld. In these scenarios, which often mimic the early stages of a patient consultation before tests can be ordered or processed, DXplain successfully identified the correct diagnosis 56% of the time. In contrast, ChatGPT’s accuracy dropped to 42%, and Gemini’s fell to 39%. Although the researchers noted that these specific results did not reach the threshold of statistical significance due to the sample size, the trend clearly favored the specialized expert system over the general-purpose language models.
The implications of these findings are significant for the future of clinical practice. Edward Hoffer, MD, a co-author of the study and a member of the LCS at MGH, noted that the current enthusiasm for large language models often overshadows the foundational successes of earlier AI applications. He emphasized that expert systems like DXplain were the first AI tools to be used successfully in the medical field, and their structured approach offers a level of reliability that is still vital today.
According to the researchers, the primary value of these systems lies in their ability to act as a safety net for the human mind. Clinical environments are often high-pressure and fast-paced, conditions under which even the most experienced physicians can succumb to cognitive biases or simple lapses in memory. An automated system can recall obscure conditions or rare manifestations of common diseases that a doctor might overlook "in the heat of the moment," as Dr. Hoffer described it. Furthermore, because these systems operate on data and logic rather than intuition, they are not susceptible to the common flaws in human reasoning that can lead to diagnostic errors.
Mitchell Feldman, MD, the corresponding author of the study and also an investigator at MGH’s LCS, highlighted that the goal of the research was not necessarily to crown a single winner, but to identify how different technologies might complement one another. He suggested that the optimal path forward involves a synergistic pairing of legacy expert systems and modern large language models.
The researchers observed that the different systems often caught diseases that the others missed. While DXplain excelled at structured diagnostic logic, the LLMs showed a superior ability to interpret and summarize the narrative text that makes up a large portion of a patient’s medical record. This observation has led to preliminary work on a hybrid approach. In this envisioned model, an LLM would be used to extract clinical findings and symptoms from a physician’s narrative notes or a patient’s spoken history. These extracted data points would then be fed into a system like DXplain, which would process them through its rigorous disease-profile database to generate a ranked list of diagnoses.
This combination would leverage the linguistic power of generative AI to handle the "input" side of the equation—turning messy, unstructured human language into clean data—while utilizing the specialized clinical logic of a DDSS to handle the "inference" side. Dr. Feldman believes that this integration will enable better automated diagnostic decision support, ultimately leading to improved patient outcomes by reducing the time to diagnosis and minimizing errors.
The study reflects a broader trend in medical informatics toward "augmented intelligence," where technology is designed not to replace the physician, but to enhance their capabilities. By providing a more robust and accurate list of potential diagnoses, these tools allow clinicians to focus their expertise on the nuances of patient care and treatment planning.
As the healthcare industry continues to grapple with the challenges of physician burnout and an increasingly complex medical landscape, the results published in JAMA Network Open serve as a reminder that the newest technology is not always the only solution. The four-decade evolution of DXplain at Massachusetts General Hospital demonstrates the value of long-term data curation and specialized engineering. By marrying this historical expertise with the conversational flexibility of modern AI, the researchers at MGH believe they can create a more reliable and effective framework for the future of medicine.
The findings also suggest that while LLMs are rapidly improving, there remains a critical need for domain-specific verification in high-stakes fields like healthcare. The structured profiles within DXplain provide a level of transparency and clinical grounding that general-purpose models currently struggle to replicate consistently. As the research continues, the team at the Laboratory of Computer Science plans to further explore how these different AI architectures can be woven together to create a seamless assistant for the next generation of medical professionals.
