The integration of artificial intelligence into the clinical environment is often discussed as a modern phenomenon, a product of the recent explosion in computing power and the advent of sophisticated neural networks. However, for the medical professionals at Massachusetts General Hospital, the use of automated systems to assist in the complex task of patient diagnosis is a practice that stretches back four decades. Long before the public consciousness was captured by the conversational fluency of generative AI, clinicians were utilizing what are known as diagnostic decision support systems (DDSSs) to navigate the intricacies of rare diseases and overlapping symptoms.
A new study led by investigators at the Massachusetts General Hospital Laboratory of Computer Science (LCS) has sought to bridge the gap between these two eras of technology. The research, recently published in JAMA Network Open, provides a rigorous comparison between the hospital’s long-standing proprietary diagnostic system, DXplain, and the current leaders of the generative AI revolution: OpenAI’s ChatGPT and Google’s Gemini. The findings suggest that while modern large language models (LLMs) possess remarkable capabilities, the structured, expert-driven logic of legacy systems still holds a significant edge in clinical accuracy.
The origins of DXplain date back to 1984, a period when the Laboratory of Computer Science at MGH—a founding member of the Mass General Brigham healthcare system—first sought to codify medical knowledge into a searchable, logical framework. Over forty years, DXplain has evolved into a massive repository of clinical intelligence, encompassing thousands of disease profiles and tens of thousands of individual clinical findings. Unlike modern LLMs, which learn patterns from vast swathes of internet data, DXplain was built as an "expert system," relying on curated data points to generate and rank potential diagnoses for use by working clinicians.
The emergence of generative AI and large language models has fundamentally shifted the landscape of medical technology. These models are prized for their ability to process natural language and provide human-like responses, making them highly accessible to healthcare providers who may not have the time to navigate complex database interfaces. Recognizing this shift, the MGH researchers aimed to determine if the newer, more versatile models could match or exceed the performance of a system specifically engineered for medical diagnostics.
In their comparative analysis, the research team utilized 36 distinct patient cases. These cases were carefully selected to represent a broad spectrum of the human experience, spanning various racial and ethnic backgrounds, age groups, and genders. This diversity was essential to ensure that the diagnostic tools were being tested against the kind of varied presentations seen in a real-world clinical setting, where demographic factors can often influence the likelihood or presentation of certain conditions.
The testing protocol was divided into two distinct phases to evaluate how each system handled different levels of information. In the first phase, the systems were provided with patient histories and physical examination findings but were denied access to laboratory data. In the second phase, the systems were given the full suite of information, including laboratory results, to see how the addition of objective data points influenced their diagnostic accuracy.
