When laboratory data was included in the case descriptions, DXplain demonstrated its enduring relevance by identifying the correct diagnosis in 72% of the cases. ChatGPT followed with a 64% accuracy rate, while Gemini correctly identified the diagnosis in 58% of the cases. While the modern models performed admirably, DXplain’s specialized architecture allowed it to maintain a lead in providing the most accurate "differential diagnosis"—the list of potential conditions that could explain a patient’s symptoms.
The disparity became even more pronounced during the phase where laboratory data was withheld. In these scenarios, which often mimic the early stages of a patient consultation before tests can be ordered or processed, DXplain successfully identified the correct diagnosis 56% of the time. In contrast, ChatGPT’s accuracy dropped to 42%, and Gemini’s fell to 39%. Although the researchers noted that these specific results did not reach the threshold of statistical significance due to the sample size, the trend clearly favored the specialized expert system over the general-purpose language models.
The implications of these findings are significant for the future of clinical practice. Edward Hoffer, MD, a co-author of the study and a member of the LCS at MGH, noted that the current enthusiasm for large language models often overshadows the foundational successes of earlier AI applications. He emphasized that expert systems like DXplain were the first AI tools to be used successfully in the medical field, and their structured approach offers a level of reliability that is still vital today.
According to the researchers, the primary value of these systems lies in their ability to act as a safety net for the human mind. Clinical environments are often high-pressure and fast-paced, conditions under which even the most experienced physicians can succumb to cognitive biases or simple lapses in memory. An automated system can recall obscure conditions or rare manifestations of common diseases that a doctor might overlook "in the heat of the moment," as Dr. Hoffer described it. Furthermore, because these systems operate on data and logic rather than intuition, they are not susceptible to the common flaws in human reasoning that can lead to diagnostic errors.
Mitchell Feldman, MD, the corresponding author of the study and also an investigator at MGH’s LCS, highlighted that the goal of the research was not necessarily to crown a single winner, but to identify how different technologies might complement one another. He suggested that the optimal path forward involves a synergistic pairing of legacy expert systems and modern large language models.
