OpenAI Model Surpasses Physicians on Clinical Reasoning Tasks in Large-Scale Study

Apr 30, 2026

Researchers led by internist and clinical AI researcher Adam Rodman published a compilation of experiments in Science on Thursday demonstrating that an OpenAI large language model outperformed physicians on case-based diagnostic and clinical reasoning evaluations, including one experiment using real-world data from a Boston emergency department. The study represents one of the largest AI-physician comparison studies to date, building on methodology from a 1959 Science paper that established criteria for evaluating clinical decision support systems. However, co-senior author Rodman warns the results should not be misconstrued as proof of AI safety and efficacy for real patient treatment, noting all experiments used simulated and historical cases rather than real-time scenarios, even as generative AI tools are being heavily marketed to patients and clinicians.

OpenAI Model's Superior Performance in Clinical Reasoning

▪Adam Rodman is the co-senior author of the Science paper on OpenAI's large language model performance.
▪Adam Rodman is an internist and clinical AI researcher.
▪One experiment in the Science publication used real-world data from a Boston emergency department.
▪A 1959 Science paper described how to determine if a clinical decision support system was capable of doing diagnosis better than humans.
▪Adam Rodman and colleagues published a compilation of experiments in Science on Thursday showing a large language model from OpenAI can outperform physicians in case-based diagnostic and clinical reasoning evaluations.

Researcher Concerns About Misinterpretation and Real-World Application

▪All experiments in the Science paper were based on simulated and historical cases rather than real-time patient treatment.
▪Adam Rodman is worried that the science experiments will be misconstrued as proof of AI's safety and efficacy when used to treat real patients.
▪Generative AI tools like chatbots are being heavily marketed to both patients and clinicians.

Perspective of Adam Rodman

▪Adam Rodman emphasizes that the Science paper experiments used simulated and historical cases rather than real-time patient treatment scenarios.
▪Adam Rodman is concerned the Science experiments could be misinterpreted as evidence that AI is safe and effective for treating real patients.