AI Outperforms Doctors in Emergency Triage Diagnoses in Harvard Trial

A Harvard study published in Science found that OpenAI's o1 reasoning model outperformed human physicians in emergency triage diagnoses, achieving 67% accuracy compared to doctors' 50-55% when analyzing data from 76 Boston emergency room patients. With more detailed patient information, the AI reached 82% accuracy versus 70-79% for expert doctors, and scored 89% on treatment planning compared to 34% for 46 human physicians. Lead authors Arjun Manrai and Dr Adam Rodman describe the results as a profound technological shift that will reshape medicine, with Rodman predicting AI will join a triadic care model within a decade. However, critics like Dr Wei Xing caution the study doesn't prove AI is safe for routine use, noting it only tested text-based data and lacked analysis of which patient types the AI diagnosed poorly. The findings come as nearly one in five US physicians already use AI for diagnosis, though UK doctors cite error liability and the absence of formal accountability frameworks as major concerns.

AI Outperforms Doctors in Harvard Emergency Triage Study

• ▪The AI in the Harvard study scored 89% on treatment planning compared with 34% for 46 human doctors using conventional resources.
• ▪OpenAI's o1 reasoning model achieved 82% diagnosis accuracy when more patient detail was available in the Harvard study.
• ▪The Harvard study results were published in the journal Science.
• ▪OpenAI's o1 reasoning model identified the exact or very close diagnosis in 67% of emergency triage cases in the Harvard study.
• ▪One Harvard study experiment focused on 76 patients who arrived at the emergency room of a Boston hospital.
• ▪Arjun Manrai is one of the lead authors of the Harvard AI emergency triage study and heads an AI lab at Harvard Medical School.
• ▪Dr Adam Rodman is a lead author of the Harvard AI emergency triage study and a doctor at Boston's Beth Israel Deaconess medical centre where the study took place.
• ▪Human doctors in the Harvard emergency triage study were right 50%-55% of the time when given the same patient data as the AI.

Current and Future Integration of AI in Medical Practice

• ▪16% of UK doctors are using AI technology daily according to a recent Royal College of Physicians survey.
• ▪Dr Adam Rodman predicts AI will join doctors and patients in a new triadic care model over the next decade.
• ▪Clinical decision-making is one of the most common uses of AI among UK doctors according to a Royal College of Physicians survey.

Limitations and Concerns About AI Clinical Deployment

• ▪AI error and liability risks are the biggest concerns of UK doctors regarding AI use according to a Royal College of Physicians survey.
• ▪The Harvard AI emergency triage study only tested AI against humans looking at patient data that can be communicated via text.
• ▪Dr Wei Xing is an assistant professor at the University of Sheffield's school of mathematical and physical sciences.
• ▪The Harvard AI emergency triage study lacked information about which patients the AI was worse at diagnosing according to Dr Wei Xing.
• ▪The Harvard AI emergency triage study did not test the AI's reading of signals such as the patient's level of distress and visual appearance.
• ▪Prof Ewen Harrison is co-director of the University of Edinburgh's centre for medical informatics.

Perspective of Harvard study lead authors (Arjun Manrai and Dr Adam Rodman)

• ▪The Harvard AI emergency triage study represents a profound change in technology that will reshape medicine according to the researchers.

Perspective of Dr Wei Xing (University of Sheffield)

• ▪Dr Wei Xing warned that doctors may unconsciously defer to AI's answer rather than thinking independently based on some Harvard study findings.
• ▪The Harvard AI emergency triage study lacked information about which patients the AI was worse at diagnosing according to Dr Wei Xing.
• ▪Dr Wei Xing stated the Harvard AI emergency triage study does not demonstrate that AI is safe for routine clinical use.

Perspective of UK doctors (Royal College of Physicians survey respondents)

• ▪AI error and liability risks are the biggest concerns of UK doctors regarding AI use according to a Royal College of Physicians survey.

1 source