Medical AI Community Grapples with Benchmarking Standards as Clinical LLMs Proliferate

As clinical and biological large language models proliferate, researchers are grappling with benchmarking their reasoning capabilities. A study evaluating general-purpose models like GPT-5.2 Pro, Gemini 3.0 Pro-Preview, and Claude Opus 4.6 reveals they can reconstruct 24% to 65% of literature-curated signaling network reactions. However, these AI-generated networks struggle with downstream regulatory pathways and show low functional accuracy, predicting perturbation responses at rates of only 5.8% to 33.3% compared to over 78% for manually curated models.

LLM-generated biochemical network models

• ▪Logic-based models derived from large language model-generated signaling networks predict responses to perturbations with accuracies ranging from 6% to 33%.
• ▪General-purpose large language models generate 24% to 65% of the reactions of literature-curated signaling networks for cardiomyocyte hypertrophy, myofibroblast activation, and mechanosignaling.
• ▪Large language models generate 64% to 91% of the reactions within the core Escherichia coli metabolic network.

Signaling network reconstruction accuracy

• ▪GPT-5.2 Pro, Gemini 3.0 Pro-Preview, and Claude Opus 4.6 reconstruct 37.70% to 63.87% of network reactions in a manually curated cardiomyocyte hypertrophy signaling network.
• ▪GPT-5.2 Pro, Gemini 3.0 Pro-Preview, and Claude Opus 4.6 perform well in reconstructing upstream ligand-receptor interactions and highly conserved pathways but struggle with downstream transcription factors and gene expression.
• ▪Large language model-generated models of fibroblast activation and mechanosignaling reconstruct 24.07% to 60.49% and 29.07% to 64.53% of their manually curated counterparts, respectively.

Metabolic network generation performance

• ▪The construction of metabolic network models historically requires manual curation of stoichiometric coefficients, reaction reversibility, and metabolite-reaction connectivity.
• ▪Large language models demonstrate highly variable accuracies in predicting substrate utilization within metabolic networks.

Perturbation prediction validation

• ▪For 114 validations where a manually curated hypertrophy network has a functional accuracy of 94.74%, large language model-generated networks achieve an accuracy of only 6.14% to 33.33%.
• ▪Across hypertrophy, fibroblast, and mechanosignaling networks, the validation accuracy of large language model-generated models for perturbation responses is lower than their accuracy for reconstructing network structure.
• ▪Large language model-generated models of fibroblast activation and mechanosignaling exhibit validation accuracies of 16.87% to 21.69% and 5.81% to 14.53%, respectively, against independent literature.

Manual curation versus AI automation

• ▪Computational models of biochemical networks typically require time-intensive manual curation to extract network mechanisms from incomplete literature.
• ▪Removing the constraint of pruning large language model-predicted interactions to the ground truth networks does not substantially improve model performance, except for the fibroblast network.

5 sources