ChatGPT-5 missed approximately half of distal radius fractures on plain x-rays when provided a single binary fracture-or-not prompt, according to a study published October 26 in Emergency Radiology. 

The study was designed to reflect the kind of simple, low-context question that busy non-radiologist clinicians might realistically pose at the point of care, with findings suggesting that advanced large language models (LLMs) should not be used as standalone diagnostic tools for fracture detection, noted a group from the Rambam Health Care Campus in Haifa, Israel. 

"At best, in their current form, such models might serve as adjunctive aids in controlled research settings or as educational tools, but they cannot be relied upon for safe clinical triage or decision-making," the researchers noted. 

General-purpose multimodal LLMs are being applied to radiology with mixed results, with most existing evaluations rely on structured, stepwise prompts or supplementary clinical context, the authors noted. Yet there is also a need to systematically assess the effectiveness of newer, more advanced LLM-based models in a minimal, binary-decision use case that closely mirrors real-world triage questions, they added. 

To that end, the researchers submitted 370 anteroposterior distal radius x-rays from the Stanford MURA dataset to ChatGPT-5 via its public web interface in a zero-shot approach. They asked, “Is there a fracture in this X-ray? Answer with yes or no.” Performance was measured against image-level labels assigned by consensus between a senior orthopedic resident and a board-certified orthopedic surgeon.

Representative ChatGPT-5 outputs against the reference standard. A True positive: distal radius fracture with visible cortical disruption; the model answered “yes”. B True negative: normal wrist; the model answered “no”. C False positive: normal wrist; the model answered “yes”. D False negative: distal radius fracture; the model answered “no”. All four are anteroposterior projections from the analyzed sample, and each was submitted in a separate temporary chat session using the identical promptEmergency RadiologyAccording to the results, fracture prevalence in the analyzed sample was 45.9% (170/370).  ChatGPT-5 missed 83 fractures, demonstrating a sensitivity of 51.2% and a specificity of 70.5%, with overall accuracy of 61.6%. False negatives significantly exceeded false positives (McNemar χ²= 4.06, p = 0.044), meaning the model was more likely to miss a fracture than to flag a normal x-ray, the authors noted. 

"Put plainly: a general-purpose chatbot should not be used to decide whether a radiograph shows a fracture," the researchers wrote. 

However, the group added that the result is specific to the study's model version, the prompt, the anatomical region, and the dataset and that it constrains claims about autonomous, minimal-prompt use of consumer multimodal LLMs for fracture detection. The study does not adjudicate the broader question of whether carefully engineered LLM-based workflows could contribute to musculoskeletal imaging, they wrote.  

"Progress in that direction will require task-specific adaptation, rigorous prompt and pipeline design, direct same-images comparison against dedicated [convolutional neural network] baselines, and evaluation across multiple anatomical regions with radiologist-adjudicated reference standards," the researchers concluded. 

The full study is available here.