First Peer-Reviewed Evaluation Shows Specialized AI Outperforms General LLMs in Radiology Impressions

Rad AI has published new peer-reviewed research in npj Digital Medicine showing that domain-specific AI models outperform general-purpose large language models (LLMs) in generating radiology report impressions—the section most critical for guiding patient care.

The study, conducted with Moffitt Cancer Center, evaluated 200 oncologic CT reports, comparing radiologist-written impressions with outputs from both a radiology-specific AI model and a general-purpose LLM. The domain-specific model, trained on institutional data from radiologists and oncologists, closely matched human performance in completeness, correctness, and conciseness. In contrast, general-purpose LLMs were consistently ranked lower by radiologists, with usability and conciseness differences ranging from about 28% to nearly 50%.

The specialized AI also produced high-quality impressions significantly faster while maintaining the concise, high-signal format clinicians prefer. Patient harm risk remained low across all outputs, scoring approximately 1.0 to 1.2 on a 3-point scale, where 1 indicates minimal risk.

“Impressions are the most critical part of the radiology report,” said Andrew Del Gaizo, MD, Chief Medical Information Officer at Rad AI and study co-author. “This study demonstrated that, in addition to accuracy, customization matters to radiologists… These findings highlight the importance of AI that's purpose-built for radiology and designed to support how clinicians actually make decisions in practice.”

The research also revealed differing preferences between specialties. Radiologists favored concise summaries, while oncologists were more receptive to detailed explanations, underscoring the need for adaptable AI tools.

“We saw meaningful variability in how radiologists and oncologists evaluated the same outputs,” said Trevor Rose, MD, MPH, a diagnostic radiologist at Moffitt Cancer Center. “Rather than optimizing for a single ‘best’ output, organizations should be prioritizing tools that can adapt to different users, workflows and clinical preferences.”

The findings highlight a broader shift in healthcare AI evaluation—from technical performance alone to real-world clinical usability. As imaging volumes grow, purpose-built AI aligned with clinical workflows may help radiologists improve efficiency while maintaining care quality.