Can LLMs help track interval changes on radiology reports?

Large-language models (LLMs) show potential for tracking interval changes on longitudinal radiology reports, according to research published April 11 in the Journal of Imaging Informatics in Medicine.

This approach would yield time savings by adding automation to a process that currently requires radiologists to manually match relevant findings.

“... the results obtained in our work represent a significant step forward towards the automated summarization of actionable findings from prior radiology reports for a structured presentation to clinicians, radiologists, and patients at the time of the next follow-up exam,” wrote the research team led by co-first authors Tejas Sudharshan Mathai, PhD, and Boah Kim, PhD, of the U.S. National Institutes of Health Clinical Center.

In the current study, the researchers retrospectively assessed the performance of privacy-preserving LLMs for matching the findings between a prior and follow-up report. With their approach, the LLM first took a sentence from the follow-up report and then discovered a matched finding in the prior report. Next, it predicted the interval change status of the matched findings.

After first determining the utility of this approach on 240 body MRI reports from the NIH, the group then validated the best-performing LLM on an external noncontrast chest CT database of 134 studies.

On the internal dataset, the LLM achieved an F1 score of 85.4% for matching findings and a 62.7% F1 score for change detection, as well as moderate agreement (κ = 0.46) with the reference standard. It also performed well on the external dataset, yielding F1 scores of 81.8% for matching findings and 77.4% for interval change detection, with a substantial agreement (κ = 0.64), according to the researchers.

In the future, the researchers plan to explore the use of vision-language models that can directly incorporate both imaging and reporting data for the task of matching findings and track interval changes.

“The results indicate that LLMs with a low computational and environmental footprint may be utilized for matching findings and tracking interval changes in radiology reports, while maintaining patient privacy by being locally run behind an institution’s network firewall,” the authors concluded.

The full study is available here.