Author: Ara Alexandrian, Jessica Ashford, Jean-Guy Belliveau, Allison Dalton, Nathan Dobranski, Krystal M. Kirby, Garrett M. Pitcher, David E. Solis, Hamlet Spears, Angela M. Stam, Sotirios Stathakis, Jason Stevens, Rodney J. Sullivan, Sean Xavier Sullivan, Natalie N. Viscariello 👨🔬
Affiliation: Louisiana State University, Mary Bird Perkins Cancer Center, The University of Alabama at Birmingham, University of Alabama at Birmingham 🌍
Purpose: To improve retrospective risk analysis in radiation oncology by leveraging Large Language Models (LLMs) to extract richly annotated data from unstructured clinical incident reports.
Methods: A process was created that automatically refines incident reports, condenses clinical information into concise summaries, and assigns informative tags. The project utilizes locally hosted LLMs on a custom-built machine with sufficient computational power to perform the necessary tensor operations in a time efficient manner. The approach incorporates multiple finely tuned models working in succession to break down complicated analysis into simpler steps. In addition, traditional Natural Language Processing (NLP) methods such as Retrieval Augmented Generation (RAG) are incorporated to provide additional context relevant to the medical physics field. Lastly, safeguards are implemented to verify no pertinent information was lost and to ensure no false data was generated. These methods help to provide accurate incident report summaries and tag assignments while maintaining the integrity of the original reports.
Results: The initial iteration of this process was tested on ROILS submissions from Mary Bird Perkins Cancer Center (1,237 submissions) and University of Alabama Birmingham (772 submissions), with a random subset (N=100) evaluated by a total of eight medical physicists across the two institutions. Preliminary results indicate a strong performance with notable room for improvement where 76.5% of summaries and 81.0% of tags were rated Okay, Good, or Exceptional. For both institutions, the evaluation ratings of the generated tag assignments ranked slightly higher than the evaluation ratings of generated summaries.
Conclusion: Our project has the potential to significantly improve retrospective risk analysis in radiation oncology by leveraging LLMs and machine learning capabilities. By refining incident reports, condensing clinical information, and assigning informative tags, we can provide physicists with accurate and actionable insights that inform evidence-based practices and improve patient outcomes.