Author: Douglas John Moseley, David M. Routman, Satomi Shiraishi, Donald C Smith, Mark R. Waddle 👨🔬
Affiliation: University of Denver, Mayo Clinic 🌍
Purpose: About 6% of patients treated for the first time in our department have received radiotherapy previously at an outside institution. We aim to provide an automatic quality assurance of identifying prior radiation to increase the safety of our treatment process.
Methods: We identified 286 patients treated in our clinic between 2019 and 2024 who had had radiotherapy (RT) before coming to our institution. We also randomly selected 266 patients who were treated for the first time during the same time frame but had not had prior radiotherapy. Patients who have received previous radiation treatment at our institution were excluded from this study as the records are easily accessible and well documented. We extracted clinical notes from our Electronic Health Record system and analyzed documents created before receiving RT in our clinic. We scan for 25 keywords that potentially indicate the mention of a patient receiving prior radiation. Sentences containing those keywords were extracted from the clinical notes and sent to Google Gemini Pro 1.5 within our institutional virtual private cloud. The prompt asked if the patient had prior radiation therapy and the justification for the decision. The result was compared against data from our prescription documents.
Results: The Gemini data extraction achieved a sensitivity of 100% and specificity of 94% with an overall accuracy of 97%. There were 16 false positives. The most common cause of false positives was the inability to distinguish if the RT happened in the past or will be happening in the near future. We also found eight true positive cases where the prescription document did not indicate prior radiation.
Conclusion: Identifying patients who had received prior RT but were not indicated in the prescription documents shows the need for a quality assurance. This study showed that a LLM solution is effective and feasible.