Author: Ramesh Boggula, Nikhil Jordan Shad 👨🔬
Affiliation: Wayne State University 🌍
Purpose: To evaluate the effectiveness of OpenAI in reviewing large volumes of radiation delivery reports from Mobius3D/FX. The goal was to assess whether automating this process could identify potential warnings or errors in the reports, reducing the time and effort required by medical physicists.
Methods: Mobius3D/FX (Varian Medical Systems) utilizes linac treatment logs to calculate and verify the 3D dose delivered to patients. A total of 296 treatment plans were chosen for this study. The comprehensive report that generated from Mobius3D/FX includes isodose, gamma analysis, dose-volume histogram, and root mean square analysis for each fraction of the patient's treatment. Each report is approximately 8 pages. The anonymized reports were processed by OpenAI to detect any warnings or errors. The first test included 266 reports with no errors or warnings, OpenAI would analyze these reports to see how many false positives it would detect. The second test included 30 reports with errors and warnings, to check how often OpenAI gives a false negative. The effectiveness of OpenAI was measured by its ability to detect these issues compared to manual review.
Results: From the 296 reports reviewed, OpenAI had an 87% accuracy for reports with no errors, and 93% accuracy for reports with errors. The AI system demonstrated time saving potential for medical physicists, assisting the review process by flagging issues. The accuracy can increase with further developments in AI models, or with other techniques involving fine tuning.
Conclusion: The integration of OpenAI in the review of radiation treatment reports from Mobius3D/FX has proven to be a promising tool for automating the error detection process. This study highlights the potential for expanding this application to other types of radiation treatment reports using AI agents, and suggests that enhancements, such as fine-tuning the AI, could improve its accuracy.