Author: Wookjin Choi, James M. Lamb, David Romanofski, David H. Thomas, Yevgeniy Vinogradskiy 👨🔬
Affiliation: Drexel, Department of Radiation Oncology, University of California, Los Angeles, Thomas Jefferson University 🌍
Purpose: To develop an intelligent Black Box Recorder for radiation therapy (RT) that monitors patient treatments using a vision language model.
Methods: The system captures synchronized screen recordings from the therapist’s record-and-verify (R&V) system and the treatment console. The standard clinical workflow is not affected. These screen recordings are synchronized and processed using an image segmentation model (FastSAM - Segment Anything). The model parameters are tuned to generate segmentation results at > 1 frame/second. The labeled images are then passed to a state-of-the-art visual language model (VLM) for image analysis. The models can be run locally on a single 12GB GPU without sharing patient data outside the system. The VLM extracts and contextualizes key events from the treatment sessions. The extracted data is compared to the R&V and logged for post-treatment review, incident learning, and workflow optimization.
Results: Initial implementation and feasibility studies demonstrate the system's ability to contextualize instructions from the R&V accurately. We assessed system feasibility to identify key treatment steps and flag deviations, including 1) identifying the correct patient, 2) identifying the prescription and anatomical site, 3) identifying set-up instructions, 4) extracting 2D axial planes, and 5) logging treatment shifts. All five feasibility items were accurately recovered. Instructions in the prescription with nuanced details such as "Port film daily and CBCT on Mondays" or “Dx bolus after 13 fx” were also accurately interpreted by the model.
Conclusion: This work introduces an intelligent monitoring system for radiation therapy. The system can interpret instructions in the record and verify systems that could be missed with current strategies. This will enable retrospective incident analysis and pave the way for real-time error prevention and decision support. The use of AI to reduce errors at the treatment console is an underexplored aspect of automation in radiation therapy.