Author: Klea Hoxha, Dylan P. O'Connell, Ricky R Savjani 👨🔬
Affiliation: Department of Radiation Oncology, University of California, Los Angeles, UCLA Radiation Oncology 🌍
Purpose: Ambiguities in physician simulation orders lead to workflow disruptions during CT simulation. Often, information that could provide
helpful context to simulation therapists and planners is already documented in the patient’s chart but variance in note structure and style between
providers makes isolating information difficult. This study assesses the effectiveness of Meta’s open-source Llama models in isolating the anatomical
treatment site(s) from pre-existing physician notes and classifying them into standardized sites.
Methods: Semi-structured interviews with radiation therapists at our institution revealed that CT simulation orders frequently lack necessary details
to perform optimal simulations. The most common concerns were incorrect/ambiguous treatment sites and setup instructions. One hundred random
patients receiving radiation treatments were selected. Llama 3.0 and Llama 3.3 models were prompted to select the most recent physician note
relevant to the CT simulation from a collection of departmental physician’s notes, identify the treatment site(s) for each patient, and classify it using a
standardized list. Results were validated by cross-referencing with dose distributions following treatment delivery.
Results: The Llama 3.3 model outperformed Llama 3.0 for note selection and treatment site identification for each patient. The Llama 3.0 model was
unable to process more than one note per patient at a time due to its smaller context window. The Llama 3.3 model identified the correct physician’s
note for each patient with an accuracy of 98.7% and isolated the correct treatment site(s) with an accuracy of 94.6%. The model classified the site
appropriately 95% of the time.
Conclusion: Identifying the correct physician’s note and isolating the accurate treatment site will eliminate some ambiguities associated with CT
simulation orders, leading to increased workflow efficiency and reduced instances of repeat CT simulations. The developed technique can be applied
to large patient datasets with a standardized list of treatment sites or other classifiers for retrospective studies.