Author: Rafe A. McBeth, Ayoola Okuribido, Rodney D. Wiersma 👨🔬
Affiliation: Department of Radiation Oncology, University of Pennsylvania, University of Pennsylvania, UCLA 🌍
Purpose: To develop a method for standardizing data collected during quality assurance checks across institutions using language models.
Background: QA procedures and data management can vary widely from institution to institution, which makes large scale data analysis difficult if not impossible. Such analysis could provide insight into potential points of failure that would be difficult to study with the limited cohort of a single institution. Standardization of data collection could open new avenues to improve patient safety.
Methods: A training set of 306 QA forms was constructed from a multi-institutional database of 3600 items. The data from each form is stored as a json text file in a mongodb database. The mongodb structure allows for each QA form to function as a bin that accepts various datatypes like DICOMs, input variables, the names of input variables, and machine tags, all as distinct text items. Each input variable can be assigned a parameter based on the appearance of certain tags. The training set is divided into different training folds, during which the model is exposed to a portion of the forms during training and another portion is held back for validation. The process is repeated by increasing the size of the validation set during each fold.All text data contained within each form will be processed as input parameters by the language model, and used to predict the input variable’s parameter. The input parameters will group integers, strings and other unrelated datatypes the same ontological root. Various language models, including reasoning models will be tested using an NVIDIA RTX 4090.
Results: An untrained sBERT's predictions were primarily in the true positive region of the ROC curve.
Conclusion: This work will serve as the foundation for developing a tool for classifying QA data amalgamated from various institutions.