Author: Arjit K. Baghwala, Sunan Cui, Jessica Fagerstrom, Eric C. Ford, Kristi Rae Gayle Hendrickson, Sharareh Koufigar, Samuel Ming Ho Luk, Bishwambhar Sengupta, Afua A. Yorke 👨🔬
Affiliation: University of Washington, Department of Radiation Oncology, Fred Hutchinson Cancer Center, University of Washington, Department of Radiation Oncology, University of Vermont Medical Center, University of Washington and Fred Hutchinson Cancer Center, Houston Methodist Hospital 🌍
Purpose: The global burden of cancer continues to rise, leading to an increased workload in radiation oncology clinics. This surge is not only due to the growing demand for treatment machines and modalities but also the increasing complexity of techniques and treatment approaches. At the same time, the field has experienced a shortage of medical physicists, limiting the availability of mentors for dedicated didactic teaching and resident training. This study evaluates whether AI-based large language models (LLMs) can assist medical physics residents in preparing for the ABR board certification exams and support their development as qualified medical physicists.
Methods: We selected three publicly available AI-based LLMs and tasked each model with (i) generating a summary of a specific AAPM Task Group (TG) report and (ii) creating ABR Part 3-style exam questions. The responses were anonymized and reviewed by 10 board-certified medical physicists (with 1 to 10+ years of post-certification experience) for factual accuracy and completeness. Additionally, the generated questions were assessed for their complexity and similarity to ABR Part 3 exam questions.
Results: Preliminary findings indicate that two out of three TG report summaries generated by the LLMs were deemed factually inaccurate or incomplete. However, all ABR Part 3-style questions were considered comparable in complexity and format to actual exam questions. Among the AI-generated answers to these questions, 80% were graded as correct, albeit with varying degrees of completeness, while one response was marked as factually incorrect.
Conclusion: These findings suggest that AI-based LLMs have the potential to supplement didactic training and assist residents in preparing for the ABR Part 3 exam. However, while they may serve as a useful revision tool, they should not be solely relied upon for comprehensive exam preparation.