Author: Rose Al Helo, Shengwen Deng, Sven L. Gallo, David W. Jordan 👨🔬
Affiliation: Department of Radiology, Radiation Safety, University Hospitals Cleveland Medical Center, University Hospitals Cleveland Medical Center, Department of Radiology, Radiation Safety, University Hospitals Cleveland Medical Center; School of Medicine, Case Western Reserve University; Department of Radiology, Louis Stokes Cleveland VA Medical Center 🌍
Purpose: From an educator perspective, preparing test questions for trainees is time-consuming and requires a lot of quality verification steps (review of stems, distractors, referencing) that can potentially be automated. To improve the efficiency of writing test items, we aim to develop and validate a prompt that preset the role as a medical physics educator and board style question generator.
Methods: We integrated principles in AAPM Medical Physicists as Educators Wiki Page and thinking Claude to define the physicist educator role, and optimize chain-of-thought processes in knowledge identification and response preparation. For board style question generation, ABR and ACR-RadExam test item writing guide were used to optimize the choice of topic, design of stems/lead-in/distractors, question quality review process and sample answer referencing style. As validation, one TG report and one AAPM presentation from three modality (CT, MR, X-ray) were used to generate questions. LLM models (GPT O1 and Claude 3.5 Sonnet) were prompted to generate multiple choice and mock oral questions, with and without this educator preset. Relevance of topic, quality of question, and quality of distractor all generated questions (with and without the educator preset) were graded (1-5 scale) blinded and independently by three board-certified physicists.
Results: A detailed prompt preset to define the role of medical physics educator for writing test item was developed. This preset can be used prior to an input of technical documents (TG Reports, regulations, AAPM presentations) to preset necessary steps to generate a board style testing question or mock oral exam question. Quality of questions were improved when using the preset compared to simple prompts.
Conclusion: A medical physics educator preset for LLM models demonstrates improved efficiency and quality in board-style test item generation. The preset provides a structured framework that accelerates the development of consistent, high-quality assessment materials with latest technical documents.