Author: Sean P. Devan, Cory S. Knill, Charles K. Matrosic, Zheng Zhang 👨🔬
Affiliation: University of Michigan 🌍
Purpose: Physicists troubleshooting machine issues during patient treatments often face high-pressure situations, balancing error codes, resource constraints, and time-sensitive decisions. To streamline access to relevant information and support decision-making, a TrueBeam troubleshooting chatbot powered by OpenAI’s large language model (LLM) was designed and tested.
Methods: Design followed an iterative process, including data collection, formatting, model parameter selection, and testing. The chatbot’s system prompt was developed to ensure relevant responses. Physicists and therapists interacted with the chatbot to resolve TrueBeam machine issues, and their qualitative feedback informed subsequent refinements.
Results: A custom Python web scraper, using BeautifulSoup, Playright, Pandas, and Google Chrome Developer tools, was created to mine 1,394 previous machine issue logs. Visual Basic scripts were used to post-process data and remove non-UTF-8 characters. This chatbot, based on a Retrieval-Augmented Generation (RAG) approach, processes input characters as tokens, with a 1000-token limit per chunk and up to 40 chunks per response. Logs were separated into individual structured text files and used to train a custom LLM embedded in the GPT-4o model. Training took approximately 15 minutes. The model was set to a temperature of 0.4 to minimize hallucinations while maintaining flexibility. A custom system prompt guided the chatbot to cite machine logs and convey information in language familiar to medical physicists and radiation therapists. The model is currently used by a subset of clinical staff, generating historical summaries and troubleshooting suggestions within 15 seconds. The chatbot excels in producing comprehensive summaries from sparse user input.
Conclusion: The GPT-4o-powered TrueBeam troubleshooting chatbot enhances clinical efficiency by offering rapid, contextually relevant support. Initial deployment highlights its potential to streamline and standardize decision-making in high-stakes environments. Future work will focus on expanding its functionality and optimizing usability to further benefit clinical workflows.