Author: Hilary P Bagshaw, Mark K Buyyounouski, Xianjin Dai, PhD, Praveenbalaji Rajendran, Lei Xing, Yong Yang 👨🔬
Affiliation: Department of Radiation Oncology, Stanford University, Massachusetts General Hospital, Harvard Medical School 🌍
Purpose: Artificial intelligence (AI)-driven methods have transformed dose prediction, streamlining the automation of radiotherapy treatment planning. However, traditional approaches depend exclusively on anatomical imaging, resulting in dose predictions that are physically optimal but not necessarily clinically optimal, often requiring manual refinements by planners and oncologists. This study aims to enable one-step clinically optimal dose prediction by integrating physical and clinical data through a large language model (LLM)-augmented learning framework.
Methods: We developed an innovative LLM-augmented vision-language model for predicting 3D dose distributions in radiotherapy. The model leverages the advanced Swin Transformer as its core architecture and incorporates an LLM to extract and integrate text-rich clinical information. Linguistic features are effectively combined with visual data through a visual-language attention module, enabling language-aware visual encoding. Retrospective radiotherapy datasets from 200 patients were collected and split into 160 cases for training, 20 for validation, and 20 for testing. The model’s performance was assessed using dose-volume histogram (DVH) analysis, and gamma evaluation.
Results: Our model demonstrated superior performance compared to conventional visual-only models, accurately predicting 3D dose distributions with greater precision. The DVH and gamma analyses reveal that the predictions generated by our model are closely aligned with those of clinically utilized plans, highlighting its potential for practical application in radiotherapy treatment planning. This alignment further substantiates our hypothesis that clinical treatment planning is inherently a complex, multi-faceted decision-making process, requiring the integration of both physical and clinical considerations for optimal outcomes.
Conclusion: By integrating imaging and clinical data through an LLM-augmented learning approach, it becomes possible to predict clinically optimal dose distributions rather than merely physically optimal ones. This method offers a pathway to streamline and automate radiotherapy treatment planning.