Author: Steve Braunstein, Yannet Interian, Hui Lin, Bo Liu, Janine Lupo, Olivier Morin, Benedict Neo đ¨âđŹ
Affiliation: Radiation Oncology, University of California San Francisco, Graduate Program in Bioengineering, University of California San Francisco-UC Berkeley, Department of Radiation Oncology, University of California San Francisco, Department of Data Science, University of San Francisco, University of San Francisco đ
Purpose: Large Language Models (LLMs) demonstrate strong general text comprehension but remain limited in oncology due to insufficient contextual alignment. We pilot embedding alignment through radiology-pathology report matching to enhance oncology-specific representations. Using Multiple Negative Ranking Loss (MNRL) and Low-Rank Adaptation (LoRA)-adapted fine-tuning of jina-embeddings-v3, we aim to create a context-aware embedding model and evaluate its efficacy and generalizability across glioma prognostication tasks, including tumor type classification, MGMT methylation prediction, and overall survival prediction.
Methods: We fine-tuned jina-embeddings-v3 on 97,398 paired radiology-pathology reports spanning nine brain tumor types. To optimize text alignment, we applied MNRL with in-batch negative sampling, ensuring the model captured fine-grained distinctions between tumor types. We evaluated model performance using stratified 5-fold cross-validation across three prognostic tasks: tumor type classification (n=4,470), MGMT methylation prediction (n=192), and overall survival prediction (n=201). We assessed accuracy, precision, recall, F1, and AUROC.
Results: Our aligned model, JINAv3F, consistently outperformed baseline models across all prognostic tasks. In tumor subtype classification, JINAv3F achieved a 2.9% increase in accuracy, 3.4% increase in precision, 5.8% increase in recall, and 5.6% increase in F1-macro, with significant effect sizes, demonstrating enhanced tumor differentiation. For MGMT methylation prediction, our model improved recall by 2.9% and F1-macro by 4.3%, capturing molecular features relevant to treatment response. In overall survival prediction, JINAv3F achieved 6% precision increase and 4.4% F1-macro increase, indicating better discrimination of high-risk patients. Furthermore, t-SNE visualizations confirmed tighter clustering of tumor subtypes compared to other medical-specific embedding models, reinforcing JINAv3Fâs improved ability to capture meaningful clinical distinctions.
Conclusion: Our clinically aligned model outperforms baselines in key glioma prognostication tasks, demonstrating the benefits of radiology-pathology alignment for oncology text embeddings. The results validate the effectiveness of our MNRL approach and LoRA adaptation strategy in enhancing oncology-specific representations, paving the way for broader adoption in neuro-oncologic applications.