Author: Kyle J. Lafata, Casey Y. Lee, Xiang Li, Megan K. Russ, Zion Sheng 👨🔬
Affiliation: Duke University, Department of Radiation Oncology, Duke University, Clinical Imaging Physics Group, Department of Radiology, Duke University Health System 🌍
Purpose:
Traditional deep learning-based cell segmentation models face limitations, such as the need for extensive training data and retraining when encountering new cell types or domains. This study proposes a novel two-step approach combining pretrained cell segmentation models with GPT-4's visual reasoning capabilities to improve lymphocyte identification in histopathological images in the case of domain shift challenges.
Methods:
The pipeline consists of an initial cell segmentation step using pretrained HoverNet for lymphocyte detection, followed by a GPT-4-based refinement step to filter false positives. The HoverNet was pretrained on internal kidney tissue samples. GPT-4 employs a structured chain-of-thought reasoning approach that analyzes tissue environment, region of interest (ROI), and cell-level characteristics. The system was evaluated using 13 tiles selected from TCGA breast, kidney, and prostate cancer tissue image database that contain domain-shift features, containing 945 cells with 162 lymphocytes verified by ground truth annotation. Text embeddings of GPT's morphological descriptions were analyzed using t-SNE visualization to assess reasoning consistency.
Results:
The initial HoverNet detection identified 367 cells as lymphocytes, of which 151 were true positives. After GPT-4 refinement, the system's performance metrics showed improvement for this group of 367 candidate lymphocytes. The precision increased from 0.17 to 0.62, recall from 0.41 to 0.59, and F1 score from 0.24 to 0.60. T-SNE visualization of text embeddings for cell morphology descriptions demonstrated that GPT-filtered lymphocytes clustered similarly, indicating consistent reasoning patterns.
Conclusion:
The integration of GPT-4 visual reasoning as a refinement step showed the potential to improve the accuracy of lymphocyte identification in histopathological images. The structured reasoning approach provided interpretable results for further verification.