Author: Ali Ammar, Quan Chen, Jingwei Duan, Yi Rong, Nathan Y. Yu, Libing Zhu π¨βπ¬
Affiliation: Mayo Clinic Arizona, University of Alabama at Birmingham π
Purpose: Clinical performance of deep learning-based auto-segmentation (DLAS) can degrade over time due to AI βagingβ from unseen data input compared to the initial model training data. This study aims to establish a performance monitoring system for DLAS quality assurance in clinical application of male pelvis cases using statistical process control (SPC).
Methods: A total of 881 prostate cancer cases were analyzed for the contour deviation between the clinical implemented DLAS model and clinical approved contours, using dice similarity coefficient (DSC), 95 percentile Hausdorff distance (HD95) and surface dice similarity coefficient (SDSC) with 2 mm tolerance. SPC control limits were calculated using the 2Ο criterion for the prostate, bladder, rectum, left femoral head (fem_head_l), and right femoral head (fem_head_r). Random sample sizes of 10, 20, 30, 40, and 50 cases were tested 100 times each to assess suitability for setting control limits to monitor DLAS performance.
Results: The lower control limits of DSC were 0.7562, 0.9193, 0.8604, 0.8742, and 0.8624 for prostate, bladder, rectum, fem_head_l and fem_head_r, respectively while they were 0.3515, 0.7867, 0.6710, 0.7624, and 0.7598 for the five evaluated organs using SDSC 2mm. The upper control limits for HD95 were 1.1025 cm, 0.9888 cm, 0.8767 cm, 1.5566 cm, and 1.3511 cm for the five organs. As sample size increased, the mean accuracy of detecting outliers improved, and variability decreased across the five organs.
Conclusion: SPC-based monitoring using multiple metrics can effectively identify out-of-distribution data, providing a reliable and easy framework for quality assurance of DLAS. Further work is needed to determine the most suitable sample size for routine clinical use.