The added value of an artificial intelligence system in assisting radiologists on indeterminate BI-RADS 0 mammograms

Objectives To investigate the value of an artificial intelligence (AI) system in assisting radiologists to improve the assessment accuracy of BI-RADS 0 cases in mammograms. Methods We included 34,654 consecutive digital mammography studies, collected between January 2011 and January 2019, among which, 1088 cases from 1010 unique patients with initial BI-RADS 0 assessment who were recalled during 2 years of follow-up were used in this study. Two mid-level radiologists retrospectively re-assessed these BI-RADS 0 cases with the assistance of an AI system developed by us previously. In addition, four entry-level radiologists were split into two groups to cross-read 80 cases with and without the AI. Diagnostic performance was evaluated using the follow-up diagnosis or biopsy results as the reference standard. Results Of the 1088 cases, 626 were actually normal (BI-RADS 1 and no recall required). Assisted by the AI system, 351 (56%) and 362 (58%) normal cases were correctly identified by the two mid-level radiologists hence can be avoided for unnecessary follow-ups. However, they would have missed 12 (10 invasive cancers and 2 ductal carcinoma in situ cancers) and 6 (invasive cancers) malignant lesions respectively as a result. These missed lesions were not highly malignant tumors. The inter-rater reliability of entry-level radiologists increased from 0.20 to 0.30 ( p < 0.005) by introducing the AI. Conclusion The AI system can effectively assist mid-level radiologists in reducing unnecessary follow-ups of mammographically indeterminate breast lesions and reducing the benign biopsy rate without missing highly malignant tumors. Key Points • The artificial intelligence system could assist mid-level radiologists in effectively reducing unnecessary BI-RADS 0 mammogram recalls and the benign biopsy rate without missing highly malignant tumors. • The artificial intelligence system was capable of detecting low suspicion lesions from heterogeneously and extremely dense breasts that radiologists tended to miss. • The use of an artificial intelligence system may improve the inter-rater reliability and sensitivity, and reduce the reading time of entry-level radiologists in assessing potential lesions in BI-RADS 0 mammograms.