Artificial intelligence for histological subtype classification of breast cancer: combining multi-scale feature maps and the recurrent attention model

Aims The aim of this study was to apply a two-stage deep model combining multi-scale feature maps and the recurrent attention model (RAM) to assist with the pathological diagnosis of breast cancer histological subtypes by the use of whole slide images (WSIs). Methods and results In this article, we propose an integrated framework combining multi-scale feature maps from Inception V3 and the recurrent attention model to classify the six histological subtypes of breast cancer. This model was trained with 194 WSIs, and on 63 validation WSIs the model achieved accuracies of 0.9030 for patch-level classification and 0.8889 for WSI-level classification. In the testing stage, a total of 65 WSIs were used to achieve an accuracy of 0.8462 without any form of data curation. The t-distributed stochastic neighbour embedding showed that features extracted by the feature network of the RAM from WSIs of the same category can cluster together after training, and the visualization of decision steps showed that the decision-making glimpses are focused on the middle tumour area of an example from test WSIs. Finally, the false classification patches indicated that the morphological similarities between tumour tissues of different subtypes or non-tumour tissues and tumour tissues in patches might contribute to misclassification. Conclusions This model can imitate the diagnostic process of pathologists, pay attention to a series of local features on the pathology image, and summarize related information, in order to accurately classify breast cancer into its histological subtypes, which is important for treatment and prognosis.