A Multi-Class Classification Algorithm Based on Hematoxylin-Eosin Staining for Neoadjuvant Therapy in Rectal Cancer: a Retrospective Study

Neoadjuvant therapy (NAT) is a major treatment option for locally advanced rectal cancer. With recent advancement of machine/deep learning algorithms, predicting the treatment response of NAT has become possible using radiological and/or pathological images. However, programs reported thus far are limited to binary classifications, and they can only distinguish the pathological complete response (pCR). In the clinical setting, the pathological NAT responses are classified as four classes: (TRG0-3), with 0 as pCR, 1 as moderate response, 2 as minimal response and 3 as poor response. Therefore, the actual clinical need for risk stratification remains unmet. By using ResNet (Residual Neural Network), we developed a multi-class classifier based on Hematoxylin-Eosin (HE) images to divide the response to three groups (TRG0, TRG1/2, and TRG3). Overall, the model achieved the AUC 0.97 at 40x magnification and AUC 0.89 at 10x magnification. For TRG0, the model under 40x magnification achieved a precision of 0.67, a sensitivity of 0.67, anda specificity of 0.95. For TRG1/2, a precision of 0.92, a sensitivity of 0.86, anda specificity of 0.89 were achieved. For TRG3, the model obtained a precision of 0.71, a sensitivity of 0.83, anda specificity of 0.88. To find the relationship between the treatment response and pathological images, we constructed a visual heat map of tiles using Class Activation Mapping (CAM). Notably, we found that tumor nuclei and tumor-infiltrating lymphocytes appeared to be potential features of the algorithm. Taken together, this multi-class classifier represents the first of its kind to predict different NAT responses in rectal cancer.