Automatic MR prostate segmentation by deep learning with holistically-nested networks.

Accurate automatic prostate magnetic resonance image (MRI) segmentation is a challenging task due to the high variability of prostate anatomic structure. Artifacts such as noise and similar signal intensity tissues around the prostate boundary inhibit traditional segmentation methods from achieving high accuracy. The proposed method performs end-to-end segmentation by integrating holistically nested edge detection with fully convolutional neural networks. Holistically-nested networks (HNN) automatically learn the hierarchical representation that can improve prostate boundary detection. Quantitative evaluation is performed on the MRI scans of 247 patients in 5-fold cross-validation. We achieve a mean Dice Similarity Coefficient of 88.70% and a mean Jaccard Similarity Coefficient of 80.29% without trimming any erroneous contours at apex and base.