VITON-Based CT Image Registration for Medical Surgery Application

Augmented Reality (AR) technology enables augmentation and information display through camera mounted on a wearable device. In medical field, it tackles asynchronization between hands and eyes and concentration decrease of the surgeon during surgery by providing medical image information by the wearable device. Inspired by virtual try-on algorithms, we propose a method based on VITON to register and display a Computed Tomography (CT) image with tumor information on the patient’s chest without markers using a camera mounted on an AR device. The proposed method consists of two stages: (1)Image Pre-process stage to prepare CT image and the corresponding CT image mask and human presentations given reconstructed CT image and reference model image; (2)Image Registration stage to register warped CT image on reference model image. Extensive experiments demonstrate that the proposed method achieves high-fidelity registration results both qualitatively and quantitatively. In particular, the generation of CT registered image of size 1024×768 enhances the ability to seek the patient’s tumor in organ area with higher accuracy during surgery, strengthening the contribution to medical field.