Virtual View Assisted Video Super-Resolution and Enhancement

3D multiview video provides users an experience different from traditional video, however, it puts huge burden on the limited bandwidth resources. Mixed-resolution video in multiview system can alleviate this problem by using different video resolutions for different views. However, in order to reduce visual uncomfortableness and to make this video format more suitable for free-viewpoint television, the low resolution views need to be super-resolved to the target full resolution. In this paper, we propose a virtual view assisted super resolution algorithm, where the inter-view similarity is used to determine whether to fill the missing pixels in the super-resolved frame by virtual view pixels or by spatial interpolated pixels. The decision mechanism is steered by the texture characteristics of the neighbors of each missing pixel. Furthermore, the inter-view similarity is used, on one hand, to enhance the quality of the virtual-view-copied pixels by compensating the luminance difference between different views, and on the other hand, to enhance the original low resolution pixels in the super-resolved frame by reducing their compression distortion. Thus, the proposed method can recover the details in regions with edges while maintaining good quality at smooth areas by properly exploiting the high quality virtual view pixels and the directional correlation of pixels. The experimental results demonstrate the effectiveness of the proposed approach with a PSNR gain of up to 3.85dB.