A Single Frame and Multi-Frame Joint Network for 360-Degree Panorama Video Super-Resolution

Spherical videos, also known as 360-degree (panorama) videos, can be viewed with various virtual reality devices such as computers and head-mounted displays. They attract a large amount of interest since awesome immersion can be experienced when watching spherical videos. However, capturing, storing and transmitting high-resolution spherical videos are extremely expensive, and the low-resolution ones are widely available. In this paper, we propose a novel single-frame and multi-frame joint network (SMFN) for recovering high-resolution spherical videos from low-resolution observations. To take advantage of pixel-level inter-frame consistency, we use deformable convolutions to eliminate the motion difference between feature maps of the target frame and its neighboring frames. A mixed attention mechanism is devised to enhance the feature representation capability. The dual learning strategy is presented to constrain the space of solutions so that a better solution can be found. A new loss function based on the weighted mean squared error is proposed to emphasize the super-resolution of the equatorial regions. This is the first attempt to settle the super-resolution of spherical videos, and we collect a new dataset from the Internet, MiG panorama video, which includes 208 videos. Experimental results on representative video clips demonstrate the efficacy of the proposed method. The dataset and our source code are available at https://github.com/lovepiano/SMFN_For_360VSR.