Block-based motion estimation for deep-learned video coding

The research on deep-learned end-to-end video compression has attracted a lot of attention over the course of recent years. A central component of many approaches is to perform motion-compensated prediction by using convolutional neural networks (CNN) which determine a compressed representation of the motion field as features. Often, this task is divided into searching motion vectors by one network and efficiently representing them by another one. However, these networks may find motion fields far from optimal because the search radius of CNNs is mainly determined by their depth and kernel size. In this paper, we apply motion estimation techniques from classical block-based hybrid video compression to search a motion field which is then fed into a variational autoencoder. These strategies include different distortion measures, different block partitions and an improved approximation of the residual bitrate. With our modifications, bitrate savings of up to 13% over the underlying end-to-end based video codec can be obtained.