VVC bitstream extraction and merging operations for multi-stream media applications

In traditional video decoding applications, the number of elementary streams that a hardware decoding platform of an end device can decode is determined at runtime by the. Upon request by the application, the decoding platform verifies whether a new decoding instance with an associated requirement in terms of data rate can fit under the current workload. Conversely, if a device can decode one 4K elementary stream in hardware, it may not be able to simultaneously decode four HD elementary streams that would each correspond to requirements in terms of data rate of 1/4 of the 4K elementary stream. Current video decoding platforms are thus designed with the assumption that each elementary stream requires the instantiation of a dedicated video decoder instance. At the same time, it has been increasingly common in new media applications such as immersive media applications to simultaneously consume several elementary streams in a synchronised fashion. The demo presents a new paradigm for media applications for which elementary streams may be consumed in such synchronised manner where the same decoder instance can be used. The demonstrator leverages on new features of the Versatile Video Coding (VVC) standard and interfaces being defined in the ongoing standardisation of MPEG-I part 13: Video Decoding Interface for Immersive Media. Stitching and cropping videos in the compressed domain can be achieved by an application via such defined interfaces. Without those interfaces, the same tasks are possible with the High Efficiency Video Coding (HEVC) standard to some extent but are tedious. In this demonstrator, we thus show how the new VVC codec can enable the decoupling of the number of elementary streams consumed by the application and the number of running video decoder instances. In addition, memory usage and CPU performance are also collected and compared with a tradition multiple decoding instance approach.