CORF: Bridging the Gap of Complex Operator Fusion for Faster DNN Inference.

In order to satisfy the accuracy requirements, Deep Neural Networks (DNNs) have become increasingly deep, with hundreds of operator layers or even more. Operator fusion is one of the key optimization techniques used to reduce the memory and time consumption in many frameworks, such as Tensor-Flow and TVM. However, current strategies do not support fusion between complex operators (Convolution, GEMM, etc.). Considering that complex operators usually take up most of the inference time, the effect of fusion is severely limited. In addition, the lack of complex operator fusion decreases the decision space of how to fuse the entire network, which has also not yet been fully investigated. To address these challenges, we begin by analyzing consecutive complex operators, with a focus on the mapping relationships between producers and consumers. Subsequently, it is determined that the root obstacle to the fusion of complex operators in the framework is inconsistent multidimensional loops that arise due to the use of an output-centric calculation strategy. To solve the above problems, we present CORF, a fusion framework constructed on the basis of the input-centric concept. CORF supports complex operator fusion by transforming the calculation strategy of complex operators and adopts a variety of optimization approaches to improve the execution efficiency of the fused complex operators. Across the entire neural network, the fusion method employed by CORF expands the search space of fusion solutions. CORF also contains a searching algorithm that leverages fusion to accelerate the entire model. Evaluation in TVM shows that CORF can obtain a speedup of 47.96% when fusing two consecutive complex operators. Moreover, for the entire DNNs, we can obtain a 40.14% improvement by means of CORF.