A quantitative insight into the role of skip connections in deep neural networks of low complexity: A case study directed at fluid flow modeling

Abstract As a deep feed-forward network with high complexity backpropagates the gradient of the loss function from final layers to earlier layers, the gradient might descend rapidly towards zero. This is known as the vanishing gradient phenomenon that stops the earlier layers from benefiting from further training. One of the most efficient techniques to solve this problem is using skip connection (shortcut) schemes. This paper presents an investigation of whether skip connections significantly affect the performance of deep neural networks of moderate complexity, or whether their inclusion has little or no effect. The analysis was conducted using Convolutional Neural Network (CNN) to predict four different multiscale basis functions for the mixed Generalized Multiscale Finite Element Method (GMsFEM). These models were applied to 249,375 samples generated in MatLab software, with the permeability field as the only input. Three skip connection schemes were added to the base structure: (Scheme 1) from the first convolutional block to the last, (Scheme 2) from the middle to the last block, and (Scheme 3) from the middle to the last and the second-to-last blocks. The results demonstrate that the third scheme produces the most effective effect, increasing the coefficient of determination (R2) value by 0.0224 to 0.044, and decreasing the Mean Squared Error (MSE) value by 0.0027 to 0.0058. Hence, enriching the last convolutional blocks by the information hidden in neighboring block is more effective than by earlier convolutional block near the input layer.