Understanding Convolutional Networks Using Linear Interpreters Extended Abstract

We are interested in the general problem of understanding the decisions made by convolutional neural networks. We focus on three popular and successful architectures. First, the VGG architecture used for image classification, that takes images into labels. Second, the EDSR architecture that upscales small resolution images into high resolution images with 16 times more pixels. And finally, the CycleGAN architecture that translates images of same size from one domain to another, for example: photos to paintings of different styles, or facades to segmentation maps. In these three cases image pixels are either reduced to scalars (classification), or increased to form more complex images (super-resolution), or kept in the same vector space but in different style domains (image-to-image translation).These three architectures are largely made of linear systems. In fact, all of their learnable parameters are contained in convolutional layers, which are linear. Furthermore, we note that the action of all the non-linear layers involved are motivated by linear operations. For example, an ReLU is easily interpreted as a 1/0-mask that multiplies the input (linear) with a mask determined by the sign of each feature pixel (non-linear). A max-pooling layer selects one among a group of pixels and allows a similar interpretation by using selection masks. An instance-normalization layer subtracts a mean and divides by a standard deviation. If we fix the mean and standard deviation, it becomes linear.Now we consider a fixed input image x(0), determine the 1/0-mask for each ReLU and max-pooling, mean and variances of instance-normalizations, and fix them. The network then becomes a large linear system, that we call the Linear Interpreter. Our purpose is to study these systems arising from popular deep learning architectures. We can use well known methods and theory of linear systems to get important insights on how these networks work.