How Important Is a Neuron?

The problem of attributing a deep networku0027s prediction to its emph{input/base} features is well-studied. We introduce the notion of emph{conductance} to extend the notion of attribution to the understanding the importance of emph{hidden} units. Informally, the conductance of a hidden unit of a deep network is the emph{flow} of attribution via this hidden unit. We use conductance to understand the importance of a hidden unit to the prediction for a specific input, or over a set of inputs. We evaluate the effectiveness of conductance in multiple ways, including theoretical properties, ablation studies, and a feature selection task. The empirical evaluations are done using the Inception network over ImageNet data, and a sentiment analysis network over reviews. In both cases, we demonstrate the effectiveness of conductance in identifying interesting insights about the internal workings of these networks.