Using SHAP to Measure Interpretability of Neuronal Feature Visualization.

Neuronal feature visualization is widely used in Explainable Artificial Intelligence (XAI). It can provide an intuitive visualization to depict the feature extraction of an individual neuron in a Convolutional Neural Network (CNN). However, it is extremely exhaustive for human users to identify highly interpretable visualizations by manually browsing massive neurons contained in a CNN. Inspired by the Shapley Value Method in coalitional game theory, this paper proposes a metric to quantitatively measure the interpretability of a neuronal feature visualization by calculating the similarity between the SHAP (SHapley Additive exPlanation) image and the visualization. This metric can help human users quickly find highly interpretable neuronal feature visualizations for understanding the classification results of a CNN.