Nowadays, the Convolutional Neural Networks (CNNs) have achieved impressive performance on many computer vision related tasks, such as object detection, image recognition, image retrieval, etc. These achievements benefit from the CNNs outstanding capability to learn the input features with deep layers of neuron structures and iterative training process. However, these learned features are hard to identify and interpret from a human vision perspective, causing a lack of understanding of the CNNs internal working mechanism. To improve the CNN interpretability, the CNN visualization is well utilized as a qualitative analysis method, which translates the internal features into visually perceptible patterns. And many CNN visualization works have been proposed in the literature to interpret the CNN in perspectives of network structure, operation, and semantic concept. In this paper, we expect to provide a comprehensive survey of several representative CNN visualization methods, including Activation Maximization, Network Inversion, Deconvolutional Neural Networks (DeconvNet), and Network Dissection based visualization. These methods are presented in terms of motivations, algorithms, and experiment results. Based on these visualization methods, we also discuss their practical applications to demonstrate the significance of the CNN interpretability in areas of network design, optimization, security enhancement, etc.

翻译：目前,革命神经网络在许多与计算机视觉有关的任务上取得了令人印象深刻的成绩,如物体探测、图像识别、图像检索等。这些成就得益于CNN在通过神经结构的深层和迭代培训过程学习输入特征的杰出能力。然而,这些学到的特征很难从人类的视觉角度辨别和解释,导致对CNN内部工作机制缺乏了解。为了改进CNN的可判读性,CNN视觉化作为一种定性分析方法得到充分利用,将内部特征转化为可见模式。在文献中提出了许多CNN视觉化工程,从网络结构、运行和语义概念的角度对CNN进行解释。在本文中,我们期望提供对有代表性的CNN视觉化方法的全面调查,包括活动最大化、网络转换、进化神经网络(DeconvNet)和基于网络分解的可视化。这些方法以动机、算法和实验结果的形式提出。根据这些可视化方法,我们还讨论其实际应用情况,以展示CNN在网络设计、改进安全性方面的意义。