The success of deep learning models has led to their adaptation and adoption by prominent video understanding methods. The majority of these approaches encode features in a joint space-time modality for which the inner workings and learned representations are difficult to visually interpret. We propose LEArned Preconscious Synthesis (LEAPS), an architecture-agnostic method for synthesizing videos from the internal spatiotemporal representations of models. Using a stimulus video and a target class, we prime a fixed space-time model and iteratively optimize a video initialized with random noise. We incorporate additional regularizers to improve the feature diversity of the synthesized videos as well as the cross-frame temporal coherence of motions. We quantitatively and qualitatively evaluate the applicability of LEAPS by inverting a range of spatiotemporal convolutional and attention-based architectures trained on Kinetics-400, which to the best of our knowledge has not been previously accomplished.

翻译：深度学习模型的成功促使其被众多主流视频理解方法所采用。这些方法大多以联合时空模态编码特征，但内部机制及学习到的表征难以通过视觉方式直接解读。我们提出**LEArned Preconscious Synthesis (LEAPS)**——一种与架构无关的方法，用于从模型的内部时空表征中合成视频。通过使用刺激视频和目标类别，我们对固定的时空模型进行预激活，并迭代优化以随机噪声初始化的视频。我们引入额外的正则化项来提升合成视频的特征多样性以及跨帧运动的时间一致性。通过在Kinetics-400上训练的一系列时空卷积及基于注意力的架构上进行逆向合成（据我们所知，此前尚未有研究实现），我们从定量与定性两个维度评估了LEAPS的适用性。