The success of deep learning models has led to their adaptation and adoption by prominent video understanding methods. The majority of these approaches encode features in a joint space-time modality for which the inner workings and learned representations are difficult to visually interpret. We propose LEArned Preconscious Synthesis (LEAPS), an architecture-agnostic method for synthesizing videos from the internal spatiotemporal representations of models. Using a stimulus video and a target class, we prime a fixed space-time model and iteratively optimize a video initialized with random noise. We incorporate additional regularizers to improve the feature diversity of the synthesized videos as well as the cross-frame temporal coherence of motions. We quantitatively and qualitatively evaluate the applicability of LEAPS by inverting a range of spatiotemporal convolutional and attention-based architectures trained on Kinetics-400, which to the best of our knowledge has not been previously accomplished.

翻译：深度学习模型的成功使其被诸多主流视频理解方法所采纳与应用。这些方法大多以联合时空模态编码特征，导致其内部运作机制与学习到的表征难以直观解释。我们提出一种与架构无关的方法——学习型前意识合成（LEAPS），旨在从模型内部时空表征中合成视频。通过刺激视频与目标类别，我们初始化一个固定时空模型，并迭代优化一个随机噪声初始化的视频。我们引入额外正则化项以提升合成视频的特征多样性及跨帧运动的时间一致性。通过在Kinetics-400数据集上反转一系列基于时空卷积与注意力机制的架构（据我们所知，此前尚未有此研究），我们从定量与定性两个维度评估了LEAPS的适用性。