Video summarization intends to produce a concise video summary by effectively capturing and combining the most informative parts of the whole content. Existing approaches for video summarization regard the task as a frame-wise keyframe selection problem and generally construct the frame-wise representation by combining the long-range temporal dependency with the unimodal or bimodal information. However, the optimal video summaries need to reflect the most valuable keyframe with its own information, and one with semantic power of the whole content. Thus, it is critical to construct a more powerful and robust frame-wise representation and predict the frame-level importance score in a fair and comprehensive manner. To tackle the above issues, we propose a multimodal hierarchical shot-aware convolutional network, denoted as MHSCNet, to enhance the frame-wise representation via combining the comprehensive available multimodal information. Specifically, we design a hierarchical ShotConv network to incorporate the adaptive shot-aware frame-level representation by considering the short-range and long-range temporal dependency. Based on the learned shot-aware representations, MHSCNet can predict the frame-level importance score in the local and global view of the video. Extensive experiments on two standard video summarization datasets demonstrate that our proposed method consistently outperforms state-of-the-art baselines. Source code will be made publicly available.

翻译：视频摘要是通过有效捕捉并融合整个视频内容中最具信息性的部分，生成简洁视频摘要的技术。现有方法将视频摘要视为逐帧关键帧选择任务，通常通过结合长程时序依赖与单模态或双模态信息构建帧级表征。然而，最优视频摘要需既能体现包含自身信息的最有价值关键帧，又能反映整体内容的语义表达能力。因此，构建更具鲁棒性的帧级表征，并以公平全面的方式预测帧级重要性得分至关重要。针对上述问题，我们提出一种多模态层级镜头感知卷积网络——MHSCNet，通过融合多维可用多模态信息增强帧级表征。具体而言，我们设计了层级化ShotConv网络，通过考虑短程与长程时序依赖生成自适应镜头感知的帧级表征。基于所学镜头感知表征，MHSCNet可从视频局部与全局视角预测帧级重要性得分。在两个标准视频摘要数据集上的实验表明，所提方法持续优于当前最优基线模型。源代码将公开提供。