Automatic video description requires the generation of natural language statements about the actions, events, and objects in the video. An important human trait, when we describe a video, is that we are able to do this with variable levels of detail. Different from this, existing approaches for automatic video descriptions are mostly focused on single sentence generation at a fixed level of detail. Instead, here we address video description of manipulation actions where different levels of detail are required for being able to convey information about the hierarchical structure of these actions relevant also for modern approaches of robot learning. We propose one hybrid statistical and one end-to-end framework to address this problem. The hybrid method needs much less data for training, because it models statistically uncertainties within the video clips, while in the end-to-end method, which is more data-heavy, we are directly connecting the visual encoder to the language decoder without any intermediate (statistical) processing step. Both frameworks use LSTM stacks to allow for different levels of description granularity and videos can be described by simple single-sentences or complex multiple-sentence descriptions. In addition, quantitative results demonstrate that these methods produce more realistic descriptions than other competing approaches.

翻译：自动视频描述需要生成关于视频中动作、事件和对象的自然语言陈述。人类描述视频时的一个重要能力是能够以不同详细程度进行描述。与此不同的是，现有自动视频描述方法主要集中于固定详细程度的单句生成。本文则针对操作动作的视频描述问题，其中需要不同详细程度来传达这些动作的层次结构信息，这一需求也与现代机器人学习方法相关。我们提出了一种混合统计框架和一种端到端框架来解决该问题。混合方法需要更少的训练数据，因为它对视频片段中的不确定性进行统计建模；而端到端方法则更依赖数据，直接将视觉编码器与语言解码器连接，无需任何中间统计处理步骤。两个框架均采用LSTM堆叠结构，支持不同的描述粒度，视频可以通过简单的单句描述或复杂的多句描述来表达。此外，定量结果表明，这些方法比其它竞争方法能生成更真实的描述。