The drastic variation of motion in spatial and temporal dimensions makes the video prediction task extremely challenging. Existing RNN models obtain higher performance by deepening or widening the model. They obtain the multi-scale features of the video only by stacking layers, which is inefficient and brings unbearable training costs (such as memory, FLOPs, and training time). Different from them, this paper proposes a spatiotemporal multi-scale model called MS-LSTM wholly from a multi-scale perspective. On the basis of stacked layers, MS-LSTM incorporates two additional efficient multi-scale designs to fully capture spatiotemporal context information. Concretely, we employ LSTMs with mirrored pyramid structures to construct spatial multi-scale representations and LSTMs with different convolution kernels to construct temporal multi-scale representations. We theoretically analyze the training cost and performance of MS-LSTM and its components. Detailed comparison experiments with twelve baseline models on four video datasets show that MS-LSTM has better performance but lower training costs.

翻译：空间与时间维度上运动的剧烈变化使得视频预测任务极具挑战性。现有RNN模型通过加深或拓宽网络来提升性能，它们仅通过堆叠层数获取视频的多尺度特征，这种方式效率低下且会带来难以承受的训练成本（如内存、FLOPs和训练时间）。与此不同，本文从多尺度视角出发，提出了一种名为MS-LSTM的时空多尺度模型。在堆叠层的基础上，MS-LSTM融入了两种额外的高效多尺度设计，以充分捕获时空上下文信息。具体而言，我们采用具有镜像金字塔结构的LSTM构建空间多尺度表征，以及采用不同卷积核的LSTM构建时间多尺度表征。我们从理论上分析了MS-LSTM及其组件的训练成本与性能。在四个视频数据集上与十二个基线模型的详细对比实验表明，MS-LSTM具有更优性能且训练成本更低。