Automatic live video commenting is with increasing attention due to its significance in narration generation, topic explanation, etc. However, the diverse sentiment consideration of the generated comments is missing from the current methods. Sentimental factors are critical in interactive commenting, and lack of research so far. Thus, in this paper, we propose a Sentiment-oriented Transformer-based Variational Autoencoder (So-TVAE) network which consists of a sentiment-oriented diversity encoder module and a batch attention module, to achieve diverse video commenting with multiple sentiments and multiple semantics. Specifically, our sentiment-oriented diversity encoder elegantly combines VAE and random mask mechanism to achieve semantic diversity under sentiment guidance, which is then fused with cross-modal features to generate live video comments. Furthermore, a batch attention module is also proposed in this paper to alleviate the problem of missing sentimental samples, caused by the data imbalance, which is common in live videos as the popularity of videos varies. Extensive experiments on Livebot and VideoIC datasets demonstrate that the proposed So-TVAE outperforms the state-of-the-art methods in terms of the quality and diversity of generated comments. Related code is available at https://github.com/fufy1024/So-TVAE.

翻译：自动直播视频评论因其在叙事生成、主题解释等方面的重要性而日益受到关注。然而，现有方法缺乏对生成评论的多样化情感考量。情感因素在交互式评论中至关重要，但目前尚缺乏相关研究。为此，本文提出了一种面向情感的基于Transformer的变分自编码器（So-TVAE）网络，该网络包含面向情感的分集编码器模块和批处理注意力模块，以实现多情感、多语义的多样化视频评论。具体而言，我们的面向情感的分集编码器巧妙结合了VAE与随机掩码机制，在情感引导下实现语义多样性，随后与跨模态特征融合生成直播视频评论。此外，本文还提出了一种批处理注意力模块，以缓解由数据不平衡导致的情感样本缺失问题——这种不平衡在直播视频中普遍存在（因视频热度各异）。在Livebot和VideoIC数据集上的大量实验表明，所提出的So-TVAE在生成评论的质量和多样性方面均优于现有最先进方法。相关代码已开源：https://github.com/fufy1024/So-TVAE。