Fueled by recent advances of self-supervised models, pre-trained speech representations proved effective for the downstream speech emotion recognition (SER) task. Most prior works mainly focus on exploiting pre-trained representations and just adopt a linear head on top of the pre-trained model, neglecting the design of the downstream network. In this paper, we propose a temporal shift module to mingle channel-wise information without introducing any parameter or FLOP. With the temporal shift module, three designed baseline building blocks evolve into corresponding shift variants, i.e. ShiftCNN, ShiftLSTM, and Shiftformer. Moreover, to balance the trade-off between mingling and misalignment, we propose two technical strategies, placement of shift and proportion of shift. The family of temporal shift models all outperforms the state-of-the-art methods on the benchmark IEMOCAP dataset under both finetuning and feature extraction settings. Our code is available at https://github.com/ECNU-Cross-Innovation-Lab/ShiftSER.

翻译：受自监督模型最新进展的驱动，预训练语音表征已被证明能有效服务于下游语音情感识别（SER）任务。以往工作主要侧重于利用预训练表征，通常仅在预训练模型上添加线性分类头，忽略了下游网络的结构设计。本文提出一种时间偏移模块，可在不引入任何参数或浮点运算量的情况下融合通道维度信息。借助该时间偏移模块，三种基础骨干模块被演化为相应的偏移变体，即ShiftCNN、ShiftLSTM和Shiftformer。此外，为平衡信息融合与特征错位之间的权衡，我们提出了两项技术策略：偏移放置位置与偏移比例。在基准数据集IEMOCAP上，基于微调和特征提取两种设置，时间偏移模型系列均全面超越了现有最优方法。代码已开源在https://github.com/ECNU-Cross-Innovation-Lab/ShiftSER。