The mainstream paradigm of speech emotion recognition (SER) is identifying the single emotion label of the entire utterance. This line of works neglect the emotion dynamics at fine temporal granularity and mostly fail to leverage linguistic information of speech signal explicitly. In this paper, we propose Emotion Neural Transducer for fine-grained speech emotion recognition with automatic speech recognition (ASR) joint training. We first extend typical neural transducer with emotion joint network to construct emotion lattice for fine-grained SER. Then we propose lattice max pooling on the alignment lattice to facilitate distinguishing emotional and non-emotional frames. To adapt fine-grained SER to transducer inference manner, we further make blank, the special symbol of ASR, serve as underlying emotion indicator as well, yielding Factorized Emotion Neural Transducer. For typical utterance-level SER, our ENT models outperform state-of-the-art methods on IEMOCAP in low word error rate. Experiments on IEMOCAP and the latest speech emotion diarization dataset ZED also demonstrate the superiority of fine-grained emotion modeling. Our code is available at https://github.com/ECNU-Cross-Innovation-Lab/ENT.

翻译：语音情感识别的主流范式是识别整个话语的单一情感标签。这类工作忽略了细时间粒度上的情感动态，且大多未能显式利用语音信号中的语言信息。本文提出情感神经换能器，通过自动语音识别联合训练实现细粒度语音情感识别。首先，我们扩展典型神经换能器，引入情感联合网络构建用于细粒度情感识别的情感格。随后，我们提出在排列格上进行格最大池化，以促进区分情感帧与非情感帧。为适配细粒度情感识别到换能器推理方式，我们进一步将自动语音识别中的特殊符号blank作为潜在情感指示符，得到分解式情感神经换能器。在典型话语级情感识别任务中，我们的ENT模型在IEMOCAP数据集上以低词错误率超越了现有最优方法。在IEMOCAP与最新语音情感分割数据集ZED上的实验同样验证了细粒度情感建模的优越性。我们的代码开源在https://github.com/ECNU-Cross-Innovation-Lab/ENT。