This paper studies modulation spectrum features ($\Phi$) and mel-frequency cepstral coefficients ($\Psi$) in joint speaker diarization and identification (JSID). JSID is important as speaker diarization on its own to distinguish speakers is insufficient for many applications, it is often necessary to identify speakers as well. Machine learning models are set up using convolutional neural networks (CNNs) on $\Phi$ and recurrent neural networks $\unicode{x2013}$ long short-term memory (LSTMs) on $\Psi$, then concatenating into fully connected layers. Experiment 1 shows models on both $\Phi$ and $\Psi$ have better diarization error rates (DERs) than models on either alone; a CNN on $\Phi$ has DER 29.09\%, compared to 27.78\% for a LSTM on $\Psi$ and 19.44\% for a model on both. Experiment 1 also investigates aleatoric uncertainties and shows the model on both $\Phi$ and $\Psi$ has mean entropy 0.927~bits (out of 4~bits) for correct predictions compared to 1.896~bits for incorrect predictions which, along with entropy histogram shapes, shows the model helpfully indicates where it is uncertain. Experiment 2 investigates epistemic uncertainties as well as aleatoric using Monte Carlo dropout (MCD). It compares models on both $\Phi$ and $\Psi$ with models trained on x-vectors ($X$), before applying Kalman filter smoothing on epistemic uncertainties for resegmentation and model ensembles. While the two models on $X$ (DERs 10.23\% and 9.74\%) outperform those on $\Phi$ and $\Psi$ (DER 17.85\%) after their individual Kalman filter smoothing, combining them using a Kalman filter smoothing method improves the DER to 9.29\%. Aleatoric uncertainties are higher for incorrect predictions. Both Experiments show models on $\Phi$ do not distinguish overlapping speakers as well as anticipated. However, Experiment 2 shows model ensembles do better with overlapping speakers than individual models do.

翻译：本文研究联合说话人分割与识别（JSID）中的调制谱特征（$\Phi$）和梅尔频率倒谱系数（$\Psi$）。JSID具有重要价值，因为单独的说话人分割在区分说话人方面对许多应用而言并不充分，通常还需要同时识别说话人。我们基于$\Phi$构建卷积神经网络（CNN）模型，基于$\Psi$构建循环神经网络——长短期记忆（LSTM）模型，并将两者特征拼接后接入全连接层。实验1表明，同时使用$\Phi$和$\Psi$的模型在分割错误率（DER）上优于单独使用任一特征的模型：基于$\Phi$的CNN的DER为29.09%，基于$\Psi$的LSTM的DER为27.78%，而联合模型的DER为19.44%。实验1还研究了偶然不确定性，结果显示联合模型对正确预测的平均熵为0.927比特（总熵4比特），对错误预测的平均熵为1.896比特，结合熵直方图形状可知，该模型能有效指示自身不确定的位置。实验2采用蒙特卡洛丢弃法（MCD）同时研究认知不确定性和偶然不确定性，对比了基于$\Phi$和$\Psi$的联合模型与基于x-向量（$X$）训练的模型，并通过卡尔曼滤波平滑认知不确定性进行重分割和模型集成。尽管两种基于$X$的模型（DER分别为10.23%和9.74%）在单独卡尔曼滤波平滑后优于基于$\Phi$和$\Psi$的模型（DER 17.85%），但采用卡尔曼滤波平滑方法将两者结合后DER提升至9.29%。错误预测的偶然不确定性更高。两个实验均表明，基于$\Phi$的模型对重叠说话人的区分能力未达预期。然而实验2显示，模型集成在处理重叠说话人方面优于单个模型。