This paper studies modulation spectrum features ($\Phi$) and mel-frequency cepstral coefficients ($\Psi$) in joint speaker diarization and identification (JSID). JSID is important as speaker diarization on its own to distinguish speakers is insufficient for many applications, it is often necessary to identify speakers as well. Machine learning models are set up using convolutional neural networks (CNNs) on $\Phi$ and recurrent neural networks $\unicode{x2013}$ long short-term memory (LSTMs) on $\Psi$, then concatenating into fully connected layers. Experiment 1 shows models on both $\Phi$ and $\Psi$ have better diarization error rates (DERs) than models on either alone; a CNN on $\Phi$ has DER 29.09\%, compared to 27.78\% for a LSTM on $\Psi$ and 19.44\% for a model on both. Experiment 1 also investigates aleatoric uncertainties and shows the model on both $\Phi$ and $\Psi$ has mean entropy 0.927~bits (out of 4~bits) for correct predictions compared to 1.896~bits for incorrect predictions which, along with entropy histogram shapes, shows the model helpfully indicates where it is uncertain. Experiment 2 investigates epistemic uncertainties as well as aleatoric using Monte Carlo dropout (MCD). It compares models on both $\Phi$ and $\Psi$ with models trained on x-vectors ($X$), before applying Kalman filter smoothing on epistemic uncertainties for resegmentation and model ensembles. While the two models on $X$ (DERs 10.23\% and 9.74\%) outperform those on $\Phi$ and $\Psi$ (DER 17.85\%) after their individual Kalman filter smoothing, combining them using a Kalman filter smoothing method improves the DER to 9.29\%. Aleatoric uncertainties are higher for incorrect predictions. Both Experiments show models on $\Phi$ do not distinguish overlapping speakers as well as anticipated. However, Experiment 2 shows model ensembles do better with overlapping speakers than individual models do.

翻译：本文研究调制频谱特征（$\Phi$）和梅尔频率倒谱系数（$\Psi$）在联合说话人日记化与识别（JSID）中的应用。JSID至关重要，因为单独的说话人日记化虽能区分说话人，但对许多应用场景而言往往不足，通常还需识别说话人身份。我们采用机器学习模型，利用卷积神经网络（CNN）处理$\Phi$，利用循环神经网络——长短期记忆网络（LSTM）处理$\Psi$，并将两者拼接至全连接层。实验一表明，同时基于$\Phi$和$\Psi$的模型其日记化错误率（DER）优于单独使用任一特征的模型；基于$\Phi$的CNN模型DER为29.09%，基于$\Psi$的LSTM模型DER为27.78%，而同时使用两者的模型DER降至19.44%。实验一还探讨了偶然不确定性，发现同时基于$\Phi$和$\Psi$的模型对正确预测的平均熵为0.927比特（最大4比特），对错误预测的平均熵为1.896比特，结合熵直方图形状表明该模型能有效指示其不确定性区域。实验二采用蒙特卡洛丢包（MCD）同时研究认知不确定性与偶然不确定性。该实验对比了基于$\Phi$和$\Psi$的模型与基于x向量（$X$）训练的模型，随后对认知不确定性应用卡尔曼滤波平滑以进行再分割和模型集成。尽管基于$X$的两个模型（DER分别为10.23%和9.74%）在其各自卡尔曼滤波平滑后优于基于$\Phi$和$\Psi$的模型（DER 17.85%），但通过卡尔曼滤波平滑方法组合这些模型可将DER改善至9.29%。错误预测的偶然不确定性更高。两个实验均显示，基于$\Phi$的模型对重叠说话人的区分能力未达预期。然而，实验二表明模型集成在处理重叠说话人时优于单个模型。