Lip reading, the process of interpreting silent speech from visual lip movements, has gained rising attention for its wide range of realistic applications. Deep learning approaches greatly improve current lip reading systems. However, lip reading in cross-speaker scenarios where the speaker identity changes, poses a challenging problem due to inter-speaker variability. A well-trained lip reading system may perform poorly when handling a brand new speaker. To learn a speaker-robust lip reading model, a key insight is to reduce visual variations across speakers, avoiding the model overfitting to specific speakers. In this work, in view of both input visual clues and latent representations based on a hybrid CTC/attention architecture, we propose to exploit the lip landmark-guided fine-grained visual clues instead of frequently-used mouth-cropped images as input features, diminishing speaker-specific appearance characteristics. Furthermore, a max-min mutual information regularization approach is proposed to capture speaker-insensitive latent representations. Experimental evaluations on public lip reading datasets demonstrate the effectiveness of the proposed approach under the intra-speaker and inter-speaker conditions.

翻译：唇语识别，即通过视觉唇部运动解读无声语音的技术，因其广泛的实际应用而日益受到关注。深度学习方法显著提升了现有唇语识别系统的性能。然而，在说话人身份变化的跨说话人场景中，由于说话人间的变异性，唇语识别面临严峻挑战。一个训练有素的唇语识别系统在面对全新说话人时可能表现不佳。为学习说话人鲁棒的唇语识别模型，关键思路在于减少不同说话人之间的视觉差异，避免模型过度拟合特定说话人。本研究基于混合CTC/注意力架构，从输入视觉线索和潜在表示两个角度出发，提出利用唇部地标引导的细粒度视觉线索替代常用的嘴部裁剪图像作为输入特征，以消除说话人特有的外观特征。此外，提出一种极大极小互信息正则化方法，用于捕获对说话人不敏感的潜在表示。在公开唇语识别数据集上的实验评估表明，所提方法在说话人内和说话人间条件下均具有有效性。