Facial motion tracking in head-mounted displays (HMD) has the potential to enable immersive "face-to-face" interaction in a virtual environment. However, current works on facial tracking are not suitable for unobtrusive augmented reality (AR) glasses or do not have the ability to track arbitrary facial movements. In this work, we demonstrate a novel system called SpecTracle that tracks a user's facial motions using two wide-angle cameras mounted right next to the visor of a Hololens. Avoiding the usage of cameras extended in front of the face, our system greatly improves the feasibility to integrate full-face tracking into a low-profile form factor. We also demonstrate that a neural network-based model processing the wide-angle cameras can run in real-time at 24 frames per second (fps) on a mobile GPU and track independent facial movement for different parts of the face with a user-independent model. Using a short personalized calibration, the system improves its tracking performance by 42.3% compared to the user-independent model.

翻译：在头戴式显示器（HMD）中实现面部运动追踪，有望在虚拟环境中营造沉浸式的“面对面”交互体验。然而，当前的面部追踪技术要么不适用于非侵入式增强现实（AR）眼镜，要么无法追踪任意的面部运动。本研究展示了一种名为SpecTracle的新型系统，该系统通过安装在Hololens面罩两侧的两个广角摄像头追踪用户的面部运动。由于避免了使用延伸至面部前方的摄像头，我们的系统显著提升了将全脸追踪集成到低剖面形态中的可行性。我们还证明，基于神经网络的模型处理广角摄像头图像时，可在移动GPU上以每秒24帧（fps）的实时速度运行，并通过用户无关模型追踪面部不同区域的独立运动。通过简短的个人化校准，该系统相较于用户无关模型的追踪性能提升了42.3%。