Recent research advance has significantly improved the visual realism of immersive 3D video communication. In this work we present a method to further enhance this immersive experience by adding the hand touch capability ("remote hand clapping"). In our system, each meeting participant sits in front of a large screen with haptic feedback. The local participant can reach his hand out to the screen and perform hand clapping with the remote participant as if the two participants were only separated by a virtual glass. A key challenge in emulating the remote hand touch is the realistic rendering of the participant's hand and arm as the hand touches the screen. When the hand is very close to the screen, the RGBD data required for realistic rendering is no longer available. To tackle this challenge, we present a dual representation of the user's hand. Our dual representation not only preserves the high-quality rendering usually found in recent image-based rendering systems but also allows the hand to reach the screen. This is possible because the dual representation includes both an image-based model and a 3D geometry-based model, with the latter driven by a hand skeleton tracked by a side view camera. In addition, the dual representation provides a distance-based fusion of the image-based and 3D geometry-based models as the hand moves closer to the screen. The result is that the image-based and 3D geometry-based models mutually enhance each other, leading to realistic and seamless rendering. Our experiments demonstrate that our method provides consistent hand contact experience between remote users and improves the immersive experience of 3D video communication.

翻译：近期研究进展显著提升了沉浸式三维视频通信的视觉真实感。本文提出一种通过增加手部触觉能力（"远程击掌"）进一步增强沉浸式体验的方法。在本系统中，每位会议参与者坐在配备触觉反馈的大屏幕前，本地参与者可将手伸向屏幕与远程参与者完成击掌动作，如同两人仅隔着一层虚拟玻璃。实现远程手部触觉的关键挑战在于：当手部接触屏幕时，需对手臂与手掌进行真实感渲染。当手部贴近屏幕时，用于真实感渲染的RGBD数据将不可用。为解决此问题，我们提出用户手部的双表征模型。该双表征不仅能保持当前基于图像渲染系统的高质量效果，还可支持手部接触屏幕。其可行性源于双表征包含基于图像的模型与基于三维几何的模型，其中后者由侧视摄像头追踪的手部骨骼驱动。此外，当手部向屏幕移动时，双表征会基于距离对两种模型进行融合。最终实现基于图像的模型与基于三维几何的模型互相增强，达成真实且无缝的渲染效果。实验证明，本方法可为远程用户提供一致的手部接触体验，并显著提升三维视频通信的沉浸感。