Supporting real-time interactions between human controllers and remote devices remains a challenging goal in the Metaverse due to the stringent requirements on computing workload, communication throughput, and round-trip latency. In this paper, we establish a novel framework for real-time interactions through the virtual models in the Metaverse. Specifically, we jointly predict the motion of the human controller for 1) proactive rendering in the Metaverse and 2) generating control commands to the real-world remote device in advance. The virtual model is decoupled into two components for rendering and control, respectively. To dynamically adjust the prediction horizons for rendering and control, we develop a two-step human-in-the-loop continuous reinforcement learning approach and use an expert policy to improve the training efficiency. An experimental prototype is built to verify our algorithm with different communication latencies. Compared with the baseline policy without prediction, our proposed method can reduce 1) the Motion-To-Photon (MTP) latency between human motion and rendering feedback and 2) the root mean squared error (RMSE) between human motion and real-world remote devices significantly.

翻译：在元宇宙中实现人类操控者与远程设备的实时交互，由于对计算负载、通信吞吐量和往返延迟的严苛要求，仍然是一个具有挑战性的目标。本文通过元宇宙中的虚拟模型，建立了一个用于实时交互的新框架。具体而言，我们联合预测人类操控者的运动，以实现：1）元宇宙中的主动渲染；2）提前生成发送给现实世界远程设备的控制命令。虚拟模型被解耦为分别用于渲染和控制的两个组件。为了动态调整渲染和控制的预测时域，我们开发了一种两步式人在回路连续强化学习方法，并利用专家策略来提高训练效率。我们构建了一个实验原型，以在不同通信延迟下验证我们的算法。与无预测的基线策略相比，我们提出的方法能够显著降低：1）人体运动与渲染反馈之间的运动到光子延迟；2）人体运动与现实世界远程设备之间的均方根误差。