In this paper, we propose a novel framework for Interactive Face Video Coding (IFVC), which allows humans to interact with the intrinsic visual representations instead of the signals. The proposed solution enjoys several distinct advantages, including ultra-compact representation, low delay interaction, and vivid expression and headpose animation. In particular, we propose the Internal Dimension Increase (IDI) based representation, greatly enhancing the fidelity and flexibility in rendering the appearance while maintaining reasonable representation cost. By leveraging strong statistical regularities, the visual signals can be effectively projected into controllable semantics in the three dimensional space (e.g., mouth motion, eye blinking, head rotation and head translation), which are compressed and transmitted. The editable bitstream, which naturally supports the interactivity at the semantic level, can synthesize the face frames via the strong inference ability of the deep generative model. Experimental results have demonstrated the performance superiority and application prospects of our proposed IFVC scheme. In particular, the proposed scheme not only outperforms the state-of-the-art video coding standard Versatile Video Coding (VVC) and the latest generative compression schemes in terms of rate-distortion performance for face videos, but also enables the interactive coding without introducing additional manipulation processes. Furthermore, the proposed framework is expected to shed lights on the future design of the digital human communication in the metaverse.

翻译：本文提出了一种交互式人脸视频编码（Interactive Face Video Coding, IFVC）的新框架，该框架允许人类与内在视觉表示而非信号本身进行交互。所提方案具有多种显著优势，包括超紧凑表示、低延迟交互以及生动的表情和头部姿态动画。具体而言，我们提出了基于内部维度增加（Internal Dimension Increase, IDI）的表示方法，在保持合理表示成本的同时，显著提升了外观渲染的保真度和灵活性。通过利用强统计规律，视觉信号可有效投影至三维空间中的可控语义（例如嘴部运动、眨眼、头部旋转与平移），并经过压缩与传输。支持语义层面自然交互的可编辑比特流，可借助深度生成模型的强大推理能力合成人脸帧。实验结果表明，所提IFVC方案具有性能优越性及应用前景。特别地，该方案不仅在率失真性能上优于最新视频编码标准通用视频编码（Versatile Video Coding, VVC）及最新生成式压缩方案，还能在不引入额外操作过程的情况下实现交互式编码。此外，本框架有望为元宇宙中数字人类通信的未来设计提供启示。