Cloud gaming is a multi-billion dollar industry. A client in cloud gaming sends its movement to the game server on the Internet, which renders and transmits the resulting video back. In order to provide a good gaming experience, a latency below 80 ms is required. This means that video rendering, encoding, transmission, decoding, and display have to finish within that time frame, which is especially challenging to achieve due to server overload, network congestion, and losses. In this paper, we propose a new method for recovering lost or corrupted video frames in cloud gaming. Unlike traditional video frame recovery, our approach uses game states to significantly enhance recovery accuracy and utilizes partially decoded frames to recover lost portions. We develop a holistic system that consists of (i) efficiently extracting game states, (ii) modifying H.264 video decoder to generate a mask to indicate which portions of video frames need recovery, and (iii) designing a novel neural network to recover either complete or partial video frames. Our approach is extensively evaluated using iPhone 12 and laptop implementations, and we demonstrate the utility of game states in the game video recovery and the effectiveness of our overall design.

翻译：云游戏是一个价值数十亿美元的产业。在云游戏中，客户端将操作指令发送至互联网上的游戏服务器，服务器渲染并回传生成的视频。为了提供良好的游戏体验，延迟需低于80毫秒。这意味着视频渲染、编码、传输、解码和显示需在此时间范围内完成，而服务器过载、网络拥塞和数据包丢失等问题使这一目标极具挑战性。本文提出了一种用于恢复云游戏中丢失或损坏视频帧的新方法。不同于传统视频帧恢复技术，本方法利用游戏状态显著提升恢复精度，并采用部分解码帧来恢复丢失区域。我们开发了一套完整系统，包括：（i）高效提取游戏状态；（ii）修改H.264视频解码器以生成掩码，标识需要恢复的视频帧区域；（iii）设计新型神经网络以恢复完整或部分视频帧。我们通过iPhone 12和笔记本电脑实现进行了全面评估，证明了游戏状态在游戏视频恢复中的实用性以及整体方案的有效性。