Cloud gaming is a multi-billion dollar industry. A client in cloud gaming sends its movement to the game server on the Internet, which renders and transmits the resulting video back. In order to provide a good gaming experience, a latency below 80 ms is required. This means that video rendering, encoding, transmission, decoding, and display have to finish within that time frame, which is especially challenging to achieve due to server overload, network congestion, and losses. In this paper, we propose a new method for recovering lost or corrupted video frames in cloud gaming. Unlike traditional video frame recovery, our approach uses game states to significantly enhance recovery accuracy and utilizes partially decoded frames to recover lost portions. We develop a holistic system that consists of (i) efficiently extracting game states, (ii) modifying H.264 video decoder to generate a mask to indicate which portions of video frames need recovery, and (iii) designing a novel neural network to recover either complete or partial video frames. Our approach is extensively evaluated using iPhone 12 and laptop implementations, and we demonstrate the utility of game states in the game video recovery and the effectiveness of our overall design.

翻译：云游戏是一个价值数十亿美元的产业。在云游戏中，客户端通过互联网向游戏服务器发送操作指令，服务器渲染并传输生成的视频流返回客户端。为提供良好的游戏体验，延迟需要控制在80毫秒以内。这意味着视频渲染、编码、传输、解码和显示都必须在此时限内完成，而服务器过载、网络拥塞和数据丢包等问题使得这一目标极具挑战。本文提出一种新的方法，用于恢复云游戏中丢失或损坏的视频帧。与传统视频帧恢复不同，我们的方法利用游戏状态显著提升恢复精度，并借助部分解码帧恢复丢失区域。我们开发了一套完整的系统，包括：(i)高效提取游戏状态；(ii)修改H.264视频解码器以生成掩码，标识视频帧中需要恢复的区域；以及(iii)设计新型神经网络，实现完整或部分视频帧的恢复。通过iPhone 12和笔记本电脑平台的广泛评估，我们验证了游戏状态在游戏视频恢复中的实用价值以及整体设计方案的有效性。