Reshaping, a point operation that alters the characteristics of signals, has been shown capable of improving the compression ratio in video coding practices. Out-of-loop reshaping that directly modifies the input video signal was first adopted as the supplemental enhancement information~(SEI) for the HEVC/H.265 without the need of altering the core design of the video codec. VVC/H.266 further improves the coding efficiency by adopting in-loop reshaping that modifies the residual signal being processed in the hybrid coding loop. In this paper, we theoretically analyze the rate-distortion performance of the in-loop reshaping and use experiments to verify the theoretical result. We prove that the in-loop reshaping can improve coding efficiency when the entropy coder adopted in the coding pipeline is suboptimal, which is in line with the practical scenarios that video codecs operate in. We derive the PSNR gain in a closed form and show that the theoretically predicted gain is consistent with that measured from experiments using standard testing video sequences.

翻译：整形作为一种改变信号特性的点操作，已被证明能够提升视频编码实践中的压缩比。最初，直接修改输入视频信号的环外整形被采纳为HEVC/H.265的补充增强信息(SEI)，且无需改变视频编解码器的核心设计。VVC/H.266进一步通过采用环内整形来提升编码效率，该技术修改了混合编码环路中待处理的残差信号。本文从理论上分析了环内整形的率失真性能，并通过实验验证了理论结果。我们证明，当编码流程中采用的熵编码器非最优时（这符合视频编解码器实际运行的场景），环内整形能够改善编码效率。我们推导出PSNR增益的闭合形式，并表明理论预测的增益与使用标准测试视频序列的实验测量结果一致。