This paper presents IP-SLT, a simple yet effective framework for sign language translation (SLT). Our IP-SLT adopts a recurrent structure and enhances the semantic representation (prototype) of the input sign language video via an iterative refinement manner. Our idea mimics the behavior of human reading, where a sentence can be digested repeatedly, till reaching accurate understanding. Technically, IP-SLT consists of feature extraction, prototype initialization, and iterative prototype refinement. The initialization module generates the initial prototype based on the visual feature extracted by the feature extraction module. Then, the iterative refinement module leverages the cross-attention mechanism to polish the previous prototype by aggregating it with the original video feature. Through repeated refinement, the prototype finally converges to a more stable and accurate state, leading to a fluent and appropriate translation. In addition, to leverage the sequential dependence of prototypes, we further propose an iterative distillation loss to compress the knowledge of the final iteration into previous ones. As the autoregressive decoding process is executed only once in inference, our IP-SLT is ready to improve various SLT systems with acceptable overhead. Extensive experiments are conducted on public benchmarks to demonstrate the effectiveness of the IP-SLT.

翻译：本文提出IP-SLT，一种简单而有效的符号语言翻译（SLT）框架。我们的IP-SLT采用循环结构，通过迭代精炼的方式增强输入手语视频的语义表示（原型）。该思想模仿了人类阅读行为——句子可被反复消化直至准确理解。在技术层面，IP-SLT由特征提取、原型初始化和迭代原型精炼三个模块组成。初始化模块基于特征提取模块提取的视觉特征生成初始原型；迭代精炼模块则利用交叉注意力机制，通过将前次原型与原始视频特征聚合来实现原型优化。经反复精炼后，原型最终收敛至更稳定精确的状态，从而生成流畅恰当的翻译。此外，为利用原型之间的序列依赖关系，我们进一步提出迭代蒸馏损失，将最终迭代的知识压缩至前期迭代中。由于自回归解码过程仅在推理时执行一次，我们的IP-SLT能以可接受的额外开销改进多种SLT系统。在公开基准数据集上的大量实验验证了IP-SLT的有效性。