Sign Language Production (SLP) is the tough task of turning sign language into sign videos. The main goal of SLP is to create these videos using a sign gloss. In this research, we've developed a new method to make high-quality sign videos without using human poses as a middle step. Our model works in two main parts: first, it learns from a generator and the video's hidden features, and next, it uses another model to understand the order of these hidden features. To make this method even better for sign videos, we make several significant improvements. (i) In the first stage, we take an improved 3D VQ-GAN to learn downsampled latent representations. (ii) In the second stage, we introduce sequence-to-sequence attention to better leverage conditional information. (iii) The separated two-stage training discards the realistic visual semantic of the latent codes in the second stage. To endow the latent sequences semantic information, we extend the token-level autoregressive latent codes learning with perceptual loss and reconstruction loss for the prior model with visual perception. Compared with previous state-of-the-art approaches, our model performs consistently better on two word-level sign language datasets, i.e., WLASL and NMFs-CSL.

翻译：手语生成（SLP）是一项将手语转化为手语视频的艰巨任务。SLP的主要目标是通过手语标注（sign gloss）生成这些视频。在本研究中，我们提出了一种无需以人体姿态作为中间步骤即可生成高质量手语视频的新方法。模型包含两个核心阶段：首先，通过学习生成器与视频的隐式特征；其次，利用另一模型对这些隐式特征的序列顺序进行建模。为提升该方法在手语视频中的表现，我们进行了多项重要改进：（i）在第一阶段，采用改进的3D VQ-GAN学习降采样后的隐式表征；（ii）在第二阶段，引入序列到序列注意力机制以更有效地利用条件信息；（iii）分离的两阶段训练会导致第二阶段中隐式编码（latent codes）失去真实视觉语义信息。为赋予隐式序列以语义信息，我们将感知损失与重构损失引入基于视觉感知的先验模型，扩展了令牌级（token-level）自回归隐式编码的学习方式。相较于先前最优方法，本模型在WLASL与NMFs-CSL两个词级手语数据集上均持续取得更优表现。