Translating written sentences from oral languages to a sequence of manual and non-manual gestures plays a crucial role in building a more inclusive society for deaf and hard-of-hearing people. Facial expressions (non-manual), in particular, are responsible for encoding the grammar of the sentence to be spoken, applying punctuation, pronouns, or emphasizing signs. These non-manual gestures are closely related to the semantics of the sentence being spoken and also to the utterance of the speaker's emotions. However, most Sign Language Production (SLP) approaches are centered on synthesizing manual gestures and do not focus on modeling the speakers expression. This paper introduces a new method focused in synthesizing facial expressions for sign language. Our goal is to improve sign language production by integrating sentiment information in facial expression generation. The approach leverages a sentence sentiment and semantic features to sample from a meaningful representation space, integrating the bias of the non-manual components into the sign language production process. To evaluate our method, we extend the Frechet Gesture Distance (FGD) and propose a new metric called Frechet Expression Distance (FED) and apply an extensive set of metrics to assess the quality of specific regions of the face. The experimental results showed that our method achieved state of the art, being superior to the competitors on How2Sign and PHOENIX14T datasets. Moreover, our architecture is based on a carefully designed graph pyramid that makes it simpler, easier to train, and capable of leveraging emotions to produce facial expressions.

翻译：将口语的书面句子翻译为一系列手动与非手动手势，对于为聋哑及听力障碍群体构建更具包容性的社会至关重要。面部表情（非手动手势）尤其负责编码待表达句子的语法结构，实现标点、代词功能或强调特定手势。这些非手动手势不仅与所表达句子的语义紧密相关，也与说话者情感的表达密不可分。然而，当前大多数手语生成方法主要集中于合成手动手势，并未对说话者表情建模给予足够关注。本文提出一种专注于手语面部表情合成的新方法，旨在通过将情感信息融入面部表情生成来提升手语生成质量。该方法利用句子情感与语义特征从有意义的表征空间中进行采样，将非手动成分的偏置整合至手语生成流程。为评估本方法，我们扩展了弗雷歇手势距离指标，提出名为弗雷歇表情距离的新度量标准，并采用一系列广泛指标对面部特定区域的质量进行评估。实验结果表明，本方法在How2Sign和PHOENIX14T数据集上达到了当前最优性能，优于现有对比方法。此外，我们的架构基于精心设计的图金字塔结构，使其更简洁、易于训练，并能有效利用情感信息生成面部表情。