While significant progress has been achieved in multimodal facial generation using semantic masks and textual descriptions, conventional feature fusion approaches often fail to enable effective cross-modal interactions, thereby leading to suboptimal generation outcomes. To address this challenge, we introduce MDiTFace--a customized diffusion transformer framework that employs a unified tokenization strategy to process semantic mask and text inputs, eliminating discrepancies between heterogeneous modality representations. The framework facilitates comprehensive multimodal feature interaction through stacked, newly designed multivariate transformer blocks that process all conditions synchronously. Additionally, we design a novel decoupled attention mechanism by dissociating implicit dependencies between mask tokens and temporal embeddings. This mechanism segregates internal computations into dynamic and static pathways, enabling caching and reuse of features computed in static pathways after initial calculation, thereby reducing additional computational overhead introduced by mask condition by over 94% while maintaining performance. Extensive experiments demonstrate that MDiTFace significantly outperforms other competing methods in terms of both facial fidelity and conditional consistency.

翻译：尽管利用语义掩码和文本描述进行多模态人脸生成已取得显著进展，但传统的特征融合方法往往无法实现有效的跨模态交互，从而导致生成效果欠佳。为解决这一挑战，我们提出了MDiTFace——一个定制的扩散Transformer框架，该框架采用统一的标记化策略来处理语义掩码和文本输入，消除了异构模态表示之间的差异。该框架通过堆叠新设计的、能同步处理所有条件的多变量Transformer块，促进了全面的多模态特征交互。此外，我们通过解耦掩码标记与时间嵌入之间的隐式依赖关系，设计了一种新颖的解耦注意力机制。该机制将内部计算分离为动态和静态路径，使得在静态路径中计算的特征在首次计算后能够被缓存和重用，从而在保持性能的同时，将掩码条件引入的额外计算开销降低了94%以上。大量实验表明，MDiTFace在人脸保真度和条件一致性方面均显著优于其他竞争方法。