Contact languages like English exhibit rich regional variations in the form of dialects, which are often used by dialect speakers interacting with generative models. However, can multimodal generative models effectively produce content given dialectal textual input? In this work, we study this question by constructing a new large-scale benchmark spanning six common English dialects. We work with dialect speakers to collect and verify over 4200 unique prompts and evaluate on 17 image and video generative models. Our automatic and human evaluation results show that current state-of-the-art multimodal generative models exhibit 32.26% to 48.17% performance degradation when a single dialect word is used in the prompt. Common mitigation methods such as fine-tuning and prompt rewriting can only improve dialect performance by small margins (< 7%), while potentially incurring significant performance degradation in Standard American English (SAE). To this end, we design a general encoder-based mitigation strategy for multimodal generative models. Our method teaches the model to recognize new dialect features while preserving SAE performance. Experiments on models such as Stable Diffusion 1.5 show that our method is able to simultaneously raise performance on five dialects to be on par with SAE (+34.4%), while incurring near zero cost to SAE performance.

翻译：接触性语言（如英语）在方言形式上表现出丰富的地域性变体，方言使用者常使用方言与生成模型进行交互。然而，多模态生成模型能否在给定方言文本输入时有效生成内容？本研究通过构建一个涵盖六种常见英语方言的大规模基准测试来探讨此问题。我们与方言使用者合作，收集并验证了超过4200条独特提示，并对17个图像与视频生成模型进行了评估。自动与人工评估结果表明，当提示中使用单个方言词汇时，当前最先进的多模态生成模型会出现32.26%至48.17%的性能下降。常见的缓解方法（如微调和提示词改写）仅能小幅提升方言性能（< 7%），同时可能导致标准美国英语（SAE）性能显著下降。为此，我们为多模态生成模型设计了一种通用的基于编码器的缓解策略。该方法使模型能够识别新的方言特征，同时保持SAE性能。在Stable Diffusion 1.5等模型上的实验表明，我们的方法能够将五种方言的性能提升至与SAE相当的水平（+34.4%），同时对SAE性能的影响近乎为零。