Large Language Models (LLMs) and Text-To-Image (T2I) models have demonstrated the ability to generate compelling text and visual stories. However, their outputs are predominantly aligned with the sensibilities of the Global North, often resulting in an outsider's gaze on other cultures. As a result, non-Western communities have to put extra effort into generating culturally specific stories. To address this challenge, we developed a visual storytelling tool called Kahani that generates culturally grounded visual stories for non-Western cultures. Our tool leverages off-the-shelf models GPT-4 Turbo and Stable Diffusion XL (SDXL). By using Chain of Thought (CoT) and T2I prompting techniques, we capture the cultural context from user's prompt and generate vivid descriptions of the characters and scene compositions. To evaluate the effectiveness of Kahani, we conducted a comparative user study with ChatGPT-4 (with DALL-E3) in which participants from different regions of India compared the cultural relevance of stories generated by the two tools. The results of the qualitative and quantitative analysis performed in the user study show that Kahani's visual stories are more culturally nuanced than those generated by ChatGPT-4. In 27 out of 36 comparisons, Kahani outperformed or was on par with ChatGPT-4, effectively capturing cultural nuances and incorporating more Culturally Specific Items (CSI), validating its ability to generate culturally grounded visual stories.
翻译:大型语言模型(LLM)和文本到图像(T2I)模型已展现出生成引人入胜的文本与视觉故事的能力。然而,其输出内容主要与全球北方地区的文化感知对齐,往往导致对其他文化采用"局外人"的视角。因此,非西方社群需要付出额外努力才能生成具有文化特定性的故事。为应对这一挑战,我们开发了一款名为Kahani的视觉叙事工具,可为非西方文化生成基于文化根基的视觉故事。我们的工具利用现成模型GPT-4 Turbo和Stable Diffusion XL(SDXL),通过思维链(CoT)与T2I提示技术,从用户提示中捕捉文化语境,并生成对角色与场景构图的生动描述。为评估Kahani的有效性,我们与ChatGPT-4(搭载DALL-E3)进行了对比用户研究,邀请来自印度不同地区的参与者比较两款工具生成故事的文化相关性。用户研究中的定性与定量分析结果表明,Kahani生成的视觉故事比ChatGPT-4更具文化细微性。在36组对比中,Kahani有27组表现优于或持平于ChatGPT-4,能有效捕捉文化细微差异并融入更多文化特定项目(CSI),验证了其生成基于文化根基的视觉故事的能力。