As the scaling of Large Language Models (LLMs) has dramatically enhanced their capabilities, there has been a growing focus on the alignment problem to ensure their responsible and ethical use. While existing alignment efforts predominantly concentrate on universal values such as the HHH principle, the aspect of culture, which is inherently pluralistic and diverse, has not received adequate attention. This work introduces a new benchmark, CDEval, aimed at evaluating the cultural dimensions of LLMs. CDEval is constructed by incorporating both GPT-4's automated generation and human verification, covering six cultural dimensions across seven domains. Our comprehensive experiments provide intriguing insights into the culture of mainstream LLMs, highlighting both consistencies and variations across different dimensions and domains. The findings underscore the importance of integrating cultural considerations in LLM development, particularly for applications in diverse cultural settings. Through CDEval, we aim to broaden the horizon of LLM alignment research by including cultural dimensions, thus providing a more holistic framework for the future development and evaluation of LLMs. This benchmark serves as a valuable resource for cultural studies in LLMs, paving the way for more culturally aware and sensitive models.

翻译：随着大型语言模型（LLMs）的规模化显著增强了其能力，如何确保其负责任和符合伦理的使用日益成为关注焦点，对齐问题也受到越来越多的重视。然而，现有的对齐工作主要集中于诸如HHH原则等普世价值，而文化这一本质上具有多元性和多样性的方面尚未得到充分关注。本研究引入了一个新的基准——CDEval，旨在评估大型语言模型的文化维度。CDEval通过结合GPT-4的自动生成和人工验证构建而成，涵盖了七个领域的六个文化维度。我们全面的实验为主流大型语言模型的文化特性提供了引人深思的见解，揭示了不同维度和领域之间的一致性与差异性。研究结果强调了在大型语言模型开发中融入文化考量的重要性，尤其是在多元文化背景下的应用。通过CDEval，我们旨在通过纳入文化维度来拓宽大型语言模型对齐研究的视野，从而为未来的模型开发与评估提供一个更全面的框架。该基准为大型语言模型的文化研究提供了宝贵资源，为开发更具文化意识和文化敏感性的模型铺平了道路。