As the scaling of Large Language Models (LLMs) has dramatically enhanced their capabilities, there has been a growing focus on the alignment problem to ensure their responsible and ethical use. While existing alignment efforts predominantly concentrate on universal values such as the HHH principle, the aspect of culture, which is inherently pluralistic and diverse, has not received adequate attention. This work introduces a new benchmark, CDEval, aimed at evaluating the cultural dimensions of LLMs. CDEval is constructed by incorporating both GPT-4's automated generation and human verification, covering six cultural dimensions across seven domains. Our comprehensive experiments provide intriguing insights into the culture of mainstream LLMs, highlighting both consistencies and variations across different dimensions and domains. The findings underscore the importance of integrating cultural considerations in LLM development, particularly for applications in diverse cultural settings. Through CDEval, we aim to broaden the horizon of LLM alignment research by including cultural dimensions, thus providing a more holistic framework for the future development and evaluation of LLMs. This benchmark serves as a valuable resource for cultural studies in LLMs, paving the way for more culturally aware and sensitive models.

翻译：随着大语言模型（LLMs）规模的扩展显著提升了其能力，确保其负责任和合乎伦理使用的对齐问题日益受到关注。尽管现有的对齐工作主要聚焦于HHH原则等普世价值，但本质上具有多元性和多样性的文化维度尚未得到充分重视。本文提出一项新基准CDEval，旨在评估大语言模型的文化维度。CDEval结合了GPT-4的自动生成与人工验证，覆盖了七个领域的六个文化维度。我们的综合实验揭示了主流大语言模型文化特征的有趣见解，凸显了不同维度与领域之间的一致性与差异性。研究结果强调了将文化考量纳入大语言模型开发的重要性，特别是针对多元文化场景的应用。通过CDEval，我们希望将文化维度纳入大语言模型对齐研究的视野，从而为未来模型开发与评估提供更全面的框架。该基准可作为大语言模型文化研究的重要资源，为构建更具文化意识与敏感度的模型铺平道路。