Multilingual large language models (LLMs) today may not necessarily provide culturally appropriate and relevant responses to its Filipino users. We introduce Kalahi, a cultural LLM evaluation suite collaboratively created by native Filipino speakers. It is composed of 150 high-quality, handcrafted and nuanced prompts that test LLMs for generations that are relevant to shared Filipino cultural knowledge and values. Strong LLM performance in Kalahi indicates a model's ability to generate responses similar to what an average Filipino would say or do in a given situation. We conducted experiments on LLMs with multilingual and Filipino language support. Results show that Kalahi, while trivial for Filipinos, is challenging for LLMs, with the best model answering only 46.0% of the questions correctly compared to native Filipino performance of 89.10%. Thus, Kalahi can be used to accurately and reliably evaluate Filipino cultural representation in LLMs.

翻译：当前的多语言大语言模型（LLM）可能无法为其菲律宾用户提供文化上恰当且相关的回应。我们推出了Kalahi，这是一个由菲律宾母语者协作创建的文化LLM评估套件。它包含150个高质量、手工制作且细致入微的提示，用于测试LLM生成与菲律宾共享文化知识和价值观相关的内容的能力。LLM在Kalahi上的优异表现表明模型能够生成与普通菲律宾人在特定情境下可能言行相似的回应。我们对支持多语言及菲律宾语的LLM进行了实验。结果显示，Kalahi对菲律宾人而言虽简单，但对LLM却具有挑战性，最佳模型仅能正确回答46.0%的问题，而菲律宾母语者的正确率为89.10%。因此，Kalahi可用于准确可靠地评估LLM中的菲律宾文化表征。