Large Language Models (LLMs) are increasingly being used to generate synthetic data for training and evaluating models. However, it is unclear whether they can generate a good quality of question answering (QA) dataset that incorporates knowledge and cultural nuance embedded in a language, especially for low-resource languages. In this study, we investigate the effectiveness of using LLMs in generating culturally relevant commonsense QA datasets for Indonesian and Sundanese languages. To do so, we create datasets for these languages using various methods involving both LLMs and human annotators. Our experiments show that the current best-performing LLM, GPT-4 Turbo, is capable of generating questions with adequate knowledge in Indonesian but not in Sundanese, highlighting the performance discrepancy between medium- and lower-resource languages. We also benchmark various LLMs on our generated datasets and find that they perform better on the LLM-generated datasets compared to those created by humans.

翻译：大型语言模型（LLM）正越来越多地被用于生成用于训练和评估模型的合成数据。然而，目前尚不清楚它们能否生成融入语言所蕴含的知识和文化细微差别的优质问答数据集，尤其是对于低资源语言。在本研究中，我们探讨了利用LLM生成印尼语和巽他语文化相关常识问答数据集的有效性。为此，我们采用涉及LLM和人工标注者的多种方法为这些语言创建数据集。实验表明，当前性能最佳的LLM——GPT-4 Turbo——能够生成具有足够印尼语知识的问题，但巽他语则不然，这凸显了中等资源和低资源语言之间的性能差异。我们还对各种LLM在我们生成的数据集上进行了基准测试，发现它们在我们生成的数据集上的表现优于人类创建的数据集。