To serve global users safely and productively, LLMs need culture-specific knowledge that might not be learned during pre-training. How do we find such knowledge that is (1) salient to in-group users, but (2) unknown to LLMs? The most common solutions are single-initiative: either researchers define challenging questions that users passively answer (traditional annotation), or users actively produce data that researchers structure as benchmarks (knowledge extraction). The process would benefit from mixed-initiative collaboration, where users guide the process to meaningfully reflect their cultures, and LLMs steer the process towards more challenging questions that meet the researcher's goals. We propose a mixed-initiative methodology called CultureCartography. Here, an LLM initializes annotation with questions for which it has low-confidence answers, making explicit both its prior knowledge and the gaps therein. This allows a human respondent to fill these gaps and steer the model towards salient topics through direct edits. We implement this methodology as a tool called CultureExplorer. Compared to a baseline where humans answer LLM-proposed questions, we find that CultureExplorer more effectively produces knowledge that leading models like DeepSeek R1 and GPT-4o are missing, even with web search. Fine-tuning on this data boosts the accuracy of Llama-3.1-8B by up to 19.2% on related culture benchmarks.

翻译：为安全高效地服务全球用户，大语言模型需要具备预训练阶段可能未习得的文化特定知识。如何发现这类既（1）对群体内用户显著，又（2）为大语言模型所未知的知识？现有解决方案多为单向模式：或由研究者定义挑战性问题供用户被动回答（传统标注），或由用户主动生成数据后由研究者构建为基准测试（知识提取）。混合主动协作模式将优化这一过程——用户可引导流程以有意义地反映其文化特征，而大语言模型则能导向更符合研究者目标的挑战性问题。我们提出名为CultureCartography的混合主动方法论：大语言模型以低置信度答案问题初始化标注，显式呈现其先验知识与知识缺口，使人类受访者能填补这些缺口并通过直接编辑引导模型聚焦显著主题。我们将该方法实现为CultureExplorer工具。相较于人类仅回答大语言模型所提问题的基线方案，CultureExplorer能更有效地生成DeepSeek R1、GPT-4o等领先模型（即使借助网络搜索）仍缺失的知识。基于此数据微调可使Llama-3.1-8B在相关文化基准测试上的准确率最高提升19.2%。