Multilingual large language models (LLMs) often exhibit emergent 'shadow' capabilities in languages without official support, yet their performance on these languages remains uneven and under-measured. This is particularly acute for morphosyntactically rich minority languages such as Scottish Gaelic, where translation benchmarks fail to capture structural competence. We introduce GaelEval, the first multi-dimensional benchmark for Gaelic, comprising: (i) an expert-authored morphosyntactic MCQA task; (ii) a culturally grounded translation benchmark and (iii) a large-scale cultural knowledge Q&A task. Evaluating 19 LLMs against a fluent-speaker human baseline ($n=30$), we find that Gemini 3 Pro Preview achieves $83.3\%$ accuracy on the linguistic task, surpassing the human baseline ($78.1\%$). Proprietary models consistently outperform open-weight systems, and in-language (Gaelic) prompting yields a small but stable advantage (+$2.4\%$). On the cultural task, leading models exceed $90\%$ accuracy, though most systems perform worse under Gaelic prompting and absolute scores are inflated relative to the manual benchmark. Overall, GaelEval reveals that frontier models achieve above-human performance on several dimensions of Gaelic grammar, demonstrates the effect of Gaelic prompting and shows a consistent performance gap favouring proprietary over open-weight models.

翻译：摘要：多语言大语言模型（LLMs）在未获得官方支持的语言中常展现出涌现的“暗含”能力，但其在这些语言上的表现仍参差不齐且缺乏充分测量。这一问题在形态句法丰富的少数民族语言（如苏格兰盖尔语）中尤为突出，因为翻译基准测试无法捕捉其结构能力。我们提出GaelEval——首个面向盖尔语的多维基准评测，包括：(i) 专家编写的形态句法多项选择问答任务；(ii) 基于文化语境的翻译基准测试；(iii) 大规模文化知识问答任务。通过对19个LLMs与流利母语者人类基线（n=30）的评估，我们发现Gemini 3 Pro Preview在语言任务上达到83.3%的准确率，超越了人类基线（78.1%）。专有模型持续优于开源模型，且使用盖尔语提示会产生虽小但稳定的优势（+2.4%）。在文化任务中，领先模型准确率超过90%，但多数系统在盖尔语提示下表现更差，且绝对得分相较于人工基准测试存在虚高。总体而言，GaelEval揭示了前沿模型在盖尔语语法的多个维度上达到超越人类水平的表现，展示了盖尔语提示的效果，并显示了专有模型始终优于开源模型的性能差距。