While large language models excel on high-resource multilingual tasks, low- and extremely low-resource Indic languages remain severely under-evaluated. We present IndicParam, a human-curated benchmark of over 13,000 multiple-choice questions covering 11 such languages (Nepali, Gujarati, Marathi, Odia as low-resource; Dogri, Maithili, Rajasthani, Sanskrit, Bodo, Santali, Konkani as extremely low-resource) plus Sanskrit-English code-mixed set. We evaluated 20 LLMs, both proprietary and open-weights, which reveals that even the top-performing \texttt{Gemini-2.5} reaches 58\% average accuracy, followed by \texttt{GPT-5} (45) and \texttt{DeepSeek-3.2} (43.1). We additionally label each question as knowledge-oriented or purely linguistic to discriminate factual recall from grammatical proficiency. Further, we assess the ability of LLMs to handle diverse question formats-such as list-based matching, assertion-reason pairs, and sequence ordering-alongside conventional multiple-choice questions. \benchmark\ provides insights into limitations of cross-lingual transfer and establishes a challenging benchmark for Indic languages. The dataset is available at https://huggingface.co/datasets/bharatgenai/IndicParam. Scripts to run benchmark are present at https://github.com/ayushbits/IndicParam.
翻译:尽管大语言模型在高资源多语言任务上表现出色,但低资源及极低资源的印度语言仍严重缺乏评估。我们提出了IndicParam,这是一个包含超过13,000道多项选择题的人工标注基准,涵盖11种此类语言(低资源语言:尼泊尔语、古吉拉特语、马拉地语、奥里亚语;极低资源语言:多格拉语、迈蒂利语、拉贾斯坦语、梵语、博多语、桑塔利语、孔卡尼语)以及一个梵语-英语语码混合集。我们评估了20个大语言模型,包括专有模型和开放权重模型,结果显示即使表现最佳的 \texttt{Gemini-2.5} 平均准确率也仅为58\%,其次是 \texttt{GPT-5}(45\%)和 \texttt{DeepSeek-3.2}(43.1\%)。此外,我们将每个问题标注为知识导向型或纯语言型,以区分事实回忆能力与语法熟练度。进一步地,我们评估了大语言模型处理多样化问题格式的能力——例如基于列表的匹配题、断言-理由对题和序列排序题——以及传统的多项选择题。\benchmark\ 揭示了跨语言迁移的局限性,并为印度语言建立了一个具有挑战性的基准。数据集可在 https://huggingface.co/datasets/bharatgenai/IndicParam 获取。运行基准的脚本位于 https://github.com/ayushbits/IndicParam。