While recent Arabic NLP benchmarks focus on scale, they often rely on synthetic or translated data which may benefit from deeper linguistic verification. We introduce ALPS (Arabic Linguistic & Pragmatic Suite), a native, expert-curated diagnostic challenge set probing Deep Semantics and Pragmatics, capabilities that complement specialized large-scale benchmarks. While broad-coverage benchmarks prioritize scale and multi-task coverage, ALPS targets the depth of linguistic understanding through 531 rigorously crafted questions across 15 tasks and 47 subtasks. We developed the dataset with deep expertise in Arabic linguistics, guaranteeing cultural authenticity and eliminating translation artifacts. Evaluating 23 diverse models (commercial, open-source, and Arabic-native) against a single-pass human performance (avg. 84.6% accuracy) and an expert-adjudicated oracle (99.2%), we reveal a critical dissociation: models achieve high fluency but fail on fundamental morpho-syntactic dependencies, with elevated error rates on morpho-syntactic dependencies (36.5% across diacritics-reliant tasks) compared to compositional semantics. While top commercial models (Gemini-3-flash at 94.2%) surpass the average single human, a substantial gap persists between commercial giants and Arabic-native models, with the best Arabic-specific model (Jais-2-70B at 83.6%) approaching but not matching human performance.

翻译：尽管近期的阿拉伯语自然语言处理基准测试侧重于规模，但它们通常依赖合成或翻译数据，这些数据可能需要更深入的语言学验证。我们推出了ALPS（阿拉伯语语言与语用测试套件），这是一个由专家精心构建的、基于母语语料的诊断性挑战集，旨在探究深层语义与语用推理能力，这些能力是对现有大规模专业基准测试的重要补充。广泛覆盖的基准测试优先考虑规模和多任务覆盖，而ALPS则通过涵盖15个任务、47个子任务的531道精心设计的问题，聚焦于语言理解的深度。我们凭借深厚的阿拉伯语语言学专业知识开发了该数据集，确保了文化真实性并消除了翻译痕迹。通过评估23个多样化模型（商业、开源及阿拉伯语原生模型），并与单次人类表现（平均准确率84.6%）及专家裁定最优结果（99.2%）进行对比，我们揭示了一个关键分离现象：模型虽能实现高流畅度，却在基础形态句法依存关系上表现不佳，在依赖变音符号的任务中形态句法依存错误率（整体36.5%）显著高于组合语义错误。尽管顶级商业模型（Gemini-3-flash达94.2%）超越了单个人类平均水平，但商业巨头与阿拉伯语原生模型之间仍存在显著差距，最佳阿拉伯语专用模型（Jais-2-70B为83.6%）接近但尚未达到人类表现水平。