Recent advancements in Large Language Models (LLMs) have significantly influenced the landscape of language and speech research. Despite this progress, these models lack specific benchmarking against state-of-the-art (SOTA) models tailored to particular languages and tasks. LAraBench addresses this gap for Arabic Natural Language Processing (NLP) and Speech Processing tasks, including sequence tagging and content classification across different domains. We utilized models such as GPT-3.5-turbo, GPT-4, BLOOMZ, Jais-13b-chat, Whisper, and USM, employing zero and few-shot learning techniques to tackle 33 distinct tasks across 61 publicly available datasets. This involved 98 experimental setups, encompassing ~296K data points, ~46 hours of speech, and 30 sentences for Text-to-Speech (TTS). This effort resulted in 330+ sets of experiments. Our analysis focused on measuring the performance gap between SOTA models and LLMs. The overarching trend observed was that SOTA models generally outperformed LLMs in zero-shot learning, with a few exceptions. Notably, larger computational models with few-shot learning techniques managed to reduce these performance gaps. Our findings provide valuable insights into the applicability of LLMs for Arabic NLP and speech processing tasks.

翻译：近年来，大型语言模型（LLM）的进展显著影响了语言与语音研究领域。然而，这些模型缺乏针对特定语言和任务的、与当前最优（SOTA）模型进行对比的专项基准测试。LAraBench填补了这一空白，聚焦阿拉伯语自然语言处理（NLP）和语音处理任务，涵盖不同领域的序列标注与内容分类。我们采用GPT-3.5-turbo、GPT-4、BLOOMZ、Jais-13b-chat、Whisper和USM等模型，结合零样本与少样本学习技术，在61个公开数据集上处理了33项不同任务。实验设置共98组，涉及约29.6万个数据点、约46小时语音数据及30句文本转语音（TTS）样本，最终完成330余组实验。我们重点分析了SOTA模型与LLM之间的性能差距。总体趋势表明，SOTA模型在零样本学习中通常优于LLM，仅存在少数例外。值得注意的是，采用少样本学习技术的大规模计算模型能够缩小这一性能差距。本研究的发现为LLM在阿拉伯语NLP及语音处理任务中的适用性提供了重要洞见。