Recent developments in large language models (LLMs) have shown promise in enhancing the capabilities of natural language processing (NLP). Despite these successes, there remains a dearth of research dedicated to the NLP problem-solving abilities of LLMs. To fill the gap in this area, we present a unique benchmarking dataset, NLPBench, comprising 378 college-level NLP questions spanning various NLP topics sourced from Yale University's prior final exams. NLPBench includes questions with context, in which multiple sub-questions share the same public information, and diverse question types, including multiple choice, short answer, and math. Our evaluation, centered on LLMs such as GPT-3.5/4, PaLM-2, and LLAMA-2, incorporates advanced prompting strategies like the chain-of-thought (CoT) and tree-of-thought (ToT). Our study reveals that the effectiveness of the advanced prompting strategies can be inconsistent, occasionally damaging LLM performance, especially in smaller models like the LLAMA-2 (13b). Furthermore, our manual assessment illuminated specific shortcomings in LLMs' scientific problem-solving skills, with weaknesses in logical decomposition and reasoning notably affecting results.

翻译：近年来，大语言模型的发展在处理自然语言处理任务方面展现出巨大潜力。尽管取得了这些成功，但针对大语言模型解决自然语言处理问题能力的研究仍然匮乏。为弥补这一领域的空白，我们提出了一个独特的基准测试数据集NLPBench，包含源自耶鲁大学历年期末考试的378个大学水平自然语言处理问题，涵盖多个自然语言处理主题。NLPBench包含带上下文的问题（多个子问题共享同一公共信息）以及多样化的题型，包括选择题、简答题和数学题。我们的评估以GPT-3.5/4、PaLM-2和LLAMA-2等大语言模型为中心，采用了链式思维和思维树等先进提示策略。研究表明，先进提示策略的效果可能不一致，有时甚至会损害大语言模型的性能，尤其是在较小模型（如LLAMA-2（13b））中表现更为明显。此外，我们的手工评估揭示了科学问题解决能力的具体缺陷，逻辑分解和推理方面的弱点显著影响了模型的表现。