Large Language Models (LLMs) play a pivotal role in both academic research and broader societal applications. LLMs are increasingly used in software testing activities such as test case generation, selection, and repair. However, several important questions remain: (1) do LLMs possess enough information about software testing principles to perform software testing tasks effectively? (2) do LLMs possess sufficient conceptual understanding of software testing to answer software testing questions under metamorphic transformations? and (3) do certain properties of software testing questions influence the performance of LLMs? To answer these questions, this study evaluates 60 multimodal language models from both commercial vendors and the open-source community. The evaluation is performed using 30 sample exams of different types (core foundation, core advanced, specialist, and expert) from the International Software Testing Qualifications Board (ISTQB), which are used to assess the competence of human testers. In total, each model is evaluated on 1,171 questions. Furthermore, to ensure sufficient conceptual understanding, the models are also tested on exam questions transformed using context-preserving metamorphic techniques. Two models passed all the certifications by scoring at least 65% in all of the 30 certification exams, with commercial models generally outperforming open-source ones. We analyze the reasons behind incorrect answers and provide recommendations for improving the design of software testing certification exams.

翻译：大型语言模型在学术研究和更广泛的社会应用中均发挥着关键作用。LLMs越来越多地被用于软件测试活动，如测试用例生成、选择和修复。然而，仍存在几个重要问题：（1）LLMs是否具备足够的软件测试原理知识，以有效执行软件测试任务？（2）LLMs是否对软件测试有足够的概念性理解，使其能够在变形变换下回答软件测试问题？（3）软件测试问题的某些属性是否会影响LLMs的性能？为回答这些问题，本研究评估了来自商业供应商和开源社区的60个多模态语言模型。评估使用了国际软件测试资格委员会（ISTQB）的30份不同类型的样题（核心基础、核心高级、专业和专家），这些样题用于评估人类测试人员的能力。总计，每个模型在1171个问题上进行了评估。此外，为确保充分的概念性理解，模型还接受了使用保持语境的变形技术转化的试题测试。两个模型在所有30份认证考试中均获得至少65%的分数，通过了所有认证，其中商业模型普遍优于开源模型。我们分析了错误答案背后的原因，并为改进软件测试认证考试的设计提供了建议。