Some of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the Language-Models-as-a-Service (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case of open-source models, such closed-off language models present specific challenges for evaluating, benchmarking, and testing them. This paper has two goals: on the one hand, we delineate how the aforementioned challenges act as impediments to the accessibility, replicability, reliability, and trustworthiness of LMaaS. We systematically examine the issues that arise from a lack of information about language models for each of these four aspects. We conduct a detailed analysis of existing solutions and put forth a number of considered recommendations, and highlight the directions for future advancements. On the other hand, it serves as a comprehensive resource for existing knowledge on current, major LMaaS, offering a synthesized overview of the licences and capabilities their interfaces offer.

翻译：当前一些最强大的语言模型属于专有系统，仅能通过（通常受限的）网页或软件编程接口进行访问，这便是语言模型即服务（LMaaS）范式。与开源模型等具备完整模型访问权限的场景不同，此类封闭式语言模型在评估、基准测试和检验方面面临着特殊挑战。本文具有双重目标：一方面，我们阐述了上述挑战如何阻碍LMaaS的可访问性、可复现性、可靠性和可信赖性，系统性地剖析了因语言模型信息缺失在这四个方面引发的问题，并对现有解决方案进行了详细分析，提出了若干可行性建议，指明了未来发展方向；另一方面，本文作为当前主流LMaaS现有知识的综合性资源，对其接口所提供的许可证和功能进行了综合概述。