The availability of Large Language Models (LLMs) which can generate code, has made it possible to create tools that improve developer productivity. Integrated development environments or IDEs which developers use to write software are often used as an interface to interact with LLMs. Although many such tools have been released, almost all of them focus on general-purpose programming languages. Domain-specific languages, such as those crucial for IT automation, have not received much attention. Ansible is one such YAML-based IT automation-specific language. Red Hat Ansible Lightspeed with IBM Watson Code Assistant, further referred to as Ansible Lightspeed, is an LLM-based service designed explicitly for natural language to Ansible code generation. In this paper, we describe the design and implementation of the Ansible Lightspeed service and analyze feedback from thousands of real users. We examine diverse performance indicators, classified according to both immediate and extended utilization patterns along with user sentiments. The analysis shows that the user acceptance rate of Ansible Lightspeed suggestions is higher than comparable tools that are more general and not specific to a programming language. This remains true even after we use much more stringent criteria for what is considered an accepted model suggestion, discarding suggestions which were heavily edited after being accepted. The relatively high acceptance rate results in higher-than-expected user retention and generally positive user feedback. This paper provides insights on how a comparatively small, dedicated model performs on a domain-specific language and more importantly, how it is received by users.

翻译：能够生成代码的大型语言模型（LLMs）的可用性，使得创建提升开发者生产力的工具成为可能。开发者编写软件时使用的集成开发环境（IDEs）通常被用作与LLMs交互的界面。尽管已有许多此类工具发布，但其中绝大多数专注于通用编程语言。领域特定语言（例如对IT自动化至关重要的语言）却鲜少受到关注。Ansible便是这样一种基于YAML的IT自动化专用语言。基于IBM Watson Code Assistant的Red Hat Ansible Lightspeed（下文简称Ansible Lightspeed）是一项专为自然语言到Ansible代码生成设计的LLM服务。本文描述了Ansible Lightspeed服务的设计与实现，并分析了数千名真实用户的反馈。我们考察了多种性能指标，这些指标根据即时使用模式和扩展使用模式以及用户情绪进行分类。分析表明，Ansible Lightspeed建议的用户接受率高于不具备编程语言特异性的同类通用工具。即使我们采用更严格的标准来定义“被接受的模型建议”（剔除接受后经大幅编辑的建议），这一结论依然成立。相对较高的接受率带来了超出预期的用户留存率及总体积极的用户反馈。本文揭示了相对小型的专用模型在领域特定语言上的表现，更重要的是，展示了用户对其的接受程度。