The availability of Large Language Models (LLMs) which can generate code, has made it possible to create tools that improve developer productivity. Integrated development environments or IDEs which developers use to write software are often used as an interface to interact with LLMs. Although many such tools have been released, almost all of them focus on general-purpose programming languages. Domain-specific languages, such as those crucial for Information Technology (IT) automation, have not received much attention. Ansible is one such YAML-based IT automation-specific language. Ansible Lightspeed is an LLM-based service designed explicitly to generate Ansible YAML, given natural language prompt. In this paper, we present the design and implementation of the Ansible Lightspeed service. We then evaluate its utility to developers using diverse indicators, including extended utilization, analysis of user edited suggestions, as well as user sentiments analysis. The evaluation is based on data collected for 10,696 real users including 3,910 returning users. The code for Ansible Lightspeed service and the analysis framework is made available for others to use. To our knowledge, our study is the first to involve thousands of users of code assistants for domain-specific languages. We are also the first code completion tool to present N-Day user retention figures, which is 13.66\% on Day 30. We propose an improved version of user acceptance rate, called Strong Acceptance rate, where a suggestion is considered accepted only if less than $50\%$ of it is edited and these edits do not change critical parts of the suggestion. By focusing on Ansible, Lightspeed is able to achieve a strong acceptance rate of 49.08\% for multi-line Ansible task suggestions. With our findings we provide insights into the effectiveness of small, dedicated models in a domain-specific context.

翻译：能够生成代码的大型语言模型（LLMs）的出现，使得开发能够提升开发者生产力的工具成为可能。开发者用于编写软件的集成开发环境（IDEs）常被用作与LLMs交互的界面。尽管此类工具已发布众多，但几乎所有工具都专注于通用编程语言。对于特定领域语言，例如信息技术（IT）自动化中至关重要的语言，则未受到足够关注。Ansible 正是一种基于 YAML 的 IT 自动化专用语言。Ansible Lightspeed 是一项基于 LLM 的服务，专门设计用于根据自然语言提示生成 Ansible YAML 代码。本文介绍了 Ansible Lightspeed 服务的设计与实现。随后，我们通过多种指标评估其对开发者的实用性，包括持续使用情况分析、用户编辑建议的分析以及用户情感分析。该评估基于从 10,696 名真实用户（包括 3,910 名回头用户）收集的数据。Ansible Lightspeed 服务及其分析框架的代码已开源供他人使用。据我们所知，本研究首次涉及数千名特定领域语言代码助手用户。我们也是首个公布 N-Day 用户留存率的代码补全工具，其第 30 天的留存率为 13.66%。我们提出了一种改进版的用户接受率，称为强接受率，其中只有当建议的编辑部分少于 $50\%$ 且这些编辑未改变建议的关键部分时，该建议才被视为被接受。通过专注于 Ansible，Lightspeed 在多行 Ansible 任务建议上实现了 49.08% 的强接受率。基于我们的发现，我们深入探讨了在特定领域背景下，小型专用模型的有效性。