The availability of Large Language Models (LLMs) which can generate code, has made it possible to create tools that improve developer productivity. Integrated development environments or IDEs which developers use to write software are often used as an interface to interact with LLMs. Although many such tools have been released, almost all of them focus on general-purpose programming languages. Domain-specific languages, such as those crucial for Information Technology (IT) automation, have not received much attention. Ansible is one such YAML-based IT automation-specific language. Ansible Lightspeed is an LLM-based service designed explicitly to generate Ansible YAML, given natural language prompt. In this paper, we present the design and implementation of the Ansible Lightspeed service. We then evaluate its utility to developers using diverse indicators, including extended utilization, analysis of user edited suggestions, as well as user sentiments analysis. The evaluation is based on data collected for 10,696 real users including 3,910 returning users. The code for Ansible Lightspeed service and the analysis framework is made available for others to use. To our knowledge, our study is the first to involve thousands of users of code assistants for domain-specific languages. We are also the first code completion tool to present N-Day user retention figures, which is 13.66% on Day 30. We propose an improved version of user acceptance rate, called Strong Acceptance rate, where a suggestion is considered accepted only if less than 50% of it is edited and these edits do not change critical parts of the suggestion. By focusing on Ansible, Lightspeed is able to achieve a strong acceptance rate of 49.08% for multi-line Ansible task suggestions. With our findings we provide insights into the effectiveness of small, dedicated models in a domain-specific context.

翻译：能够生成代码的大型语言模型（LLMs）的出现，使得开发提升开发者生产力的工具成为可能。开发者用于编写软件的集成开发环境（IDEs）常被用作与LLMs交互的界面。尽管已发布许多此类工具，但几乎所有工具都专注于通用编程语言。而领域特定语言，例如对信息技术（IT）自动化至关重要的语言，尚未得到太多关注。Ansible就是一种基于YAML的IT自动化专用语言。Ansible Lightspeed是一项专门设计用于根据自然语言提示生成Ansible YAML的基于LLM的服务。本文介绍了Ansible Lightspeed服务的设计与实现。随后，我们通过多种指标评估其对开发者的实用性，包括持续使用情况分析、用户编辑建议的分析以及用户情感分析。该评估基于10,696名真实用户（其中包括3,910名回头用户）收集的数据。Ansible Lightspeed服务及其分析框架的代码已开源供他人使用。据我们所知，我们的研究首次涉及了数千名使用领域特定语言代码助手的用户。我们也是首个公布N日用户留存率的代码补全工具，其第30日留存率为13.66%。我们提出了一种改进版的用户接受率，称为强接受率，其中只有当建议被编辑的部分少于50%且这些编辑未改变建议的关键部分时，该建议才被视为被接受。通过专注于Ansible，Lightspeed在生成多行Ansible任务建议时实现了49.08%的强接受率。基于我们的发现，我们深入探讨了在领域特定背景下小型专用模型的有效性。