With their exceptional natural language processing capabilities, tools based on Large Language Models (LLMs) like ChatGPT and Co-Pilot have swiftly become indispensable resources in the software developer's toolkit. While recent studies suggest the potential productivity gains these tools can unlock, users still encounter drawbacks, such as generic or incorrect answers. Additionally, the pursuit of improved responses often leads to extensive prompt engineering efforts, diverting valuable time from writing code that delivers actual value. To address these challenges, a new breed of tools, built atop LLMs, is emerging. These tools aim to mitigate drawbacks by employing techniques like fine-tuning or enriching user prompts with contextualized information. In this paper, we delve into the lessons learned by a software development team venturing into the creation of such a contextualized LLM-based application, using retrieval-based techniques, called CodeBuddy. Over a four-month period, the team, despite lacking prior professional experience in LLM-based applications, built the product from scratch. Following the initial product release, we engaged with the development team responsible for the code generative components. Through interviews and analysis of the application's issue tracker, we uncover various intriguing challenges that teams working on LLM-based applications might encounter. For instance, we found three main group of lessons: LLM-based lessons, User-based lessons, and Technical lessons. By understanding these lessons, software development teams could become better prepared to build LLM-based applications.

翻译：凭借其卓越的自然语言处理能力，基于大型语言模型（LLM）的工具（如ChatGPT和Co-Pilot）已迅速成为软件开发人员工具包中不可或缺的资源。尽管近期研究表明这些工具能带来的潜在生产力提升，用户仍会遇到诸如泛泛或错误答案等缺陷。此外，为追求更优回答往往需要投入大量提示词工程工作，从而挤占了编写真正有价值代码的宝贵时间。为应对这些挑战，一种基于LLM构建的新型工具正在兴起。这类工具通过采用微调或用情境化信息丰富用户提示词等技术手段来缓解缺陷。本文深入探讨了一个软件开发团队在尝试使用基于检索技术创建此类情境化LLM应用（名为CodeBuddy）过程中总结的经验教训。尽管团队此前缺乏LLM应用开发的专业经验，仍历时四个月从零构建了该产品。在初始版本发布后，我们与负责代码生成组件的开发团队进行了交流。通过访谈和对应用问题追踪系统的分析，我们揭示了工作于LLM应用的团队可能遇到的各类有趣挑战。例如，我们发现主要存在三类经验教训：基于LLM的经验教训、基于用户的经验教训以及技术层面的经验教训。理解这些经验教训有助于软件开发团队更好地构建基于LLM的应用。