In recent years, large language models (LLMs) have seen rapid advancements, significantly impacting various fields such as natural language processing, and software engineering. These LLMs, exemplified by OpenAI's ChatGPT, have revolutionized the way we approach language understanding and generation tasks. However, in contrast to traditional software development practices, LLM development introduces new challenges for AI developers in design, implementation, and deployment. These challenges span different areas (such as prompts, APIs, and plugins), requiring developers to navigate unique methodologies and considerations specific to LLM development. Despite the profound influence of LLMs, to the best of our knowledge, these challenges have not been thoroughly investigated in previous empirical studies. To fill this gap, we present the first comprehensive study on understanding the challenges faced by LLM developers. Specifically, we crawl and analyze 29,057 relevant questions from a popular OpenAI developer forum. We first examine their popularity and difficulty. After manually analyzing 2,364 sampled questions, we construct a taxonomy of challenges faced by LLM developers. Based on this taxonomy, we summarize a set of findings and actionable implications for LLM-related stakeholders, including developers and providers (especially the OpenAI organization).

翻译：近年来，大型语言模型（LLMs）取得了快速进展，显著影响了自然语言处理、软件工程等多个领域。以OpenAI的ChatGPT为代表的这些LLMs，彻底改变了我们处理语言理解与生成任务的方式。然而，与传统软件开发实践相比，LLM开发为AI开发者在设计、实现和部署方面带来了新的挑战。这些挑战涉及不同领域（如提示词、API和插件），要求开发者掌握LLM开发特有的方法论和注意事项。尽管LLMs影响深远，但据我们所知，这些挑战尚未在以往的实证研究中得到深入探讨。为填补这一空白，我们首次开展了全面研究以理解LLM开发者面临的挑战。具体而言，我们从热门的OpenAI开发者论坛中爬取并分析了29,057个相关问题。我们首先考察了这些问题的流行度和难度。在人工分析2,364个抽样问题后，我们构建了LLM开发者挑战的分类体系。基于该分类体系，我们总结了一系列研究发现，并为LLM相关利益方（包括开发者和供应商，特别是OpenAI组织）提出了可操作的启示。