Large Language Model (LLM) assistants, such as ChatGPT, have emerged as potential alternatives to search methods for helping users navigate complex, feature-rich software. LLMs use vast training data from domain-specific texts, software manuals, and code repositories to mimic human-like interactions, offering tailored assistance, including step-by-step instructions. In this work, we investigated LLM-generated software guidance through a within-subject experiment with 16 participants and follow-up interviews. We compared a baseline LLM assistant with an LLM optimized for particular software contexts, SoftAIBot, which also offered guidelines for constructing appropriate prompts. We assessed task completion, perceived accuracy, relevance, and trust. Surprisingly, although SoftAIBot outperformed the baseline LLM, our results revealed no significant difference in LLM usage and user perceptions with or without prompt guidelines and the integration of domain context. Most users struggled to understand how the prompt's text related to the LLM's responses and often followed the LLM's suggestions verbatim, even if they were incorrect. This resulted in difficulties when using the LLM's advice for software tasks, leading to low task completion rates. Our detailed analysis also revealed that users remained unaware of inaccuracies in the LLM's responses, indicating a gap between their lack of software expertise and their ability to evaluate the LLM's assistance. With the growing push for designing domain-specific LLM assistants, we emphasize the importance of incorporating explainable, context-aware cues into LLMs to help users understand prompt-based interactions, identify biases, and maximize the utility of LLM assistants.

翻译：大语言模型（LLM）助手（如ChatGPT）已作为搜索方法的潜在替代方案出现，用于帮助用户操作复杂且功能丰富的软件。LLM利用来自领域特定文本、软件手册和代码仓库的海量训练数据，模拟类人交互，提供包括逐步指令在内的定制化帮助。在本研究中，我们通过一项包含16名参与者的受试者内实验及后续访谈，调查了LLM生成的软件指导。我们将基线LLM助手与针对特定软件上下文优化的LLM（SoftAIBot）进行了比较，后者还提供了构建合适提示的指南。我们评估了任务完成度、感知准确性、相关性和信任度。令人惊讶的是，尽管SoftAIBot的表现优于基线LLM，但结果显示，无论是否提供提示指南及领域上下文集成，LLM的使用率和用户感知均无显著差异。大多数用户难以理解提示文本与LLM响应之间的关联，并常常逐字遵循LLM的建议，即使这些建议是错误的。这导致在将LLM建议应用于软件任务时出现困难，任务完成率较低。我们的详细分析还显示，用户对LLM响应中的不准确性毫无察觉，表明其缺乏软件专业知识与评估LLM帮助能力之间存在差距。随着设计领域特定LLM助手的呼声日益高涨，我们强调将可解释的、上下文感知的线索纳入LLM的重要性，以帮助用户理解基于提示的交互、识别偏差，并最大化LLM助手的效用。