Despite demonstrating robust capabilities in performing tasks related to general-domain data-operation tasks, Large Language Models (LLMs) may exhibit shortcomings when applied to domain-specific tasks. We consider the design of domain-specific AI-powered data analysis tools from two dimensions: interaction and user agency. We implemented two design probes that fall on the two ends of the two dimensions: an open-ended high agency (OHA) prototype and a structured low agency (SLA) prototype. We conducted an interview study with nine data scientists to investigate (1) how users perceived the LLM outputs for data analysis assistance, and (2) how the two test design probes, OHA and SLA, affected user behavior, performance, and perceptions. Our study revealed insights regarding participants' interactions with LLMs, how they perceived the results, and their desire for explainability concerning LLM outputs, along with a noted need for collaboration with other users, and how they envisioned the utility of LLMs in their workflow.

翻译：尽管大语言模型（LLMs）在通用领域数据操作任务中展现出强大能力，但在应用于领域特定任务时仍可能显现不足。我们从交互模式与用户自主权两个维度出发，设计了针对特定领域的人工智能数据分析工具。基于这两个维度的两端，我们构建了两种设计原型：开放式高自主权（OHA）原型与结构化低自主权（SLA）原型。通过对九位数据科学家进行访谈研究，我们考察了：（1）用户如何感知LLM提供的数据分析辅助结果；（2）OHA与SLA两种测试设计原型如何影响用户行为、表现与认知。研究揭示了参与者与LLM的交互方式、他们对结果的认知模式、对LLM输出可解释性的需求、与其他用户协作的明确诉求，以及他们如何设想LLM在工作流程中的实际效用。