Bioscientists frequently seek to visualize the biological systems they have empirically characterized and reported in the literature. Realizing such visualizations requires biological structure modeling, an inherently complex process that demands both biological and geometric understanding. This paper addresses the problem of constructing such 3D models for visualization. In this paper, we introduce a novel agent framework that mitigates the challenges of operating 3D modeling software by transforming user inputs, including natural language descriptions, research publication content, and textual descriptions of the existing objects and structures in the current scene, into modeling operations in a structured JSON format and final 3D results. The major technical contribution lies in the collaborative agent design that simultaneously supports model planning, execution, and novel user interaction design, such as interactive modeling execution and dynamic widget generation that fuse text and mouse interaction within the chat window. The framework further incorporates a customized modeling memory to enhance user interaction, featuring components such as personalized memory management, feedback collection, and skill library design. This modeling memory is leveraged to enable improved 3D modeling performance over time. The quantitative evaluation on our collected dataset showcases the effectiveness of our framework. We also develop a prototype tool, Chat Modeling, and demonstrate its usage through two modeling case studies. Our user study and expert interviews highlight the potential of our approach for use in scientific workflows.

翻译：摘要：生物科学家常需将其实验表征并在文献中报道的生物系统进行可视化。实现此类可视化需要生物结构建模——这一过程本质复杂，既需要生物学理解，又需要几何学认知。本文聚焦于构建用于可视化的三维模型问题。我们提出了一种新型智能体框架，通过将用户输入（包括自然语言描述、研究论文内容、当前场景中现有对象与结构的文本描述）转化为结构化JSON格式的建模操作及最终三维结果，从而缓解操作三维建模软件的挑战。主要技术贡献在于协作式智能体设计，该设计同时支持模型规划、执行以及新型用户交互设计（如交互式建模执行与动态控件生成，可在聊天窗口内融合文本与鼠标交互）。该框架进一步整合了定制化建模记忆以增强用户交互，其组件包括个性化记忆管理、反馈收集及技能库设计。这种建模记忆被用于随时间推移提升三维建模性能。在我们收集的数据集上的定量评估展示了该框架的有效性。我们还开发了原型工具Chat Modeling，并通过两个建模案例研究演示其用法。用户研究与专家访谈突显了该方法在科学工作流中的应用潜力。