Text prompt is the most common way for human-generative AI (GenAI) communication. Though convenient, it is challenging to convey fine-grained and referential intent. One promising solution is to combine text prompts with precise GUI interactions, like brushing and clicking. However, there lacks a formal model to capture synergistic designs between prompts and interactions, hindering their comparison and innovation. To fill this gap, via an iterative and deductive process, we develop the Interaction-Augmented Instruction (IAI) model, a compact entity-relation graph formalizing how the combination of interactions and text prompts enhances human-GenAI communication. With the model, we distill twelve recurring and composable atomic interaction paradigms from prior tools, verifying our model's capability to facilitate systematic design characterization and comparison. Four usage scenarios further demonstrate the model's utility in applying, refining, and innovating these paradigms. These results illustrate the IAI model's descriptive, discriminative, and generative power for shaping future GenAI systems.

翻译：文本提示是人类与生成式AI（GenAI）沟通最常用的方式。虽然便捷，但传达细粒度和指代性意图具有挑战性。一种有前景的解决方案是将文本提示与精确的图形用户界面交互（如刷选和点击）相结合。然而，目前缺乏一个正式模型来捕捉提示与交互之间的协同设计，阻碍了它们的比较与创新。为填补这一空白，通过迭代与演绎过程，我们提出了交互增强指令（IAI）模型——一个紧凑的实体-关系图，用于形式化交互与文本提示的组合如何增强人类与GenAI的沟通。利用该模型，我们从现有工具中提炼出十二种可复现且可组合的原子交互范式，验证了我们的模型在促进系统性设计表征与比较方面的能力。四个应用场景进一步展示了该模型在应用、优化和创新这些范式方面的实用性。这些结果说明了IAI模型在塑造未来GenAI系统方面所具有的描述性、判别性和生成性能力。