Node graph systems are used ubiquitously for material design in computer graphics. They allow the use of visual programming to achieve desired effects without writing code. As high-level design tools they provide convenience and flexibility, but mastering the creation of node graphs usually requires professional training. We propose an algorithm capable of generating multiple node graphs from different types of prompts, significantly lowering the bar for users to explore a specific design space. Previous work was limited to unconditional generation of random node graphs, making the generation of an envisioned material challenging. We propose a multi-modal node graph generation neural architecture for high-quality procedural material synthesis which can be conditioned on different inputs (text or image prompts), using a CLIP-based encoder. We also create a substantially augmented material graph dataset, key to improving the generation quality. Finally, we generate high-quality graph samples using a regularized sampling process and improve the matching quality by differentiable optimization for top-ranked samples. We compare our methods to CLIP-based database search baselines (which are themselves novel) and achieve superior or similar performance without requiring massive data storage. We further show that our model can produce a set of material graphs unconditionally, conditioned on images, text prompts or partial graphs, serving as a tool for automatic visual programming completion.

翻译：节点图系统在计算机图形学的材质设计中广泛应用。它们通过可视化编程实现预期效果而无需编写代码。作为高级设计工具，节点图系统兼具便捷性与灵活性，但掌握节点图的创建通常需要专业培训。我们提出一种能从不同类型提示生成多个节点图的算法，显著降低了用户探索特定设计空间的门槛。此前研究局限于无条件生成随机节点图，使得生成预期材质面临挑战。我们提出一种多模态节点图生成神经架构，用于高质量程序化材质合成，该架构基于CLIP编码器，可接收不同输入（文本或图像提示）的条件约束。同时，我们创建了大幅扩充的材质图数据集，这对提升生成质量至关重要。最终，通过正则化采样过程生成高质量图样本，并利用可微优化对排名靠前的样本进行匹配质量改进。我们将方法与基于CLIP的数据库搜索基线（该基线本身具有创新性）进行比较，在无需海量数据存储的情况下取得了更优或相当的性能。进一步研究表明，我们的模型能够无条件生成一组材质图，或基于图像、文本提示、局部图进行条件生成，可作为自动可视化编程补全工具。