Procedural Content Generation (PCG) algorithms enable the automatic generation of complex and diverse artifacts. However, they don't provide high-level control over the generated content and typically require domain expertise. In contrast, text-to-3D methods allow users to specify desired characteristics in natural language, offering a high amount of flexibility and expressivity. But unlike PCG, such approaches cannot guarantee functionality, which is crucial for certain applications like game design. In this paper, we present a method for generating functional 3D artifacts from free-form text prompts in the open-world game Minecraft. Our method, DreamCraft, trains quantized Neural Radiance Fields (NeRFs) to represent artifacts that, when viewed in-game, match given text descriptions. We find that DreamCraft produces more aligned in-game artifacts than a baseline that post-processes the output of an unconstrained NeRF. Thanks to the quantized representation of the environment, functional constraints can be integrated using specialized loss terms. We show how this can be leveraged to generate 3D structures that match a target distribution or obey certain adjacency rules over the block types. DreamCraft inherits a high degree of expressivity and controllability from the NeRF, while still being able to incorporate functional constraints through domain-specific objectives.

翻译：程序化内容生成（PCG）算法能够自动生成复杂多样的内容构件，但无法对生成内容提供高级控制，且通常需要领域专业知识。相比之下，文本到三维（Text-to-3D）方法允许用户用自然语言指定期望特征，具有高度灵活性和表现力。然而与PCG不同，此类方法无法保证功能性——这对游戏设计等特定应用至关重要。本文提出一种方法，可在开放世界游戏Minecraft中根据自由形式文本提示生成功能性3D构件。我们的方法DreamCraft通过训练量化神经辐射场（NeRF）来表征构件，使其在游戏内呈现时与给定文本描述相匹配。研究发现，DreamCraft生成的游戏内构件比基于后处理非约束NeRF输出的基线方法更为对齐。借助环境的量化表征，可通过专用损失项整合功能性约束。我们展示了如何利用这一特性生成符合目标分布或遵循特定方块类型邻接规则的3D结构。DreamCraft继承了NeRF的高表现力和可控性，同时通过领域特定目标函数融入功能性约束。