Humans communicate with graphical sketches apart from symbolic languages. Primarily focusing on the latter, recent studies of emergent communication overlook the sketches; they do not account for the evolution process through which symbolic sign systems emerge in the trade-off between iconicity and symbolicity. In this work, we take the very first step to model and simulate this process via two neural agents playing a visual communication game; the sender communicates with the receiver by sketching on a canvas. We devise a novel reinforcement learning method such that agents are evolved jointly towards successful communication and abstract graphical conventions. To inspect the emerged conventions, we define three fundamental properties -- iconicity, symbolicity, and semanticity -- and design evaluation methods accordingly. Our experimental results under different controls are consistent with the observation in studies of human graphical conventions. Of note, we find that evolved sketches can preserve the continuum of semantics under proper environmental pressures. More interestingly, co-evolved agents can switch between conventionalized and iconic communication based on their familiarity with referents. We hope the present research can pave the path for studying emergent communication with the modality of sketches.

翻译：人类除了符号语言外，还使用图形草图进行交流。近期关于涌现沟通的研究主要关注后者，忽视了图形草图；它们未能解释符号系统在象似性与象征性权衡中涌现的演化过程。在本研究中，我们首次通过两个在视觉交流游戏中互动的智能体来建模并模拟这一过程：发送者在画布上绘制草图与接收者进行沟通。我们设计了一种新颖的强化学习方法，使得智能体在成功沟通与抽象图形约定中协同演化。为审视涌现的约定，我们定义了三个基本属性——象似性、象征性和语义性，并据此设计了相应的评估方法。在不同控制条件下的实验结果与人类图形约定研究中的观察结论一致。值得注意的是，我们发现演化出的草图在适当的环境压力下能够保持语义的连续性。更有趣的是，协同演化的智能体能够根据对所指对象的熟悉程度，在约定俗成与象似性沟通之间灵活切换。我们期望本研究能为探索基于草图模态的涌现沟通研究奠定基础。