Stroke-based rendering aims to recreate an image with a set of strokes. Most existing methods render complex images using an uniform-block-dividing strategy, which leads to boundary inconsistency artifacts. To solve the problem, we propose Compositional Neural Painter, a novel stroke-based rendering framework which dynamically predicts the next painting region based on the current canvas, instead of dividing the image plane uniformly into painting regions. We start from an empty canvas and divide the painting process into several steps. At each step, a compositor network trained with a phasic RL strategy first predicts the next painting region, then a painter network trained with a WGAN discriminator predicts stroke parameters, and a stroke renderer paints the strokes onto the painting region of the current canvas. Moreover, we extend our method to stroke-based style transfer with a novel differentiable distance transform loss, which helps preserve the structure of the input image during stroke-based stylization. Extensive experiments show our model outperforms the existing models in both stroke-based neural painting and stroke-based stylization. Code is available at https://github.com/sjtuplayer/Compositional_Neural_Painter

翻译：基于笔触的渲染旨在用一组笔触重建图像。现有方法大多采用均匀分块策略渲染复杂图像，这会导致边界不一致的伪影问题。为解决该问题，我们提出组合式神经绘画器（Compositional Neural Painter），这是一种新颖的基于笔触的渲染框架，能够基于当前画布动态预测下一个绘画区域，而非将图像平面均匀划分为绘画区域。我们从空白画布开始，将绘画过程分为多个步骤。在每个步骤中，首先由经过阶段式强化学习（phasic RL）策略训练的组合器网络预测下一个绘画区域，再由经过WGAN判别器训练的绘画器网络预测笔触参数，最后由笔触渲染器将笔触绘制到当前画布的绘画区域上。此外，我们将该方法扩展到基于笔触的风格迁移，提出一种新颖的可微距离变换损失函数，该函数有助于在基于笔触的风格化过程中保持输入图像的结构。大量实验表明，我们的模型在基于笔触的神经绘画和基于笔触的风格化任务中均优于现有模型。代码已开源至https://github.com/sjtuplayer/Compositional_Neural_Painter