Planning from demonstrations has shown promising results with the advances of deep neural networks. One of the most popular real-world applications is automated handwriting using a robotic manipulator. Classically it is simplified as a two-dimension problem. This representation is suitable for elementary drawings, but it is not sufficient for Japanese calligraphy or complex work of art where the orientation of a pen is part of the user expression. In this study, we focus on automated planning of Japanese calligraphy using a three-dimension representation of the trajectory as well as the rotation of the pen tip, and propose a novel deep imitation learning neural network that learns from expert demonstrations through a combination of images and pose data. The network consists of a combination of variational auto-encoder, bi-directional LSTM, and Multi-Layer Perceptron (MLP). Experiments are conducted in a progressive way, and results demonstrate that the proposed approach is successful in completion of tasks for real-world robots, overcoming the distribution shift problem in imitation learning. The source code and dataset will be public.

翻译：通过深度神经网络的进步，基于示教的规划方法已展现出令人瞩目的成果。其中最具代表性的实际应用之一是利用机械臂实现自动化手写。传统上，该问题被简化为二维平面问题。这种表示方法适用于基础绘图，但对于需要体现笔尖方向作为用户表达要素的日本书法或复杂艺术品而言并不充分。本研究聚焦于采用三维轨迹表征与笔尖端旋转信息的日本书法自动化规划方法，提出一种新颖的深度模仿学习神经网络，该网络通过图像与姿态数据的融合学习专家示教。网络架构由变分自编码器、双向长短期记忆网络与多层感知机组合而成。通过渐进式实验验证，结果表明所提方法能成功完成真实机器人任务，有效克服模仿学习中的分布偏移问题。相关源代码与数据集将予以公开。