Batching is an essential technique to improve computation efficiency in deep learning frameworks. While batch processing for models with static feed-forward computation graphs is straightforward to implement, batching for dynamic computation graphs such as syntax trees or social network graphs is challenging due to variable computation graph structure across samples. Through simulation and analysis of a Tree-LSTM model, we show the key trade-off between graph analysis time and batching effectiveness in dynamic batching. Based on this finding, we propose a dynamic batching method as an extension to MXNet Gluon's just-in-time compilation (JIT) framework. We show empirically that our method yields up to 6.25 times speed-up on a common dynamic workload, a tree-LSTM model for the semantic relatedness task.

翻译：捆绑是提高深层学习框架计算效率的一项必要技术。 虽然静态进料前计算图表模型的批量处理简单易行,但由于各样本之间可变的计算图表结构,如语法树或社交网络图表等动态计算图表的批量处理具有挑战性。通过模拟和分析树-LSTM模型,我们展示了图表分析时间与动态批量效率之间的关键权衡。基于这一发现,我们提出了动态批量方法,作为MXNet Gluon即时汇编(JIT)框架的延伸。我们从经验上表明,我们的方法在共同的动态工作量上是树-LSTM模型(树-LSTM模型)速度的6.25倍,用于与语义相关的任务。