Over past few years afterward the birth of ResNet, skip connection has become the defacto standard for the design of modern architectures due to its widespread adoption, easy optimization and proven performance. Prior work has explained the effectiveness of the skip connection mechanism from different perspectives. In this work, we deep dive into the model's behaviors with skip connections which can be formulated as a learnable Markov chain. An efficient Markov chain is preferred as it always maps the input data to the target domain in a better way. However, while a model is explained as a Markov chain, it is not guaranteed to be optimized following an efficient Markov chain by existing SGD-based optimizers which are prone to get trapped in local optimal points. In order to towards a more efficient Markov chain, we propose a simple routine of penal connection to make any residual-like model become a learnable Markov chain. Aside from that, the penal connection can also be viewed as a particular model regularization and can be easily implemented with one line of code in the most popular deep learning frameworks~\footnote{Source code: \url{https://github.com/densechen/penal-connection}}. The encouraging experimental results in multi-modal translation and image recognition empirically confirm our conjecture of the learnable Markov chain view and demonstrate the superiority of the proposed penal connection.

翻译：自ResNet问世以来的过去数年间，跳跃连接因其广泛采用、易于优化和已验证的性能，已成为现代架构设计中事实上的标准。先前的研究已从不同角度解释了跳跃连接机制的有效性。在本工作中，我们深入探究了可被建模为可学习马尔可夫链的跳跃连接模型的行为特性。高效的马尔可夫链是更优选择，因其总能以更佳方式将输入数据映射到目标域。然而，当模型被解释为马尔可夫链时，现有基于随机梯度下降的优化器（易陷入局部最优点）无法保证模型能遵循高效马尔可夫链进行优化。为趋近更优的马尔可夫链，我们提出一种简单的惩罚连接方案，使任意残差类模型成为可学习的马尔可夫链。此外，惩罚连接还可视为一种特定的模型正则化方法，且可通过一行代码在主流深度学习框架中轻松实现。在多模态翻译和图像识别任务中的优异实验结果，经验证了我们对可学习马尔可夫链视角的猜想，并展示了所提惩罚连接的优越性。