A recent paper proposed Dynamic Tanh (DyT) as a drop-in replacement for Layer Normalization. Although the method is empirically well-motivated and appealing from a practical point of view, it lacks a theoretical foundation. In this work, we derive DyT mathematically and show that a well-defined approximation is needed to do so. By dropping said approximation, an alternative element-wise transformation is obtained, which we call Elementwise Layer Normalization (ELN). We demonstrate that ELN resembles Layer Normalization more accurately than DyT does.

翻译：近期一篇论文提出动态Tanh（DyT）作为层归一化的即插即用替代方案。尽管该方法在实证层面具有充分动机且从实用角度颇具吸引力，但其缺乏理论基础。在本工作中，我们从数学角度推导了DyT，并证明需要借助一个明确定义的近似才能实现该推导。通过舍弃该近似，我们得到了一种替代性的逐元素变换方法，称之为逐元素层归一化（ELN）。我们证明ELN比DyT更精确地模拟了层归一化的特性。