Equivariant Transformers such as Equiformer have demonstrated the efficacy of applying Transformers to the domain of 3D atomistic systems. However, they are still limited to small degrees of equivariant representations due to their computational complexity. In this paper, we investigate whether these architectures can scale well to higher degrees. Starting from Equiformer, we first replace $SO(3)$ convolutions with eSCN convolutions to efficiently incorporate higher-degree tensors. Then, to better leverage the power of higher degrees, we propose three architectural improvements -- attention re-normalization, separable $S^2$ activation and separable layer normalization. Putting this all together, we propose EquiformerV2, which outperforms previous state-of-the-art methods on the large-scale OC20 dataset by up to $12\%$ on forces, $4\%$ on energies, offers better speed-accuracy trade-offs, and $2\times$ reduction in DFT calculations needed for computing adsorption energies.

翻译：等变Transformer（如Equiformer）已证明将Transformer应用于三维原子系统领域的效果。然而，由于计算复杂度限制，这类架构仍局限于低阶等变表示。本文研究这些架构能否有效扩展至更高阶表示。以Equiformer为基础，我们首先用eSCN卷积替代$SO(3)$卷积以高效整合更高阶张量。随后为更好利用高阶表示能力，提出三项架构改进——注意力重归一化、可分离$S^2$激活函数以及可分离层归一化。综合以上改进，我们提出EquiformerV2，在大规模OC20数据集上相较此前最优方法实现力预测提升12%、能量预测提升4%，同时实现更优的速度-精度权衡，并减少计算吸附能所需的DFT计算量达2倍。